Qt-interest Archive, October 2007
Scheduled http gets
Message 1 in thread
Hi all,
I'm doing Amazon requests using their rest api. I've implemented an
amazon fetcher which gets information about products and wishlists.
However, given a wishlist id, which might have hundreds of products, I
get the products id in the wishlist and using a fetcher for products I
fetch all the products, since the calls for get are asynchronous I
setup hundreds of gets in a very short time. Amazon started to send me
error 503 pages. Well, it seems Amazon license only allows 1 request
per second per IP address, which means I need to schedule each of my
request, on a 1 request per second basis.
I wonder, what would be the best strategy to implement this. Any ideas
are welcome.
By thinking about it for a few moments (and without knowing if the
underlying Qt framework helps me with this) I get to think that the
best way is to have a queue of requests timestamped with the soonest
time they could be done and have a cycle that while the queue is not
empty executes the front of the queue as soon as possible given its
timestamp. This would probably have to be done in a separate thread or
something and dealing with threads really is not my passion but unless
there's other solution...
Any ideas on this or better implementations?
Cheers,
--
[ signature omitted ]
Message 2 in thread
Interesting.
Not sure if this helps, but...
You could create a class named GetScheduler, to send HTTP get requests at specified intervals.
GetScheduler has an addGetRequest(const QString &) function, which adds new URLs to a QStringList named _getUrls.
A function named doGetRequest() is called every one second (or whatever). This does a HTTP get request for the first/last URL in _getUrl, then removes it.
If _getUrls is emptied, the timer to call doGetRequest() is stopped -- and vice versa.
Obviously, if you create lots of QHttp objects you'll need to delete them after processing the response in the slot connected to the done() signal.
Sam Dutton
SAM DUTTON
SENIOR SITE DEVELOPER
200 GRAY'S INN ROAD
LONDON
WC1X 8XZ
UNITED KINGDOM
T +44 (0)20 7430 4496
F
E SAM.DUTTON@xxxxxxxxx
WWW.ITN.CO.UK
P Please consider the environment. Do you really need to print this email?
-----Original Message-----
From: pocmatos@xxxxxxxxx [mailto:pocmatos@xxxxxxxxx] On Behalf Of Paulo J. Matos
Sent: Tuesday 09 October 2007 23:53
To: Qt-interest
Subject: Scheduled http gets
Hi all,
I'm doing Amazon requests using their rest api. I've implemented an amazon fetcher which gets information about products and wishlists.
However, given a wishlist id, which might have hundreds of products, I get the products id in the wishlist and using a fetcher for products I fetch all the products, since the calls for get are asynchronous I setup hundreds of gets in a very short time. Amazon started to send me error 503 pages. Well, it seems Amazon license only allows 1 request per second per IP address, which means I need to schedule each of my request, on a 1 request per second basis.
I wonder, what would be the best strategy to implement this. Any ideas are welcome.
By thinking about it for a few moments (and without knowing if the underlying Qt framework helps me with this) I get to think that the best way is to have a queue of requests timestamped with the soonest time they could be done and have a cycle that while the queue is not empty executes the front of the queue as soon as possible given its timestamp. This would probably have to be done in a separate thread or something and dealing with threads really is not my passion but unless there's other solution...
Any ideas on this or better implementations?
Cheers,
--
[ signature omitted ]
