Solved Max requests at the same time with QNetworkAccessManager
-
@ambershark said in Max requests at the same time with QNetworkAccessManager:
My thoughts on what you're doing .... I would have a list of urls that you remove the url from while processing. That way you can check for an empty list to know when you're done.
I have already previously suggested this to the OP for this issue, but the thread seems to have been deleted by someone....
-
@JonB said in Max requests at the same time with QNetworkAccessManager:
@ambershark said in Max requests at the same time with QNetworkAccessManager:
My thoughts on what you're doing .... I would have a list of urls that you remove the url from while processing. That way you can check for an empty list to know when you're done.
I have already previously suggested this to the OP for this issue, but the thread seems to have been deleted by someone....
The OP himself deleted the thread.
@Mr-Gisa, when you receive the answer you need, please mark the topic as "Solved" instead of deleting it. This way, other people can find the posts in the future and benefit from the discussion.
-
@JonB @JKSH I deleted the topic myself and the reason is that it was confused even for myself. I created the topic asking for help with the algorithm but I saw that it was really confusing and it wasn't solved so I decided to create a topic explaining things better and with the code I have, this way when solved it would be better for me and for the others. I don't delete topics that are solved or at least the ones I find that is going to help others, which is the case of the deleted topic mentioned by you guys.
-
@ambershark I made the changes you pointed out and it looks way better, the problem is still the same as the first one tho, it's emitting finished
3
times.QueuedRequester::QueuedRequester(int concurrent, QObject *parent) : mConcurrent(concurrent), QObject(parent) { mManager.setRedirectPolicy(QNetworkRequest::NoLessSafeRedirectPolicy); } void QueuedRequester::setUrls(const QList<QUrl> &urls) { mQueue.append(urls); } void QueuedRequester::start() { for (int i = 0; i < mConcurrent; ++i) { nextUrl(); } } void QueuedRequester::abort() { for (auto reply : mManager.findChildren<QNetworkReply *>()) { reply->abort(); } } void QueuedRequester::requestUrl(const QUrl &url) { auto request = QNetworkRequest(url); auto reply = mManager.get(request); connect(reply, &QNetworkReply::finished, this, &QueuedRequester::handlePage); } void QueuedRequester::nextUrl() { if (mQueue.empty()) { return finished(); } requestUrl(mQueue.dequeue()); } void QueuedRequester::handlePage() { auto reply = qobject_cast<QNetworkReply *>(sender()); reply->deleteLater(); auto status = reply->attribute(QNetworkRequest::HttpStatusCodeAttribute).toInt(); qDebug() << status; nextUrl(); }
-
@Mr-Gisa
You are callingnextUrl();
in bothQueuedRequester::start()
&QueuedRequester::handlePage()
. Why is that?Are the urls safely in
mQueue
(viasetUrls()
), andmCurrent
correctly set correspondingly, by the timeQueuedRequester::start()
is called?QueuedRequester::nextUrl()
emitsfinished();
each & every time the queue is empty, not just once. So you have to be real careful about how many times it's called.I don't know exactly, and I'm not going to follow each line of code, but it's a couple of things you can check. Maybe if you try with just a few Urls (not your 100), and print out what Url is for each start & finish, you'll spot where it's going wrong...?
-
You are calling nextUrl(); in both QueuedRequester::start() & QueuedRequester::handlePage(). Why is that?
Because as I need to start the amount of requests that is defined in
mConcurrents
I need to callnextUrl
in the loop.
And in thehandlePage
because when a request is finished it will request the next one.Are the urls safely in mQueue (via setUrls()), and mCurrent correctly set correspondingly, by the time QueuedRequester::start() is called?
mCurrent
doesn't exist here anymore as I changed fromQList
toQQueue
as @ambershark pointed out. And the urls are in themQueue
when added viasetUrls
.QueuedRequester::nextUrl() emits finished(); each & every time the queue is empty, not just once. So you have to be real careful about how many times it's called.
In theory it should work, I mean, start
3
requests, when one is finished starts the next one, then we will always have3
requests going on, until there's no more urls to request.
The thing is that I have to check when there's no more urls andnextUrl
is the only place that I thought.I don't know exactly, and I'm not going to follow each line of code, but it's a couple of things you can check. Maybe if you try with just a few Urls (not your 100), and print out what Url is for each start & finish, you'll spot where it's going wrong...?
I'm doing with
6
urls and3
concurrent, that is the output:200
200
200
200
finished
200
finished
200
finishedIt always emit
finished
the amount ofmConcurrent
, like when the last3
requests are being made, when they finish, they try to see if there's more urls and it's empty and so it emitfinished
. -
@Mr-Gisa To not to emit finished() more than once you could use a boolean variable which is set to true when emitting finished(). And finished() is only emitted if it is false.
-
Hi,
@Mr-Gisa said in Max requests at the same time with QNetworkAccessManager:void QueuedRequester::nextUrl()
{
if (mQueue.empty()) {
return finished();
}requestUrl(mQueue.dequeue());
}I think the culprit is here.
You only check if the mQueue is empty. Each of your concurrent runs is checking after its done it work, if the queue is empty or if there's still work to be done. 3 Concurrents 3 checks if empty 3 times result true 3 times finished signal.
Simple solution, create a
QVector<bool> m_NetworkrequestsOngoing
each concurrent sets one state of the vector to true, when its doing work and back to false, when its done.Than check for mQueue.empty() and m_NetworkrequestsOngoing.contains(true)
void QueuedRequester::nextUrl() { if (mQueue.empty() && !m_NetworkrequestsOngoing.contains(true) { return finished(); } requestUrl(mQueue.dequeue()); }
-
@J.Hilk I'm kinda confused with that, how am I going to know which request set to true in order to set it to false as it is a
QVector
and I can only insert abool
in it? -
@Mr-Gisa What I meant when I talked about removing items from a list as you processed them was that when the list was empty you were truly finished. However you are removing the items before processing on the item is complete, so you are getting 3 finishes rather than just one because you are checking for an empty queue "to be processed" rather than a "finished being processed" list.
There are a couple ways to handle this...
-
You can create another list of "processing" items and when that list is empty emit the finish. So right before you process the item, push it on the list. Then when processing is finished or abort is called, pop it off the list. After the pop check the list for isEmpty and if it is you're truly finished (and only once).
-
You can just use an
int
member in your class and when it hits 0 it's finished. When you start processing increment it by 1, when you finish processing decrement it. When it's 0, you're done, emit finished.
-