Multiple requests in the same method

[Mr Gisa]

I come from languages that are not events driven like Qt and sometimes when I want to create a kind of API wrapper or something like that I can create multiple requests in the same method and wait for their return in order to process the next one. I can request for login, request a page and download a file in the same method, waiting the other requests to finish before going forward to the next one. In Qt its a bit complicated for me as it uses signals and slots, so I was wondering, how can I create a method that can do multiple network actions?

[mrjj]

Hi
http://doc.qt.io/qt-5/qnetworkaccessmanager.html#details
is already asynchronous so you can have up to 6 request out at same time.
You cannot in any good way use it synchronous so doing it all from same function will prove difficult.
So you better make a small handling class that controls the downloading and
first send off next request when the one before is finished if that is the goal.

[Mr Gisa]

The question is about the possibility to make multiple requests at the same time but the requirement to make one after the other when its finished. For example:

I need to create a class that have a method search, in that search method I need to send a post request to a form, get the returned json (this is one request), with the json in hand, I need to parse that json and get a code, with that code I need to send another request to another url (this is the second request), with that second request I need to parse and return a string to the user. This in the same method.
The thing is, in order to return a string to the user I need to make two requests before that. In PHP, Python or other language I can just assign the return of the request to a variable and we are good to go, but In Qt its more complicated cause its events.

[mrjj]

Hi
So it should have downloaded all request before processing them ?
You need a queue or similar to handle that, so when each one comes in you put the data in a list. Then when list is complete enough, you can hand it off to a thread for processing if its heavy. If its ok to parse in GUI thread ( it will hang while parsing) then no thread needed.
I tried to find sample for this but didn't find any good ones as the the "task" handling is very app dependant.

[Mr Gisa]

I don't care about the method being async, because I can just use the Qt Concurrent later with the run method in order to solve the freezing problem of the GUI
I was reading the signal and slots documentation and I saw that I can use lambdas instead of assigning a method as a slot. You think that this would work?
Yes, one request depends on the other and the method needs to return something, so I can't really rely on signals and slots in this case.

[mrjj]

@Mr-Gisa
Hi, using lambdas wont help more than using traditional slots but does the code more compact and local.
You need to structure your code so it does not depends on blocking behaviour as for loops.
So you need a way to send out request, collect data,when comes in and then
process them when data is ready.
I dont see that being in a single function without some kind of state machine running it.

Maybe others have some trick to do it synchronous without ugly local event loops and such hacks.

[Mr Gisa]

Yes, most of the codes I found on the internet is exiting the event loop and then executing again after when its finished. But I think that the best way to go here is lambdas as I can't see another way to create this kind of project. I depend on other information so I really don't know what to do knowing that Qt only works with async requests.

[mrjj]

Hi
Normally people use a small state machine design so code can work async.
( a queue and some signals to switch states )
Im not sure lambdas will help unless you use the local event loop trick.

[Mr Gisa]

That is so weird, I will make some tests with lambda, if it doesn't work I will have to use the event loop hack thing or use a third party library to make the http requests.

[Chris Kawa]

The problem is you're trying to impose a blocking style on something that is asynchronous in nature.
If I understand you correctly you're thinking like this:

void func() {
    result  = doSomething();              //blocks until finished
    result2 = doSomethingElse(result);    //blocks until finished
    result3 = doYetAnotherThing(result2)  //blocks until finished
}

This is doable. You just need to make the calls blocking. You could wrap them in something like this:

QByteArray blockingGet(QNetworkRequest* req)
{
    auto reply = qnam.get(req);  
    
    while(!reply->isFinished())
        qApp->processEvents();
    
    reply->deleteLater();
    return reply->readAll();
}

but this is kinda horrible.

The way to do it more Qtish is to think about how can you split the functionalities you need into objects that signal their state. To use your search example. You could make a Searcher class like this (simplified):

class Searcher : public QObject
{
    Q_OBJECT    
public:
    void search(const QString& what); // posts a request and connects `finished()` to `processJson`
    
signals:
    void searchFinished(const QString& result);
    
private:
    void processJson(); // reads code from json, posts another request and connects `finished()` to `processSearchResult`
    void processSearchResult(); // parses the response and emits searchFinished with the result
};

Then you would make a connection like:

connect(searcher, &Searcher::searchFinished, some_class, &SomeClass::showSearchResultsToTheUser);

and initialize search (e.g. on user input) like this:

searcher->search(something);

For the simple case of two requests this would be enough. For more complex request schemes you would create a state machine (as @mrjj suggested). You can take a look at QStateMachine.

You're saying you need it in a single function. That's the thing - you probably don't. You're just used to doing it this way and trying to force it here.

This might look like more code than the blocking style but don't be fooled. More code doesn't mean worse code. It decouples your network code from the main thread and separates responsibilities a lot nicer.

[Mr Gisa]

@Chris-Kawa Its amazing, you made me see this in a whole new way, seriously. And about the state machine thing, I don't know if I will need that, but I will tell you what I want to do.

The class is an API wrapper, meaning that it will have a login, logout and search methods. I can only search if the user is logged in. The search method is the most complex one that I need to create multiple requests, and it also has pages so I will have to loop over the pages and get requests for each one. Like looping over a pagination and put the results in a list.

[Chris Kawa]

Depending on the api of that service looping over pages is also easy. Just don't think about looping in terms of c++ for.
It's more like:

void requestFirstPage(); //makes a request and connects to parsePage()
void parsePage();// parses page, if there's more pages makes another request and connects co parsePage() again, otherwise emits some finishing signal

[Mr Gisa]

What about the result list, I should put it as a member of the class or emit it together with the signal?

[Chris Kawa]

If you're getting results as you make more requests you'll need a member anyway to store them. After that either make a finishing signal something like searchFinished(const QStringList& results) or just searchFinished() and another method like QStringList searchResults() const so the caller can retrieve results. Or you can combine both, whatever you like more.

If you want to update some ui with the results in realtime you could also add a signal like searchResultsAvailable after each request processed so the caller can show partial results as they come.

[Mr Gisa]

Amazing amazing, thank you for making me see this in a whole new way. I will be playing with this in order to learn more.

[Mr Gisa]

@Chris-Kawa I was playing around and I have two questions:

1 - Is it better for me to connect with the QNetworkAccessManager finished in order to send the QNetworkReply to another method such as processJson or I can have one QNetworkReply pointer member and reuse that for other requests?

2 - What about error handling for each step?

[Chris Kawa]

The answer to both questions is "whatever you want or will integrate nicely with the rest of your code".
To give it a little more substance:

1 - It's generally suggested to use single QNAM instance. Since you mentioned you can have multiple requests in flight at the same time connecting to the QNAM's finished signal might be inconvenient. You would have to somehow distinguish which series of requests it is and which "stage" of it it is. That's of course doable. You could for example do something like this:

auto response = qnam.get(request);
response->setProperty("request_type", foo); //foo could be an enum indicating login, logout, search etc.
response->setProperty("request_stage", bar); // bar could be an enum indicating json retrieval or the following request

and then in the slot connected to QNAM's finished(QNetworkReply*) signal you could check these properties and do a big "switch". This would work but can get kinda messy as those switches tend to grow over time and gather unrelated functionalities.

Another approach is to connect not to the QNAM's finished signal but to the response's finished signal. This way you can "string" the requests together and you don't have to do a big switch to see which one it is. every method knows what it receives and can start the next stage.
To get the response object you can use a little lambda on connection site:

void Searcher::search(const QString& what)
{
   auto reply = qnam.get( /* build the request from what*/ );
   connect(reply, &QNetworkReply::finished, [=]{ processJson(reply); });
}

void Searcher::processJson(QNetworkReply* reply) { ... }

Or, if you want to spare yourself the lambda, you can also do it like this:

void Searcher::search(const QString& what)
{
   auto reply = qnam.get( /* build the request from what*/ );
   connect(reply, &QNetworkReply::finished, this, &Searcher::processJson);
}

void Searcher::processJson()
{
   auto reply =  qobject_cast<QNetworkReply*>(sender());
   ...
}

Some people will tell you that sender() is bad and you shouldn't use it (even the docs says that), but if you keep methods that use it private I consider that internal piping and a valid use for sender(). It is considered bad because you never know how the slot is called if it's public, but if it's private you, as the designer of the class should know and be responsible for calling it right.
Anyway, lambda or not, this is one way to do it.

2 - That one is pretty simple really. Apart from connecting to finished signal of the reply connect also to the error and sslErrors (if you use https). If you don't care what particular error happened just emit searchFinished with an empty list of results. If you do care (e.g. want to show some message to the user) instead of emitting searchFinished you could emit something like searchFailed(QString& reason) or you could add another param to the searchFinished signal e.g. an enum that would indicate what type of error occurred or a message string where empty string would indicate a successful search and non-empty would contain the error message.

[Mr Gisa]

@Chris-Kawa So in this case I can use the same QNetworkReply with another request right? But where do I delete the reply? I see in the docs that they usually call deleteLater in the finish slot, if I want to reuse the reply where to delete it?

[Chris Kawa]

You can't reuse a reply. A new reply object is created by QNAM and returned to you by each call to get(), post() etc.
You can store and reuse a pointer to it but IMO this can lead to bugs if you mix up your requests. I wouldn't recommend that. It's better to just connect to what you need and be done with it.

You can call deleteLater in the slot connected to finished(). As said before - you can't reuse the reply object, just the pointer to it and this is the trouble I mentioned - it's easy to mess up and delete the wrong thing or at the wrong time. I'd suggest not to store it at all and rely on either function param or sender() in the slot connected to finished() signal.

[Mr Gisa]

@Chris-Kawa Thank you very much

[Mr Gisa]

@Chris-Kawa Playing around I noticed that I needed extra information on my search method, two more arguments as QString. The thing is that I will be needing that two extra arguments later in another method/slot.

More or less like this:

search > processJson > processProducts

I will need the extra arguments in the processProducts.

The option I thought was:

1 - store the values passed to search in a member and get later in another method.

Is there a better alternative for that?

[Chris Kawa]

If one instance of your class does only one search at a time then sounds like a reasonable thing to store the extra state as members. No point in complicating things.

[Mr Gisa]

Yes, it will be one search per instance, because I will create a QTabWidget and each QWidget of the tab will have an instance of the Searcher. I mean, the new tab will be created after the user fills a line edit and click on the search button.

[Mr Gisa]

@Chris-Kawa I have a few more questions:

The searcher will also be a downloader.

The idea behind this application is that the user can search for a product and the product will appear in a tree view. I will explain the tree view.

I want to do something like this:

1 - The user fills a line edit with the product he wants to search and press the search button.
2 - A new tab will be created in the QTabWidget with a tree view and an instance of the Searcher.
3 - When the search is done and Searcher emits a signal that the search is done I want to display the results in a tree view.
4 - The results that Searcher will return is a collection of Products, a struct with a few information such as the name, color and other information, but the most important one, it will have a link to a zip file.
5 - I want to create a button on the tree view that is called Download in each Product which contains the zip link.
6 - When the user clicks on that button I want to download that zip file and get the file names and put as children to the download clicked item on the tree view.

For me its kinda complicated, any ideas of what I can do to archive that?

The Product B has a zip link so it will have a download button on the tree view. Imagine that the user clicked on the button, it downloaded the zip file, got its the file names and put as children for ProductB.