QNetworkAccessManager and caching POST request data

ALTinners

Hi all,

I've been using QNetworkAccessManager to fire POST requests to an internal REST API. It didn't work initially, and I eventually found that making sequential POST requests with a JSON payload on the same AccessManager ended up inserting the previous's request onto the current request.

Something like

Request 1 @ http://localhost:1234/urlA
{
    "resourceA": "the contents of request A"
}

Request 2 @ http://localhost:1234/urlB
{
    "resourceA": "the contents of request A"
}
{
    "resourceB": "the contents of request B"
}

I've turned off as many of the caching options as I can find on both QNetworkAccessManager and the QNetworkRequests I instantiate, and can confirm that the QNetworkReplies that are generated get destroyed. The only workaround I've found is that using clearConnectionCache() on NetworkAccessManager - but that also destroys any inflight network requests. From what I can see, the NetworkRequest's data is buffered in NetworkAccessManager, but for some reason this never gets released.

Does anyone know how to make the old JSON payload not get cached - or otherwise how to force NetworkAccessManager to forget a particular request's data? I can see an internal cache which gets cleared when clearConnectionCache() is called, but that is private in NetworkAccessManager.

Edit: using 5.9.1

ambershark

@ALTinners This doesn't sound like normal behavior at all. I don't think it's being cached. My guess would be something to do with not flushing your network buffer or a reused variable that has the old data in it. Something along those lines.

I can't tell without seeing code but that's what my guess would be based on the information provided.

There is little to no reason that a POST request would get cached since they rarely have the same data. That would break many websites. ;)

ALTinners

@ambershark Should have put code up to start with :).

void BaseInterfaceFunctor::sendHttpRequest(const int& commandId,
                                            const HttpVerb& verb,
                                            function<bool(CommandResponse&)> callback,
                                            const QHash<QString, QString>& arguments,
                                            const QJsonObject& data)
{
    QString str = "http://" + m_address.toString();
    QUrl url{str};
    url.setPath(m_path);
    url.setPort(SwarmbotInterface::DEFAULT_SWARMBOT_HTTP_PORT);

    if (arguments.size() > 0)
    {
        QUrlQuery query;
        QHash<QString, QString>::const_iterator itr = arguments.constBegin();
        while (itr != arguments.constEnd())
        {
            query.addQueryItem(itr.key(), itr.value());
            ++itr;
        }
        url.setQuery(query);
    }

    QString payload = QJsonDocument(data).toJson();

    qDebug() << "Payload is " << payload;   //This output shows the correct data

    QNetworkRequest request;
    request.setUrl(url);
    request.setHeader(QNetworkRequest::ContentTypeHeader, "application/json");
    request.setHeader(QNetworkRequest::ContentLengthHeader, payload.size());
    request.setAttribute(QNetworkRequest::CacheLoadControlAttribute, QNetworkRequest::AlwaysNetwork);
    request.setAttribute(QNetworkRequest::CacheSaveControlAttribute, false);
    request.setAttribute(QNetworkRequest::DoNotBufferUploadDataAttribute, true);
    request.setAttribute(QNetworkRequest::HttpPipeliningAllowedAttribute, false);

///Lock so that async requests on the same functor do not try to access the AccessManager simultaneously
    m_httpMutex.lock();
    QNetworkReply* reply = NULL;
    if (verb == HEAD)
        reply = m_httpClient->head(request);
    else if (verb == GET)
        reply = m_httpClient->get(request);
    else if (verb == PUT)
        reply = m_httpClient->put(request, payload.toUtf8());
    else if (verb == POST)
        reply = m_httpClient->post(request, payload.toUtf8());
    else if (verb == DELETE)
        reply = m_httpClient->deleteResource(request);

    ///A container to hold the reply, its context and callback when it eventually fires finished.
    ///Also contains a timer which is owned by the reply which sets the timeout limit
    m_requests.push_back( ReplyContainer(reply, request, commandId, callback) );
    connect(reply, &QNetworkReply::finished, this, &BaseInterfaceFunctor::processHttpReply);
    m_httpMutex.unlock();


}

ambershark

@ALTinners Ok so from that code I noticed you have a mutex to prevent multiple async requests... Is it possible that requests are happening close enough together to not have flushed properly through the tcp/ip buffer?

What happens if you send request #2 a few seconds after req #1? Does it have the same behavior?

Can you watch your rest/server side and see what is being sent at what times?

ALTinners

@ambershark Yep, definitely not the mutex - sending requests close together or far apart (relatively speaking, say 3 secs) still appends old data onto new ones.

My thinking is that QNAM is waiting for a signal from the NetworkReply being destroyed before it removes that payload buffer from its own internal buffer. I know that NetworkReplies should be kept until they fire finished() (and then deleted using deleteLater() ) - is there a similar requirement for NetworkResponses?

I attached a slot to the NetworkReply's destroyed() signal and can see replies gets destroyed after being handled - should this be then clearing the Reply from QNAM, and also clearing whatever buffer data is associated with that reply?

I'm going to also try a more minimal example to see if its something with keeping NetworkReply and NetworkResponses around.

ambershark

@ALTinners Yea I would suggest simplifying and see if you can duplicate this issue with an easy example.

If you can get it simple enough provide the code and I will build and test it.

I don't see why one QNetworkRequest or QNetworkReply would have anything to do with another one. It's just weird behavior. That would have like no useful reason to exist inside Qt. At least none I can think of right now.

So something weird is going on. You could make a quick echo server (take the REST out of it) and see what is being posted to it from your application.

ALTinners

@ambershark So - I modified the HTTP Example from the Qt Examples to simply send a POST with the same params as my client code, and was able to replicate the results.

This has got me thinking that it could be something with the server not closing requests properly and keeping data around when interacting with Qt's code. I couldn't replicate the bug when using Curl, so that made me sniff out Qt as a potential issue - however it could be when Qt clears the connection cache it is closing some outstanding request with the server.

I've tested the minimal example I made with http://httpbin.org/ and it doesn't double up dispatches - so I think this is a bad server issue, not a client side issue.

Cheers @ambershark !