How to limit upload speed?

TheSHEEEP · wrote on 29 Jun 2017, 07:59

Hello,

I am using QNetworkAccessManager to upload a file (and some data) like this:

QNetworkRequest request(<some URL>);
QHttpMultiPart *multiPart = new QHttpMultiPart(QHttpMultiPart::FormDataType);
<multipart is filled with single parts>

networkAccessManager->post(request, multiPart);

It works well.
However, users have reported that the upload influences their upload bandwidth. Which... makes sense, of course ;)

Now, I would like to implement something to limit the upload speed of this post request. However, after looking around for a while, I found absolutely nothing to do this in any simple way.

There is setReadBufferSize for the QNetworkReply, and I read that this can be used to limit the download speed. But the description doesn't make it look like it would work for uploads as well. Also, there is a years old bug report for that function which doesn't exactly inspire confidence in it working at all (QTBUG-15065).

Is there any way to achieve an upload throttle at all without having to rewrite Qt sources?
A way to "globally" limit the upload speed of the application would be acceptable as well.

I have found this ancient thing: https://doc.qt.io/archives/qq/qq17-ratecontrol.html
It looks like that could be used, but since I do not only transfer binary data, but all the other HTML multi part stuff as well, it would mean a significant amount of work to pull it off correctly.

TheSHEEEP · 4 Jul 2017, 09:03

After trying for many days, I have to conclude that limiting upload speed simply isn't possible.
Even by heavily adjusting the torrent example with the rate control class, it cannot work.

The main problem is that QIODevice's read & write functions are not virtual, you can overwrite QNetworkReply as much as you want, it won't help when it comes to upload limiting. Your own read/write functions won't be called and there are no signals that would allow you to interrupt the writing process.

For download, there is at least the setReadBufferSize, and it works not perfectly (for some not at all, it seems), but it is there.
Due to a lack of a setWriteBufferSize function, it is simply not possible.

Other than by re-implementing the complete network side yourself, that is. Or changing Qt's sources, which is just impractical for many reasons.

Should I make a report or feature request for this?

kshegunov · wrote on 4 Jul 2017, 09:03

@TheSHEEEP said in How to limit upload speed?:

The main problem is that QIODevice's read & write functions are not virtual

They are, the relevant ones are protected: QIODevice::readData and QIODevice::writeData, however this might not be the thing to do in your case.

Should I make a report or feature request for this?

Yes, posting a feature request is what I'd do.

TheSHEEEP · 7 Apr 2017, 12:56

@kshegunov said in How to limit upload speed?:

@TheSHEEEP said in How to limit upload speed?:

The main problem is that QIODevice's read & write functions are not virtual

They are, the relevant ones are protected: QIODevice::readData and QIODevice::writeData, however this might not be the thing to do in your case.

Huh! Indeed they are. Must've looked int the wrong place...
So maybe this is possible after all.

What I am doing is using a class derived from QNetworkAccessManager, where I overwrote the createReply method like this:

QNetworkReply*
LimitingNetworkAccessManager::createRequest(QNetworkAccessManager::Operation op,
                                            const QNetworkRequest& req,
                                            QIODevice* outgoingData)
{
    // Create our custom reply (by passing on the reply created by original Qt)
    QNetworkReply* qtReply = QNetworkAccessManager::createRequest(op, req, outgoingData);
    LimitingNetworkReply* reply = new LimitingNetworkReply(qtReply, this);

    // Add this reply to the limiter
    _limiter->addReplyToLimit(reply);

    // Return
    return reply;
}

However, my own LimitingNetworkReply class' read/writeData functions are never called. Despite this createRequest function returning my own reply. How is that possible?
Is the original QNetworkReply created by QNetworkAccessManager::createRequest still being used for reading/writing?

I knew using the original QNetworkAccessManager::createRequest function was risky here. But I just see no way of implementing my own QNetworkReply creation fully, since the original function is stock full with D-pointers and usage of private classes, so that's not something I can do when deriving.

I feel like I'd have to completely write my own networking classes when there really should be a simple functionality to limit upload speed - which would be very easy to do by just using a timer when sending data over sockets...

kshegunov · 7 Jul 2017, 10:32

@TheSHEEEP said in How to limit upload speed?:

Is the original QNetworkReply created by QNetworkAccessManager::createRequest still being used for reading/writing?

That'd be my guess, yes. The QNetworkAccessManager::createRequest method is rather complex and does all kinds of stuff, so calling it like this probably connects the created (Qt) reply object and thus your own object's methods aren't invoked.

I feel like I'd have to completely write my own networking classes when there really should be a simple functionality to limit upload speed - which would be very easy to do by just using a timer when sending data over sockets...

Well, the thing is these classes are "high-level", meaning they are there to simplify your life most of the time. Incidentally this also means that they're designed with a specific use case, and modifying that isn't at all trivial. For http NAM and friends thread the operations internally (and there are a lot of support classes that are used privately) so I'm not convinced you'd be able to do what you want in a simple fashion.

Off the top of my head, a simple, albeit not exactly nice, workaround would be to connect the downloadProgress signal from the reply to a dedicated thread that manages the reception speed through Qt::BlockingQueuedConnection and intentionally blocking the event loop for some specific time. Depending on the internal implementation of QNAM it may provide ability to control the rate at which the download proceeds, however the best solution is still going to be to post a feature request.

TheSHEEEP · 7 Jul 2017, 10:59

@kshegunov That sounds a bit hackish, indeed, and I don't think it would work. Wouldn't that lead to rather unpredictable phases of high upload speed vs none instead of having a nice average upload speed?

Either way, I did create a feature request and "solved" the problem in the meantime by using libcurl instead of QNetworkAccessManager.

kshegunov · 7 Jul 2017, 11:00

@TheSHEEEP said in How to limit upload speed?:

Wouldn't that lead to rather unpredictable phases of high upload speed vs none instead of having a nice average upload speed?

Possibly, it was the first thing I could think of.

PS.
Please post the ticket here, so others searching for such a solution could find it through this thread.

TheSHEEEP · wrote on 10 Jul 2017, 12:17

Sorry, I forgot to post the link:
https://bugreports.qt.io/browse/QTBUG-61787