How to pass QNetworkReply‘s data without copying it into memory

Creaperdown

Hey, I have a QNetworkReply whose data is a rather big pdf file. I need to pass that data to a function, which is supposed to write it to a file on the local file system, but I want to avoid copying the data into memory.

Is there a way for me to pass the backing file of the QNetworkReply and read from there?

Thanks in advance

JonB

@Creaperdown
If you use QDataStream you will not just be writing content bytes to the file, you will be adding extra information. And the file will only then be readable by another application using QDataStream. If you expect to use the file with other applications that is probably not what you want.

Your first attempt with void saveToFile(const QByteArray& data) will write from the original QByteArray without doing any copying.

sierdzio

@Creaperdown by default, Qt is using implicit sharing (copy-on-write) so as long as you pass data as const without altering it, no copying will occur.

Also, QNetworkReply is a subclass of QIODevice so you can use QDataStream to parse it or copy to file etc.

Creaperdown

@sierdzio So I could use something like:

void saveToFile(const QByteArray& data)
{
    QFile file(…);
    file.open(…);
    file.write(data);
}

…
const QByteArray data = myResponse.readAll();
saveToFile(data);

And it would not copy the data into memory, but write it directly to the file?

Creaperdown

Or should I use something like:

void copyToFile(QDataStream& data)
{
    QFile file(…);
    file.open(…);
 
    QDataStream out(&file);
    out << data;
}

…
QDataStream data(myResponse);
copyToFile(data);

JonB

@Creaperdown
If you use QDataStream you will not just be writing content bytes to the file, you will be adding extra information. And the file will only then be readable by another application using QDataStream. If you expect to use the file with other applications that is probably not what you want.

Your first attempt with void saveToFile(const QByteArray& data) will write from the original QByteArray without doing any copying.

Creaperdown

@JonB So the first way is the one I‘d need?

JonB

@JonB said in How to pass QNetworkReply‘s data without copying it into memory:

Your first attempt with void saveToFile(const QByteArray& data) will write from the original QByteArray without doing any copying.

jsulm

@Creaperdown If you pass "by reference" nothing is copied anyway in C++...

Creaperdown

@jsulm Yes but my fear was that it will be loaded into memory, if I use readAll to get a QByteArray

JonB

@Creaperdown
That will indeed happen. Let's be clear: you can avoid taking an extra copy in memory by using the stuff above, but you can't somehow get a network reply and never copy it into memory in the first place in order to write it to file.

If you mean: as it stands the code reads the whole response into memory and then writes it to file. You could reduce how much memory you use if you used QIODevice::read() to read chunks at a time and QFile::write() them as you go.

Creaperdown

@JonB I see, that's probably what I'll go for then. I hoped there might be a possibility to copy the data directly from the file that QNetworkReply is based on to a new file, but it seems like this is not possible

jsulm

@Creaperdown said in How to pass QNetworkReply‘s data without copying it into memory:

file that QNetworkReply

There is no file. The incoming data is put into a buffer from which you then read it. Ideally you simply connect a slot to https://doc.qt.io/qt-6/qiodevice.html#readyRead of your QNetworkReply and read the data every time this slot is called.