Saving data in a file is too slow

[rahuman]

Hi,
Reading data continously from server,and using qDebug() i can able to see data with high speed .But while saving in a file is getting slow.Here is the sample code .

QByteArray RxBuffer = socket->readAll();
qDebug() <<"Reading data......" <<RxBuffer;
QFile file ("C:/123.bin");
if(file.open(QIODevice::WriteOnly | QIODevice::Append | QIODevice::Text))

    {
         QTextStream out(&file);
         out <<RxBuffer;  
   }

[jsulm]

@rahuman You could open the file once instead of opening it every time you want to write to it.

[rahuman]

@jsulm
yes,I tried that also.But only 1 mb is increasing.

[SGaist]

Hi,

What is the data volume ?
What is the speed you expect ?
What is your hard drive speed ?

[rahuman]

@SGaist
Data volume : Using E(drive),Available space :350GB
My required data size is 10GB to be download.
Speed: Required 40 Mbyte/Sec
Hard drive speed:7200 rpm(Toshiba 1 TB (DT01ACA100))

Note:While doing Usb data transmission also i faced the same issue.But in python data is saving with the required speed.

[jsulm]

@rahuman Did you check whether data transfer continues after 1MB?

[rahuman]

@jsulm
Yes,Continously i can recieve data, but with less speed.
For ex: 1 Min it was reading 10Mb data

[jsulm]

@rahuman And did you check that you are actually sending at 40MB/sec?

[rahuman]

@jsulm
No, i didn't check of sending speed .As of now i am just reading data.

[JonB]

@rahuman said in Saving data in a file is too slow:

@jsulm
yes,I tried that also.But only 1 mb is increasing.

I do not know what "But only 1 mb is increasing." might mean.

Please make 100% clear whether you are presently calling QFile::open() every time you get signal readyRead()/call readAll(), or not. Continually re-opening a file each time you want to append to it, for "small" chunks, is not optimal.

[rahuman]

@JonB

1 Mb increasing means previously written like this.

QByteArray RxBuffer = socket->readAll();
qDebug() <<"Reading data......" <<RxBuffer;
QFile file ("C:/123.bin");
if(file.open(QIODevice::WriteOnly | QIODevice::Append | QIODevice::Text))
{
QTextStream out(&file);
out <<RxBuffer;
}

And changed like

QByteArray RxBuffer = socket->readAll();
qDebug() <<"Reading data......" <<RxBuffer;
QFile file ("C:/123.bin");
file.open(QIODevice::WriteOnly | QIODevice::Append | QIODevice::Text)
QTextStream out(&file);
out <<RxBuffer;

[JonB]

@rahuman
This says nothing about 1MB, one way or the other.

If the second "changed like" is supposed to address "Continually re-opening a file each time you want to append to it", it does not.

[Christian Ehrlicher]

Why do you use a QTextStream which converts your QByteArray into a QString and then back to a QByteArray (when it's written to disk). Simply using QFile::write() should be enough here.

[Kent-Dorfman]

If you learn nothing else, understand that file IO speeds are not guaranteed. If is dependent upon the OS level IO your system supports, availaiblity of buffer/cache, and physical write speeds of the media. Waht happens when you compare the write speeds of rotational disk to an SSD and then to a ramdisk?...and like others said...don't open/close the file on every transaction