Optimal buffer size

[Qt Enthusiast]

Hi All
I am using following

class myScintilla:public QScintilla {
public:
readFile();

private:

};

myScintilla:: readFile() {

if (FILE* fp = f:open(ofilename.toLatin1(), "r")) {
QTextStream ts(fp, QIODevice::ReadOnly);
int bufferSize =1024*1024;
do {
QString s = ts.read(bufferSize);
append(s);
} while(!ts.atEnd());

}

Since we have some upper limit for memory of QString . I want to know what is limit of memory for QStrign on a 32 bit machine and

I want to know what is the optimal bufferSize in the above code so that we do not get any runtime issue .

My Experiments says

for 800 MB file : for bufferSize size of 1MB (1024*1024;) it took 13 seconds
For 800 MB file . for bufferSize size of 8KB it took 27 seconds

Can anyone suggest optimal bufferSize in the above code so that we do not get any runtime issu

[kshegunov]

@Qt-Enthusiast
Hello,

Since we have some upper limit for memory of QString . I want to know what is limit of memory for QStrign on a 32 bit machine

Theoretically the whole addressable memory (4GB). In practice its somewhat smaller though.

I want to know what is the optimal bufferSize in the above code so that we do not get any runtime issue .

Generally bigger buffers tend to perform faster, but there's saturation to that effect. It depends on the application itself, the OS, the drivers and ultimately the hardware, so there's no "correct" size. For files of the size up to 1-2 GB I'd think 1MB is reasonable, for larger files I'd consider making the buffer larger as well.

Kind regards.

[Qt Enthusiast]

Also it will be helpful what is maximum limit for QTextStream

regards
Roshni

[kshegunov]

@Qt-Enthusiast

Also it will be helpful what is maximum limit for QTextStream

There's no such thing. There's an internal buffer in QTextStream but it bears no consequence here. It'll be flushed on device close or on QTextStream::endl/QTextStream::flush. So the stream has no notion of size, it's a stream.

Kind regards.

[Qt Enthusiast]

Just to clear my understanding what if , if I have a file of size 20 GB and I do

FILE *fp = fopen("myfile.txt","r");
QTextStream ts(fp, QIODevice::ReadOnly);

is there internal buffer has memory limit to hold the data

[Qt Enthusiast]

Hi
I tried this to calculate the buffer size/ Can any one comment on it

// myEditor.h
class myEditor : QScintilla {
public:
readFile();
};

#include "myEditor.h"
// myEditor.cc
myEditor::readFile() {
FILE* fp = fopen("mynew.v","r"):
QTextStream ts(fp, QIODevice::ReadOnly);
/* reading the text stream buffer by buffer
bufferSize is calculated using following formula
2 to power(k) * n = 2 to power 31*
where n is size of each block in linux filesystem
n is 4096 in my case and
since ts.read returns QString and return value of QString::size is int
thus 31 is for that limit */

int bufferSize =(1024* 1024)/2;
do {
QString s = ts.read(bufferSize);
append(s);
} while(!ts.atEnd());
}

Please comment

[kshegunov]

@Qt-Enthusiast

is there internal buffer has memory limit to hold the data

There's an internal buffer, but it's for temporary storage and will grow when needed. The data itself is on the disk until it's read.

tried this to calculate the buffer size

I have no clue what this code is supposed to do, or why are you insistent on knowing how large the text stream buffer is.

[Qt Enthusiast]

@kshegunov said:

I have no clue what this code is supposed to do, or why are you insistent on knowing how large the text stream buffer is.

Because in our case File sizes can go to 2 -3 GBS . so I am insistent so that I want to know that the tool does not crash even for the following line

QTextStream ts(fp, QIODevice::ReadOnly);

[kshegunov]

@Qt-Enthusiast
Sorry for the late reply, I somehow missed the notification.

Because in our case File sizes can go to 2 -3 GBS

The problem is not with the QTextStream in this case. It's with how much of the file you want to read. On a 32 bit OS this is close to impossible, because you'd be pushing the memory limit reading a file that big in one go. An alternative to reading the whole file is to process it in parts instead. Perhaps the QFile's memory mapping API might be of use in your case.

I want to know that the tool does not crash even for the following line

QTextStream ts(fp, QIODevice::ReadOnly);

This does next to nothing. It will create an object that wraps around the QIODevice/FILE * and will have an empty buffer. As I said it's a stream, it doesn't read anything by itself, and it has no notion of size. Anything you read from the file will be returned, anything you write to the stream will be buffered and actually written when the stream is flushed. That's all. There's no big secret with the buffer or its size. It's just put there for a minor optimization (and has no bearing on your problem here, as I stated before).

Kind regards.