Optimal buffer size
-
Hi All
I am using followingclass myScintilla:public QScintilla {
public:
readFile();private:
};
myScintilla:: readFile() {
if (FILE* fp = f:open(ofilename.toLatin1(), "r")) {
QTextStream ts(fp, QIODevice::ReadOnly);
int bufferSize =1024*1024;
do {
QString s = ts.read(bufferSize);
append(s);
} while(!ts.atEnd());}
Since we have some upper limit for memory of QString . I want to know what is limit of memory for QStrign on a 32 bit machine and
I want to know what is the optimal bufferSize in the above code so that we do not get any runtime issue .
My Experiments says
for 800 MB file : for bufferSize size of 1MB (1024*1024;) it took 13 seconds
For 800 MB file . for bufferSize size of 8KB it took 27 secondsCan anyone suggest optimal bufferSize in the above code so that we do not get any runtime issu
-
@Qt-Enthusiast
Hello,Since we have some upper limit for memory of QString . I want to know what is limit of memory for QStrign on a 32 bit machine
Theoretically the whole addressable memory (4GB). In practice its somewhat smaller though.
I want to know what is the optimal bufferSize in the above code so that we do not get any runtime issue .
Generally bigger buffers tend to perform faster, but there's saturation to that effect. It depends on the application itself, the OS, the drivers and ultimately the hardware, so there's no "correct" size. For files of the size up to 1-2 GB I'd think 1MB is reasonable, for larger files I'd consider making the buffer larger as well.
Kind regards.
-
Also it will be helpful what is maximum limit for QTextStream
regards
Roshni -
Also it will be helpful what is maximum limit for QTextStream
There's no such thing. There's an internal buffer in
QTextStream
but it bears no consequence here. It'll be flushed on device close or onQTextStream::endl
/QTextStream::flush
. So the stream has no notion of size, it's a stream.Kind regards.
-
Just to clear my understanding what if , if I have a file of size 20 GB and I do
FILE *fp = fopen("myfile.txt","r");
QTextStream ts(fp, QIODevice::ReadOnly);is there internal buffer has memory limit to hold the data
-
Hi
I tried this to calculate the buffer size/ Can any one comment on it// myEditor.h
class myEditor : QScintilla {
public:
readFile();
};#include "myEditor.h"
// myEditor.cc
myEditor::readFile() {
FILE* fp = fopen("mynew.v","r"):
QTextStream ts(fp, QIODevice::ReadOnly);
/* reading the text stream buffer by buffer
bufferSize is calculated using following formula
2 to power(k) * n = 2 to power 31*
where n is size of each block in linux filesystem
n is 4096 in my case and
since ts.read returns QString and return value of QString::size is int
thus 31 is for that limit */int bufferSize =(1024* 1024)/2;
do {
QString s = ts.read(bufferSize);
append(s);
} while(!ts.atEnd());
}Please comment
-
is there internal buffer has memory limit to hold the data
There's an internal buffer, but it's for temporary storage and will grow when needed. The data itself is on the disk until it's read.
tried this to calculate the buffer size
I have no clue what this code is supposed to do, or why are you insistent on knowing how large the text stream buffer is.
-
@kshegunov said:
I have no clue what this code is supposed to do, or why are you insistent on knowing how large the text stream buffer is.
Because in our case File sizes can go to 2 -3 GBS . so I am insistent so that I want to know that the tool does not crash even for the following line
QTextStream ts(fp, QIODevice::ReadOnly);
-
@Qt-Enthusiast
Sorry for the late reply, I somehow missed the notification.Because in our case File sizes can go to 2 -3 GBS
The problem is not with the
QTextStream
in this case. It's with how much of the file you want to read. On a 32 bit OS this is close to impossible, because you'd be pushing the memory limit reading a file that big in one go. An alternative to reading the whole file is to process it in parts instead. Perhaps theQFile
's memory mapping API might be of use in your case.I want to know that the tool does not crash even for the following line
QTextStream ts(fp, QIODevice::ReadOnly);
This does next to nothing. It will create an object that wraps around the
QIODevice
/FILE *
and will have an empty buffer. As I said it's a stream, it doesn't read anything by itself, and it has no notion of size. Anything you read from the file will be returned, anything you write to the stream will be buffered and actually written when the stream is flushed. That's all. There's no big secret with the buffer or its size. It's just put there for a minor optimization (and has no bearing on your problem here, as I stated before).Kind regards.