Unsolved Crash in reading a big file
-
I have tried this code , it does not crash the tool hangs do we have solution so that I can remove the hang and the files load incrementatlly on moviing the scrollbar so that user is able to see some actions
about an hour ago Reputation: 1 | Posts: 144
-
My guess is that your test at the end of while loop does not work, because the string has been initialized before.
Change
} while (!line.isNull());
to
} while ( ! ts.atEnd() );
-
One more question why does QScintilla crashes if we are consuming 1 GB of memory for this code
class myClass: public QsciScintilla {
public:
void readFile();
}
myClass::readFile() {
if (FILE* fp = UFile::open(ofilename.toLatin1(), "r")) {
QTextStream ts(fp, QIODevice::ReadOnly);
setText(ts.readAll());----------------- Crash
setModified(FALSE);
UFile::close(fp);
d_filename = ofilename;
emit fileNameChanged(ofilename);
}
} -
It still hangs even after this change
hange
} while (!line.isNull());
to} while ( ! ts.atEnd()
-
Hi,
Out of curiosity, why not use QFile ?
As for the loop, you have an example of it in QTextStream detailed documentation.
-
I have now changes the code from
if (FILE* fp = MYFile::open(ofilename.toLatin1(), "r")) { QTextStream ts(fp, QIODevice::ReadOnly); setText(ts.readAll()); setModified(FALSE); MyFile::close(fp); d_filename = ofilename; emit fileNameChanged(ofilename); }
to
QFile data(ofilename.toLatin1().constData()); if (data.open(QFile::ReadOnly)) { QTextStream ts(&data); setText(ts.readAll()); setModified(FALSE); d_filename = ofilename; emit fileNameChanged(ofilename); }
and the hang goes . Can someone explain in details why does hang go in this case
-
You do realize that you are reading the whole file in one go with that code ?
-
Do you mean
in this codeif (FILE* fp = MYFile::open(ofilename.toLatin1(), "r")) {------------------------------
QTextStream ts(fp, QIODevice::ReadOnly);----------------------------
setText(ts.readAll());
setModified(FALSE);
MyFile::close(fp);
d_filename = ofilename;
emit fileNameChanged(ofilename);
}We are reading a big file in one go vs
and the following code we are not reading whole big file in one go
QFile data(ofilename.toLatin1().constData());
if (data.open(QFile::ReadOnly)) { -------------------------------------in this code the
QTextStream ts(&data);
setText(ts.readAll());
setModified(FALSE);
d_filename = ofilename;
emit fileNameChanged(ofilename);
} -
-
This post is deleted! -
After debugging further , I found you that the crash is happening when QTextStream .readAll is assigned to QString
if (FILE* fp = UFile::open(ofilename.toLatin1(),"r")) {
QTextStream ts(fp, QIODevice::ReadOnly);
QString s = ts.readAll();
} */but if we assign it to then the crash goes away
QFile data(ofilename.toLatin1().constData());
QString s = data.readAll();Whyy readAll from QFile and assigning to Qstring does not crash and why does QString s = ts.readAll(); crashes
-
Did you read the docs for QTextStream::readAll() ?
It says:
QString QTextStream::readAll() Reads the entire content of the stream, and returns it as a QString. Avoid this function when working on large files, as it will consume a significant amount of memory. Calling readLine() is better if you do not know how much data is available.
Possibly the implementation is consuming too much space during reading. There are always different ways to implement things.
Just reading the crystal ball. FILE is a C construct. Certainly you can easily use it in C++. However, you just have a file handle to read from. AFAIK all information especially file size is available after you have read the complete file. Basically you need to read and fill the buffer which size is not known at the start. Some containers have an issue. String containers rely on allocation of continuous memory, because of pointer access to whole container. Consequently the routine will read a chunk of data until buffer is full. In order to read more it has to allocate a new larger chunk. A common technique is doubling the already allocated memory in order to avoid too many allocation. However, this has a disadvantage.
Assume first size equals 1 MB
at next it will allocate 2 MB,
copy the 1 MB,
release the 1 MB
continue and read another 1 MB.
In between it uses 1.5 of the final size. Since you are really at the system limits with your file size, this might be the case. Just a single byte to read in addition the routine triple at least for a moment the required allocated memory and you are gone.QFile knows probably more about the file than FILE*. It knows the file size and readAll is simply able to allocate the proper size without guessing.
BTW you should be able to see this in a task manager for windows for instance. Since the reading time is considerable you should be able to follow the memory allocation process.