Crash in reading a big file

Qt Enthusiast

The second version you have posted reads line by line and adds the stuff to your storage. Assuming a pretty long line of 1 kB this is peanuts on each read. When you have read your stuff you have increased your memory requirement by appr 480 MB. So you are increasing your memory step by step.

When you are working with computer with 32bit OS (respectively 32bit Qt libs) you are temporarily pretty close to the limits with the first approach. The second approach looks saver from the part of code you are posting.

This is not

I have tried this code , it does not crash the tool hangs do we have solution so that I can remove the hang and the files load incrementatlly on moviing the scrollbar so that user is able to see some actions

Qt Enthusiast

I have tried this code , it does not crash the tool hangs do we have solution so that I can remove the hang and the files load incrementatlly on moviing the scrollbar so that user is able to see some actions

about an hour ago Reputation: 1 | Posts: 144

koahnig

@Qt-Enthusiast

My guess is that your test at the end of while loop does not work, because the string has been initialized before.

Change

 } while (!line.isNull());

to

} while ( ! ts.atEnd() );

Qt Enthusiast

One more question why does QScintilla crashes if we are consuming 1 GB of memory for this code

class myClass: public QsciScintilla {
public:
void readFile();
}
myClass::readFile() {
if (FILE* fp = UFile::open(ofilename.toLatin1(), "r")) {
QTextStream ts(fp, QIODevice::ReadOnly);
setText(ts.readAll());----------------- Crash
setModified(FALSE);
UFile::close(fp);
d_filename = ofilename;
emit fileNameChanged(ofilename);
}
}

Qt Enthusiast

It still hangs even after this change

hange

} while (!line.isNull());
to

} while ( ! ts.atEnd()

SGaist

Hi,

Out of curiosity, why not use QFile ?

As for the loop, you have an example of it in QTextStream detailed documentation.

Qt Enthusiast

I have now changes the code from

 if (FILE* fp = MYFile::open(ofilename.toLatin1(), "r")) {
     QTextStream ts(fp, QIODevice::ReadOnly);
     setText(ts.readAll());
     setModified(FALSE);
     MyFile::close(fp);
     d_filename = ofilename;
     emit fileNameChanged(ofilename);
     }

to

  QFile data(ofilename.toLatin1().constData());
  if (data.open(QFile::ReadOnly))  {
    QTextStream ts(&data);
    setText(ts.readAll());
    setModified(FALSE);
    d_filename = ofilename;
    emit fileNameChanged(ofilename);
    }

and the hang goes . Can someone explain in details why does hang go in this case

SGaist

You do realize that you are reading the whole file in one go with that code ?

Qt Enthusiast

Do you mean
in this code

if (FILE* fp = MYFile::open(ofilename.toLatin1(), "r")) {------------------------------
QTextStream ts(fp, QIODevice::ReadOnly);----------------------------
setText(ts.readAll());
setModified(FALSE);
MyFile::close(fp);
d_filename = ofilename;
emit fileNameChanged(ofilename);
}

We are reading a big file in one go vs

and the following code we are not reading whole big file in one go

QFile data(ofilename.toLatin1().constData());
if (data.open(QFile::ReadOnly)) { -------------------------------------in this code the
QTextStream ts(&data);
setText(ts.readAll());
setModified(FALSE);
d_filename = ofilename;
emit fileNameChanged(ofilename);
}

koahnig

@Qt-Enthusiast

In both versions you have ts.readAll().

Therefore you do!

Qt Enthusiast

This post is deleted!

Qt Enthusiast

After debugging further , I found you that the crash is happening when QTextStream .readAll is assigned to QString

if (FILE* fp = UFile::open(ofilename.toLatin1(),"r")) {
QTextStream ts(fp, QIODevice::ReadOnly);
QString s = ts.readAll();
} */

but if we assign it to then the crash goes away

QFile data(ofilename.toLatin1().constData());
QString s = data.readAll();

Whyy readAll from QFile and assigning to Qstring does not crash and why does QString s = ts.readAll(); crashes

koahnig

@Qt-Enthusiast

Did you read the docs for QTextStream::readAll() ?

It says:

QString QTextStream::readAll()

Reads the entire content of the stream, and returns it as a QString. Avoid this function when working on large files, as it will consume a significant amount of memory.

Calling readLine() is better if you do not know how much data is available.

Possibly the implementation is consuming too much space during reading. There are always different ways to implement things.

Just reading the crystal ball. FILE is a C construct. Certainly you can easily use it in C++. However, you just have a file handle to read from. AFAIK all information especially file size is available after you have read the complete file. Basically you need to read and fill the buffer which size is not known at the start. Some containers have an issue. String containers rely on allocation of continuous memory, because of pointer access to whole container. Consequently the routine will read a chunk of data until buffer is full. In order to read more it has to allocate a new larger chunk. A common technique is doubling the already allocated memory in order to avoid too many allocation. However, this has a disadvantage.
Assume first size equals 1 MB
at next it will allocate 2 MB,
copy the 1 MB,
release the 1 MB
continue and read another 1 MB.
In between it uses 1.5 of the final size. Since you are really at the system limits with your file size, this might be the case. Just a single byte to read in addition the routine triple at least for a moment the required allocated memory and you are gone.

QFile knows probably more about the file than FILE*. It knows the file size and readAll is simply able to allocate the proper size without guessing.

BTW you should be able to see this in a task manager for windows for instance. Since the reading time is considerable you should be able to follow the memory allocation process.