reading file line by line crashes

mourad_bilog

Hello everyone,

I've a program that open a file (380 Mo) into a QTextStream and read data line by line with the readLine function.
Recenly a have encoutred a crash problem when reading data and I've tried some manipulation by the crash persists.
Note that every line contains some big data and lines ends with \n\r.

Below my code :

QFile srfile(mScriptFileName);  

	if (!srfile.open(QIODevice::ReadOnly | QIODevice::Text))
	{
		mExistMajData = false;	
		mIsError = true;
		return ;
	}

	QTextStream in(&srfile);

	while (!in.atEnd() && !mIsError) 
	{
		qApp->processEvents(QEventLoop::WaitForMoreEvents);

		//lecture des données lignes par ligne		
		try 
		{ 
			line_data = in.readLine(); // it crashes here (eg. PJ)
			record_data = line_data.split("<|>");
			qDebug() << "Fin lecture d'une ligne n°" << QString::number(++nb_cmd) << record_data[3] << QTime::currentTime().toString("hh:mm") << endl ;
		} 
		catch ( const std::bad_alloc & ) 
		{ 
			 std::cerr << "Erreur : mémoire insuffisante.\n"; 
		} 
		catch ( const std::out_of_range & ) 
		{ 
			 std::cerr << "Erreur : débordement de mémoire.\n"; 
		} 
		catch ( const std::exception & e ) 
		{ 
			std::cerr << e.what(); 
		}

The call stack indicated the following :

 	msvcr80d.dll!memcpy(unsigned char * dst=0x28446f80, unsigned char * src=0x01b26ffa, unsigned long count=32768)  Ligne 188	Asm
 	QtCored4.dll!QString::append(const QString & str={...})  Ligne 1244 + 0x2d octets	C++
 	QtCored4.dll!QString::operator+=(const QString & s={...})  Ligne 225 + 0x13 octets	C++
 	QtCored4.dll!QTextStreamPrivate::fillReadBuffer(__int64 maxBytes=-1)  Ligne 568 + 0x5b octets	C++
 	QtCored4.dll!QTextStreamPrivate::scan(const QChar * * ptr=0x0014a42c, int * length=0x0014a434, int maxlen=0, QTextStreamPrivate::TokenDelimiter delimiter=EndOfLine)  Ligne 730 + 0x2a octets	C++
 	QtCored4.dll!QTextStream::readLine(__int64 maxlen=0)  Ligne 1588 + 0x16 octets	C++
>	TheriaqueUpdater.exe!TheriaUpdater::NewUpdateBase()  Ligne 3377 + 0x14 octets	C++
 	TheriaqueUpdater.exe!TheriaUpdater::start()  Ligne 1479	C++
 	TheriaqueUpdater.exe!TheriaUpdater::Update()  Ligne 1199	C++
 	TheriaqueUpdater.exe!TheriaUpdater::qt_metacall(QMetaObject::Call _c=InvokeMetaMethod, int _id=8, void * * _a=0x0014b154)  Ligne 96 + 0x8 octets	C++
 	QtCored4.dll!QMetaObject::activate(QObject * sender=0x01aa4888, int from_signal_index=29, int to_signal_index=30, void * * argv=0x0014b154)  Ligne 3063 + 0x46 octets	C++
 	QtCored4.dll!QMetaObject::activate(QObject * sender=0x01aa4888, const QMetaObject * m=0x657a2374, int from_local_signal_index=2, int to_local_signal_index=3, void * * argv=0x0014b154)  Ligne 3142 + 0x15 octets	C++
 	QtGuid4.dll!QAbstractButton::clicked(bool _t1=false)  Ligne 180 + 0x17 octets	C++
 	QtGuid4.dll!QAbstractButtonPrivate::emitClicked()  Ligne 517	C++
 	QtGuid4.dll!QAbstractButtonPrivate::click()  Ligne 509	C++
 	QtGuid4.dll!QAbstractButton::mouseReleaseEvent(QMouseEvent * e=0x0014b788)  Ligne 1086	C++
 	QtGuid4.dll!QWidget::event(QEvent * event=0x0014b788)  Ligne 6064	C++
 	QtGuid4.dll!QAbstractButton::event(QEvent * e=0x0014b788)  Ligne 1048	C++
 	QtGuid4.dll!QPushButton::event(QEvent * e=0x0014b788)  Ligne 651	C++
 	QtGuid4.dll!QApplicationPrivate::notify_helper(QObject * receiver=0x01aa4888, QEvent * e=0x0014b788)  Ligne 3541 + 0x11 octets	C++
 	QtGuid4.dll!QApplication::notify(QObject * receiver=0x01aa4888, QEvent * e=0x0014b788)  Ligne 3240 + 0x2f octets	C++
 	QtCored4.dll!QCoreApplication::notifyInternal(QObject * receiver=0x01aa4888, QEvent * event=0x0014b788)  Ligne 516	C++
 	QtCored4.dll!QCoreApplication::sendSpontaneousEvent(QObject * receiver=0x01aa4888, QEvent * event=0x0014b788)  Ligne 188 + 0x38 octets	C++

I've reinstalled the VC redistribuable. Rebuil all the project. Search if I can free QTextStream buffer but the problem persists

Can anyone tell me what wrong et what I must to do to resolve this. Some

Thanks in advance

jsulm

@mourad_bilog said in reading file line by line crashes:

crash problem

What kind of crash is it? Out of memory? Or something else?

J.Hilk

hi @mourad_bilog

Even if this doesn't crash your application, you should check if recordData has indeed more than 3 entries!

qDebug() << "Fin lecture d'une ligne n°" << QString::number(++nb_cmd) << record_data[3] << QTime::currentTime().toString("hh:mm") << endl ;

besides that, I see this

qApp->processEvents(QEventLoop::WaitForMoreEvents);

potentially causing serious issues, do not spin the event loop manually!

mourad_bilog

First, Thanks for reply.

@jsulm it's a memory crash. Below the error occured :

Exception de première chance à 0x0fcbd5be (msvcr80d.dll) dans TheriaqueUpdater.exe : 0xC0000005: Violation d'accès lors de l'écriture à l'emplacement 0x2804e000.
Exception non gérée à 0x0fcbd5be (msvcr80d.dll) dans TheriaqueUpdater.exe : 0xC0000005: Violation d'accès lors de l'écriture à l'emplacement 0x2804e000.

@J-Hilk the crash occus on in.readLine. The recod data has 6 entries and it's shown evey time.
I've commented the eventLoop but the application still crashes.

Note that line data lengths can be around 1 milion carachtars.

jsulm

@mourad_bilog said in reading file line by line crashes:

Note that line data lengths can be around 1 milion carachtars

How long is the line you're reading when the app is crashing?

J.Hilk

@mourad_bilog said in reading file line by line crashes:

@J-Hilk the crash occus on in.readLine. The recod data has 6 entries and it's shown evey time.

do not base this solely on faith:

if(record_data.size()> 3)
qDebug() << "Fin lecture d'une ligne n°" << QString::number(++nb_cmd) << record_data[3] << QTime::currentTime().toString("hh:mm") << endl ;
 else
 qDebug() << "would have crashed";

mourad_bilog

@jsulm said in reading file line by line crashes:

How long is the line you're reading when the app is crashing?

About 1 200 000 charactars.

mourad_bilog

@J-Hilk
I've commented that line but the same occurs.

J.Hilk

@mourad_bilog alight thanks thumbs up

question , is this Qt4?
and is this a multi threaded project, and is do you read and write to that file from different places ?

mourad_bilog

@J.Hilk
question , is this Qt4? yes. but the program runs correctly for some years.

and is this a multi threaded project, : No, it's one thread.

and is do you read and write to that file from different places : No, I just read from that file. I just open it in "ReadOnly" mode.

mourad_bilog

I've read in a post how has similar problem that he has solved the problem using file.readLine no QTextStream.readLine(). What is the difference between the 2 functions ?

Christian Ehrlicher

@mourad_bilog said in reading file line by line crashes:

1 200 000

really one line?

mourad_bilog

@Christian-Ehrlicher there's many lines witch everyone exceed 1 milion charactars.
Is this a problem ? That can exceed what QTextStream buffer contains ?

Christian Ehrlicher

At least I would consider this - it's a 2.4MB huge QString, should be ok but who knows. I would simply try QFile::readLine(), split the QByteArray and convert to QString as late as possible (if needed at all)

mourad_bilog

Checking Windows events journal, it indicates that msvcr causes this crash, that's true ?

Nom de l’application défaillante Updater.exe, version : 2.7.4.0, horodatage : 0x5a018f88
Nom du module défaillant : MSVCR80.dll, version : 8.0.50727.9445, horodatage : 0x5a7bc74c
Code d’exception : 0xc0000005
Décalage d’erreur : 0x0001508e
ID du processus défaillant : 0x19a0
Heure de début de l’application défaillante : 0x01d4d8a5bb99dd52
Chemin d’accès du module défaillant: C:\WINDOWS\WinSxS\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.9445_none_d08c58b4442ba54f\MSVCR80.dll
ID de rapport : 6c470b7e-a3b2-4584-b404-3d7957545ead
Nom complet du package défaillant : 
ID de l’application relative au package défaillant :

I've re-installed Visual redistribuable 2005 but the problem persists. Note that I dont found msvcr80 in the system32 not like other versions msvcr71, msvcr10, etc.

Does this the problem ?

SGaist

Hi,

Did you monitor your application RAM usage ?

mourad_bilog

Finally I changed reading mode to read directly from the data file. It's most faster and just I was care to encode data when reading to keep special charcatars.

Thanks all for your ideas