reading file line by line crashes



  • Hello everyone,

    I've a program that open a file (380 Mo) into a QTextStream and read data line by line with the readLine function.
    Recenly a have encoutred a crash problem when reading data and I've tried some manipulation by the crash persists.
    Note that every line contains some big data and lines ends with \n\r.

    Below my code :

    QFile srfile(mScriptFileName);  
    
    	if (!srfile.open(QIODevice::ReadOnly | QIODevice::Text))
    	{
    		mExistMajData = false;	
    		mIsError = true;
    		return ;
    	}
    
    	QTextStream in(&srfile);
    
    	while (!in.atEnd() && !mIsError) 
    	{
    		qApp->processEvents(QEventLoop::WaitForMoreEvents);
    
    		//lecture des données lignes par ligne		
    		try 
    		{ 
    			line_data = in.readLine(); // it crashes here (eg. PJ)
    			record_data = line_data.split("<|>");
    			qDebug() << "Fin lecture d'une ligne n°" << QString::number(++nb_cmd) << record_data[3] << QTime::currentTime().toString("hh:mm") << endl ;
    		} 
    		catch ( const std::bad_alloc & ) 
    		{ 
    			 std::cerr << "Erreur : mémoire insuffisante.\n"; 
    		} 
    		catch ( const std::out_of_range & ) 
    		{ 
    			 std::cerr << "Erreur : débordement de mémoire.\n"; 
    		} 
    		catch ( const std::exception & e ) 
    		{ 
    			std::cerr << e.what(); 
    		}
    

    The call stack indicated the following :

     	msvcr80d.dll!memcpy(unsigned char * dst=0x28446f80, unsigned char * src=0x01b26ffa, unsigned long count=32768)  Ligne 188	Asm
     	QtCored4.dll!QString::append(const QString & str={...})  Ligne 1244 + 0x2d octets	C++
     	QtCored4.dll!QString::operator+=(const QString & s={...})  Ligne 225 + 0x13 octets	C++
     	QtCored4.dll!QTextStreamPrivate::fillReadBuffer(__int64 maxBytes=-1)  Ligne 568 + 0x5b octets	C++
     	QtCored4.dll!QTextStreamPrivate::scan(const QChar * * ptr=0x0014a42c, int * length=0x0014a434, int maxlen=0, QTextStreamPrivate::TokenDelimiter delimiter=EndOfLine)  Ligne 730 + 0x2a octets	C++
     	QtCored4.dll!QTextStream::readLine(__int64 maxlen=0)  Ligne 1588 + 0x16 octets	C++
    >	TheriaqueUpdater.exe!TheriaUpdater::NewUpdateBase()  Ligne 3377 + 0x14 octets	C++
     	TheriaqueUpdater.exe!TheriaUpdater::start()  Ligne 1479	C++
     	TheriaqueUpdater.exe!TheriaUpdater::Update()  Ligne 1199	C++
     	TheriaqueUpdater.exe!TheriaUpdater::qt_metacall(QMetaObject::Call _c=InvokeMetaMethod, int _id=8, void * * _a=0x0014b154)  Ligne 96 + 0x8 octets	C++
     	QtCored4.dll!QMetaObject::activate(QObject * sender=0x01aa4888, int from_signal_index=29, int to_signal_index=30, void * * argv=0x0014b154)  Ligne 3063 + 0x46 octets	C++
     	QtCored4.dll!QMetaObject::activate(QObject * sender=0x01aa4888, const QMetaObject * m=0x657a2374, int from_local_signal_index=2, int to_local_signal_index=3, void * * argv=0x0014b154)  Ligne 3142 + 0x15 octets	C++
     	QtGuid4.dll!QAbstractButton::clicked(bool _t1=false)  Ligne 180 + 0x17 octets	C++
     	QtGuid4.dll!QAbstractButtonPrivate::emitClicked()  Ligne 517	C++
     	QtGuid4.dll!QAbstractButtonPrivate::click()  Ligne 509	C++
     	QtGuid4.dll!QAbstractButton::mouseReleaseEvent(QMouseEvent * e=0x0014b788)  Ligne 1086	C++
     	QtGuid4.dll!QWidget::event(QEvent * event=0x0014b788)  Ligne 6064	C++
     	QtGuid4.dll!QAbstractButton::event(QEvent * e=0x0014b788)  Ligne 1048	C++
     	QtGuid4.dll!QPushButton::event(QEvent * e=0x0014b788)  Ligne 651	C++
     	QtGuid4.dll!QApplicationPrivate::notify_helper(QObject * receiver=0x01aa4888, QEvent * e=0x0014b788)  Ligne 3541 + 0x11 octets	C++
     	QtGuid4.dll!QApplication::notify(QObject * receiver=0x01aa4888, QEvent * e=0x0014b788)  Ligne 3240 + 0x2f octets	C++
     	QtCored4.dll!QCoreApplication::notifyInternal(QObject * receiver=0x01aa4888, QEvent * event=0x0014b788)  Ligne 516	C++
     	QtCored4.dll!QCoreApplication::sendSpontaneousEvent(QObject * receiver=0x01aa4888, QEvent * event=0x0014b788)  Ligne 188 + 0x38 octets	C++
    
    

    I've reinstalled the VC redistribuable. Rebuil all the project. Search if I can free QTextStream buffer but the problem persists

    Can anyone tell me what wrong et what I must to do to resolve this. Some

    Thanks in advance


  • Qt Champions 2018

    @mourad_bilog said in reading file line by line crashes:

    crash problem

    What kind of crash is it? Out of memory? Or something else?


  • Moderators

    hi @mourad_bilog

    Even if this doesn't crash your application, you should check if recordData has indeed more than 3 entries!

    qDebug() << "Fin lecture d'une ligne n°" << QString::number(++nb_cmd) << record_data[3] << QTime::currentTime().toString("hh:mm") << endl ;

    besides that, I see this

    qApp->processEvents(QEventLoop::WaitForMoreEvents);

    potentially causing serious issues, do not spin the event loop manually!



  • First, Thanks for reply.

    @jsulm it's a memory crash. Below the error occured :

    Exception de première chance à 0x0fcbd5be (msvcr80d.dll) dans TheriaqueUpdater.exe : 0xC0000005: Violation d'accès lors de l'écriture à l'emplacement 0x2804e000.
    Exception non gérée à 0x0fcbd5be (msvcr80d.dll) dans TheriaqueUpdater.exe : 0xC0000005: Violation d'accès lors de l'écriture à l'emplacement 0x2804e000.
    

    @J-Hilk the crash occus on in.readLine. The recod data has 6 entries and it's shown evey time.
    I've commented the eventLoop but the application still crashes.

    Note that line data lengths can be around 1 milion carachtars.


  • Qt Champions 2018

    @mourad_bilog said in reading file line by line crashes:

    Note that line data lengths can be around 1 milion carachtars

    How long is the line you're reading when the app is crashing?


  • Moderators

    @mourad_bilog said in reading file line by line crashes:

    @J-Hilk the crash occus on in.readLine. The recod data has 6 entries and it's shown evey time.

    do not base this solely on faith:

    if(record_data.size()> 3)
    qDebug() << "Fin lecture d'une ligne n°" << QString::number(++nb_cmd) << record_data[3] << QTime::currentTime().toString("hh:mm") << endl ;
     else
     qDebug() << "would have crashed";
    


  • @jsulm said in reading file line by line crashes:

    How long is the line you're reading when the app is crashing?

    About 1 200 000 charactars.



  • @J-Hilk
    I've commented that line but the same occurs.


  • Moderators

    @mourad_bilog alight thanks thumbs up

    question , is this Qt4?
    and is this a multi threaded project, and is do you read and write to that file from different places ?



  • @J.Hilk
    question , is this Qt4? yes. but the program runs correctly for some years.

    and is this a multi threaded project, : No, it's one thread.

    and is do you read and write to that file from different places : No, I just read from that file. I just open it in "ReadOnly" mode.



  • I've read in a post how has similar problem that he has solved the problem using file.readLine no QTextStream.readLine(). What is the difference between the 2 functions ?


  • Qt Champions 2018

    @mourad_bilog said in reading file line by line crashes:

    1 200 000

    really one line?



  • @Christian-Ehrlicher there's many lines witch everyone exceed 1 milion charactars.
    Is this a problem ? That can exceed what QTextStream buffer contains ?


  • Qt Champions 2018

    At least I would consider this - it's a 2.4MB huge QString, should be ok but who knows. I would simply try QFile::readLine(), split the QByteArray and convert to QString as late as possible (if needed at all)



  • Checking Windows events journal, it indicates that msvcr causes this crash, that's true ?

    Nom de l’application défaillante Updater.exe, version : 2.7.4.0, horodatage : 0x5a018f88
    Nom du module défaillant : MSVCR80.dll, version : 8.0.50727.9445, horodatage : 0x5a7bc74c
    Code d’exception : 0xc0000005
    Décalage d’erreur : 0x0001508e
    ID du processus défaillant : 0x19a0
    Heure de début de l’application défaillante : 0x01d4d8a5bb99dd52
    Chemin d’accès du module défaillant: C:\WINDOWS\WinSxS\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.9445_none_d08c58b4442ba54f\MSVCR80.dll
    ID de rapport : 6c470b7e-a3b2-4584-b404-3d7957545ead
    Nom complet du package défaillant : 
    ID de l’application relative au package défaillant : 
    

    I've re-installed Visual redistribuable 2005 but the problem persists. Note that I dont found msvcr80 in the system32 not like other versions msvcr71, msvcr10, etc.

    Does this the problem ?


  • Lifetime Qt Champion

    Hi,

    Did you monitor your application RAM usage ?



  • Finally I changed reading mode to read directly from the data file. It's most faster and just I was care to encode data when reading to keep special charcatars.

    Thanks all for your ideas