Important: Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

How to make QByteArray read Full Text file?



  • Hello to everyone, im trying to read full text files line by line using QByteArray. It does work but it only reads half of the text file. I have text files that have thousands of lines, but QByteArray dosent read all the lines, only a few. I've managed to read large files but using QByteArray only shows half of the file.. My code snippet is below... Thanks

    //your code here 
              QByteArray data;
    	  data.reserve(5000000);
              if (!file.open(QIODevice::ReadOnly | QIODevice::Text)) {
    		//QMessageBox::warning(this,"...","error in opening keyword file");
    		return;
    	}
    	else {
                         if (fileExt == "txt")
    		   {
    			 data = file.read(file.size());
    			 fileLine.append(data);
    			*fileList = fileLine.split(QRegExp("[\r\n]"), QString::SkipEmptyParts);
    			          if(fileList->size() >1000)
    			          {
    				           QMessageBox::warning(this, "...", "Max keyword list size: 1000");
    			         }
                                    else
    			        { 
    				             ui->lineEdit_Keyword_List_File_Location->setText(file.fileName());
    				            for (int i = 0; i <fileList->size(); i++) 
                                                 {
    					      options[4]->keywordLoadListOptions[fileList->at(i)] = 0;
    				             }
    			      }		
             }
    

  • Moderators

    How do you know it only reads half the file? I can't see what fileLine is, but if it's a string you have an issue there. QByteArrays are binary, so there can be data before a 0 or null terminator. If you get a 0 in a QString it's done. That's the full string regardless of what might be after that in the QByteArray.

    Anyway I don't have enough information to really diagnose it.

    Here is some simple code to read a whole text file:

    QFile f("myfile", QIODevice::ReadOnly | QIODevice::Text);
    QByteArray ba = f.readAll();
    qDebug() << "read " << ba.size << "bytes";
    

    That will read the whole file and tell you if it could read it all into a byte array. It basically does the same as data = file.read(file.size()). After that verify the bytes match. Then look at your fileLine.split code to see if it's losing data. Also that could be a LOT of data for split to work on. It will have some cpu impact and could hang your gui for a bit.

    If so your problem is more than likely something like I mentioned above.



  • This post is deleted!


  • @ambershark yes, *fileLine is a string pointer thats being initialized right above the other code. So I cant append a QByteArray to a string without any issues? Using your modified code Its giving me a bit more data but its still loosing data. I tried the following ::
    QString fileList = new QString();
    QByteArray ba = file.readAll();
    fileLine.append(ba);
    *fileList = fileLine.split(QRegExp("[\r\n]"), QString::SkipEmptyParts);
    QByteArray ba = file.readAll();
    qDebug() << ba.size(); // 617782
    qDebug() << file.size(); //647254

    The QByteArray object is loosing data, verses the actual file size data. Since im using *fileList as a string that holds the QByteArray data, is there possibly another alternative to get the QByteArray data as a string and split it without loosing any data.



  • @ambershark From your advice I was able to use QByteArrayList which gives me the full text file only when I use qDebug(). When I loop through the QbyteArrayList im able to see the full text file, and each line of the data only in qDebug(), but not in my gui. I have a text file of 7,000 lines of, and its only giving me 3,000. My code snippet is below

    QByteArray ba = file.readAll();
    QByteArrayList qba = ba.split('\n');
    for (int j = 0; j<qba.size(); j++)
    {
    QByteArray ba_data = qba.at(j);
    std::string stdString(ba_data.constData(), ba_data.length());
    options[4]->keywordLoadListOptions[QString::fromStdString(stdString)] = 0;
    qDebug() << QString::fromStdString(stdString);
    }


  • Lifetime Qt Champion

    @tshoats If it is a text file then why do you read it as QByteArray? You should use http://doc.qt.io/qt-5/qtextstream.html



  • @jsulm when i use QByteArray it dosent block or freeze my gui, since im working with large text files to read. The only problem is It only shows half of the text file data. I tried using QTextStream but it blocks my gui unlike QByteArray. Below is a snippet of the QTextStream example thats blocking my gui..

    if (!file.open(QIODevice::ReadOnly | QIODevice::Text)) {
    	//QMessageBox::warning(this,"...","error in opening keyword file");
    	return;
    }
    //QByteArray ba = file.readAll();
    QTextStream ts(&file);
    while (!ts.atEnd()) {
    	strings.append(ts.readLine());
    	list = strings.split(QRegExp("[\r\n]"), QString::SkipEmptyParts);
    	qDebug() << list;
    
    	for (int i = 0; i <list.size(); i++) {
    	    hashKeywordLoadList[list.at(i)] = 0;
    	options[4]->keywordLoadListOptions = hashKeywordLoadList;
    	}
    }
    

    file.close();



  • @tshoats

    • It won't be "using QByteArray versus QTextStream blocks the GUI", it will depend on what/how you do with it.
    • For example, you chose to go QByteArray ba = file.readAll(); but QTextStream::readLine(). These are very different. You could have used QTextStream::readAll() for comparison. So you're not comparing like-for-like.
    • If you're going to read line-by-line, and do something with the lines, and the file is large, you could put in judicious processEvents() so that the UI does not "freeze".
    • Your current example code seems (assuming strings is a QStringList variable?) to read one line, append it to an ever-growing list of strings, split that whole string, and keep examining the list as a whole. If that's true it's slow. Depends where strings comes from.

  • Moderators

    @tshoats said in How to make QByteArray read Full Text file?:

    @ambershark From your advice I was able to use QByteArrayList which gives me the full text file only when I use qDebug(). When I loop through the QbyteArrayList im able to see the full text file, and each line of the data only in qDebug(), but not in my gui. I have a text file of 7,000 lines of, and its only giving me 3,000. My code snippet is below

    QByteArray ba = file.readAll();
    QByteArrayList qba = ba.split('\n');
    for (int j = 0; j<qba.size(); j++)
    {
    QByteArray ba_data = qba.at(j);
    std::string stdString(ba_data.constData(), ba_data.length());
    options[4]->keywordLoadListOptions[QString::fromStdString(stdString)] = 0;
    qDebug() << QString::fromStdString(stdString);
    }

    So based on this your data should have been read properly. This means the problem is probably in your GUI code.

    Can you share that or at least describe what you are trying to do and what kind of data is in the text file/how are you trying to display it.

    If you give me an idea of what you are trying to do I could write you a quick example.

    Also, I didn't mean you can't put a QByteArray into a QString, was just mentioning that since QByteArray is binary and QString is not you could end up with a 0 in your byte array prematurely ending your string. If you are sure that the data read is all text data then you're pretty safe to put it in a QString.

    Moving it from a QString to a std::string is not what I meant to do. Drop the std::string and use QString if you want strings. In your code above you created a std::string that was a copy of the bytearray and then created a QString from the std::string. So you now have triple the memory allocated for a very large string. Not to mention the processor time required to copy those strings.


  • Moderators

    Wrote a quick little test and there should not be any need for a QByteArrayList. My test (shown below) reads a 1.3mb text file which is about 13k lines with no problem. If you are still reading wrong sizes in a single BA then I'm guessing there is actually a problem with your text file. Here is the test:

    #include <QCoreApplication>
    #include <QByteArray>
    #include <QFile>
    #include <QDebug>
    
    int main(int ac, char **av)
    {
        QCoreApplication app(ac, av);
        
        QFile f("/home/mike/last_backup.log");
        if (!f.open(QIODevice::ReadOnly | QIODevice::Text))
        {
            qDebug() << "failed to open";
            return 1;
        }
        
        QByteArray ba = f.readAll();
        qDebug() << "read" << ba.size();
        qDebug() << "file size is" << f.size();
        
        return 0;
    }
    

    And here is the output:

    read 1224526
    file size is 1224526
    

    One other thing I noticed, when you are reading via a QTextStream, which is a nice way to go line by line, you are appending to a strings variable, which seems wrong. Wouldn't you want strings = ts.readLine() rather than an append? Append would continually add line after line to that variable which would test keywords over and over (unintentionally?). Here is the code I'm talking about:

    QTextStream ts(&file);
    while (!ts.atEnd()) {
    	strings.append(ts.readLine());   <-- this line here
    	list = strings.split(QRegExp("[\r\n]"), QString::SkipEmptyParts);
    	qDebug() << list;
    
    	for (int i = 0; i <list.size(); i++) {
    	    hashKeywordLoadList[list.at(i)] = 0;
    	options[4]->keywordLoadListOptions = hashKeywordLoadList;
    	}
    }
    

    Edit:

    Tested with a 59mb file just to be sure everything was happy even on quite large text files and QByteArray was fine with that too:

    read 61226300
    file size is 61226300
    

    658850 lines in that one.



  • @ambershark , and @JonB ,thanks sooo much!!! You've been a great help. Yup you were totally right about strings and binary issues. I forgot in my header file, I was using a QHash<QString, int> keywordLoadListOptions, with QString as a type. This is where keywordLoadListOptions come into play. I changed my code to use QTextStream to read the file using file.readLine(). I then used processEvents() inside the while loop to prohibit my GUI from freezing. Some where along the lines my QString within my Qlist keywordLoadListOptions was ending my string. Below is what is currently working. I've heard good and bad things about using processEvents(), so I will keep testing.

    QString fileName = QFileDialog::getOpenFileName(this, "Open text file", "");
    QFile file(fileName);
    QFileInfo fi(file.fileName());
    QString fileExt = fi.completeSuffix();;
    QString strings;
    QString str;
    
    if (!file.open(QIODevice::ReadOnly)) {
    	//QMessageBox::warning(this,"...","error in opening keyword file");
    	return;
    }
    QTextStream ts(&file);
    while (!ts.atEnd()) {
    	QApplication::processEvents();
    	str = ts.readLine();
    	*fileList << str;
    }
    
    
    for (int row = 0; row < fileList->size(); row++)
    {
    	for (int col = 0; col < 2; col++)
    	{
    		if (col == 0) 
            {
    		      ui->tableWidget_Keywords_Queue->setItem(row, col, new QTableWidgetItem(fileList->at(row)));
    
    		}
    		if (col == 1)
                    {
    			ui->tableWidget_Keywords_Queue->setItem(row, col, new QTableWidgetItem(""));
    		}
    	}
    }
      qDebug() << fileList->size();
      file.close();

  • Lifetime Qt Champion

    hi @tshoats,

    processEvents() is a crutch, you should avoid it.

    if reading your file lasts more then a few milliseconds, you have two options:

    1. use a QTimer with small timeout and read some lines from the file in the slot. when the timer fires again, read the next lines.
    2. put the whole file reading in a thread

    both will keep your app responsive and you can even have a progress bar.

    just my two cent
    regards



  • @tshoats
    @ambershark is a touch harsh on processEvents() :) It's lazy and has issues but it's often employed!

    If you do stick with it, you are calling it for each line read. It needs to be called, but not that often, and it has a performance penalty. Put in a counter and call it every 10 or 100 lines read, enough so that you retain responsiveness but not too often.

    If all you are doing with the lines is populating some data structure, and not directly doing anything to the GUI (e.g. not putting them into a widget, you just store them in your read loop and only later put them into your table), @ambershark's suggestion of read in its own thread is not hard to implement and is neatest.


  • Moderators

    @aha_1980 said in How to make QByteArray read Full Text file?:

    processEvents() is a crutch, you should avoid it.

    would you be surprised, if I told you, that using processEvents() is an advice I once got from the Qt support team?


  • Lifetime Qt Champion

    @J.Hilk

    I still think if you have a better solution, you should avoid it.

    you may have good reasons to use it anyway, but you should know what you are doing. therefore I wouldn't recommend it in the forum or elsewhere.


  • Moderators

    @aha_1980
    very true, in the End I came up, in my humble opinion :-), with a smart(er) alternative.


  • Moderators

    @JonB said in How to make QByteArray read Full Text file?:

    @tshoats
    @ambershark is a touch harsh on processEvents() :) It's lazy and has issues but it's often employed!

    Well to be fair that wasn't me that said that. ;) However harsh or not I do agree with @aha_1980. I think if it lasts long enough to freeze your GUI, you are much better offloading it to a thread.

    Qt makes this super easy with QObject::moveToThread().

    I have nothing against processEvents, but I wouldn't really use it personally. :)



  • @ambershark
    Whoops, sorry, it was @aha_1980. It's your faults for having the same starting character in your name when this forum auto-completes ;-)


  • Moderators

    @JonB Lol totally, I take full responsibility for that and apologize. ;)


Log in to reply