How to make QByteArray read Full Text file?

ambershark

Wrote a quick little test and there should not be any need for a QByteArrayList. My test (shown below) reads a 1.3mb text file which is about 13k lines with no problem. If you are still reading wrong sizes in a single BA then I'm guessing there is actually a problem with your text file. Here is the test:

#include <QCoreApplication>
#include <QByteArray>
#include <QFile>
#include <QDebug>

int main(int ac, char **av)
{
    QCoreApplication app(ac, av);
    
    QFile f("/home/mike/last_backup.log");
    if (!f.open(QIODevice::ReadOnly | QIODevice::Text))
    {
        qDebug() << "failed to open";
        return 1;
    }
    
    QByteArray ba = f.readAll();
    qDebug() << "read" << ba.size();
    qDebug() << "file size is" << f.size();
    
    return 0;
}

And here is the output:

read 1224526
file size is 1224526

One other thing I noticed, when you are reading via a QTextStream, which is a nice way to go line by line, you are appending to a strings variable, which seems wrong. Wouldn't you want strings = ts.readLine() rather than an append? Append would continually add line after line to that variable which would test keywords over and over (unintentionally?). Here is the code I'm talking about:

QTextStream ts(&file);
while (!ts.atEnd()) {
	strings.append(ts.readLine());   <-- this line here
	list = strings.split(QRegExp("[\r\n]"), QString::SkipEmptyParts);
	qDebug() << list;

	for (int i = 0; i <list.size(); i++) {
	    hashKeywordLoadList[list.at(i)] = 0;
	options[4]->keywordLoadListOptions = hashKeywordLoadList;
	}
}

Edit:

Tested with a 59mb file just to be sure everything was happy even on quite large text files and QByteArray was fine with that too:

read 61226300
file size is 61226300

658850 lines in that one.

tshoats

@ambershark , and @JonB ,thanks sooo much!!! You've been a great help. Yup you were totally right about strings and binary issues. I forgot in my header file, I was using a QHash<QString, int> keywordLoadListOptions, with QString as a type. This is where keywordLoadListOptions come into play. I changed my code to use QTextStream to read the file using file.readLine(). I then used processEvents() inside the while loop to prohibit my GUI from freezing. Some where along the lines my QString within my Qlist keywordLoadListOptions was ending my string. Below is what is currently working. I've heard good and bad things about using processEvents(), so I will keep testing.

QString fileName = QFileDialog::getOpenFileName(this, "Open text file", "");
QFile file(fileName);
QFileInfo fi(file.fileName());
QString fileExt = fi.completeSuffix();;
QString strings;
QString str;

if (!file.open(QIODevice::ReadOnly)) {
	//QMessageBox::warning(this,"...","error in opening keyword file");
	return;
}
QTextStream ts(&file);
while (!ts.atEnd()) {
	QApplication::processEvents();
	str = ts.readLine();
	*fileList << str;
}


for (int row = 0; row < fileList->size(); row++)
{
	for (int col = 0; col < 2; col++)
	{
		if (col == 0) 
        {
		      ui->tableWidget_Keywords_Queue->setItem(row, col, new QTableWidgetItem(fileList->at(row)));

		}
		if (col == 1)
                {
			ui->tableWidget_Keywords_Queue->setItem(row, col, new QTableWidgetItem(""));
		}
	}
}
  qDebug() << fileList->size();
  file.close();

aha_1980

hi @tshoats,

processEvents() is a crutch, you should avoid it.

if reading your file lasts more then a few milliseconds, you have two options:

use a QTimer with small timeout and read some lines from the file in the slot. when the timer fires again, read the next lines.
put the whole file reading in a thread

both will keep your app responsive and you can even have a progress bar.

just my two cent
regards

JonB

@tshoats
@ambershark is a touch harsh on processEvents() :) It's lazy and has issues but it's often employed!

If you do stick with it, you are calling it for each line read. It needs to be called, but not that often, and it has a performance penalty. Put in a counter and call it every 10 or 100 lines read, enough so that you retain responsiveness but not too often.

If all you are doing with the lines is populating some data structure, and not directly doing anything to the GUI (e.g. not putting them into a widget, you just store them in your read loop and only later put them into your table), @ambershark's suggestion of read in its own thread is not hard to implement and is neatest.

J.Hilk

@aha_1980 said in How to make QByteArray read Full Text file?:

processEvents() is a crutch, you should avoid it.

would you be surprised, if I told you, that using processEvents() is an advice I once got from the Qt support team?

aha_1980

@J.Hilk

I still think if you have a better solution, you should avoid it.

you may have good reasons to use it anyway, but you should know what you are doing. therefore I wouldn't recommend it in the forum or elsewhere.

J.Hilk

@aha_1980
very true, in the End I came up, in my humble opinion :-), with a smart(er) alternative.

ambershark

@JonB said in How to make QByteArray read Full Text file?:

@tshoats
@ambershark is a touch harsh on processEvents() :) It's lazy and has issues but it's often employed!

Well to be fair that wasn't me that said that. ;) However harsh or not I do agree with @aha_1980. I think if it lasts long enough to freeze your GUI, you are much better offloading it to a thread.

Qt makes this super easy with QObject::moveToThread().

I have nothing against processEvents, but I wouldn't really use it personally. :)

JonB

@ambershark
Whoops, sorry, it was @aha_1980. It's your faults for having the same starting character in your name when this forum auto-completes ;-)

ambershark

@JonB Lol totally, I take full responsibility for that and apologize. ;)