How to make QByteArray read Full Text file?

tshoats · 17 Jul 2018, 05:20

@ambershark From your advice I was able to use QByteArrayList which gives me the full text file only when I use qDebug(). When I loop through the QbyteArrayList im able to see the full text file, and each line of the data only in qDebug(), but not in my gui. I have a text file of 7,000 lines of, and its only giving me 3,000. My code snippet is below

QByteArray ba = file.readAll();
QByteArrayList qba = ba.split('\n');
for (int j = 0; j<qba.size(); j++)
{
QByteArray ba_data = qba.at(j);
std::string stdString(ba_data.constData(), ba_data.length());
options[4]->keywordLoadListOptions[QString::fromStdString(stdString)] = 0;
qDebug() << QString::fromStdString(stdString);
}

jsulm · 17 Jul 2018, 11:59

@tshoats If it is a text file then why do you read it as QByteArray? You should use http://doc.qt.io/qt-5/qtextstream.html

tshoats · 17 Jul 2018, 13:30

@jsulm when i use QByteArray it dosent block or freeze my gui, since im working with large text files to read. The only problem is It only shows half of the text file data. I tried using QTextStream but it blocks my gui unlike QByteArray. Below is a snippet of the QTextStream example thats blocking my gui..

if (!file.open(QIODevice::ReadOnly | QIODevice::Text)) {
	//QMessageBox::warning(this,"...","error in opening keyword file");
	return;
}
//QByteArray ba = file.readAll();
QTextStream ts(&file);
while (!ts.atEnd()) {
	strings.append(ts.readLine());
	list = strings.split(QRegExp("[\r\n]"), QString::SkipEmptyParts);
	qDebug() << list;

	for (int i = 0; i <list.size(); i++) {
	    hashKeywordLoadList[list.at(i)] = 0;
	options[4]->keywordLoadListOptions = hashKeywordLoadList;
	}
}

file.close();

JonB · wrote on 17 Jul 2018, 13:30

@tshoats

It won't be "using QByteArray versus QTextStream blocks the GUI", it will depend on what/how you do with it.
For example, you chose to go QByteArray ba = file.readAll(); but QTextStream::readLine(). These are very different. You could have used QTextStream::readAll() for comparison. So you're not comparing like-for-like.
If you're going to read line-by-line, and do something with the lines, and the file is large, you could put in judicious processEvents() so that the UI does not "freeze".
Your current example code seems (assuming strings is a QStringList variable?) to read one line, append it to an ever-growing list of strings, split that whole string, and keep examining the list as a whole. If that's true it's slow. Depends where strings comes from.

ambershark · wrote on 17 Jul 2018, 19:42

@tshoats said in How to make QByteArray read Full Text file?:

@ambershark From your advice I was able to use QByteArrayList which gives me the full text file only when I use qDebug(). When I loop through the QbyteArrayList im able to see the full text file, and each line of the data only in qDebug(), but not in my gui. I have a text file of 7,000 lines of, and its only giving me 3,000. My code snippet is below

QByteArray ba = file.readAll();
QByteArrayList qba = ba.split('\n');
for (int j = 0; j<qba.size(); j++)
{
QByteArray ba_data = qba.at(j);
std::string stdString(ba_data.constData(), ba_data.length());
options[4]->keywordLoadListOptions[QString::fromStdString(stdString)] = 0;
qDebug() << QString::fromStdString(stdString);
}

So based on this your data should have been read properly. This means the problem is probably in your GUI code.

Can you share that or at least describe what you are trying to do and what kind of data is in the text file/how are you trying to display it.

If you give me an idea of what you are trying to do I could write you a quick example.

Also, I didn't mean you can't put a QByteArray into a QString, was just mentioning that since QByteArray is binary and QString is not you could end up with a 0 in your byte array prematurely ending your string. If you are sure that the data read is all text data then you're pretty safe to put it in a QString.

Moving it from a QString to a std::string is not what I meant to do. Drop the std::string and use QString if you want strings. In your code above you created a std::string that was a copy of the bytearray and then created a QString from the std::string. So you now have triple the memory allocated for a very large string. Not to mention the processor time required to copy those strings.

ambershark · 17 Jul 2018, 23:19

Wrote a quick little test and there should not be any need for a QByteArrayList. My test (shown below) reads a 1.3mb text file which is about 13k lines with no problem. If you are still reading wrong sizes in a single BA then I'm guessing there is actually a problem with your text file. Here is the test:

#include <QCoreApplication>
#include <QByteArray>
#include <QFile>
#include <QDebug>

int main(int ac, char **av)
{
    QCoreApplication app(ac, av);
    
    QFile f("/home/mike/last_backup.log");
    if (!f.open(QIODevice::ReadOnly | QIODevice::Text))
    {
        qDebug() << "failed to open";
        return 1;
    }
    
    QByteArray ba = f.readAll();
    qDebug() << "read" << ba.size();
    qDebug() << "file size is" << f.size();
    
    return 0;
}

And here is the output:

read 1224526
file size is 1224526

One other thing I noticed, when you are reading via a QTextStream, which is a nice way to go line by line, you are appending to a strings variable, which seems wrong. Wouldn't you want strings = ts.readLine() rather than an append? Append would continually add line after line to that variable which would test keywords over and over (unintentionally?). Here is the code I'm talking about:

QTextStream ts(&file);
while (!ts.atEnd()) {
	strings.append(ts.readLine());   <-- this line here
	list = strings.split(QRegExp("[\r\n]"), QString::SkipEmptyParts);
	qDebug() << list;

	for (int i = 0; i <list.size(); i++) {
	    hashKeywordLoadList[list.at(i)] = 0;
	options[4]->keywordLoadListOptions = hashKeywordLoadList;
	}
}

Edit:

Tested with a 59mb file just to be sure everything was happy even on quite large text files and QByteArray was fine with that too:

read 61226300
file size is 61226300

658850 lines in that one.

tshoats · 18 Jul 2018, 01:45

@ambershark , and @JonB ,thanks sooo much!!! You've been a great help. Yup you were totally right about strings and binary issues. I forgot in my header file, I was using a QHash<QString, int> keywordLoadListOptions, with QString as a type. This is where keywordLoadListOptions come into play. I changed my code to use QTextStream to read the file using file.readLine(). I then used processEvents() inside the while loop to prohibit my GUI from freezing. Some where along the lines my QString within my Qlist keywordLoadListOptions was ending my string. Below is what is currently working. I've heard good and bad things about using processEvents(), so I will keep testing.

QString fileName = QFileDialog::getOpenFileName(this, "Open text file", "");
QFile file(fileName);
QFileInfo fi(file.fileName());
QString fileExt = fi.completeSuffix();;
QString strings;
QString str;

if (!file.open(QIODevice::ReadOnly)) {
	//QMessageBox::warning(this,"...","error in opening keyword file");
	return;
}
QTextStream ts(&file);
while (!ts.atEnd()) {
	QApplication::processEvents();
	str = ts.readLine();
	*fileList << str;
}


for (int row = 0; row < fileList->size(); row++)
{
	for (int col = 0; col < 2; col++)
	{
		if (col == 0) 
        {
		      ui->tableWidget_Keywords_Queue->setItem(row, col, new QTableWidgetItem(fileList->at(row)));

		}
		if (col == 1)
                {
			ui->tableWidget_Keywords_Queue->setItem(row, col, new QTableWidgetItem(""));
		}
	}
}
  qDebug() << fileList->size();
  file.close();

aha_1980 · 18 Jul 2018, 07:42

hi @tshoats,

processEvents() is a crutch, you should avoid it.

if reading your file lasts more then a few milliseconds, you have two options:

use a QTimer with small timeout and read some lines from the file in the slot. when the timer fires again, read the next lines.
put the whole file reading in a thread

both will keep your app responsive and you can even have a progress bar.

just my two cent
regards

JonB · 19 Jul 2018, 01:25

@tshoats
@ambershark is a touch harsh on processEvents() :) It's lazy and has issues but it's often employed!

If you do stick with it, you are calling it for each line read. It needs to be called, but not that often, and it has a performance penalty. Put in a counter and call it every 10 or 100 lines read, enough so that you retain responsiveness but not too often.

If all you are doing with the lines is populating some data structure, and not directly doing anything to the GUI (e.g. not putting them into a widget, you just store them in your read loop and only later put them into your table), @ambershark's suggestion of read in its own thread is not hard to implement and is neatest.

J.Hilk · wrote on 18 Jul 2018, 07:42

@aha_1980 said in How to make QByteArray read Full Text file?:

processEvents() is a crutch, you should avoid it.

would you be surprised, if I told you, that using processEvents() is an advice I once got from the Qt support team?

aha_1980 · 18 Jul 2018, 08:41

@J.Hilk

I still think if you have a better solution, you should avoid it.

you may have good reasons to use it anyway, but you should know what you are doing. therefore I wouldn't recommend it in the forum or elsewhere.

J.Hilk · wrote on 18 Jul 2018, 08:41

@aha_1980
very true, in the End I came up, in my humble opinion :-), with a smart(er) alternative.

ambershark · 19 Jul 2018, 07:10

@JonB said in How to make QByteArray read Full Text file?:

@tshoats
@ambershark is a touch harsh on processEvents() :) It's lazy and has issues but it's often employed!

Well to be fair that wasn't me that said that. ;) However harsh or not I do agree with @aha_1980. I think if it lasts long enough to freeze your GUI, you are much better offloading it to a thread.

Qt makes this super easy with QObject::moveToThread().

I have nothing against processEvents, but I wouldn't really use it personally. :)

JonB · 20 Jul 2018, 22:29

@ambershark
Whoops, sorry, it was @aha_1980. It's your faults for having the same starting character in your name when this forum auto-completes ;-)

ambershark · J JonB 19 Jul 2018, 07:10

@JonB Lol totally, I take full responsibility for that and apologize. ;)