Is there a direct way to know the lines of a text file ?

cyberpunker

Instead of:

QFile file( "datum.txt" );
if ( file.open( IO_ReadWrite ) )
{
QTextStream stream( &file );

while ( !stream.atEnd() )
{
count += 1;
line = stream.readLine( );
if(lineNo == count)
printf( "%3d: %s\n", count, line.latin1() );

}
file.close();

I am going to show show app's running log in the QTextEdit . The size of the log is unpredictable and changeable . So if I load a file into the QTextEdit , the time and efficency are variable, and if the file is too big to show , the GUI maybe dead.

I also try to use a thread to deal with the text file , then send the readlines to the QTextEdit by SIGNAL~SLOT , and it seem very clumsy.

So I going to get the line number of the text first , and the I can read the specific line out and show according to the scrollbar's draging . The api is QString QTextStream::readLine(qint64 maxlen = 0)

Or may use "memory map file "?

JonB

@cyberpunker
You cannot get to a particular line number in a text file without somehow reading all the lines (up to the desired one at least) and counting, period.

If you're finding it freezes the UI, you'll have to push the reading into a thread. Or, put your own slot on the scrolling and read incrementally only as necessary.

KroMignon

@cyberpunker said in Is there a direct way to know the lines of a text file ?:

o if I load a file into the QTextEdit , the time and efficency are variable, and if the file is too big to show , the GUI maybe dead.

As @JonB suggest to you, move your search in to another thread to no lock the GUI thread.
This can be done easily with QtConcurrent::run()

sankarapandiyan

@cyberpunker You may try like this

  QString filename1="path of the text file ";
      QFile file(filename1);
      if(!file.exists()){
          qDebug() << "NO FILE "<<filename1;
      }else{
          qDebug() << filename1<<" ...";
      }
      ui->sender->clear();
      if (file.open(QIODevice::ReadOnly | QIODevice::Text)){ ui->sender->setText(file.readAll()); }// here you may display in text edit or by using qdebug you display in the output

JonB

@sankarapandiyan
@cyberpunker knows this. The problem is that the file.readAll(), or the processing of its content, is too slow for the responsiveness of the GUI, that is what he is asking about.

artwaw

Hi,
if you have enough time on your hands you might want to design a model with lazy population - like QSqlTableModel does - that would just store the visible part of the file + some buffer around it.

JonB

@cyberpunker

@JonB said in Is there a direct way to know the lines of a text file ?:

Or, possibly, implement the scroll drag slot to incrementally read the file as required.

That is what @artwaw is suggesting. It is the neatest, most efficient way to read in only what it is needed, but it does require some work. Also, if for the sake of argument the user scrolls in one go to the end of file, you would have to read the whole file, which is what you do not like on the main thread as it will cause UI freeze.

RonaldViscarraL

Hi, try this:

How to load large data from txt file in Qt
https://stackoverflow.com/questions/46753753/how-to-load-large-data-from-txt-file-in-qt

JonB

@RonaldViscarraL
The trouble is, the approach there which supposedly is the quickest is simply data = readFile.readAll();. If the OP here says he has tried readAll() and it's too slow for his size file, it doesn't really solve.

@cyberpunker
You are using stream.readLine( ). You could use some stream.read( ) and do your own line counting, there might be an improvement there.

Just how big/how many lines is your log file going to grow to? Have you actually tried any realistic testing of the read delay?

cyberpunker

@JonB

The api : QString QTextStream::readLine(qint64 maxlen = 0) ?

I also try to use a thread to deal with the text file , then send the readlines to the QTextEdit by SIGNAL~SLOT , and it seems very clumsy.

JonB

@cyberpunker
I am suggesting you might try not using QTextStream::readLine(), and instead QTextStream::read() or QTextStream::readAll(), plus then do your own in-memory searching for \ns to count lines. The theory is that may be faster than QTextStream::readLine()s. Frankly I'm dubious how much difference it might make, but you could give it a go.

I did ask you just how big this file is, in bytes & lines? Have you actually tried some code with timings?

Your original question was, is it possible to know where a line is in a text file without reading each line, and the answer is, no it is not.

So, to prevent GUI freeze you are left with either doing something in a separate thread, or doing something where the user scrolling causes the file to be read in incrementally. at that time.

ODБOï

hi,

@JonB said in Is there a direct way to know the lines of a text file ?:

original question was, is it possible to know where a line is in a text file without reading each line

on windows you can use FINDSTR command with qprocess

FINDSTR /N "the word i search" file.txt

JonB

@LeLev
That just reads each line in the file, so why should that be any more efficient than, say, QTextStream::readAll(), which the OP is saying is/might be too slow?

ODБOï

@JonB its just an option. and because is is build in windows i thought it could be little faster somehow

cyberpunker

@JonB

In fact , it is the glog file , the file size is variable , I want to parse the log file and show its content according to a time period. The time range may may spans several log files , and the start time or stop time can be any position of the log file . So I need a mechanics which is efficecy , resource save, and human kind .

JonB

@cyberpunker
So I have given you the options. You will have to read all of the log file(s) in order to accomplish this, there is no shortcut. Have you looked at/answered where I keep asking you " Have you actually tried some code with timings?"?

Psnarf

@cyberpunker To reinterpret your question, you want to know when another application appends another line to datum.txt? Perhaps QIODevice::waitForReadyRead() may be one way to determine when datum.txt changes size?
[https://doc.qt.io/qt-5/qiodevice.html#waitForReadyRead](link url)

JonB

@Psnarf
I have not tried it out, but do you have any evidence that QIODevice::waitForReadyRead() will be triggered if another process writes to a file which this process has open? My suspicion is that it will not, but I could be wrong. I would expect to have to use https://doc.qt.io/qt-5/qfilesystemwatcher.html#fileChanged if I wanted to be notified of a file content change from an external program.