Viewing huge files with a Qt app

deisik

What should I use to view huge files such as logs with their size over 1Gb (no editing required)?

QTextEdit is useless for anything larger than a few MBs

JonB

@deisik
Use QPlainTextEdit. If required, implement a "dynamic scrolling" mechanism which only reads further blocks from the file when the user scrolls down the document (or searches). If memory consumption must be limited, dispose earlier blocks previously read.

Something like https://stackoverflow.com/questions/57457371/load-huge-text-buffer-into-qplaintextedit (even maybe https://interest.qt-project.narkive.com/Z1XhnVtv/qtextedit-is-too-slow-hints-needed) or other Google results for qplaintextedit large file may be of interest.

deisik

Wouldn't QPlainText be even better (without editing)? Is there such a thing?

JonB

@deisik
Don't know what you asking. I pointed you to QPlainTextEdit (which you can set readonly if you wish). If you mean is there a class QPlainText then you can discover for yourself there is not by searching the documentation/Googling.

deisik

As far as I'm familiar with QTextEdit, you have first to "select" a part of the text in order to remove it programmatically

This feels cumbersome to me, so I wondered if there's a parent class with which you can do stuff like this in a more direct way

SGaist

Hi,

Do you mean something like QTextDocument ?

deisik

@SGaist

Can it handle a 1GB log file?

SGaist

@deisik it's the underpinning of the QTextEdit and QPlainTextEdit classes.

deisik

@SGaist

So it all comes down to dynamic loading and unloading data. I'm looking for something as simple and easy as adding/removing a row in a table

SGaist

In that case, one thing you can consider is using the model view paradigm to show your file so you can use a QListView to show each line of your file.
You can either load your whole file as a QStringListModel but it will likely be too slow for your needs. Or create your own model that will have a certain amount of lines in memory to show and scroll and will load/unload what needed when the user scrolls.

deisik

@SGaist

The problem with QListView (or QTableView, for that matter) is that you won't be able to select text across multiple lines (rows) in a regular way like with QTextEdit

JonB

@deisik
As suggested, why don't you at least try the QPlainTextEdit way?

deisik

@JonB

It didn't change much, if anything at all. Tried to open a 90Mb file and had to euthanize it

JonB

@deisik
Then as previously suggested you probably need to read only a portion of the file into memory for display and handle scrolling programmatically. Which is the same principle if you used a model.

Old thread on this forum https://forum.qt.io/topic/4519/how-do-i-open-large-file-on-qtextedit seems to indicate this kind of approach.

BTW: for your 90MB, was it taking a long time on reading the data from file (should not be) or on displaying it in the QPlainTextEdit on screen?

deisik

@JonB

It freezes reading, not displaying. Actually, it doesn't freeze, but will likely take a couple hours to process the file contents

It is not the part about reading a portion of the file which is cumbersome, it is the part where you have to select the text programmatically to erase it which just doesn't feel right

JonB

@deisik
You are big on "things don't feel right" when they seem fine to me :)

BTW if you have some inbuilt aversion to "using the cursor" to delete/insert, you don't have to. Just re-use setPlainText() to set the whole of the document to the newly calculated range. I don't know, but in practice this may be just as fast as using the cursor. Also BTW you could try just putting the text in a QLabel (probably wrapped in a scroll area), that does not use a cursor. You have to set the complete text, there is no insert/remove.

I suggest there is something wrong if your file reading code "takes a couple of hours" to read in 90MB, don't you think...? It should take like, I don't know, a few seconds.... This is fundamental no matter which approach you take to display it.

deisik

@JonB said in Viewing huge files with a Qt app:

@deisik
You are big on "things don't feel right" when they seem fine to me :)

BTW if you have some inbuilt aversion to "using the cursor" to delete/insert, you don't have to. Just re-use setPlainText() to set the whole of the document to the newly calculated range. I don't know, but in practice this may be just as fast as using the cursor. Also BTW you could try just putting the text in a QLabel (probably wrapped in a scroll area), that does not use a cursor. You have to set the complete text, there is no insert/remove.

I'm thinking about it. QLabel is probably the way to go. How can I make it look like a QTextEdit widget?

I suggest there is something wrong if your file reading code "takes a couple of hours" to read in 90MB, don't you think...? It should take like, I don't know, a few seconds.... This is fundamental no matter which approach you take to display it.

I have to parse the file (at least, would have the part to display)

And while we are at it, just switching to a tab (window) containing a big enough QTextEdit object takes a few minutes

JonB

@deisik said in Viewing huge files with a Qt app:

I have to parse the file (at least, would have the part to display)

Sorry, but if whatever parsing you do on a 90MB file takes "a couple of hours" as you state, or even if that's on 1GB, something is not right.

just switching to a tab (window) containing a big enough QTextEdit object takes a few minutes

Which is why you should use a QPlainTextEdit not a QTextEdit. Your timing may be affected by options, for example word wrap mode.

deisik

@JonB said in Viewing huge files with a Qt app:

@deisik said in Viewing huge files with a Qt app:

I have to parse the file (at least, would have the part to display)

Sorry, but if whatever parsing you do on a 90MB file takes "a couple of hours" as you state, or even if that's on 1GB, something is not right

I assume that it would take a couple of hours or so. I need real time

Anyway, why don't you try it out and check for yourself?

JonB

@deisik
You think it would take a couple of hours to do your parsing on a 90MB/1GB file?!

Anyway, why don't you try it out and check for yourself?

Sorry, you would like me to write and test what you want? Not to mention, without even knowing what your "parsing" entails? That's not how it works: YOU can "try it out and check for yourself?" since it's your issue, not me.