How do I open large file on QTextEdit?

yeaiping · wrote on 21 Mar 2011, 09:50

[quote author="Gerolf" date="1300692329"]Hi yeaiping,

do I understand correctly?
You want us to write the code?
[/quote]

I'm sorry to say that, my meaning is give me a hint, not complete code.
Thanks for your reply. I'm trying to use my new method to resolve this question.

yeaiping · wrote on 21 Mar 2011, 09:53

[quote author="Andre" date="1300692416"][quote author="yeaiping" date="1300690631"]Please posts the resulting code. Thanks[/quote]
First of all, this is not a very "smart question":http://www.catb.org/~esr/faqs/smart-questions.html to ask. Don't do this. Do your own work, just like we do ours.
[/quote]

Sorry, this means "give me a hint, not code", and thanks your suggestion. :)

yeaiping · wrote on 21 Mar 2011, 10:15

[quote]
What kind of file is this text file anyway? A huge log file perhaps?[/quote]

Yes, a heap of huge log files.

andre · wrote on 21 Mar 2011, 10:26

[quote author="yeaiping" date="1300702504"]
[quote]
What kind of file is this text file anyway? A huge log file perhaps?[/quote]

Yes, a heap of huge log files.[/quote]

Well... might I do a suggestion then to make it a bit more manageable for both the user and for you?

I assume this log file spans a certain time period. For instance, a couple of months. If so, why not present the user with a layout where you present something like a list or a tree view on the left that contains the different days you have data on (and if you use a tree, you might group that into sections like "this week", "last week", "last month", "Januari 2011", "December 2010", etc.) The text view with the actual log can be filtered to only display the data belonging to the selected section(s) in this view.

You can fill the view by scanning over the log file in a background thread, and keeping track of the relevant offsets in the file for each relevant time point that you put in your view. Perhaps you can even store this index, so you can re-use the stored index the next time you view the file.

A setup like this would make navigating the log much easier for your user, and rid you of the problem of loading and displaying a multi-megabyte text file all at once in a performant way.

Of course, the feasibility of such a setup depends on your exact requirements.

yshurik · wrote on 21 Mar 2011, 10:37

We used next approach - subclassed QPlainTextEdit, but load only ~1000 lines portion. Also suppress native scrollbar and inserted own which is linked to position of 1K lines window in file. As results - opening any GB files, scroll them, own search through them etc. Of course file is not loaded fully in memory, but that's even better for our cases.

yeaiping · wrote on 21 Mar 2011, 13:28

[quote]

Well... might I do a suggestion then to make it a bit more manageable for both the user and for you?

I assume this log file spans a certain time period. For instance, a couple of months.[/quote]

Yes, I used the RollingFileAppender to the logger(but the file size must larger than 50M, the size region [50M, 100M])

[quote]
If so, why not present the user with a layout where you present something like a list or a tree view on the left that contains the different days you have data on (and if you use a tree, you might group that into sections like "this week", "last week", "last month", "Januari 2011", "December 2010", etc.) The text view with the actual log can be filtered to only display the data belonging to the selected section(s) in this view.

You can fill the view by scanning over the log file in a background thread, and keeping track of the relevant offsets in the file for each relevant time point that you put in your view. Perhaps you can even store this index, so you can re-use the stored index the next time you view the file.

A setup like this would make navigating the log much easier for your user, and rid you of the problem of loading and displaying a multi-megabyte text file all at once in a performant way.

Of course, the feasibility of such a setup depends on your exact requirements.

[/quote]

Your suggestion is another way to resolve this trouble thing. But sometimes we used the "Debug" level log, the message very huge, and DailyRollingAppender method is useless.

yeaiping · wrote on 21 Mar 2011, 13:42

[quote author="yshurik" date="1300703841"]We used next approach - subclassed QPlainTextEdit, but load only ~1000 lines portion. Also suppress native scrollbar and inserted own which is linked to position of 1K lines window in file. As results - opening any GB files, scroll them, own search through them etc. Of course file is not loaded fully in memory, but that's even better for our cases.[/quote]

The 50M log file of mine about 500000 lines( :( ) I'm not only read them to QTextEdit, also processing the string eg. select the fatal message.

And there are some opensource Editor(eg. Vi, UltraEdit), They open the large file fast. So I think use QT to open the large file not a problem :)
I'll try

giesbert · wrote on 21 Mar 2011, 14:11

It depends how they open a file and how you open it.
perhaps they just open the file and work with seek, so they don't really load tit to memory?
Or they just check the size and create a big buffer and load it to a pre allocated buffer.

I'm not sure how QTextStream::readAll is working...

andre · wrote on 21 Mar 2011, 14:13

I would guess they map the file into memory.

yeaiping · wrote on 21 Mar 2011, 14:30

[quote author="Andre" date="1300716815"]I would guess they map the file into memory.[/quote]

I agree with you. Th QFile has map() method.
I think, the question to read large file not only depend on QTextEdit, but QFile, use a delegate between view(QTextEdit) and file(file on disk), and estimate the pre or next file blocks. If process the data, use temp file to save new data etc.

goetz · wrote on 21 Mar 2011, 18:45

The warning in the docs is clear:

bq. "QTextStream::readAll() ":http://doc.qt.nokia.com/4.7/qtextstream.html#readAll
Reads the entire content of the stream, and returns it as a QString. Avoid this function when working on large files, as it will consume a significant amount of memory.
Calling readLine() is better if you do not know how much data is available.
[Emphasis by me]

map() cannot work with QTextStream's readAll(). QTextStream must read and analyze the stream of bytes to do the decoding stuff. And it does nothing more than

@
QString QTextStream::readAll()
{
Q_D(QTextStream);
CHECK_VALID_STREAM(QString());

return d->read(INT_MAX);

}
@

Read itself does the actual decoding of the data.