How do I open large file on QTextEdit?
-
Hi, all
I have some files, which size > 50M. I open the file use QFile, and show the text use QTextEdit, but It's terrible? My open file code:
@QFile file(m_strLogFullPathName);
if(!file.open(QFile::ReadOnly | QFile::Text))
{
QMessageBox::warning(this, tr("Log Reader"),
tr("Cannot read file %1:\n%2")
.arg(m_strLogFullPathName)
.arg(file.errorString()));
return;
}
QTextStream in(&file);
QApplication::setOverrideCursor(Qt::WaitCursor);
ui->logTextEdit->setPlainText(in.readAll());
QApplication::restoreOverrideCursor();@
I search on Web, there are some resolve methods:- Use a work thread to pre-read some blocks
- Use QScrollBar, and calc the viewport and pre-read size of view.
But these methods not details. Please posts the resulting code. Thanks
-
Hi yeaiping,
do I understand correctly?
You want us to write the code?You already have a solution:
make a background thread for reading, only present parts of the text bny building an own view.
A QTextEdit is a widget for displaying texts in general.50 MB of text by the way is a really huge text file.
Did you try to use QPlainTextEdit instead? It does less layouting stuff and might be faster.
-
[quote author="yeaiping" date="1300690631"]Please posts the resulting code. Thanks[/quote]
First of all, this is not a very "smart question":http://www.catb.org/~esr/faqs/smart-questions.html to ask. Don't do this. Do your own work, just like we do ours.[quote]Hi, all
I have some files, which size > 50M. I open the file use QFile, and show the text use QTextEdit, but It's terrible? [/quote]
Only you or your users can tell you. Does it perform well enough for your use case? If so, it is not terrible. But it would not be my choice if I could avoid it...[quote]
My open file code:
(snip)I search on Web, there are some resolve methods:
- Use a work thread to pre-read some blocks
- Use QScrollBar, and calc the viewport and pre-read size of view.
But these methods not details. [/quote]
You should first ask yourself if your users really need to see the whole 50MB of text in a single view. If that makes sense at all. How are they going to navigate through such a huge block of text? Are your users really going to read only the first couple of lines, or do you really expect them to need all of it? Is a scroll bar going to be sufficient means of navigation then, since it's resolution is going to be insufficient. Remember, if a text area is 1000 pixels high, and your text file contains about 500,000 lines (50MB @ 100 characters per line), then every pixel in your scrollbar represents 500 lines of text. Not a very nice way of navigating...
If you, after much deliberation, thinking and reflection, come to the conclusion that you really, really need to put all that text in a single text view, then it starts to make sense to think about how you are going to achieve that in a performant way.
What kind of file is this text file anyway? A huge log file perhaps?
-
[quote author="Gerolf" date="1300692329"]Hi yeaiping,
do I understand correctly?
You want us to write the code?
[/quote]I'm sorry to say that, my meaning is give me a hint, not complete code.
Thanks for your reply. I'm trying to use my new method to resolve this question. -
[quote author="Andre" date="1300692416"][quote author="yeaiping" date="1300690631"]Please posts the resulting code. Thanks[/quote]
First of all, this is not a very "smart question":http://www.catb.org/~esr/faqs/smart-questions.html to ask. Don't do this. Do your own work, just like we do ours.
[/quote]Sorry, this means "give me a hint, not code", and thanks your suggestion. :)
-
[quote author="yeaiping" date="1300702504"]
[quote]
What kind of file is this text file anyway? A huge log file perhaps?[/quote]Yes, a heap of huge log files.[/quote]
Well... might I do a suggestion then to make it a bit more manageable for both the user and for you?
I assume this log file spans a certain time period. For instance, a couple of months. If so, why not present the user with a layout where you present something like a list or a tree view on the left that contains the different days you have data on (and if you use a tree, you might group that into sections like "this week", "last week", "last month", "Januari 2011", "December 2010", etc.) The text view with the actual log can be filtered to only display the data belonging to the selected section(s) in this view.
You can fill the view by scanning over the log file in a background thread, and keeping track of the relevant offsets in the file for each relevant time point that you put in your view. Perhaps you can even store this index, so you can re-use the stored index the next time you view the file.
A setup like this would make navigating the log much easier for your user, and rid you of the problem of loading and displaying a multi-megabyte text file all at once in a performant way.
Of course, the feasibility of such a setup depends on your exact requirements.
-
We used next approach - subclassed QPlainTextEdit, but load only ~1000 lines portion. Also suppress native scrollbar and inserted own which is linked to position of 1K lines window in file. As results - opening any GB files, scroll them, own search through them etc. Of course file is not loaded fully in memory, but that's even better for our cases.
-
[quote]
Well... might I do a suggestion then to make it a bit more manageable for both the user and for you?
I assume this log file spans a certain time period. For instance, a couple of months.[/quote]
Yes, I used the RollingFileAppender to the logger(but the file size must larger than 50M, the size region [50M, 100M])
[quote]
If so, why not present the user with a layout where you present something like a list or a tree view on the left that contains the different days you have data on (and if you use a tree, you might group that into sections like "this week", "last week", "last month", "Januari 2011", "December 2010", etc.) The text view with the actual log can be filtered to only display the data belonging to the selected section(s) in this view.You can fill the view by scanning over the log file in a background thread, and keeping track of the relevant offsets in the file for each relevant time point that you put in your view. Perhaps you can even store this index, so you can re-use the stored index the next time you view the file.
A setup like this would make navigating the log much easier for your user, and rid you of the problem of loading and displaying a multi-megabyte text file all at once in a performant way.
Of course, the feasibility of such a setup depends on your exact requirements.
[/quote]
Your suggestion is another way to resolve this trouble thing. But sometimes we used the "Debug" level log, the message very huge, and DailyRollingAppender method is useless.
-
[quote author="yshurik" date="1300703841"]We used next approach - subclassed QPlainTextEdit, but load only ~1000 lines portion. Also suppress native scrollbar and inserted own which is linked to position of 1K lines window in file. As results - opening any GB files, scroll them, own search through them etc. Of course file is not loaded fully in memory, but that's even better for our cases.[/quote]
The 50M log file of mine about 500000 lines( :( ) I'm not only read them to QTextEdit, also processing the string eg. select the fatal message.
And there are some opensource Editor(eg. Vi, UltraEdit), They open the large file fast. So I think use QT to open the large file not a problem :)
I'll try -
It depends how they open a file and how you open it.
perhaps they just open the file and work with seek, so they don't really load tit to memory?
Or they just check the size and create a big buffer and load it to a pre allocated buffer.I'm not sure how QTextStream::readAll is working...
-
[quote author="Andre" date="1300716815"]I would guess they map the file into memory.[/quote]
I agree with you. Th QFile has map() method.
I think, the question to read large file not only depend on QTextEdit, but QFile, use a delegate between view(QTextEdit) and file(file on disk), and estimate the pre or next file blocks. If process the data, use temp file to save new data etc. -
The warning in the docs is clear:
bq. "QTextStream::readAll() ":http://doc.qt.nokia.com/4.7/qtextstream.html#readAll
Reads the entire content of the stream, and returns it as a QString. Avoid this function when working on large files, as it will consume a significant amount of memory.
Calling readLine() is better if you do not know how much data is available.
[Emphasis by me]map() cannot work with QTextStream's readAll(). QTextStream must read and analyze the stream of bytes to do the decoding stuff. And it does nothing more than
@
QString QTextStream::readAll()
{
Q_D(QTextStream);
CHECK_VALID_STREAM(QString());return d->read(INT_MAX);}
@Read itself does the actual decoding of the data.
-
J JonB referenced this topic on