Best practices for displaying millions of loglines (colored and different line heights) using Qt (QTableView,...)

markus.liebe

Hi there,

The task
I have to be able to present millions of log lines to the user.
The lines consists of several columns like [timestamp][log level][...][log message]
The log message itself can contain several newlines.
Those log lines have to be presented in the form of a table.
The user can resize the window and the logmessage has to wrap the lines which makes the loglines grow in height.
The user must be able to filter the data according to different attributes like loglevel.

The current solution
At the moment I use a QTableView and a QSortFilterProxyModel to display the data.
It does the job and works fine - just not for very large datasets.
As the amount of lines grows into several millions the program gets slower - which is understandable, because of all the allocated data (e.g 1,3GB worth of text data inserted into the model, and then displayed)

Your proposals?
I am interested in different approaches to tackle the task of displaying the logdata using Qt.
And in discussing them.
Be it an approach using QtQuick or Widgets or even using Web technology.

What kind of solutions would you propose?

Best regards,
Markus

VRonin

Hard loading 1.3GB of text in memory is madness, you need to find a smart way to handle it. The first thing that comes to mind is dropping those logs in a SQL database (SQLite?) and then use QSqlQueryModel to read it. The advantage is that you can perform the filtering by changing the query which it's infinitely faster than QSortFilterProxyModel on large datasets.
QSqlQueryModel also implements lazy loading (canFetchMore/fetchMore) which should help (even if not solve) the performance issue

markus.liebe

thank you for your proposal!

In fact I already had a SQLite based solution running, which i ditched again, because I also have to cope with live data (think "tail -f" mode) and always auto scroll to the latest lines at the bottom of the view.

The QSqlQueryModel provides the API setQuery to set and execute the query.
As soon as new loglines arrive (maybe only a few rows) I did have to call setQuery again, which then causes the view to be cleared and filled again with the complete result.
I failed to come up with a solution that would do that in an incremental fashion.
The approach using the QSqlQueryModel led to a unpleasant experience because of the constantly updating tableview and thus a constantly changing scrollbar. Where in fact I would like to have a possibility to somehow reevaluate the query and update the view with the delta (diff).

SGaist

Hi,

You should consider implementing a moving window that only shows a fixed number of record, you can update the content of that "window" when scrolling back and fort. And if the user is at the top of the list, then you would only need to retrieve that reduced number of records.

JonB

@markus.liebe

At the moment I use [...] and a QSortFilterProxyModel to display the data.

One thing: do you allow the user to sort the data as well as filter it? Receiving new rows and deciding whether to display them is nasty if you allow sorting by anything other than latest datetime (or column directly related to that) in your paging algorithm.

VRonin

@SGaist said in Best practices for displaying millions of loglines (colored and different line heights) using Qt (QTableView,...):

You should consider implementing a moving window that only shows a fixed number of record

Just anecdotally, it was one of my first "to do utilities" I tried to implement but then gave up because it looked horrible.
You have to basically break the separation (proxy)model/view and as soon as the model becomes a tree the whole thing is close to madness

JonB

@VRonin
But as @SGaist said, at the end of the day you have to implement a "moving window" in some shape or form to achieve the desired behaviour.... Maybe you gave up too easily? ;-)

JonB

@markus.liebe , @VRonin , @SGaist
It seems to me that conceptually what you need to do is:

Forget using QSql... for the moment.
Use a QStandardItemModel.
Initially fill it with whatever rows from the database.
Then periodically issue a new database query to just fetch whatever new rows have been appended. (If you can't do that, you'll have to re-fetch them all, and then remove those which are presently in your model so that you are left with the new ones --- slow.)
At this point you can use QStandardItemModel::insertRows() to insert the new ones, in the right place. (You'll probably want to compensate by using QStandardItemModel::removeRows() to get rid of some at the beginning of the old data, to keep the overall number of rows down.)
This should allow your view to update "efficiently" for the new rows.

Now, the QSql... classes do not allow you to add your own rows (I believe). So you might, for example, have to use two models to achieve this: an "invisible" QSql... to do the queries nicely, and then a QStandardItemModel with the view attached into which you copy rows from the QSql... as necessary. You'll have to check on the copying speed for this....

VRonin

@JonB said in Best practices for displaying millions of loglines (colored and different line heights) using Qt (QTableView,...):

Then periodically issue a new database query to just fetch whatever new rows have been appended. (If you can't do that, you'll have to re-fetch them all, and then remove those which are presently in your model so that you are left with the new ones --- slow.)

You can actually set up an sql trigger and use QSqlDriver::subscribeToNotification to receive a signal when new rows are inserted. This way you can keep the filtering on the SQL side for better performance

JonB

@VRonin
OOI, how is the QSqlDriver::subscribeToNotification implemented at the SQL side? I'm familiar in the past with MS SQL Server, though from Qt I now use MySQL. Which SQL servers have that QSqlDriver::hasFeature(), and how do they implement it, e.g. is it periodic or immediate at the server side?

VRonin

@JonB said in Best practices for displaying millions of loglines (colored and different line heights) using Qt (QTableView,...):

how is the QSqlDriver::subscribeToNotification implemented at the SQL side?

It's really db specific. For SQL server you can use CREATE EVENT NOTIFICATION in postgre you need to create a trigger on insert that sends an event notification

JonB

@VRonin
What about for MySQL, please?

VRonin

Looks like the feature is available in Oracle DB but not in its little brother, sorry

JonB

@VRonin
Yep, I Googled, and also i tried hasFeature(QtSql.QSqlDriver.EventNotifications) and QMYSQL returns false. Ho-hum :(

markus.liebe

@SGaist Thanks for the suggestion.
In fact this was another approach that I tried: I implemented something I called a "SliceModel" which was essentially what you are suggesting:
I used a QSortFilterProxyModel that did only accept rows with row numbers in between a certain range of rows.
A Window of rows was defined by the amount of lines that fit on a screen + some extra lines.

I had to add another scrollbar to the TableView then, which displayed the amount of rows in the sourceModel.
When the user moved the scrollbar, this resulted in adjusting the windows start and end row and so the view itself displayed the desired range, but contained only so many lines as specified by the window range.

The original scrollbar of the view had to be hidden, because it was tied to the proxy models data.

The complete concept was quite nice, because I had a very smooth scrolling experience, even when using millions of lines.
The problem with this approach was, that it did break the typical Model -> View concept and it got quite difficult to get a decent user experience.