Best practices for displaying millions of loglines (colored and different line heights) using Qt (QTableView,...)
-
Hi,
You should consider implementing a moving window that only shows a fixed number of record, you can update the content of that "window" when scrolling back and fort. And if the user is at the top of the list, then you would only need to retrieve that reduced number of records.
Interested in AI ? www.idiap.ch
Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct -
Hi there,
The task
I have to be able to present millions of log lines to the user.
The lines consists of several columns like [timestamp][log level][...][log message]
The log message itself can contain several newlines.
Those log lines have to be presented in the form of a table.
The user can resize the window and the logmessage has to wrap the lines which makes the loglines grow in height.
The user must be able to filter the data according to different attributes like loglevel.The current solution
At the moment I use a QTableView and a QSortFilterProxyModel to display the data.
It does the job and works fine - just not for very large datasets.
As the amount of lines grows into several millions the program gets slower - which is understandable, because of all the allocated data (e.g 1,3GB worth of text data inserted into the model, and then displayed)Your proposals?
I am interested in different approaches to tackle the task of displaying the logdata using Qt.
And in discussing them.
Be it an approach using QtQuick or Widgets or even using Web technology.What kind of solutions would you propose?
Best regards,
MarkusAt the moment I use [...] and a QSortFilterProxyModel to display the data.
One thing: do you allow the user to sort the data as well as filter it? Receiving new rows and deciding whether to display them is nasty if you allow sorting by anything other than latest datetime (or column directly related to that) in your paging algorithm.
-
Hi,
You should consider implementing a moving window that only shows a fixed number of record, you can update the content of that "window" when scrolling back and fort. And if the user is at the top of the list, then you would only need to retrieve that reduced number of records.
@SGaist said in Best practices for displaying millions of loglines (colored and different line heights) using Qt (QTableView,...):
You should consider implementing a moving window that only shows a fixed number of record
Just anecdotally, it was one of my first "to do utilities" I tried to implement but then gave up because it looked horrible.
You have to basically break the separation (proxy)model/view and as soon as the model becomes a tree the whole thing is close to madness"La mort n'est rien, mais vivre vaincu et sans gloire, c'est mourir tous les jours"
~Napoleon BonaparteOn a crusade to banish setIndexWidget() from the holy land of Qt
-
@SGaist said in Best practices for displaying millions of loglines (colored and different line heights) using Qt (QTableView,...):
You should consider implementing a moving window that only shows a fixed number of record
Just anecdotally, it was one of my first "to do utilities" I tried to implement but then gave up because it looked horrible.
You have to basically break the separation (proxy)model/view and as soon as the model becomes a tree the whole thing is close to madness -
thank you for your proposal!
In fact I already had a SQLite based solution running, which i ditched again, because I also have to cope with live data (think "tail -f" mode) and always auto scroll to the latest lines at the bottom of the view.
The QSqlQueryModel provides the API setQuery to set and execute the query.
As soon as new loglines arrive (maybe only a few rows) I did have to call setQuery again, which then causes the view to be cleared and filled again with the complete result.
I failed to come up with a solution that would do that in an incremental fashion.
The approach using the QSqlQueryModel led to a unpleasant experience because of the constantly updating tableview and thus a constantly changing scrollbar. Where in fact I would like to have a possibility to somehow reevaluate the query and update the view with the delta (diff).@markus.liebe , @VRonin , @SGaist
It seems to me that conceptually what you need to do is:- Forget using
QSql...for the moment. - Use a
QStandardItemModel. - Initially fill it with whatever rows from the database.
- Then periodically issue a new database query to just fetch whatever new rows have been appended. (If you can't do that, you'll have to re-fetch them all, and then remove those which are presently in your model so that you are left with the new ones --- slow.)
- At this point you can use
QStandardItemModel::insertRows()to insert the new ones, in the right place. (You'll probably want to compensate by usingQStandardItemModel::removeRows()to get rid of some at the beginning of the old data, to keep the overall number of rows down.) - This should allow your view to update "efficiently" for the new rows.
Now, the
QSql...classes do not allow you to add your own rows (I believe). So you might, for example, have to use two models to achieve this: an "invisible"QSql...to do the queries nicely, and then aQStandardItemModelwith the view attached into which you copy rows from theQSql...as necessary. You'll have to check on the copying speed for this.... - Forget using
-
@markus.liebe , @VRonin , @SGaist
It seems to me that conceptually what you need to do is:- Forget using
QSql...for the moment. - Use a
QStandardItemModel. - Initially fill it with whatever rows from the database.
- Then periodically issue a new database query to just fetch whatever new rows have been appended. (If you can't do that, you'll have to re-fetch them all, and then remove those which are presently in your model so that you are left with the new ones --- slow.)
- At this point you can use
QStandardItemModel::insertRows()to insert the new ones, in the right place. (You'll probably want to compensate by usingQStandardItemModel::removeRows()to get rid of some at the beginning of the old data, to keep the overall number of rows down.) - This should allow your view to update "efficiently" for the new rows.
Now, the
QSql...classes do not allow you to add your own rows (I believe). So you might, for example, have to use two models to achieve this: an "invisible"QSql...to do the queries nicely, and then aQStandardItemModelwith the view attached into which you copy rows from theQSql...as necessary. You'll have to check on the copying speed for this....@JonB said in Best practices for displaying millions of loglines (colored and different line heights) using Qt (QTableView,...):
Then periodically issue a new database query to just fetch whatever new rows have been appended. (If you can't do that, you'll have to re-fetch them all, and then remove those which are presently in your model so that you are left with the new ones --- slow.)
You can actually set up an sql trigger and use QSqlDriver::subscribeToNotification to receive a signal when new rows are inserted. This way you can keep the filtering on the SQL side for better performance
"La mort n'est rien, mais vivre vaincu et sans gloire, c'est mourir tous les jours"
~Napoleon BonaparteOn a crusade to banish setIndexWidget() from the holy land of Qt
- Forget using
-
@JonB said in Best practices for displaying millions of loglines (colored and different line heights) using Qt (QTableView,...):
Then periodically issue a new database query to just fetch whatever new rows have been appended. (If you can't do that, you'll have to re-fetch them all, and then remove those which are presently in your model so that you are left with the new ones --- slow.)
You can actually set up an sql trigger and use QSqlDriver::subscribeToNotification to receive a signal when new rows are inserted. This way you can keep the filtering on the SQL side for better performance
@VRonin
OOI, how is theQSqlDriver::subscribeToNotificationimplemented at the SQL side? I'm familiar in the past with MS SQL Server, though from Qt I now use MySQL. Which SQL servers have thatQSqlDriver::hasFeature(), and how do they implement it, e.g. is it periodic or immediate at the server side? -
@VRonin
OOI, how is theQSqlDriver::subscribeToNotificationimplemented at the SQL side? I'm familiar in the past with MS SQL Server, though from Qt I now use MySQL. Which SQL servers have thatQSqlDriver::hasFeature(), and how do they implement it, e.g. is it periodic or immediate at the server side?@JonB said in Best practices for displaying millions of loglines (colored and different line heights) using Qt (QTableView,...):
how is the QSqlDriver::subscribeToNotification implemented at the SQL side?
It's really db specific. For SQL server you can use CREATE EVENT NOTIFICATION in postgre you need to create a trigger on insert that sends an event notification
"La mort n'est rien, mais vivre vaincu et sans gloire, c'est mourir tous les jours"
~Napoleon BonaparteOn a crusade to banish setIndexWidget() from the holy land of Qt
-
@JonB said in Best practices for displaying millions of loglines (colored and different line heights) using Qt (QTableView,...):
how is the QSqlDriver::subscribeToNotification implemented at the SQL side?
It's really db specific. For SQL server you can use CREATE EVENT NOTIFICATION in postgre you need to create a trigger on insert that sends an event notification
-
Looks like the feature is available in Oracle DB but not in its little brother, sorry
"La mort n'est rien, mais vivre vaincu et sans gloire, c'est mourir tous les jours"
~Napoleon BonaparteOn a crusade to banish setIndexWidget() from the holy land of Qt
-
Hi,
You should consider implementing a moving window that only shows a fixed number of record, you can update the content of that "window" when scrolling back and fort. And if the user is at the top of the list, then you would only need to retrieve that reduced number of records.
@SGaist Thanks for the suggestion.
In fact this was another approach that I tried: I implemented something I called a "SliceModel" which was essentially what you are suggesting:
I used a QSortFilterProxyModel that did only accept rows with row numbers in between a certain range of rows.
A Window of rows was defined by the amount of lines that fit on a screen + some extra lines.I had to add another scrollbar to the TableView then, which displayed the amount of rows in the sourceModel.
When the user moved the scrollbar, this resulted in adjusting the windows start and end row and so the view itself displayed the desired range, but contained only so many lines as specified by the window range.The original scrollbar of the view had to be hidden, because it was tied to the proxy models data.
The complete concept was quite nice, because I had a very smooth scrolling experience, even when using millions of lines.
The problem with this approach was, that it did break the typical Model -> View concept and it got quite difficult to get a decent user experience.
