Model for a list of large data in QML
-
Thank you all for your comments.
I have not strict limitation about what kind of list or table to use. But I think a TableView will not let me to implement things like the expandable/collapsable details or adding "fancy" things. So I think I should go for a ListView instead of a TableView.
I have actually implemented kind of a test app using QSqlQueryModel but when the data changes I call setQuery again...this I think is very slow and the control is re-filled so I lost the location where the user was in the list. Also with these approach I don't know how to keep data that is not coming from the DB...for instance the status of a row (expanded/collapsed).
@XDePedro said in Model for a list of large data in QML:
I have actually implemented kind of a test app using QSqlQueryModel but when the data changes I call setQuery again...this I think is very slow and the control is re-filled
Yes, this is a big problem for you to think through. With your data:
a. Do you only want to re-fetch incrementally this values which have been added?
b. When you do re-fetch, do you wish all the old, existing rows to still be in the model/view, or do you want those to be removed and only the new stuff now shown?so I lost the location where the user was in the list
This one is a minor point: if necessary, note the position in the list (via something unique in the data, e.g. a primary key) and restore after re-populate.
-
My - perhaps crazy - thought: use the QSql model to read your DB (side note: perhaps some other db would work better, like psql or mariadb. SQLite tends to be rather slower than others), preferably in a separate thread. When DB data changes, emit a signal to notify the upper layers. Then set up another model, like QAbstractListModel or model proxy which will:
- remember which list elements were expanded/ collapsed (btw. to display these list items, I'd recommend implementing some custom QML delegate). You can simply extend the data object you get from SQL model and add a boolean variable to mark which items were expanded
- present the "last read" info from your DB. So, while your SQL model will be busy reading new data (after some background data change), your proxy model will still hold and display old data. Once all is read, it will get the signal from SQL model and update itself
(Z(:^
-
My - perhaps crazy - thought: use the QSql model to read your DB (side note: perhaps some other db would work better, like psql or mariadb. SQLite tends to be rather slower than others), preferably in a separate thread. When DB data changes, emit a signal to notify the upper layers. Then set up another model, like QAbstractListModel or model proxy which will:
- remember which list elements were expanded/ collapsed (btw. to display these list items, I'd recommend implementing some custom QML delegate). You can simply extend the data object you get from SQL model and add a boolean variable to mark which items were expanded
- present the "last read" info from your DB. So, while your SQL model will be busy reading new data (after some background data change), your proxy model will still hold and display old data. Once all is read, it will get the signal from SQL model and update itself
-
@sierdzio
This sounds reasonable to me.
The only thing is: given the OP's "crazy" ( ;-) ) requirement to have "at least 100k" items in his lists, it sounds like the approach will require yet another copy of all the data rows! :(@JonB said in Model for a list of large data in QML:
@sierdzio
This sounds reasonable to me.
The only thing is: given the OP's "crazy" ( ;-) ) requirement to have "at least 100k" items in his lists, it sounds like the approach will require yet another copy of all the data rows! :(That's why it will be crucial to implement @dheerendra's idea of windowed view. Or at the very least use lazy initialization (
canFetchMore()andfetchMore()).(Z(:^
-
@JonB said in Model for a list of large data in QML:
@sierdzio
This sounds reasonable to me.
The only thing is: given the OP's "crazy" ( ;-) ) requirement to have "at least 100k" items in his lists, it sounds like the approach will require yet another copy of all the data rows! :(That's why it will be crucial to implement @dheerendra's idea of windowed view. Or at the very least use lazy initialization (
canFetchMore()andfetchMore()).@sierdzio
Yes, but independent of whether lazy reading is involved, my comment was that OP will have one ("large") list of rows in hisQSqlQueryModeland another ("large", potentially very similar) list in his newQAbstractListModel. It may be inevitable, but it was just an observation. -
Oh, right, that's true.
-
@XDePedro said in Model for a list of large data in QML:
I have actually implemented kind of a test app using QSqlQueryModel but when the data changes I call setQuery again...this I think is very slow and the control is re-filled
Yes, this is a big problem for you to think through. With your data:
a. Do you only want to re-fetch incrementally this values which have been added?
b. When you do re-fetch, do you wish all the old, existing rows to still be in the model/view, or do you want those to be removed and only the new stuff now shown?so I lost the location where the user was in the list
This one is a minor point: if necessary, note the position in the list (via something unique in the data, e.g. a primary key) and restore after re-populate.
@JonB
It might be a minor problem but I am still trying to solve it. I have to signals: beginDBChange and endDBChange. I use the first one to store the index of the selected item and then the second one to set it as a current item index. It works but I still have some annoying scrollings to set the item at the beginning or the end of the table. I would like this to be completely transparent to the user. -
@sierdzio
This sounds reasonable to me.
The only thing is: given the OP's "crazy" ( ;-) ) requirement to have "at least 100k" items in his lists, it sounds like the approach will require yet another copy of all the data rows! :( -
@JonB
It might be a minor problem but I am still trying to solve it. I have to signals: beginDBChange and endDBChange. I use the first one to store the index of the selected item and then the second one to set it as a current item index. It works but I still have some annoying scrollings to set the item at the beginning or the end of the table. I would like this to be completely transparent to the user.@XDePedro
Not an area I know about; can only make possible suggestion:If you really cannot solve that by (somehow) pre-scrolling before new list is shown --- because list is being completely repopulated from new data --- ISTM you will have to come up with whatever solution to only append new rows to visual list without complete reset/redraw, whatever that might involve.
-
Sounds crazy :P but I think I'll give it a try.
Does someone know if the QSqlQuery use any kind of lazy initialization? or when you execute a query the data is being loaded regardless nobody ask for it??
@XDePedro said in Model for a list of large data in QML:
Does someone know if the QSqlQuery use any kind of lazy initialization?
Like all Q*Model classes, there is canFetchMore() and fetchMore().
Whether any further laziness is implemented in the actual loading - I don't know.
(Z(:^
-
@XDePedro said in Model for a list of large data in QML:
Does someone know if the QSqlQuery use any kind of lazy initialization?
Like all Q*Model classes, there is canFetchMore() and fetchMore().
Whether any further laziness is implemented in the actual loading - I don't know.
@sierdzio
I have been wondering about just this forQSqlQueryModeletc. for a while now!Do we know how the "fetchMore" is actually implemented? I presume this is a driver implementation (
QMYSQL)? For example, does it actually use a SQLCURSORon a still on-going query (I hope not! but maybe it does), does it simply mean it fetches another packet from some complete SQL resultset, or what??This makes a huge difference to how I might implement efficiency in my SQL code....
-
@sierdzio
Thanks for that!Unfortunately for me, I do all my work in PyQt. That means I can't step through the code in debugger to follow everything, as I would if I used C++ and had the sources.
Although the page's hyperlinks are good, they're not nearly nearly the same as following through stepping.
A brief look doesn't tell me much. Though:
QSqlQueryModel::fetchMore()
Fetches more rows from a database.
This only affects databases that don't report back the size of a query
(see QSqlDriver::hasFeature()).At a guess MySQL will return "size of a query" (perhaps SQLite does not), so it may be all irrelevant in my situation?
In any case, as you'll see I don't think the implementation will be in
QSql.... That will rely on the abstraction level of the driver it's using,QMYSQL, and I'm thinking that is where what I would like to know is buried...But thanks anyway.
