QAbstractItemModel::Index(...) and Internal pointers when Data in persistant memory not in RAM

raven-worx

@Kofr
the view can request an index at any time (for whatever reason it needs it), until the model signals that the index became invalid / has moved / etc.

Kofr

@raven-worx
Do I understand correctly that there is no way to avoid keeping in RAM all internal pointers by model?

raven-worx

@Kofr
as long as the model index points to valid data the pointer must also remain valid.
If thats a problem you may want to change your design. For example you could implement some sort of paging and only display a max amount of rows.

Kofr

@raven-worx , dear, can you advice some of paging technics or how can I overcome this QAbstractItemModel limitation with adressing morethan 2^32 items in a model with minimal resource usage?

raven-worx

@Kofr
with paging i meant do not download all the data into the modle. But only 1000, next page the next 1000 and so on.
You are using a database right?

Kofr

@raven-worx Yes, I use QSLite number of tables, where I have top items table and number of tables which bontain children of different items.
So, one of columns in table keeps name of the table with children.
I do not get what to you mean how to load some number of items from table because data may be shown from very different tables.

So far I got that QQuickTreeModelAdaptor keeps only visible items cached, and thus I think I may try to keep model's data in RAM just by knowing what data do I need.
However yet I do not find solution to access TreeModelAdaptor to obtain list of visible items.

raven-worx

@Kofr said in QAbstractItemModel::Index(...) and Internal pointers when Data in persistant memory not in RAM:

So far I got that QQuickTreeModelAdaptor keeps only visible items cached, and thus I think I may try to keep model's data in RAM just by knowing what data do I need.
However yet I do not find solution to access TreeModelAdaptor to obtain list of visible items.

The model data may also be accessed for not visible items. Since you do not know the purpose for which the data is requested.
And still when using QQuickTreeModelAdaptor, the modle is responsible for the internal pointers. So this doesn't help you in any with your current problem.

Even if your model is used by different tables. Then create a model base class with only queries X number of entries. This model has a properties to define the data range from the DB table.
Then for each table create a new model and set this range for each table on it's own.

kshegunov

@Kofr said in QAbstractItemModel::Index(...) and Internal pointers when Data in persistant memory not in RAM:

how can I overcome this QAbstractItemModel limitation with adressing morethan 2^32 items in a model with minimal resource usage?

It isn't practical to think you can display 4 billion items and even if you could it wouldn't make much sense. Taking this in mind and the numbers you mentioned earlier, not only won't you be able to display those items, but you can't even hold them in memory. So for all intents and purposes void * is more than enough to hold reference to anything.

And another note: with those parent-child tables it sounds to me you've denormalized the native table structure for no apparent reason.

Kofr

@kshegunov I am not intended to keep in RAM 2^32 items, I just want to be able to address with internal pointer of QModelIndex more items.
QQuickTreeModelAdaptor which is in use with QML's TreeView calls QAbstractItemModel::data(..) and need to pass eligible 16 byte data encrypted in QModelIndex::InternalPointer somehow.
According to tour point about my solution with multiple tables for every parent item, I believe that access to table by name directly is much faster than designating filtering condition for select statement. And I have benefits of easy working with ordering items.

kshegunov

@Kofr said in QAbstractItemModel::Index(...) and Internal pointers when Data in persistant memory not in RAM:

@kshegunov I am not intended to keep in RAM 2^32 items

This contradicts this post you've made earlier:

In worst case I will have to keep in RAM about 4e+38 of 16 bytes pointers, sounds so so.

I just want to be able to address with internal pointer of QModelIndex more items.

The internal addressing of QModelIndex allows you to address up to 2^flavor elements, where flavor is your OS's flavor - 32 bit or 64 bit.
All this means with a 64 bit OS you can address up to 2^64 elements, isn't this enough?

QQuickTreeModelAdaptor which is in use with QML's TreeView calls QAbstractItemModel::data(..) and need to pass eligible 16 byte data encrypted in QModelIndex::InternalPointer somehow.

Just remap the identifiers in the model. Here:

typedef char Db16ByteId[16];
QVector<Db16ByteId> idMapping;

that's it. Then QModelIndex will contain the index in this idMapping vector.

I believe that access to table by name directly is much faster than designating filtering condition for select statement.

I believe you believe wrong.

And I have benefits of easy working with ordering items.

Well, it's your code, I just wouldn't do it like this.

? Offline

Hi! Just to be sure I understood this correctly: You have a single DB with a humongous number of identically structured tables? Do you really believe you can store 2^128 bits / rows / tables / whatever in your database / filesystem / datacenter?

In worst case I will have to keep in RAM about 4e+38 of 16 bytes pointers

Even that is way more rows than what a SQLite table can hold (see: https://www.sqlite.org/limits.html).