QAbstractItemModel::Index(...) and Internal pointers when Data in persistant memory not in RAM

[Kofr]

I want to work with my QAbstractitem model derived class as a adapter for SQLite database and Do not have model's items in RAM,
The internal pointer size is not enough to keep 16 bytes Id of item.
I have quite complex SQL database structure where children of the parent with the id = "123' are contained in discrete database table with name "123". So I need to have this 16 byte Id to locate items across SQL tadabase. I also did not get a logic when and in what order views call Index(...) from model.
The database is huge, I have to use 16 bytes (BLOB) values to provide a unique id for every item.
How can I implement QModelIndex QAbstractItemModel::index(int row, int column, const QModelIndex &parent = QModelIndex()) const in my class so I can provide a 16 bytes value as an internal pointer or internal id of returned QModelIndex in createIndex(row, column, internalPointer) ?
comment: model provides the tree structure (with child - parent relationship)

[raven-worx]

@Kofr
i am not quite sure if i got your question right, but the internal pointer can point to anything you like. So you can store your QString id, your custom struct, or whatever you want to the internal pointer.

[Kofr]

@raven-worx I thought this way.
But I come into the problem that I do not know when to delete this key. I could not track when this value is used.
For example 'Key *key = getKey(parentKey)', next, 'return createIndex(row, column, key)'
But be because I do not know when it is used, I can not be sure when to do 'delete key'

The only idea is to overload like this 'Key &operator*() {addPointerForDeletionList(this); return *this; }'
So, delete this pointer in 1 sec for example.

[raven-worx]

@Kofr
The pointer should be owned by the model. Since the model normally also holds the data.
The internal pointer of course should only be valid as long as the index (or it's data it points to) is also valid.
So actually you need to store all pointers as long as the data is valid.

[Kofr]

@raven-worx In worst case I will have to keep in RAM about 4e+38 of 16 bytes pointers, sounds so so.
How to figure out when exactly index() is called. I believe that the model can request indexes on demand, not once for a runtime.

[raven-worx]

@Kofr
the view can request an index at any time (for whatever reason it needs it), until the model signals that the index became invalid / has moved / etc.

[Kofr]

@raven-worx
Do I understand correctly that there is no way to avoid keeping in RAM all internal pointers by model?

[raven-worx]

@Kofr
as long as the model index points to valid data the pointer must also remain valid.
If thats a problem you may want to change your design. For example you could implement some sort of paging and only display a max amount of rows.

[Kofr]

@raven-worx , dear, can you advice some of paging technics or how can I overcome this QAbstractItemModel limitation with adressing morethan 2^32 items in a model with minimal resource usage?

[raven-worx]

@Kofr
with paging i meant do not download all the data into the modle. But only 1000, next page the next 1000 and so on.
You are using a database right?

[Kofr]

@raven-worx Yes, I use QSLite number of tables, where I have top items table and number of tables which bontain children of different items.
So, one of columns in table keeps name of the table with children.
I do not get what to you mean how to load some number of items from table because data may be shown from very different tables.

So far I got that QQuickTreeModelAdaptor keeps only visible items cached, and thus I think I may try to keep model's data in RAM just by knowing what data do I need.
However yet I do not find solution to access TreeModelAdaptor to obtain list of visible items.

[raven-worx]

@Kofr said in QAbstractItemModel::Index(...) and Internal pointers when Data in persistant memory not in RAM:

So far I got that QQuickTreeModelAdaptor keeps only visible items cached, and thus I think I may try to keep model's data in RAM just by knowing what data do I need.
However yet I do not find solution to access TreeModelAdaptor to obtain list of visible items.

The model data may also be accessed for not visible items. Since you do not know the purpose for which the data is requested.
And still when using QQuickTreeModelAdaptor, the modle is responsible for the internal pointers. So this doesn't help you in any with your current problem.

Even if your model is used by different tables. Then create a model base class with only queries X number of entries. This model has a properties to define the data range from the DB table.
Then for each table create a new model and set this range for each table on it's own.

[kshegunov]

@Kofr said in QAbstractItemModel::Index(...) and Internal pointers when Data in persistant memory not in RAM:

how can I overcome this QAbstractItemModel limitation with adressing morethan 2^32 items in a model with minimal resource usage?

It isn't practical to think you can display 4 billion items and even if you could it wouldn't make much sense. Taking this in mind and the numbers you mentioned earlier, not only won't you be able to display those items, but you can't even hold them in memory. So for all intents and purposes void * is more than enough to hold reference to anything.

And another note: with those parent-child tables it sounds to me you've denormalized the native table structure for no apparent reason.

[Kofr]

@kshegunov I am not intended to keep in RAM 2^32 items, I just want to be able to address with internal pointer of QModelIndex more items.
QQuickTreeModelAdaptor which is in use with QML's TreeView calls QAbstractItemModel::data(..) and need to pass eligible 16 byte data encrypted in QModelIndex::InternalPointer somehow.
According to tour point about my solution with multiple tables for every parent item, I believe that access to table by name directly is much faster than designating filtering condition for select statement. And I have benefits of easy working with ordering items.

[kshegunov]

@Kofr said in QAbstractItemModel::Index(...) and Internal pointers when Data in persistant memory not in RAM:

@kshegunov I am not intended to keep in RAM 2^32 items

This contradicts this post you've made earlier:

In worst case I will have to keep in RAM about 4e+38 of 16 bytes pointers, sounds so so.

I just want to be able to address with internal pointer of QModelIndex more items.

The internal addressing of QModelIndex allows you to address up to 2^flavor elements, where flavor is your OS's flavor - 32 bit or 64 bit.
All this means with a 64 bit OS you can address up to 2^64 elements, isn't this enough?

QQuickTreeModelAdaptor which is in use with QML's TreeView calls QAbstractItemModel::data(..) and need to pass eligible 16 byte data encrypted in QModelIndex::InternalPointer somehow.

Just remap the identifiers in the model. Here:

typedef char Db16ByteId[16];
QVector<Db16ByteId> idMapping;

that's it. Then QModelIndex will contain the index in this idMapping vector.

I believe that access to table by name directly is much faster than designating filtering condition for select statement.

I believe you believe wrong.

And I have benefits of easy working with ordering items.

Well, it's your code, I just wouldn't do it like this.

[A Former User]

Hi! Just to be sure I understood this correctly: You have a single DB with a humongous number of identically structured tables? Do you really believe you can store 2^128 bits / rows / tables / whatever in your database / filesystem / datacenter?

In worst case I will have to keep in RAM about 4e+38 of 16 bytes pointers

Even that is way more rows than what a SQLite table can hold (see: https://www.sqlite.org/limits.html).