Simplest way to write M-to-N (eg. aggregating) proxy models?

[ssokolow]

I have several projects where I need to display a QTreeView sidebar that acts as a filter for a main view. However, unlike in a file manager, the source data is flat and entirely in memory.

I've decided to implement populating the QTreeView as a proxy model and I did find this Qt Center thread for group/aggregate models, but it doesn't cover hooking up the signals and slots to react to changing data and that's an area where I don't feel confident in my understanding.

Can someone point me to a simple example of how to write a properly updating proxy model which isn't limited to 1-to-1 mappings? (Or, failing that, the clearest possible guide on how to properly hook up the various signals and slots related to mutating data in a model)

One of my projects does require many-to-many mappings (M tags, grouped by type, to N tagged objects) but, if there's a simpler option for one-to-many (eg. click a folder to see files within) mappings, I'd like to know for the projects to which it applies.

I'm prototyping in Python but may rewrite in C++ once the design stabilizes, so any code will need to either be C++ with PyQt5-compatible Python bindings or be simple enough that it's not a hassle to port it myself.

[VRonin]

I'm not sure I understand what you are trying to do...

Is it something like converting a db-like table structure in a table?

ID	Region	Client	Amount
1	EU	Merkel	100
2	EU	Hollande	50
3	EU	Merkel	20
4	USA	Trump	1
5	USA	Clinton	2

should become (bold numbers are in a different column)

• EU 170
  • Merkel 120
    • 1 100
    • 3 20
  • Hollande 50
    • 2 100
• USA 3
  • Trump 1
    • 4 1
  • Clinton 2
    • 5 2

Is this correct?

[ssokolow]

@VRonin

No. While I may have a use for that at some point, right now, my most pressing use case has a source model capable of producing a tabular view like this:

Title	Provider
[icon] Game Name 1	<processed string>
[icon] Game Name 2	<processed string>
[icon] Game Name 3	<processed string>

...and, importantly, Qt::UserRole on any column returns that row's raw object from the underlying data model, which has members like this:

• Game Name 1:
  • sources: {'heuristics', 'xdg'}
  • launchers: {<launcher: role: play>}
  • ...
• Game Name 2:
  • sources: {'heuristics', 'gog.com'}
  • launchers: {<launcher: role: play>, <launcher: role: install>, etc.}
  • ...
• Game Name 3:
  • sources: {'scummvm'}
  • launchers: {<launcher: role: play>, etc.}
  • ...

And the result I'm trying to get from the filter model is a QTreeView that looks like this:

• Sources
  • Filesystem Heuristics
  • GOG.com
  • ScummVM
  • System Launchers
• Install Status
  • Downloaded
  • Playable

(The human-readable names come from a separate set of objects)

Then, for example, if you click "Downloadable", "selection changed" signal-slot connections should cause another filter model to limit another view to a list of records you see things with a role=install launcher but no role=play launcher.

Since a picture really is worth 1000 words in this case, here's what I have so far, with everything but the sidebar functional and the sidebar using a very minimal temporary hack to populate partial data for visual effect.

[VRonin]

This looks tatally duable using 2 QSortFilterProxyModel subclasses...
you just need a way to link "installed" to "<launcher: role: install>"

[ssokolow]

@VRonin

Linking data and metadata together is no problem. (<launcher: role: install> is just my sloppy approximation of the debugging string the Launcher instances return if you print() a reference to them in the Python prototype)

I was actually already planning to subclass and override QSortFilterProxyModel::filterAcceptsRow for filtering the central QListView/QTableView widget.

(Given that I have complex multi-select behaviour planned, such as "(GOG OR ScummVM) AND (Downloaded OR Playable)", combined with the aforementioned "Downloaded excludes install+play, but GOG includes heuristics+gog", I'm obviously going to need to code my own "Do we want this?" routine.)

What has always been the problem is performing the complex M-to-N conversion to feed the QTreeView with as little wheel-reinvention as possible, and, even with you saying it should be possible, I have no idea how I'd do that by extending QSortFilterProxyModel (or even QAbstractProxyModel) with less effort than by subclassing QAbstractItemModel and reinventing everything.

From my reading of the docs, QSortFilterProxyModel and QAbstractProxyModel seem pretty wedded to the idea of a one-to-one mapping between input and output rows.

[VRonin]

@ssokolow I don't know how you link metadata to human readable strings but if its something like a "dictionary" you can use that directly as a tree

[ssokolow]

@VRonin

This looks tatally duable using 2 QSortFilterProxyModel subclasses...

I don't know how you link metadata to human readable strings but if its something like a "dictionary" you can use that directly as a tree

While the project I need this for most immediately may allow me to solve this by forcing a QStandardItemModel to be the authoritative copy of the sidebar's data, ensuring the data is in fifth normal form, and then just overriding filterAcceptsRow to let the search field control the sidebar visibility, I asked my question the way I did because, even in this project, that's a bit of a contortion and not all of the projects I have coming can be solved that way.

Sometimes, I can't just normalize the data before it enters Qt's purview and, when entirely new "sidebar" entries must come into existence as a side-effect of new denormalized data arriving in the main panel, that brings this back to what I wrote initially:

Can someone point me to a simple example of how to write a properly updating proxy model which isn't limited to 1-to-1 mappings? (Or, failing that, the clearest possible guide on how to properly hook up the various signals and slots related to mutating data in a model)

Emphasis mine. The former point (updating) is beyond the scope of this Qt Center thread and the latter (not limited to 1-to-1 mappings) makes QAbstractProxyModel and its decendants ineligible because mapFromSource and mapToSource take and return QModelIndex.

Again...

1. Let's assume I know what I'm talking about and I really do need to dynamically normalize aspects of data from one model to feed a supplementary view and react as the source model changes.
2. I am perfectly capable of reinventing the wheel... it's just that...
   • Maybe I have too much faith in Qt, but doing this from scratch just because I need an M-to-N mapping feels like reinventing QAbstractProxyModel to implement multi-column filtering because I wasn't aware of QSortFilterProxyModel::filterAcceptsRow.
   • I don't feel confident in my understanding of which signals/slots I need to use when in other to ensure that the two views don't get out of sync... hence my request for example code.

[VRonin]

Sorry, I didn't want to come off as dismissive.
From your answers in other posts I am firmly convinced you know your stuff very well, apologies again if my tone was disrespectful on that side.

Without knowing how your data is stored it's difficult however to give you any concrete answer.

Could you give us a 2 or 3 rows example of your data (including what data is stored in what role)?

[ssokolow]

@VRonin

I'm a bit skeptical that 2 or 3 rows of example data would be relevant to developing a solution I can apply to the other projects I have waiting in the wings (hence my asking about a general solution), but I'll give as much as I can.

The Qt side of the data pipeline is still somewhat in flux (partly because I have yet to implement persistent column hide/show for the tabular view and certain custom item delegates, which means that much of the data is retrieved by retrieving the raw data objects via UserRole, bypassing the intended way of doing things.)

However, here's how it currently works:

1. The authoritative data is in the form of a list of objects which my backend test code dumps like this (with null entries removed):

   [
     {
       "name": "A Boy And His Blob",
       "provider": ["GOG.com"], 
       "icon": "/mnt/buffalo_ext/games/a_boy_and_his_blob_2.1.0.2/support/icon.png", 
       "base_path": "/mnt/buffalo_ext/games/a_boy_and_his_blob_2.1.0.2", 
       "commands": [
         {
           "name": "Play", 
           "role": Roles.play, 
           "provider": "GOG.com", 
           "sort_key": [ Roles.play,  ["/mnt/buffalo_ext/games/a_boy_and_his_blob_2.1.0.2/start.sh",  "--start"]], 
           "tryexec": "/mnt/buffalo_ext/games/a_boy_and_his_blob_2.1.0.2/start.sh", 
           "argv": ["/mnt/buffalo_ext/games/a_boy_and_his_blob_2.1.0.2/start.sh",  "--start"], 
           "categories": [
             "Game", 
             "ActionGame"
           ], 
   
         }, 
     }, 
   ]
   

   (Note: I synthesized the categories list since, at this point in time, the GOG.com backend doesn't set it.)

2. The main model wraps that data (without proper dynamic updating due to my aforementioned lack of confidence in my understanding of how to hook up the signals and slots) and maps each row as follows:

   | |Title|Providers|
   |-|-|-|
   DisplayRole|entry.name|', '.join(x.backend_name for x in entry.provider)|
   DecorationRole|icon_provider.get_icon(entry.icon, ICON_SIZE)||
   ToolTipRole|entry.summarize()||
   UserRole|entry|entry|

3. While it doesn't yet, the sidebar should produce a tree based on these transforms, with entries hidden if they contain no results (eg. "Not Downloadable" would only be visible if at least one manually installed game was removed after the system started keeping records for it):

   |Top-level|Child-node|Criteria for inclusion|
   |-|-|-|
   Install Status||all|
   |Not Downloadable|Not in any other category|
   |Downloadable|Has no Roles.install but provider supports downloading|
   |Downloaded|Has Roles.install but not Roles.play|
   |Installed|Has Roles.play|
   Play Status||<created-by-default entry in user-defined categories system>|
   |Unplayed|<created-by-default entry in user-defined categories system>|
   |Started|<created-by-default entry in user-defined categories system>|
   |Beaten|<created-by-default entry in user-defined categories system>|
   |Complete|<created-by-default entry in user-defined categories system>|
   |Endless|<created-by-default entry in user-defined categories system>|
   Source||all|
   |<dynamic>|One for each provider plugin|
   Genres||all|
   |<dynamic>|aggregated and deduplicated from entry.categories|
   <dynamic>||User-defined categories system allows users to create their own tag classes|
   |<dynamic>|User-defined categories system allows users to create and apply their own tags|

Clicking on a sidebar entry should filter the main view, so the pipeline to reach the QListView or QTableView should look like this:

http://imgur.com/a/K9UhI

[VRonin]

Ok, now I get what is going on. Sorry for being this dull.

What you are trying to do should not be done via a proxy model. proxy models should only modify the structure of the data, it shouldn't add any data to the original model.

CategoriesModel should be a different model populated and changed when GameListModel changes (use the dataChanged signal to detect them). they should not however be proxies of the same model as they contain different data (even if only partially different)

[ssokolow]

@VRonin

To be fair to my past self, I'd forgotten about the plan for custom categories when I made my initial post and the play status was still envisioned as a hard-coded enum like Install Status. (And, when seen like that, it is purely an M-to-N aggregation of data already present in a source model that could be seen as a denormalized SQL-like table where some columns contain delimited lists.)

That said, unless the docs are wrong, dataChanged isn't sufficient.

This signal is emitted whenever the data in an existing item changes.

Emphasis mine.

This is why, even when I was asking about a high-level API for building M-to-N proxies, I was saying "or failing that, example code or reference information (more focused than the API docs) on how all of the 'something was added/removed/changed' signals and slots are intended to fit together"

Even the source GameListModel is effectively a proxy (since it follows rather than holds the authoritative copy of the data) similar in concept to QFileSystemModel or QSqlQueryModel and both GameListModel and CategoriesModel need to be enhanced with proper change notification. (Currently, I'm just throwing out GamesListModel and generating a whole new one whenever the user hits F5... which is clearly not optimal, if for no other reason than doing so resets view state like the selection.)

I've produced an updated data flow diagram to acknowledge that aspect and to leave myself a reminder that CategoriesModel isn't a pure proxy filter:

https://imgur.com/a/1E9x8

(I'm going to assume that, for the other projects which do require pure "normalize/aggregate tabular data at runtime" behaviour, there's still no high-level helper for doing M-to-N proxying, so the answers I need to build this should also be what I need to write some kind of AbstractAggregatingProxyModel for those.)

[VRonin]

The closest thing I found to what you are trying to do is KCategorizedSortFilterProxyModel from KDE but it's still 1 to 1 mapping, the view also only supports lists (1 column and no children). Nevertheless it might be a good place to start given apparently this will not be something you'll find out of the box.

An example using KCategorizedSortFilterProxyModel can be:

#include <QApplication>
#include <KCategorizedView>
#include <KCategoryDrawer>
#include <QStandardItemModel>
#include <KCategorizedSortFilterProxyModel>
#include <QPixmap>
#include <QIcon>
int main(int argc, char **argv)
{
    QApplication app(argc,argv);
    QPixmap bluePix(48,48);
    bluePix.fill(Qt::blue);
    QStandardItemModel baseModel;
    baseModel.insertRows(0, 3);
    baseModel.insertColumns(0, 2);
    baseModel.setHeaderData(0, Qt::Horizontal, "Col1");
    baseModel.setHeaderData(1, Qt::Horizontal, "Col2");
    baseModel.setData(baseModel.index(0, 0), "0.0");
    baseModel.setData(baseModel.index(0, 0), QIcon(bluePix), Qt::DecorationRole);
    baseModel.setData(baseModel.index(0, 1), "0.1");
    baseModel.setData(baseModel.index(1, 0), "1.0");
    baseModel.setData(baseModel.index(1, 1), "1.1");
    baseModel.setData(baseModel.index(2, 0), "2.0");
    baseModel.setData(baseModel.index(2, 1), "2.1");
    baseModel.setData(baseModel.index(2, 0),5,KCategorizedSortFilterProxyModel::CategorySortRole);
    baseModel.setData(baseModel.index(2, 1),5,KCategorizedSortFilterProxyModel::CategorySortRole);
    baseModel.setData(baseModel.index(2, 0),"Test",KCategorizedSortFilterProxyModel::CategoryDisplayRole);
    baseModel.setData(baseModel.index(2, 1),"Test",KCategorizedSortFilterProxyModel::CategoryDisplayRole);
    KCategorizedSortFilterProxyModel proxyModel;
    proxyModel.setSourceModel(&baseModel);
    proxyModel.setCategorizedModel(true);
    KCategorizedView mainView;
    mainView.setCategoryDrawer(new KCategoryDrawer(&mainView));
    mainView.setModel(&proxyModel);
    mainView.show();
    return app.exec();
}

[ssokolow]

@VRonin

I appreciate the suggestion, but KCategorizedSortFilterProxyModel is 1-to-1 because it subclasses QSortFilterProxyModel, which means it provides no benefit over just poking through the source of QAbstractProxyModel to figure out how all of the "thing X changed in manner Y" signals and slots are supposed to be hooked up.

(Unfortunately, I'm not familiar with Qt internals and I've been having trouble finding the code that does said hook-ups.)

Hell, if I can just get confident in hooking up those signals and slots, I could probably split the "Games" input on CategoriesModel out into a QSortFilterProxyModel subclass with a custom filterAcceptsRow and plumb CategoriesModel directly into things like the game provider backends in order to add the non-user-defined categories.

[VRonin]

KCategorizedSortFilterProxyModel is 1-to-1 because it subclasses QSortFilterProxyModel, which means it provides no benefit over just poking through the source of QAbstractProxyModel

It does as it adds the category that is an extra piece of data not present on the original model but agree it's not great.

to figure out how all of the "thing X changed in manner Y" signals and slots are supposed to be hooked up.

dataChanged is emitted anytime setData is called so while not being sufficient in general (adding an empty line will not trigger it) it is in practice (as soon as you populate that empty line dataChanged will be emitted) the other signals to keep an eye on are rowsRemoved, modelReset and possibly columnsRemoved

[ssokolow]

@VRonin

dataChanged is emitted anytime setData is called so while not being sufficient in general (adding an empty line will not trigger it) it is in practice (as soon as you populate that empty line dataChanged will be emitted) the other signals to keep an eye on are rowsRemoved, modelReset and possibly columnsRemoved

That makes no sense. I'm using QAbstractItemModel rather than QStandardItemModel, which means I don't just magically get insertRows for free, which means that the signals you listed can't be all there is to it.

I get the impression you're still be operating under the assumption that Qt's model holds the authoritative copy of the data in some sense of the word, rather than simply being a proxy that takes a non-Qt "model" as its source.

I have to get to bed but, once I get back tomorrow, why don't I just write the kind of summary I was hoping to find and then you can tell me if I got anything obviously wrong.

[VRonin]

That makes no sense. I'm using QAbstractItemModel rather than QStandardItemModel, which means I don't just magically get insertRows for free

If you don't, you are subclassing QAbstractItemModel the wrong way. http://doc.qt.io/qt-5/qabstractitemmodel.html#subclassing clearly specifies:

The dataChanged() and headerDataChanged() signals must be emitted explicitly when reimplementing the setData() and setHeaderData() functions, respectively.

An insertRows() implementation must call beginInsertRows() before inserting new rows into the data structure, and endInsertRows() immediately afterwards.

and endInsertRows takes care of emitting rowsInserted. below you find the implementation

void QAbstractItemModel::endInsertRows()
{
    Q_D(QAbstractItemModel);
    QAbstractItemModelPrivate::Change change = d->changes.pop();
    d->rowsInserted(change.parent, change.first, change.last);
    emit rowsInserted(change.parent, change.first, change.last, QPrivateSignal());
}

These mechanisms are relied upon by the views to work that's why they are so strict

[ssokolow]

@VRonin

Yes, but the docs for insertRows also say:

Note: The base class implementation of this function does nothing and returns false.

If you implement your own model, you can reimplement this function if you want to support insertions. Alternatively, you can provide your own API for altering the data. In either case, you will need to call beginInsertRows() and endInsertRows() to notify other components that the model has changed.

In other words, I have two choices for supporting unsolicited insertion events from the backend:

• I can maintain the backend's internal model separate from the frontend's model (in which case, I might as well use QStandardItemModel) and write unappealing sync code to manually mirror changes back and forth.

• I can write my QAbstractItemModel subclass such that the only state it keeps is cached results for expensive-to-derive fields and manually call beginInsertRows and endInsertRows when a new row arrives from the backend without warning.

To rephrase what I said before, we're obviously not on the same page because the only way your advice, as stated, makes sense, is if I'm taking the former approach, which I don't consider acceptable.

When I'm free tomorrow, I'll try to summarize my understanding of when to call which begin* and end* functions and you can tell me if I've missed or misunderstood any.

[ssokolow]

Things went bad and I only got about 3 or 4 hours of sleep. I'll summarize as soon as I've had a good night's sleep.

[ssokolow]

Sorry for the lack of response. Shortly after my last message, I got bombarded with obligations and it's taken me the last two weeks to get back to where I was before.

I may or may not have a proper response in the next couple of days. The rest of the family is just getting over a debilitating flu and I've started to show faint symptoms.

[VRonin]

That's often the result of meddling with model/view programming :)

Hope you get better