QTreeView + sort model + QStandardItemModel, the most efficient way to update the model?

Violet Giraffe

So , I have my own QTreeView subclass, which gets the data from my QSortFIlterProxyModel sublcass, which gets the data from my QStandardItemModel subclass. Here's what I do when I need to update the model with new data:

sortModel->setSourceModel(nullptr) - to avoid sorting + view updates as I update the model;
model->clear();
create all the items for the model based on the new data;
set the item for each column and each row by calling model->setItem;
set the model back into the chain: sortModel->setSourceModel(model);

Is there a more effective, faster way to do this? I'm currently testing with 3000 items. Step 4 is fast, but step 5 is slow - almost 1000 ms, must be sorting + displaying the items in the view. Assuming all the items really do change, meaning incremental refresh instead of complete refresh won't be too beneficial, is there anything else I can optimize?

P. S. Interestingly, one of my view columns has an icon, but it barely makes any performance impact at all (just removed the setIcon call and compared times). That gives me a hint that the actual rendering of the new items by the view happens asynchronously, so that's not what affects step 5 time.

antis

@Violet-Giraffe Please ask yourself: Did I really understand the model/view concept? You need to differ between the terms "item" and "model index". The view is a complete separate thing and usually you don't even need to inherit that, unless you do really fancy stuff, like e.g. implement your own paint() method.

As a general rule: If you experience problems in performance with a data source <100.000 items, there's something (badly) wrong with your code/architecture.

Tip: Depending on your goal, consider the use of QAbstractItemModel::sort() instead inheriting QSortFilterProxyModel.

In other words:
Resetting (clearing) the model and disabling stuff is a last resort (!) and no solution to your problem. The only reason to rebuild the model is, when your data source completely is replaced. For example, closing file A and opening file B. Basically you need to delete your steps 1, 2 and 5 and insert the items before sorting them.

And ... don't care about the view in the first place. It is well designed and faster than you think :)!

Violet Giraffe

I do some input events management and selection management in my tree view. I only mentioned that because if I need to alter something in the tree view - I'm ready to.
I implemented a QSortFilterProxyModel to implement my custom lessThan logic for natural sorting of strings.
Now, I really didn't understand your suggestion. Are you saying clearing the model is not required? Do I only need to add / delete rows according to the new number of items, and call setItem for each? But calling setItem forces update of the whole chain - re-sorting items in the sort model, repaint of the view. Does it not? I'm removing and re-setting the model to kinda do it in batch mode - update the data then refresh everything once, instead of refreshing after each setItem call (of which, btw, there are 3k x 4 columns = 12k).

You didn't really explain why setSourceModel costs ~800 ms for 3k items.

Violet Giraffe

If I simply remove steps 1 and 5 and leave everything else the same, it takes so long I didn't wait for the completion. Over 10 seconds.
I think the model API is missing a batch setItem method. This setModel trick really shouldn't be necessary.

JKSH

Hi @Violet-Giraffe,

If you want a high-performance model, don't use QStandardItemModel. That class is designed for easy programming, not for performance.

Subclass a QAbstractTableModel instead.

antis

Sorry, I didn't use QStandardItemModel for quite a while and simply overread that fact - assuming you inherited QAbstractItemModel. I agree to @JKSH: The best suggestion I can give you is to learn about model/view programming and inherit QAbstractItemModel.

I wrote some quick and dirty test application to your problem, to give you some inspiration of how not to do it :). The relevant part is here:

MainWindow::MainWindow(QWidget *parent) :
    QMainWindow(parent)
{
    setupUi(this);

    tableView->setModel( &mModel );
    tableView->setSortingEnabled(true);
}

void MainWindow::createModel(int count)
{
    // THIS IS DIRTY CODING!
    // DON'T DO IT IN REAL LIFE!
    mModel.clear();
    mModel.setRowCount(count);
    mModel.setColumnCount(2);
    // to further speed up, you can disable the view update
    tableView->setUpdatesEnabled(false);

    QStringList listOne = QStringLiteral("A B C D E F G H I J K L M N O P Q R S T U V W X Y Z")
                          .split(QLatin1Char(' '));

    for ( int row = 0; row < count; row++ ) {
        QList<QStandardItem*> columns;
        columns << new QStandardItem( QString::fromUtf8("#%1").arg(row, 4, 10, QLatin1Char('0')) )
                << new QStandardItem( listOne[row % listOne.size()] );

        // usual approach should be this:
        //mModel.appendRow(columns);

        for ( int col = 0; col < columns.size(); col++) {
            mModel.setItem( row, col, columns[col] );
        }
    }

    tableView->setUpdatesEnabled(true);
    tableView->update();
}

void MainWindow::on_btnCreateItems_clicked()
{
    const int count = spinBox->value();
    QTime stopWatch;
    stopWatch.start();

    qDebug("Creating %d items ...", count);
    createModel( count );
    qDebug( "Created %d items in %d ms", count, stopWatch.elapsed() );

    // one time sort of items in model
    qDebug("Sorting items ...");
    stopWatch.restart();
    mModel.sort( 1 );
    qDebug("Sorted %d items by column 1 in %dms", count, stopWatch.elapsed());
}

You should also avoid the use of QSortFilterProxyModel, which serves a special purpose. For simple sorting tasks (sort by column), you are better off by QStandardItemModel::sort() / QAbstractItemModel::sort(). In the item based approach, this will change the order of your QStandardItems. In the model based approach, the item order stays always untouched!

Basically you write your own model like this:

class MyModel : public QAbstractItemModel
{
public:
  // initialize the model with a tree like structure
  void initModel(MyItem* item);

  QVariant data(const QModelIndex& index, int role) const;
  
  // important: create a model index for items (usually requested by a view update) 
  QModelIndex index(int row, int col, QModelIndex& parentIndex) {
     // you need to find the item by row (not column!)
     // In a tree structure, row starts with 0 on every tree level!
     return createIndex(row, col, item);
  }
  QModelIndex parent(int row, int col, const QModelIndex& index) {
    // Simplified: for a tree, return a model index with the pointer to it's parent item
    return item->parent() ? createIndex(row, 0, item->parent() : QModelIndex();
  }

  // to insert items, you need to voerride "insertRow"
  // You can sort the model by column(s) overriding sort. Underlying data structure is not touched!

private:
  // for a tree, you provide the root item 
  MyItem* root; 
};

If you don't require filtering and want to sort by column, I'd suggest to remove the QSortFilterProxyModel and use the default QAbstractItemModel::sort() instead.

Finally, let me answer your question:

Are you saying clearing the model is not required? Do I only need to add / delete rows according to the new number of items, and call setItem for each?

Exactly! You need to insert/delete the items/indexes. Still, the QStandardItemModel is slow, because if you don't badly hack it, it creates a QStandardItem for each data field, plus a more or less invisible, but lightweight QModelIndex. Basically the item based approach is complete crap and you would be better off by using a QTreeWidget, if you look for "simplicity" ... sorry :(.

Clearing/rebuilding the model means, all the model indexes are recalculated. In case of sorting, this is giving you the bad performance issues.

This explanation turned out long. Hope, this gives you the information you need.

Violet Giraffe

Thanks!

I have started by using a QTreeWidget, but the number of features I was unable to implement with it has long ago reached the critical value. So I rewrote half my UI for QTreeView, created the models etc.
I'm definitely NOT going to subclass a QAbstractItemModel in any near future. That was the first thing I did after moving to tree view from tree widget, and it was a horrible experience. It was unclear which methods I must implement, and for some methods it was unclear what they should actually do. The Qt documentation in this part is lacking at best. Besides, my current model is simple in its basic code, but complicated in additional features (which I'm not looking forward to rewriting), and it just WORKS. If speed is the price of functionality, I guess I can live with it.

I need filtering so I assume I still need to keep the QSortFIlterProxyModel in the hierarchy, but I suppose I can remove my own subclass if the same can be achieved by QAbstractItemModel::sort(). All I need is to supply a different operator< for each column.

Do I understand correctly that my current implementation is the best possible for QStandardItemModel? The only improvement that comes to mind is leave everything the same, but replace model.clear() with model.setColumnCount().

JKSH

@Violet-Giraffe said:

I'm definitely NOT going to subclass a QAbstractItemModel in any near future. That was the first thing I did after moving to tree view from tree widget, and it was a horrible experience.

If you don't need a tree, subclassing QAbstractTableModel is much nicer to use.

Do I understand correctly that my current implementation is the best possible for QStandardItemModel? The only improvement that comes to mind is leave everything the same, but replace model.clear() with model.setColumnCount().

Yes. setColumnCount() lets you keep the existing items (you can just modify their contents, instead of deleting everything and then reconstructing them all again).

Note that if the new column count is bigger than before, I think you need to explicitly construct new QStandardItems to put into the columns (I can't remember; I haven't used QStandardItemModel in a long long time)

Finally, if you only want to add new items, use addColumn()/appendColumn().

Violet Giraffe

Thanks. I didn't use table view because I thought it has less functionality even when there are no children items. Was I wrong?

JKSH

@Violet-Giraffe said:

Thanks. I didn't use table view because I thought it has less functionality even when there are no children items. Was I wrong?

If you don't have a tree model, then there is no benefit in using a tree view.

But anyway, I was talking about the model, not the view.

A QAbstractTableModel is a subclass of QAbstractItemModel, so it inherits all functionality. However, QAbstractTableModel is easier to use, because it already implements lots of things needed by QAbstractItemModel, so you don't have to implement them yourself.

antis

Hey guys,

I wrote a little application demonstrating the usage of QAbstractTableModel with generic table-like data. It can be found here and can easily handle 500.000+ items. As long as there's enough memory, there's no real maximum border.

Unless building a customized sort system, QAbstractItemModel::sort() changes the original item order - which usually comes with a lot of problems. Thus, I decided to use also a QSortFilterProxyModel approach. This can be improved as well, but is fast enough for the demo.

It is noteworthy, that item creation is about at least twice as fast, as with the tweaked QStandardItemModel approach. Memory footprint should also be much smaller.

@Violet-Giraffe Hope this helps to improve your understanding of Qt-MV programming.

Violet Giraffe

Thanks a lot, this looks interesting! Definitely a very, very useful example.
I expect a whole bunch of problems from switching tree view to table view and from QAbstractItemModel to QAbstractTableModel, but I'll give it a try!

Violet Giraffe

I've just tried QTableView and immediately recalled why I didn't use it in the first place. It just looks bad:

http://i.imgur.com/DQusulq.png

Two problems with it:

The top and bottom margins of every item are too huge, at least 5 times bigger than in a QTreeView;
despite the selection mode being set to "Select rows", the current item cursor (dotted frame) is only drawn around a specific column, not the whole row. This also isn't a problem with QTreeView. Here's a tree view screen shot for comparison: http://i.imgur.com/ICp2gNA.png

And I'd say the general visual style of the tree view (selection etc.) is better and looks more native to Windows. But the table does indeed work a bit faster.

antis

To 1: Take a look here or better try the following:

tableView->verticalHeader()->setSectionResizeMode(QHeaderView::ResizeToContents);
tableView->setStyleSheet( QStringLiteral("QAbstractItemView::item{ margin: 0; padding: 0; }") );

To 2: What's wrong with that? Same applies to QTreeView. This depends on the selection model used. You can simply set behaviour:

tableView->selectionModel().setSelectionBehaviour( QAbstractItemView::SelectRows );

Btw.: I don't want to confuse you, but do you know there's a QDirModel prepared for you? This seems pretty much like what you're looking for. Found a nice tutorial.

Violet Giraffe

Very interesting, I'll try your suggestions, shortly, thanks.
On 2: I have set selection behavior to "Select rows" for the view, but not the model. It worked for tree view, but not for table view, so it confused me.

I have heard of the QDirModel. I'm making a file manager, so I need a flexible and powerful solution. Which means I have to write it all myself. Besides, I need to separate the model (UI level code) from the actual data (core level).

Violet Giraffe

Wait, QItemSelectionModel doesn't have a setSelectionBehaviour member! So I can't do

tableView->selectionModel().setSelectionBehaviour( QAbstractItemView::SelectRows );

Violet Giraffe

Another observation: doing

verticalHeader()->setSectionResizeMode(QHeaderView::ResizeToContents);
setStyleSheet("QAbstractItemView::item{ margin: 0; padding: 0; }");

Totally kills performance. Looks like the view does a LOT of relayouting, or maybe just calculating font metrics for each item. Something like that, judging from the call stack. Either way, the UI thread is 100% busy and the event loop is barely executing.

antis

Wait, QItemSelectionModel doesn't have a setSelectionBehaviour member! So I can't do

Right, but QAbstractItemView has:

tableView->setSelectionBehaviour( QAbstractItemView::SelectRows );

antis

@Violet-Giraffe Just try without setting resize mode.

Further, the css rule might not be right. Sorry, you need to figure that out by yourself.

Violet Giraffe

@antis said:

Right, but QAbstractItemView has:
tableView->setSelectionBehaviour( QAbstractItemView::SelectRows );

That it does! And like I said, I have set this option. the table view does select a whole row, but it does not span the cursor across the whole row. While the tree view does both no problem.