QTreeView memory consumption increasing

jinming

Hi, all, I read the qt example code(D:\Qt\Examples\Qt-6.5.1\widgets\itemviews\simpletreemodel). And I want to test the MVC memory consumption, so I change the function(TreeModel::TreeModel(const QString &data, QObject *parent)) to add more data.below is the code:

TreeModel::TreeModel(const QString &data, QObject *parent)
    : QAbstractItemModel(parent)
{
    //rootItem = new TreeItem({tr("Title"), tr("Summary")});
    //setupModelData(data.split('\n'), rootItem);
    rootItem = new TreeItem({ tr("column0"), tr("column1"), ("column2"), ("column3"), ("column4"), ("column5"), ("column6"), ("column7") });

    for (int i = 0; i < 100000; i++)
    {
        TreeItem* item = new TreeItem({ tr("column0"), tr("column1"), ("column2"), ("column3"), ("column4"), ("column5"), ("column6"), ("column7") }, rootItem);
        rootItem->appendChild(item);

        if (i == 0)
        {
            TreeItem* item1 = new TreeItem({ tr("column0"), tr("column1"), ("column2") }, item);
            item->appendChild(item1);
        }
    }

}

I add 100000 children item the the rootItem.When the program run,I run windows task manager to watch the program memory consumption.First when the Simple Tree Model runs, it causes 975.3 MB. and then I scroll down sliding bar to watch more data, and I find the program is increasing large to 3341.1MB .Below is the picture.

I search the internet, and I find some people also have the problem, but it seems to not resolve, like:
https://forum.qt.io/topic/142830/how-to-create-a-qtreeview-that-can-handle-very-large-amount-of-data or https://www.qtcentre.org/archive/index.php?t-30796.html
Why ??? And how to solve this problem. Thanks a lot.

sierdzio

@jinming Please keep in mind that Task Manager does not provide reliable memory usage data. The program can actually need less RAM, some of the reported value is memory that your app used to occupy but has since freed. It is up to your operating system to decide if and when to reassign that memory to free pool or to some other process.

With that said, the way to limit memory usage and increase performance is to use lazy loading canFetchMore() and fetchMore().

jinming

@sierdzio Hi, I change the the qt example code(D:\Qt\Examples\Qt-6.5.1\widgets\itemviews\simpletreemodel) to use canFetchMore() and fetchMore(), and I find some strange problem.This time, I use the Visual Studio to monitor the program memory increasing.First time, I set a break in the code (TreeModel init function) to watch 500000 item memory using.In the Visual Studio diagnostic window, it uses about 675MB, then I continue to run the code.When the UI windows appears,
the program uses memory about 1001MB,the memory uses is too big. Now I scroll the bar to watch more item data, it works normally without memory increasing, but in the items end, it works strangely,the memory is always slowly increasing, and the UI windows is blocking, which like the program is in some loop without return.Below is my code, which is reference by the qt example "simpletreemodel" and "fetchmore". Can you help me to look where is the problem? Thanks a lot.

treeitem.h

// Copyright (C) 2016 The Qt Company Ltd.
// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR BSD-3-Clause

#ifndef TREEITEM_H
#define TREEITEM_H

#include <QVariant>
#include <QList>
#include <QStyledItemDelegate>

//! [0]
class TreeItem
{
public:
    explicit TreeItem(const QList<QVariant> &data, TreeItem *parentItem = nullptr);
    ~TreeItem();

    void appendChild(TreeItem *child);

    TreeItem *child(int row);
    int childCount() const;
    int columnCount() const;
    QVariant data(int column) const;
    int row() const;
    TreeItem *parentItem();

private:
    QVector<TreeItem *> m_childItems;
    QVector<QVariant> m_itemData;
    TreeItem *m_parentItem;
};
//! [0]

#endif // TREEITEM_H

treeitem.cpp

// Copyright (C) 2016 The Qt Company Ltd.
// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR BSD-3-Clause

/*
    treeitem.cpp

    A container for items of data supplied by the simple tree model.
*/

#include "treeitem.h"
#include "treemodel.h"

//! [0]
TreeItem::TreeItem(const QList<QVariant> &data, TreeItem *parent)
    : m_itemData(data), m_parentItem(parent)
{}
//! [0]

//! [1]
TreeItem::~TreeItem()
{
    qDeleteAll(m_childItems);
}
//! [1]

//! [2]
void TreeItem::appendChild(TreeItem *item)
{
    m_childItems.append(item);
}
//! [2]

//! [3]
TreeItem *TreeItem::child(int row)
{
    if (row < 0 || row >= m_childItems.size())
        return nullptr;
    return m_childItems.at(row);
}
//! [3]

//! [4]
int TreeItem::childCount() const
{
    return m_childItems.count();
}
//! [4]

//! [5]
int TreeItem::columnCount() const
{
    return m_itemData.count();
}
//! [5]

//! [6]
QVariant TreeItem::data(int column) const
{
    if (column < 0 || column >= m_itemData.size())
        return QVariant();
    return m_itemData.at(column);
}
//! [6]

//! [7]
TreeItem *TreeItem::parentItem()
{
    return m_parentItem;
}
//! [7]

//! [8]
int TreeItem::row() const
{
    if (m_parentItem)
        return m_parentItem->m_childItems.indexOf(const_cast<TreeItem*>(this));

    return 0;
}
//! [8]

treemodel.h

// Copyright (C) 2016 The Qt Company Ltd.
// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR BSD-3-Clause

#ifndef TREEMODEL_H
#define TREEMODEL_H

#include <QAbstractItemModel>
#include <QModelIndex>
#include <QVariant>

class TreeItem;

//! [0]
class TreeModel : public QAbstractItemModel
{
    Q_OBJECT

public:
    explicit TreeModel(const QString &data, QObject *parent = nullptr);
    ~TreeModel();

    QVariant data(const QModelIndex &index, int role) const override;
    Qt::ItemFlags flags(const QModelIndex &index) const override;
    QVariant headerData(int section, Qt::Orientation orientation,
                        int role = Qt::DisplayRole) const override;
    QModelIndex index(int row, int column,
                      const QModelIndex &parent = QModelIndex()) const override;
    QModelIndex parent(const QModelIndex &index) const override;
    int rowCount(const QModelIndex &parent = QModelIndex()) const override;
    int columnCount(const QModelIndex &parent = QModelIndex()) const override;

protected:
    bool canFetchMore(const QModelIndex& parent) const override;
    void fetchMore(const QModelIndex& parent) override;

public:
    TreeItem *rootItem;
    int loadCount = 0;

private:
    void setupModelData(const QStringList &lines, TreeItem *parent);


};
//! [0]

#endif // TREEMODEL_H

treemodel.cpp

// Copyright (C) 2016 The Qt Company Ltd.
// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR BSD-3-Clause

/*
    treemodel.cpp

    Provides a simple tree model to show how to create and use hierarchical
    models.
*/

#include "treemodel.h"
#include "treeitem.h"

#include <QStringList>

//TreeItem* TreeModel::rootItem = nullptr;

//! [0]
TreeModel::TreeModel(const QString &data, QObject *parent)
    : QAbstractItemModel(parent)
{
    //rootItem = new TreeItem({tr("Title"), tr("Summary")});
    //setupModelData(data.split('\n'), rootItem);
    static int sCnt = 0;

    rootItem = new TreeItem({ "column0", tr("column1"), ("column2"), ("column3"), ("column4"), ("column5"), ("column6"), ("column7")});

    for (int i = 0; i < 500000; i++)
    {
        QString message = tr("column%1.").arg(sCnt++);
        TreeItem* item = new TreeItem({ message, tr("column1"), ("column2"), ("column3"), ("column4"), ("column5"), ("column6"), ("column7") }, rootItem);
        rootItem->appendChild(item);

        if (i == 0)
        {
            TreeItem* item1 = new TreeItem({ tr("column0"), tr("column1"), ("column2") }, item);
            item->appendChild(item1);
        }
    }
}
//! [0]

//! [1]
TreeModel::~TreeModel()
{
    delete rootItem;
}
//! [1]

//! [2]
int TreeModel::columnCount(const QModelIndex &parent) const
{

    if (parent.isValid())
    {
        return static_cast<TreeItem*>(parent.internalPointer())->columnCount();
    }
    return rootItem->columnCount();
}
bool TreeModel::canFetchMore(const QModelIndex& parent) const
{
    if (parent.isValid())
        return false;
    if (rootItem == nullptr)
        return false;
    return (loadCount < rootItem->childCount());
}
#include "qminmax.h"
static const int batchSize = 10000;
void TreeModel::fetchMore(const QModelIndex& parent)
{
    if (parent.isValid())
        return;
    if (rootItem == nullptr)
        return;

    const int start = loadCount;
    const int remainder = int(rootItem->childCount()) - start;
    const int itemsToFetch = qMin(batchSize, remainder);

    if (itemsToFetch <= 0)
        return;

    beginInsertRows(QModelIndex(), start, start + itemsToFetch - 1);

    loadCount += itemsToFetch;

    endInsertRows();
}
//! [2]

//! [3]
QVariant TreeModel::data(const QModelIndex &index, int role) const
{
    if (!index.isValid())
        return QVariant();

    if (role != Qt::DisplayRole)
        return QVariant();

    TreeItem *item = static_cast<TreeItem*>(index.internalPointer());

    return item->data(index.column());
}
//! [3]

//! [4]
Qt::ItemFlags TreeModel::flags(const QModelIndex &index) const
{
    if (!index.isValid())
        return Qt::NoItemFlags;

    return QAbstractItemModel::flags(index);
}
//! [4]

//! [5]
QVariant TreeModel::headerData(int section, Qt::Orientation orientation,
                               int role) const
{
    if (orientation == Qt::Horizontal && role == Qt::DisplayRole)
        return rootItem->data(section);

    return QVariant();
}
//! [5]

//! [6]
QModelIndex TreeModel::index(int row, int column, const QModelIndex &parent) const
{
    if (!hasIndex(row, column, parent))
        return QModelIndex();

    TreeItem *parentItem;

    if (!parent.isValid())
        parentItem = rootItem;
    else
        parentItem = static_cast<TreeItem*>(parent.internalPointer());

    TreeItem *childItem = parentItem->child(row);
    if (childItem)
        return createIndex(row, column, childItem);
    return QModelIndex();
}
//! [6]

//! [7]
QModelIndex TreeModel::parent(const QModelIndex &index) const
{
    if (!index.isValid())
        return QModelIndex();

    TreeItem *childItem = static_cast<TreeItem*>(index.internalPointer());
    TreeItem *parentItem = childItem->parentItem();

    if (parentItem == rootItem)
        return QModelIndex();

    return createIndex(parentItem->row(), 0, parentItem);
}
//! [7]

//! [8]
int TreeModel::rowCount(const QModelIndex &parent) const
{
    TreeItem *parentItem;

    if (parent.column() > 0)
        return 0;

    if (!parent.isValid())
    {
        parentItem = rootItem;
        return loadCount;
    }
    else
    {
        parentItem = static_cast<TreeItem*>(parent.internalPointer());
        return parentItem->childCount();
    }
}
//! [8]

void TreeModel::setupModelData(const QStringList &lines, TreeItem *parent)
{
    QList<TreeItem *> parents;
    QList<int> indentations;
    parents << parent;
    indentations << 0;

    int number = 0;

    while (number < lines.count()) {
        int position = 0;
        while (position < lines[number].length()) {
            if (lines[number].at(position) != ' ')
                break;
            position++;
        }

        const QString lineData = lines[number].mid(position).trimmed();

        if (!lineData.isEmpty()) {
            // Read the column data from the rest of the line.
            const QStringList columnStrings =
                lineData.split(QLatin1Char('\t'), Qt::SkipEmptyParts);
            QList<QVariant> columnData;
            columnData.reserve(columnStrings.count());
            for (const QString &columnString : columnStrings)
                columnData << columnString;

            if (position > indentations.last()) {
                // The last child of the current parent is now the new parent
                // unless the current parent has no children.

                if (parents.last()->childCount() > 0) {
                    parents << parents.last()->child(parents.last()->childCount()-1);
                    indentations << position;
                }
            } else {
                while (position < indentations.last() && parents.count() > 0) {
                    parents.pop_back();
                    indentations.pop_back();
                }
            }

            // Append a new item to the current parent's list of children.
            parents.last()->appendChild(new TreeItem(columnData, parents.last()));
        }
        ++number;
    }
}

main.cpp

// Copyright (C) 2016 The Qt Company Ltd.
// SPDX-License-Identifier: LicenseRef-Qt-Commercial OR BSD-3-Clause

#include "treemodel.h"

#include <QApplication>
#include <QFile>
#include <QTreeView>

#include "treeitem.h"

int main(int argc, char *argv[])
{
    Q_INIT_RESOURCE(simpletreemodel);

    QApplication app(argc, argv);

    QFile file(":/default.txt");
    file.open(QIODevice::ReadOnly);
    TreeModel model(file.readAll());
    file.close();

    QTreeView view;
    view.setUniformRowHeights(true);

    view.setModel(&model);
    view.setWindowTitle(QObject::tr("Simple Tree Model"));
    view.show();
    //view.setS
    return app.exec();
}

Finally, I use the Visual Studio to load simpletreemodel/CMakeLists.txt CMake project.Below is the test picture:

In the end the memory is slowly increasing

JonB

@jinming
Displaying half a million records in a treeview is not a great idea.

JoeCFD

@jinming Jon is right. Better to display on demand. It is unnecessary to add all of them all in once.

#include <QApplication>
#include <QMainWindow>
#include <QTreeWidget>
#include <QTreeWidgetItem>
#include <QScrollBar>
#include <QVBoxLayout>

class MyWindow : public QMainWindow {
   Q_OBJECT

public:
   MyWindow(QWidget *parent = nullptr) : QMainWindow(parent) {
       treeWidget = new QTreeWidget(this);
       treeWidget->setColumnCount(1);
       treeWidget->setHeaderLabels({"Items"});

       setCentralWidget(treeWidget);

       // Connect the itemEntered signal to the slot
       connect(treeWidget, &QTreeWidget::itemEntered, this, &MyWindow::onItemEntered);

       // Populate the initial items
       populateTree();

       // Connect the scroll bar value changed signal to the slot
       connect(treeWidget->verticalScrollBar(), &QScrollBar::valueChanged, this, &MyWindow::onVerticalScroll);
   }

   void populateTree() {
       // Add some top-level items
       for (int i = 0; i < 50; ++i) {
           QTreeWidgetItem *topItem = new QTreeWidgetItem(treeWidget, QStringList() << "Top Item " + QString::number(i));
           // Add a dummy child item to make the parent item expandable
           new QTreeWidgetItem(topItem, QStringList() << "Loading...");
       }
   }

public slots:
   void onItemEntered(QTreeWidgetItem *item, int column) {
       // Check if the item already has child items (the dummy item for loading)
       if (item->child(0) && item->child(0)->text(0) == "Loading...") {
           // Remove the dummy item
           delete item->takeChild(0);
           // Load and add the actual child items
           for (int i = 0; i < 3; ++i) {
               new QTreeWidgetItem(item, QStringList() << "Child Item " + QString::number(i));
               // Add more child items or load data as needed
           }
       }
   }

   void onVerticalScroll(int value) {
       // Iterate through visible items and trigger onItemEntered for each
       QTreeWidgetItemIterator it(treeWidget);
       while (*it) {
           if (treeWidget->visualItemRect(*it).intersects(treeWidget->viewport()->rect())) {
               // The item is currently visible
               onItemEntered(*it, 0);
           }
           ++it;
       }
   }

private:
   QTreeWidget *treeWidget;
};

int main(int argc, char *argv[]) {
   QApplication a(argc, argv);
   MyWindow w;
   w.show();
   return a.exec();
}

sierdzio

You are not loading anything in fetchMore(), you still do full loading in the constructor of your model. So you will not gain anything this way.

jinming

@JonB said in QTreeView memory consumption increasing:

@jinming
Displaying half a million records in a treeview is not a great idea.

You are right,but my app is always reading data from USB device like CAN Card.I need alway display the last receive data in a UI, so people can know what is last data in CAN bus,.You know the data from USB device is growing by the time,so I think the MVC model is suitable and I select TreeView,but TreeView uses large memory likely.

JonB

@jinming
I was just making an observation that (a) if you show half a million records in a tree view it's likely to cost quite some memory and (b) I wonder how accessible/useful that many records in a widget is to the end user.

Maybe you will have to "prune" some of the older records as you go along and the new ones arrive to keep the total number lower. Or only create the (deeper) tree nodes "on demand" as the user actually interacts with the tree and expands parent nodes.

jinming

@JoeCFD You are right,but my app is always reading data from USB device like CAN Card.I need alway display the last receive data in a UI, so people can know what is last data in CAN bus,.You know the data from USB device is growing by the time,so I think the MVC model is suitable and I select TreeView,but TreeView uses large memory likely.
You give me a idea about each time display 50 item's data in large data.The problem is when vertical scroll bar changing, I need update the 50 items display.

jinming

@JonB said in QTreeView memory consumption increasing:

@jinming
I was just making an observation that (a) if you show half a million records in a tree view it's likely to cost quite some memory and (b) I wonder how accessible/useful that many records in a widget is to the end user.

Maybe you will have to "prune" some of the older records as you go along and the new ones arrive to keep the total number lower. Or only create the (deeper) tree nodes "on demand" as the user actually interacts with the tree and expands parent nodes.

Yes, I need make some policy on QTreeView, for example, QTreeView only displays 1024 item, I need custom vertical scroll bar to select 1024 item data from the large data.

jsulm

@jinming said in QTreeView memory consumption increasing:

I need alway display the last receive data in a UI

But not ALL the data received so far, right?

jinming

@sierdzio said in QTreeView memory consumption increasing:

You are not loading anything in fetchMore(), you still do full loading in the constructor of your model. So you will not gain anything this way.

In the constructor, I just make the ready data and lazy loading by vertical scroll bar.
In the rowCount function, I just return the loadCount.I think it works.

int TreeModel::rowCount(const QModelIndex &parent) const
{
    TreeItem *parentItem;

    if (parent.column() > 0)
        return 0;

    if (!parent.isValid())
    {
        parentItem = rootItem;
        return loadCount;
    }
    else
    {
        parentItem = static_cast<TreeItem*>(parent.internalPointer());
        return parentItem->childCount();
    }
}

jinming

@jsulm said in QTreeView memory consumption increasing:

@jinming said in QTreeView memory consumption increasing:

I need alway display the last receive data in a UI

But not ALL the data received so far, right?

It depends on the CAN baudrate,for example, if the baudrate is 500Kb/s, we will recive 4000 frames which DLC is 8 in every second. I need find a efficient way to display large data without the UI blocking. Maybe I need make some policy on QTreeView, for example, QTreeView only displays 1024 item, I need custom vertical scroll bar to select 1024 item data from the large data.

sierdzio

@jinming said in QTreeView memory consumption increasing:

In the constructor, I just make the ready data

That is exactly the issue :-) You create all the objects in the constructor, thus taking a long time and a lot of RAM upfront.

With lazy loading the point is to defer (delay in time) initialising of the data until it is actually needed - when user scrolls close to it.

jinming

@sierdzio said in QTreeView memory consumption increasing:

@jinming said in QTreeView memory consumption increasing:

In the constructor, I just make the ready data

That is exactly the issue :-) You create all the objects in the constructor, thus taking a long time and a lot of RAM upfront.

With lazy loading the point is to defer (delay in time) initialising of the data until it is actually needed - when user scrolls close to it.

The constructor is model, it is called once. My pointer is to test how memory the QTreeView uses to display large data, so I scroll the vertical scroll bar to watch the QTreeView memory increasing by Visual Studio IDE

JoeCFD

@jinming if loading blocks UI, send loading to a thread.