QML ListView and QAbstractListModel and prefetch part of the data from 10000 records

[Frenk21]

Thanks for the reply @jeremy_k. I will just stop on the first part as I stumble here:

@jeremy_k said in QML ListView and QAbstractListModel and prefetch part of the data from 10000 records:

@Frenk21 said in QML ListView and QAbstractListModel and prefetch part of the data from 10000 records:

I am trying to achive this, but no luck at the moment.

@jeremy_k I think I cant use DataLoader as you have suggested because my loader constructor depends on IPC connection class. So I have to use qmlRegisterUncreatedType in the QML. Or I am maybe missing something? I cant expose the IPC connection class to the QML.

It sounds like there is a misunderstanding. The DataLoader I proposed is used as a creatable type. IPC connections can be handled in C++, and invisible to QML.

I have a class DataLoader, but this one uses IPC connections which is actually other C++ class called IPCConnection. For the moment I pass this class as a reference to the DataLoader constructor. So this is the reason I cant use qmlRegisterType.
I could probably make IPCConnection as singleton and then use something like getInstance(), will revise the code if this will be ok. But how it is implemented now the qmlRegisterType will fail, except if I expose IPCConnection to the QML, but I dont want to do this.
If I want to internally use IPCConnection with signals/slots and connect() approach I there also need to pass an instance.
Or is there another way to achieve that a QML exported c++ class using the qmlRegisterType can use another c++ class without exposing it to the QML?

[jeremy_k]

I have a class DataLoader, but this one uses IPC connections which is actually other C++ class called IPCConnection. For the moment I pass this class as a reference to the DataLoader constructor. So this is the reason I cant use qmlRegisterType.

It's true that allowing the QML engine to instantiate objects means giving up some control of the constructor invocation. The same is true of most interfaces. There are other ways to influence the configuration. A singleton is one. A property identifying the parameter via a string, integer, or reference to another object is another.

Or is there another way to achieve that a QML exported c++ class using the qmlRegisterType can use another c++ class without exposing it to the QML?

Registering a type makes it available to QML, and dictates how an instance can be instantiated. It doesn't control what that type can use in its implementation, or force exposing additional classes.

The important part is code that works and is maintainable by the people working with the it. If imperative use of a loader object is easier to work with, that beats any discussion forum solution.

[Frenk21]

@jeremy_k
As again thanks for your great effort in explaining the unknown. Will try to do something in the c++ code as your solution seems really simple and efficient :-)

[Frenk21]

@jeremy_k

I have a reader which on read data always emits the same signal, but with different data. So at the moment all my objects are being updated after read operation is finished and all have at the end the same data. I assume that in your example:

void setRowId(int id) {
   reply = getRow(id);
   connect(reply, &Reply::finished, this, &DataLoader::setData);
 }

 void setData(Reply *reply) {
   this->picture = reply->picture;
   this->name = reply->name;
   delete request;
   emit pictureChanged();
   emit nameChanged();
 }

You made some list of Request/Response connect pairs that are unique? But if the reader returns always the same signal, how do you then distinct which slot must be called?

[jeremy_k]

@Frenk21 said in QML ListView and QAbstractListModel and prefetch part of the data from 10000 records:

@jeremy_k

I have a reader which on read data always emits the same signal, but with different data. So at the moment all my objects are being updated after read operation is finished and all have at the end the same data.

Something needs to demultiplex the data. It's more efficient to do that before it hits the delegate instance.

The alternative is to pass an identifier along with the data and let each delegate compare that id to its own. Doing so will lead to N * N notifications for N delegates, which isn't ideal.

I assume that in your example:

[ see https://forum.qt.io/post/684038 DataLoader::setRowId() and setData()]

You made some list of Request/Response connect pairs that are unique?

That wasn't the intention. Each delegate makes its own request, connects the reply to the delegate's reply handler, and returns. The connect statement should have used a lambda to capture the reply. Cleaning up finished replies was also omitted to focus on the core idea.

[Frenk21]

@jeremy_k

For the moment each DataLoader has a object Message that passes it to the Reader. DataLoader slot is connected to the Message signal. And when the Reader finish with reading it emits the Message.signal. And this is working fine. I am not sure if this is a good solution as I dont know if its ok that one class emit signal from the other class.

I can also assume that when I scroll the items in the ListView, the ListView will not load all of them. It will probably try to load/create visible, but as soon they become hidden it will unload/delete them. For example if I scroll fast from 0 to the row 1000 the ListView would not load all the items.

Is there any performance impact on using this approach of prefetching/caching. As now it seems ListView first creates delegates for the items, then update the values after I read them in compare to using just the Model and fetchMore()?

[jeremy_k]

@Frenk21 said in QML ListView and QAbstractListModel and prefetch part of the data from 10000 records:

@jeremy_k

For the moment each DataLoader has a object Message that passes it to the Reader. DataLoader slot is connected to the Message signal. And when the Reader finish with reading it emits the Message.signal. And this is working fine. I am not sure if this is a good solution as I dont know if its ok that one class emit signal from the other class.

Without code, it's hard to visualize. The description involves more steps than I imagined, but I don't know the details of the backend. Working is a good sign. Unfortunately it isn't proof of correctness in the way that not working is proof of incorrectness.

I have a quick proof of concept implementation of the pattern which might be of use. I'll post it after this message.

I can also assume that when I scroll the items in the ListView, the ListView will not load all of them. It will probably try to load/create visible, but as soon they become hidden it will unload/delete them. For example if I scroll fast from 0 to the row 1000 the ListView would not load all the items.

That's a ListView delegate management detail. cacheBuffer and reuseItems influence the management, but you're still handing over control in exchange for convenience. 0 to 1000 could be interpreted as going from 0-9 to 991-1000 with nothing between, or it could be just enough time in each segment to allow the corresponding delegate to be constructed before destruction. Performance testing can help decide which is more likely and tune accordingly.

Is there any performance impact on using this approach of prefetching/caching. As now it seems ListView first creates delegates for the items, then update the values after I read them in compare to using just the Model and fetchMore()?

Using the correct range of values from the beginning is going to take less computation and power than starting with a placeholder which is updated later. That's a cost imposed by a fluid/responsive user interface. fetchMore doesn't sound like a reasonable solution given the memory constraints and model size.

[jeremy_k]

main.cpp:

#include <QGuiApplication>
#include <QQmlApplicationEngine>
#include <QNetworkAccessManager>
#include <QNetworkReply>
#include <QObject>
#include <QTemporaryFile>
#include <QChar>
#include <QLibraryInfo>

class ImageLoader : public QObject {
    Q_OBJECT
    Q_PROPERTY(QUrl url MEMBER m_url WRITE setUrl NOTIFY urlChanged)
    Q_PROPERTY(QUrl image READ image NOTIFY imageChanged)
    QUrl m_url;
    QUrl m_imageFile;
    static QUrl m_defaultImage;

signals:
    void urlChanged();
    void imageChanged();

private:
    void handleReply(QNetworkReply *reply)
    {
        if (reply->error() != QNetworkReply::NoError)
            return;
        QVariant contentTypeVariant = reply->header(QNetworkRequest::ContentTypeHeader);
        if (! contentTypeVariant.isValid())
            return;
        QStringList contentType = contentTypeVariant.toString().split(QChar('/'));
        if (contentType.length() != 2)
            return;
        if (contentType.at(0) != QString("image"))
            return;
        QByteArray data = reply->readAll();
        {
            QTemporaryFile file(QString("example_XXXXXX.%1").arg(contentType.at(1)));
            file.setAutoRemove(false);
            file.open();
            file.write(data);
            m_imageFile = QUrl::fromLocalFile(file.fileName());
        }
        emit imageChanged();
    }

    void setUrl(const QUrl &newUrl)
    {
        if (m_url == newUrl)
            return;
        m_url = newUrl;
        QQmlEngine *engine = qmlEngine(this);
        if (!engine)
            return;
        QNetworkAccessManager *manager = engine->networkAccessManager();
        QNetworkRequest request(m_url);
        request.setAttribute(QNetworkRequest::AutoDeleteReplyOnFinishAttribute, true);
        QNetworkReply *reply = manager->get(request);
        QObject::connect(this, &QObject::destroyed, reply, &QNetworkReply::abort);
        QObject::connect(reply, &QNetworkReply::finished, this, [reply, this](){ handleReply(reply); });
        emit urlChanged();
    }

    const QUrl &image() const { return m_imageFile; }

public:
    ImageLoader(QObject *parent = nullptr) : QObject(parent), m_imageFile(ImageLoader::m_defaultImage) { }

    ~ImageLoader() {
        if (m_imageFile != m_defaultImage)
            QFile::remove(m_imageFile.toLocalFile());
    }
};

QUrl ImageLoader::m_defaultImage =
        QUrl::fromLocalFile(QLibraryInfo::location(QLibraryInfo::Qml2ImportsPath) + "/QtQuick/Dialogs/images/question.png");

int main(int argc, char *argv[])
{
    QGuiApplication app(argc, argv);

    qmlRegisterType<ImageLoader>("Example", 1, 0, "ImageLoader");

    QQmlApplicationEngine engine;
    const QUrl url(QStringLiteral("qrc:/main.qml"));
    QObject::connect(&engine, &QQmlApplicationEngine::objectCreated, &app,
    [url] (QObject *obj, const QUrl &objUrl) {
        if (!obj && url == objUrl)
            QCoreApplication::exit(-1);
    }, Qt::QueuedConnection);
    engine.load(url);

    return app.exec();
}

#include "main.moc"

main.qml:

import QtQuick 2.15
import QtQuick.Window 2.15
import Example 1.0

Window {
    width: 640
    height: 480
    visible: true

    ListView {
        anchors.fill: parent
        model: [ /* A list of image file URLs, eg "https://doc.qt.io/style/qt-logo-documentation.svg" */ ]
        delegate: Column {
            ImageLoader {
                id: loader
                url: modelData
            }
            Text {
                text: loader.url
            }
            Image {
                width: 100
                height: 100
                fillMode: Image.PreserveAspectFit
                source: loader.image
            }
        }
    }
}

[Frenk21]

@jeremy_k

Hello and sorry for delay.

Thanks for the example, but as the low communication part is already implemented and does not use QNetwork... I had to add my own read queue and demultiplex the signals. The code is now working perfectly :-) I really appreciate all the support and advices.

There is one observation that I found out. Sometimes when I scroll one object with the same index is apparently created, destroyed and immediate after that created once more. Even if the object/item is always visible.

So this post can be marked as solved :-)

Will try to put some rough example, very similar to yours, but it does not use QtNetwork..

[jeremy_k]

@Frenk21 said in QML ListView and QAbstractListModel and prefetch part of the data from 10000 records:

There is one observation that I found out. Sometimes when I scroll one object with the same index is apparently created, destroyed and immediate after that created once more. Even if the object/item is always visible.

The view management sometimes exhibits unexpected behavior. reuseItems is particular source of surprises.

Will try to put some rough example, very similar to yours, but it does not use QtNetwork..

I wouldn't bother going into the communication details, unless it's a point of concern and deserves its own question. Good examples should be a useful illustration for a wide audience, with documented public APIs. Nobody needs Hello, World for its own sake.