Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. [SOLVED] Asynchronous data loading for QFileSystemModel
Forum Updated to NodeBB v4.3 + New Features

[SOLVED] Asynchronous data loading for QFileSystemModel

Scheduled Pinned Locked Moved General and Desktop
11 Posts 2 Posters 4.7k Views 2 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • C Offline
    C Offline
    chabo
    wrote on last edited by chabo
    #1

    I would like to display meta-data and thumbnail stored in file(own format). Because loading and parsing these data can take some time I would like to do it on separate thread. I implemented it using QtConcurrent::run and emitting dataChanged signal.

    Code:

    class MyModel : public QFileSystemModel
    {
      Q_OBJECT
     
    public:
      MyModel(QObject *parent = nullptr);
      virtual ~MyModel();
     
      QVariant data(const QModelIndex& index, int role) const override;
     
    signals:
      void thumbnailLoaded(const QString &);
    
    protected slots:
      void updateThumbnail(const QString &);
    
    protected:
      void loadThumbnail(const QString & path, const QDateTime & time);
     
    protected:
      mutable QMutex _mutex;
     
      using Thumbnails = QHash<QString, QDateTime>;
      mutable Thumbnails _thumbnails;
      mutable QPixmapCache _pixmapCache;
    };
     
     MyModel::MyModel(QObject * parent)
      : QFileSystemModel(parent)
    {
      // FIX
      connect(this, SIGNAL(thumbnailLoaded(const QString &)), this, SLOT(updateThumbnail(const QString &)), Qt::QueuedConnection);
    }
    
    MyModel::~MyModel()
    {
      QMutexLocker lock(&_mutex);
    
      _thumbnails.clear();
    }
    
    QVariant MyModel ::data(const QModelIndex& index, int role) const
    {
      if (role == Qt::DecorationRole)
      {
        QFileInfo info = fileInfo(index);
     
        if (info.isFile() && info.suffix() == "my")
        {
    /*
          // Synchronous
          QPixmap pixmap;
     
          auto it = _thumbnails.find(info.filePath());
          if (it == _thumbnails.end() || it.value() < info.lastModified() || !_pixmapCache.find(info.filePath(), &pixmap))
          {
            if (auto screenshot = LoadScreenShotFromFile(info.filePath()))
            {
              pixmap.loadFromData(screenshot->GetData(), screenshot->GetSize());
     
              _thumbnails[info.filePath()] = info.lastModified();
              _pixmapCache.insert(info.filePath(), pixmap);
            }
          }
    */
          // Asynchronous
          QPixmap pixmap;
          bool load = false;
     
          {
            QMutexLocker lock(&_mutex);
     
            auto it = _thumbnails.find(info.filePath());
            if (it != _thumbnails.end())
            {
              if (it.value().isValid())
              {
                if (it.value() == info.lastModified())
                {
                  load = !_pixmapCache.find(info.filePath(), &pixmap);
                }
                else
                {
                  load = true;
                }
              }
            }
            else
            {
              _thumbnails[info.filePath()] = QDateTime();
     
              load = true;
            }
          }
         
          if (load)
          {
            QtConcurrent::run(const_cast<MyModel *>(this), &MyModel::loadThumbnail, info.filePath(), info.lastModified());
          }
     
          return pixmap;
        }
      }
     
      return QFileSystemModel::data(index, role);
    }
     
     
    void MyModel::loadThumbnail(const QString & path, const QDateTime & time)
    {
      QPixmap pixmap;
      bool loaded = false;
     
      if (auto screenshot = LoadScreenShotFromFile(info.filePath()))
      {
          loaded = pixmap.loadFromData(screenshot->GetData(), screenshot->GetSize());
      }
    
      bool changed = false;
      {
        QMutexLocker lock(&_mutex);
    
        auto it = _thumbnails.find(path);
        if (it != _thumbnails.end() && it.value() <= time)
        {
          if (loaded)
          {
            *it = time;
            _pixmapCache.insert(path, pixmap);
          }
          else
          {
            _thumbnails.erase(it);
            _pixmapCache.remove(path);
          }
    
          changed = true;
        }
      }
    
      if (changed)
      {
        // FIX
        emit thumbnailLoaded(path);
      }
    }
    
    void MyModel::updateThumbnail(const QString & path)
    {
      QModelIndex i = index(path);
      
      if (i.isValid())
      {
        emit dataChanged(i, i, QVector<int>{QFileSystemModel::FileIconRole});
      }
    }
    

    The problem is that sometime view displaying this model is corrupted, only small icons without file names or only file names are displayed.
    Code should be thread safe. Signal from the thread pool is run in GUI thread. Only problem I see is const_cast.

    Am I missing something? Thanks for any help.

    SOLVED

    C 1 Reply Last reply
    0
    • SGaistS Offline
      SGaistS Offline
      SGaist
      Lifetime Qt Champion
      wrote on last edited by
      #2

      Hi and welcome to devnet,

      QFileSystemModel is already thread so you might be creating interference currently. You should maybe start creating your thumbnails only after directoryLoaded has been emitting so you are sure that all the information of the folder you are accessing is already available to the model.

      Hope it helps

      Interested in AI ? www.idiap.ch
      Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

      C 1 Reply Last reply
      0
      • SGaistS SGaist

        Hi and welcome to devnet,

        QFileSystemModel is already thread so you might be creating interference currently. You should maybe start creating your thumbnails only after directoryLoaded has been emitting so you are sure that all the information of the folder you are accessing is already available to the model.

        Hope it helps

        C Offline
        C Offline
        chabo
        wrote on last edited by
        #3

        @SGaist

        Hi, thanks for answer, but I already tried it and it didn't help.

        1 Reply Last reply
        0
        • SGaistS Offline
          SGaistS Offline
          SGaist
          Lifetime Qt Champion
          wrote on last edited by
          #4

          Rather than using a mutex, why not return a "busy" image while you create the thumbnail ? Once it's created emitting data changed will trigger the update and you can return the thumbnail

          Interested in AI ? www.idiap.ch
          Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

          C 1 Reply Last reply
          0
          • SGaistS SGaist

            Rather than using a mutex, why not return a "busy" image while you create the thumbnail ? Once it's created emitting data changed will trigger the update and you can return the thumbnail

            C Offline
            C Offline
            chabo
            wrote on last edited by chabo
            #5

            @SGaist
            Could you point me to some documentation about "busy" image please. I am new to Qt.
            I thought returning empty QPixmap while thumbnail is loading is enough, but maybe this is causing the problem.
            After thumbnail is loaded I am emitting dataChanged.

            1 Reply Last reply
            0
            • SGaistS Offline
              SGaistS Offline
              SGaist
              Lifetime Qt Champion
              wrote on last edited by
              #6

              By "busy image" I meant a non null pixmap containing e.g. a clock or whatever you see fit to make your user understand that some processing is occurring

              Interested in AI ? www.idiap.ch
              Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

              1 Reply Last reply
              0
              • C Offline
                C Offline
                chabo
                wrote on last edited by chabo
                #7

                Ok, I've tried to return some preloaded image while thumbnail is loading. After real thumbnail was loaded and dataChanged was emitted, element size in QListView keeps the same size as preloaded image. I've also tried to return QColor which is also acceptable as result by DecorationRole in data() function. In this case it looks same as when I was returning empty QPixmap. Elements was small.
                So I think problem is somewhere else. It looks like QListView doesn't react properly when element size is changed by DecorationRole in model data() function. Or I am missing some function call.
                One more observation, this happens only when widget with QListView is hidden and then it shows up. I have QListView on QTabWidget. Same implementation for view and model, but problem occurs on tab which is hidden and then showed up.
                Window resize fixes the problem when resizeMode is equal QListView::Adjust

                1 Reply Last reply
                0
                • SGaistS Offline
                  SGaistS Offline
                  SGaist
                  Lifetime Qt Champion
                  wrote on last edited by
                  #8

                  Can you share the setup of your QListView ?

                  Interested in AI ? www.idiap.ch
                  Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

                  1 Reply Last reply
                  0
                  • C Offline
                    C Offline
                    chabo
                    wrote on last edited by chabo
                    #9
                     model = new MyModel(this);
                     model->setRootPath(QDir::currentPath());
                     model->setReadOnly(false);
                     model->setFilter(QDir::Files | QDir::Dirs | QDir::NoDot);
                     model->setNameFilters(QStringList{ "*.my" });
                     model->setNameFilterDisables(false);
                    
                     ui->listView->setViewMode(QListView::IconMode);
                     ui->listView->setModel(model);
                     ui->listView->setRootIndex(_model->index(QDir::currentPath()));
                    
                      connect(ui->horizontalSliderIconSize, &QSlider::valueChanged, [=](int value)
                      {
                        ui->listView->setIconSize(QSize(value, value));
                      });
                      ui->horizontalSliderIconSize->setValue(128);
                    

                    One more observation, there is difference depends on what you return from data() function.
                    If I return QIcon instead of QPixmap so code in data looks like:

                    .
                    .
                    if (load)
                    {
                      QtConcurrent::run(const_cast<MyModel*>(this), &MyModel::loadThumbnail, info.filePath(), info.lastModified());
                      return QColor(Qt::gray);
                    }
                    else
                    {
                      return QIcon(pixmap);
                    }
                    

                    the problem is not so visible, but when you slide horizontalSliderIconSize a little bit you can see that decoration of elements changed

                    1 Reply Last reply
                    0
                    • C chabo

                      I would like to display meta-data and thumbnail stored in file(own format). Because loading and parsing these data can take some time I would like to do it on separate thread. I implemented it using QtConcurrent::run and emitting dataChanged signal.

                      Code:

                      class MyModel : public QFileSystemModel
                      {
                        Q_OBJECT
                       
                      public:
                        MyModel(QObject *parent = nullptr);
                        virtual ~MyModel();
                       
                        QVariant data(const QModelIndex& index, int role) const override;
                       
                      signals:
                        void thumbnailLoaded(const QString &);
                      
                      protected slots:
                        void updateThumbnail(const QString &);
                      
                      protected:
                        void loadThumbnail(const QString & path, const QDateTime & time);
                       
                      protected:
                        mutable QMutex _mutex;
                       
                        using Thumbnails = QHash<QString, QDateTime>;
                        mutable Thumbnails _thumbnails;
                        mutable QPixmapCache _pixmapCache;
                      };
                       
                       MyModel::MyModel(QObject * parent)
                        : QFileSystemModel(parent)
                      {
                        // FIX
                        connect(this, SIGNAL(thumbnailLoaded(const QString &)), this, SLOT(updateThumbnail(const QString &)), Qt::QueuedConnection);
                      }
                      
                      MyModel::~MyModel()
                      {
                        QMutexLocker lock(&_mutex);
                      
                        _thumbnails.clear();
                      }
                      
                      QVariant MyModel ::data(const QModelIndex& index, int role) const
                      {
                        if (role == Qt::DecorationRole)
                        {
                          QFileInfo info = fileInfo(index);
                       
                          if (info.isFile() && info.suffix() == "my")
                          {
                      /*
                            // Synchronous
                            QPixmap pixmap;
                       
                            auto it = _thumbnails.find(info.filePath());
                            if (it == _thumbnails.end() || it.value() < info.lastModified() || !_pixmapCache.find(info.filePath(), &pixmap))
                            {
                              if (auto screenshot = LoadScreenShotFromFile(info.filePath()))
                              {
                                pixmap.loadFromData(screenshot->GetData(), screenshot->GetSize());
                       
                                _thumbnails[info.filePath()] = info.lastModified();
                                _pixmapCache.insert(info.filePath(), pixmap);
                              }
                            }
                      */
                            // Asynchronous
                            QPixmap pixmap;
                            bool load = false;
                       
                            {
                              QMutexLocker lock(&_mutex);
                       
                              auto it = _thumbnails.find(info.filePath());
                              if (it != _thumbnails.end())
                              {
                                if (it.value().isValid())
                                {
                                  if (it.value() == info.lastModified())
                                  {
                                    load = !_pixmapCache.find(info.filePath(), &pixmap);
                                  }
                                  else
                                  {
                                    load = true;
                                  }
                                }
                              }
                              else
                              {
                                _thumbnails[info.filePath()] = QDateTime();
                       
                                load = true;
                              }
                            }
                           
                            if (load)
                            {
                              QtConcurrent::run(const_cast<MyModel *>(this), &MyModel::loadThumbnail, info.filePath(), info.lastModified());
                            }
                       
                            return pixmap;
                          }
                        }
                       
                        return QFileSystemModel::data(index, role);
                      }
                       
                       
                      void MyModel::loadThumbnail(const QString & path, const QDateTime & time)
                      {
                        QPixmap pixmap;
                        bool loaded = false;
                       
                        if (auto screenshot = LoadScreenShotFromFile(info.filePath()))
                        {
                            loaded = pixmap.loadFromData(screenshot->GetData(), screenshot->GetSize());
                        }
                      
                        bool changed = false;
                        {
                          QMutexLocker lock(&_mutex);
                      
                          auto it = _thumbnails.find(path);
                          if (it != _thumbnails.end() && it.value() <= time)
                          {
                            if (loaded)
                            {
                              *it = time;
                              _pixmapCache.insert(path, pixmap);
                            }
                            else
                            {
                              _thumbnails.erase(it);
                              _pixmapCache.remove(path);
                            }
                      
                            changed = true;
                          }
                        }
                      
                        if (changed)
                        {
                          // FIX
                          emit thumbnailLoaded(path);
                        }
                      }
                      
                      void MyModel::updateThumbnail(const QString & path)
                      {
                        QModelIndex i = index(path);
                        
                        if (i.isValid())
                        {
                          emit dataChanged(i, i, QVector<int>{QFileSystemModel::FileIconRole});
                        }
                      }
                      

                      The problem is that sometime view displaying this model is corrupted, only small icons without file names or only file names are displayed.
                      Code should be thread safe. Signal from the thread pool is run in GUI thread. Only problem I see is const_cast.

                      Am I missing something? Thanks for any help.

                      SOLVED

                      C Offline
                      C Offline
                      chabo
                      wrote on last edited by
                      #10

                      Problem was emitting dataChanged signal from different thread.
                      After using Qt::QueuedConnection everything works as intended.

                      1 Reply Last reply
                      0
                      • SGaistS Offline
                        SGaistS Offline
                        SGaist
                        Lifetime Qt Champion
                        wrote on last edited by
                        #11

                        Sorry, I didn't saw my answer wasn't sent.

                        Indeed, dataChanged will trigger GUI update which is not possible from thread other than the GUI thread.

                        Interested in AI ? www.idiap.ch
                        Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

                        1 Reply Last reply
                        0

                        • Login

                        • Login or register to search.
                        • First post
                          Last post
                        0
                        • Categories
                        • Recent
                        • Tags
                        • Popular
                        • Users
                        • Groups
                        • Search
                        • Get Qt Extensions
                        • Unsolved