Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. Could Someone Give me Advice on Optimizing QTableView with Large Data Sets?
Forum Updated to NodeBB v4.3 + New Features

Could Someone Give me Advice on Optimizing QTableView with Large Data Sets?

Scheduled Pinned Locked Moved Unsolved General and Desktop
3 Posts 3 Posters 266 Views
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • H Offline
    H Offline
    Hadley
    wrote on last edited by Hadley
    #1

    Hello there,

    I am working on a project where I need to display and interact with a large dataset approximately 1 million rows using QTableView in a desktop application. The dataset is sourced dynamically from a database, and while I have managed to load it into the model, performance issues are becoming a bottleneck.

    Currently; I am using QSqlTableModel as the underlying model for QTableView. While this works for smaller datasets; scrolling; searching; and sorting become sluggish as the dataset grows. I suspect that rendering such a large amount of data at once is not the most efficient approach; but I am unsure about the best way to optimize this.

    Is there a recommended way to implement lazy loading or data fetching on demand for QTableView? I have read about QAbstractItemModel; but I am not sure if that is the best approach here.

    Would implementing a custom model improve performance significantly compared to using QSqlTableModel? If so; could anyone provide a basic example or point me to relevant documentation?

    What are some efficient techniques to handle searching and sorting within a large dataset? Would it be better to offload these tasks to the database instead of doing it within the application?

    Thank you in advance for your help and assistance.

    JonBJ 1 Reply Last reply
    0
    • H Hadley

      Hello there,

      I am working on a project where I need to display and interact with a large dataset approximately 1 million rows using QTableView in a desktop application. The dataset is sourced dynamically from a database, and while I have managed to load it into the model, performance issues are becoming a bottleneck.

      Currently; I am using QSqlTableModel as the underlying model for QTableView. While this works for smaller datasets; scrolling; searching; and sorting become sluggish as the dataset grows. I suspect that rendering such a large amount of data at once is not the most efficient approach; but I am unsure about the best way to optimize this.

      Is there a recommended way to implement lazy loading or data fetching on demand for QTableView? I have read about QAbstractItemModel; but I am not sure if that is the best approach here.

      Would implementing a custom model improve performance significantly compared to using QSqlTableModel? If so; could anyone provide a basic example or point me to relevant documentation?

      What are some efficient techniques to handle searching and sorting within a large dataset? Would it be better to offload these tasks to the database instead of doing it within the application?

      Thank you in advance for your help and assistance.

      JonBJ Offline
      JonBJ Offline
      JonB
      wrote on last edited by
      #2

      @Hadley
      I presume you have already come across fetchMore() which implements "paging" when filling the model from the SQL result set. See when that is called to determine whether QTableView pre-fetches everything from the start or only as needed as you move through the table.

      You might interpose some QAbstractProxyModel between the SQL model and the table view or do your own QAbstractTableModel to implement some kind of paging.

      https://stackoverflow.com/questions/72909009/displaying-large-amount-of-data-in-qtableview
      https://forum.qt.io/topic/116231/load-big-database-into-model-to-be-shown-in-table-view
      You might Google qtableview large model or similar.

      Regardless of the table view it will of course be hugely faster and cost vastly less memory if you do searching/filtering/sorting at the database side instead of the client's in-memory model.

      1 Reply Last reply
      1
      • S Offline
        S Offline
        SamiV123
        wrote on last edited by SamiV123
        #3

        One million rows is nothing, but the trick is of course to have a data model that is well suited for it.

        In my system where I'm throwing hundreds of thousands to millions rows of data at a QTableView I have:

        • Data that has been organized for quick loading ahead of time
        • System that provides quick access to data without creating objects or having allocate stuff with new.

        Basically the data is split into chunks that are stored on disk. Each chunk is then memory mapped and each chunk contains "records". Since the records can have variable size each data chunk has a header and and offset table which stores a starting offset for reach record in the data part of the chunk. (You can think of this as a jump table)

        So when a QTableView needs to display any particular row it all essentially boils down to taking a base pointer to a chunk of data and then reading data at offsets relative to the base pointer.

        The system is extremely fast.

        1 Reply Last reply
        0

        • Login

        • Login or register to search.
        • First post
          Last post
        0
        • Categories
        • Recent
        • Tags
        • Popular
        • Users
        • Groups
        • Search
        • Get Qt Extensions
        • Unsolved