QTableModel memory keep growing
-
I have created a new project with only one interface, which contains a QTableView. Then I inherited from QAbstractTableModel and overwritten the data() function as follows: if (role = Qt::DisplayRole) return QString::number(index.row()). In the mainwindow.cpp, I created a timer that inserts rows when it times out. At this point, I only increase the number of rows and refresh the interface without adding any variables. However, even after running this project, the memory keeps growing. why? I have not add any data into memory. 
-
I have created a new project with only one interface, which contains a QTableView. Then I inherited from QAbstractTableModel and overwritten the data() function as follows: if (role = Qt::DisplayRole) return QString::number(index.row()). In the mainwindow.cpp, I created a timer that inserts rows when it times out. At this point, I only increase the number of rows and refresh the interface without adding any variables. However, even after running this project, the memory keeps growing. why? I have not add any data into memory. 
-
@vinHuang
Not sure what you are asking/expecting. If you keep adding new, even blank, rows, and displaying the table with them, then it will increase how much memory it uses. -
@JonB I hope to scroll and display a lot of rows in the table, but I will get the data from a local file. According to what you said, is it impossible to continuously increase the number of rows in any way, unless rows are dynamically deleted?
@vinHuang
Again, I don't know what you are saying or expecting. I did not say it is impossible to continuously increase the number of rows, you are already doing so. If you delete some rows at the same time as inserting new ones, you may reduce the amount of extra memory required. Though frankly I doubt it, as C++ does not guarantee to re-use existing allocated memory as soon as it is deallocated. Rather the hope is that over time the memory allocator will re-use certain freed areas instead of allocating anew. But when this will happen is not specified.The memory usage of
QTableVieworQAbstractTableModelis what it is, and I suggest you just accept it. (If you are thinking Qt code will unintentionally "leak" during model/view operations I would suggest it will not, it has been well tested and used in many places.) Yes, you would use a lot of space if you keep millions of rows in memory at the same time. If that bothers you, do not do so, e.g. only keep "chunks" or use a database for the backend. -
@vinHuang
Again, I don't know what you are saying or expecting. I did not say it is impossible to continuously increase the number of rows, you are already doing so. If you delete some rows at the same time as inserting new ones, you may reduce the amount of extra memory required. Though frankly I doubt it, as C++ does not guarantee to re-use existing allocated memory as soon as it is deallocated. Rather the hope is that over time the memory allocator will re-use certain freed areas instead of allocating anew. But when this will happen is not specified.The memory usage of
QTableVieworQAbstractTableModelis what it is, and I suggest you just accept it. (If you are thinking Qt code will unintentionally "leak" during model/view operations I would suggest it will not, it has been well tested and used in many places.) Yes, you would use a lot of space if you keep millions of rows in memory at the same time. If that bothers you, do not do so, e.g. only keep "chunks" or use a database for the backend. -
@vinHuang
Usually if you "add rows" that means you now have extra rows in memory, so that is some increase on footprint. When you callQAbstractItemModel::insertRows()I assume or it would not surprise me that these is some overhead for its bookkeeping which might increase memory used. If all you implement is to increaserowCount()and always return fixed, calculated data likereturn QString::number(index.row())then that looks like no extra space in your model and any increased Qt model memory might be small.When you ask
QTableViewto display more rows, or even different rows, again it would not surprise me if there is some memory overhead which this increases.However, I would expect these increases to be "small" if your model is as small as you indicate. I don't know what you are measuring how, but don't get hung up on statistics from small amounts. Try it with decent amounts of real data and see how it goes.
Yes, if you want to keep (a lot of) data out of memory then a backed which stores it on disk, such as a SQL server via
QSql....should reduce the memory footprint, compared against holding all the data in memory. -
@vinHuang
Usually if you "add rows" that means you now have extra rows in memory, so that is some increase on footprint. When you callQAbstractItemModel::insertRows()I assume or it would not surprise me that these is some overhead for its bookkeeping which might increase memory used. If all you implement is to increaserowCount()and always return fixed, calculated data likereturn QString::number(index.row())then that looks like no extra space in your model and any increased Qt model memory might be small.When you ask
QTableViewto display more rows, or even different rows, again it would not surprise me if there is some memory overhead which this increases.However, I would expect these increases to be "small" if your model is as small as you indicate. I don't know what you are measuring how, but don't get hung up on statistics from small amounts. Try it with decent amounts of real data and see how it goes.
Yes, if you want to keep (a lot of) data out of memory then a backed which stores it on disk, such as a SQL server via
QSql....should reduce the memory footprint, compared against holding all the data in memory. -
@JonBThank you for your detailed explanation. It seems that my understanding of QTablemodel is incorrect. I need to make changes to the plan. Thank you again
@vinHuang
Do you wish to explain what your understanding was? All I have said is that new data has to be stored somewhere, which is common sense.QTableModel+QTableViewprobably has some overhead for extra rows, even if they are "blank". But I would not expect it to be "much" from what you describe for your current situation. Measuring memory usage can be tricky if you expect it to be accurate. -
@vinHuang
Do you wish to explain what your understanding was? All I have said is that new data has to be stored somewhere, which is common sense.QTableModel+QTableViewprobably has some overhead for extra rows, even if they are "blank". But I would not expect it to be "much" from what you describe for your current situation. Measuring memory usage can be tricky if you expect it to be accurate.@JonB Sure, let me describe my requirement. For example, I'm creating a UI to display parsed data. I have a receiving thread that continuously receives and parses data, and then saves it to a local file. I want to display the parsed data using QTabelView + QTableModel, and the amount of data may be very large (tens of GB or even hundreds of GB). Previously, I always thought that as long as I put the data in a local file, and read the data from the file in the data() function by overwriting QTabelModel, I could continuously add new rows to display data without storing data in memory. However, from your reply and my verification (even adding blank lines will continue to increase memory usage), this solution is not feasible. It seems that I still need to display the data in segments.
-
@JonB Sure, let me describe my requirement. For example, I'm creating a UI to display parsed data. I have a receiving thread that continuously receives and parses data, and then saves it to a local file. I want to display the parsed data using QTabelView + QTableModel, and the amount of data may be very large (tens of GB or even hundreds of GB). Previously, I always thought that as long as I put the data in a local file, and read the data from the file in the data() function by overwriting QTabelModel, I could continuously add new rows to display data without storing data in memory. However, from your reply and my verification (even adding blank lines will continue to increase memory usage), this solution is not feasible. It seems that I still need to display the data in segments.
@vinHuang said in QTbaleModel memory keep growing:
and read the data from the file in the data() function by overwriting QTabelModel, I could continuously add new rows to display data without storing data in memory.
That would be extraordinarily slow.
QTableViewcan potentially ask for content from the model at any time. It also redraws/refreshes from time to time, so could ask about any data it might display even if that was requested in the past. The model is intended to hold in memory whatever is being worked on or displayed in a view.If you have a lot of data you would typically be expected to access it from a (SQL) database model. Even then it will read "chunks" of data rows into memory. See
canFetchMore()&fetchMore()inQAbstractItemModelandQSqlQueryModel. "Reimplement this if you are populating your model incrementally.". If you don't want to use SQL you could perhaps implement it for your file based data, but it would likely be hard/inefficient.If you take this approach, and only retain "chunks" of data in your memory model data, then views will only be able to display what it currently has. You might introduce a "paging" mechanism to allow the user to move to other chunks of data which you re-read, but you have to write that code.
-
I am not fully convinced that the memory will grow indefinitely. From my understanding, your current model does not access any real data yet, but just returns the row index on the fly. Which is why you expect the model to not use up any memory. Right? Just test this by only creating the model and not attaching it to a table view. If you add rows, does it still increase memory? My expectation is that it doesn't. Otherwise we should start here and you should provide the code of your test model for us to investigate.
Together with the table view, in the beginning adding more items will increase memory usage as it has to create new table items for display. Once the full visible size of the table is used up table items could be reused in an ideal implementation. My understanding is that Qt is actually doing this. Maybe an optimized version has a few extra table items around for performance reasons which are not visible. But, at some point Qt should not allocate memory for the table view anymore.
In that respect you are right that there should come a point when the memory should stop growing while adding more rows to your model and thus to your table view.
