Overcoming the memory limitation of QGraphicsView

[j_omega]

I am working on a custom map project. My software currently uses QGraphicsScene to load the map data set into memory and then QGraphicsView to render the data. I have been very happy with this solution, but it has scaling problems. Right now, I am only rendering a small subset of my full data set. The full data set is >1GB and my target platform only has 512MB of RAM. Obviously this won't work due to RAM size. I am trying to figure out the best way to approach this problem.

Is there a way to make QGraphicsView render from a database file instead of loading everything into memory?

I am wondering if I have hit the limits of QGraphicsView and need to abandon it for a custom tiling approach. If so, are there any recommendations for this?

[mrjj]

Hi
The limit is not QGraphicsView in terms of items possible -but the very low
memory you have to work with.
http://doc.qt.io/qt-5/qtwidgets-graphicsview-chip-example.html

What is this data ?
How much memory does each piece of data require ?

[j_omega]

@mrjj I am using data from the TIGER geodatabase. These files are translated from shapefiles into QPolygons which are then loaded into a QGraphicsScene. The size per dataset can be upwards of 100MB

Is there a different approach I can take to limit the memory usage of QGraphicsScene?

[SGaist]

Hi,

One thing that comes to mind is to limit the number of elements you present on screen.

Do you really need to load them all ? Can you display only parts of them and then use some kind of window to always only show a subset of them ?

[Buckwheat]

@SGaist said in Overcoming the memory limitation of QGraphicsView:

ally need to load them all ? Can you display only parts of them and then use some kind of window to always only show a subset of them ?

Hi @j_omega!

I agree with SGaist. I had a similar situation where I was loading navigational map databases. I could only keep a 9x9 (6x6 visible) matrix of images in memory at any given time. This presented a problem because at different ranges, the window size of the data changed.

I ended up solving the problem by making a server style class that would monitor my position and feed me file/rectangle to load areas for each element of the matrix.

[j_omega]

@Buckwheat So maybe I could do something with QGraphicsItemGroup. I can think of my map as a matrix of "tiles". Then I would query my geo-database for items in nearby tiles. Each tile would loaded into its own QGraphicsItemGroup. Then these groups could be easily added or removed to the scenegraph as the map moves.

[Asperamanca]

The question is: Are you ever in a situation where the user wants an overview over the whole data ("zoom out"), then see every detail in a certain spot ("zoom in")? In that case, a combination of tiles, level of detail (LOD) and smart data caching should work.

• For tiling, I would create a custom QGraphicsItem that can draw a part of your scene.
• For level of detail, see the 40000 chips demo.
• Smart caching: Make the single items a lightweight shell that can draw a tile at different level of details, without the need to have all data stored in memory. Add a cache so the currently visible tiles have the data they need for the current level of detail.
• If the user zooms in, the tiles that are no longer visible can empty their caches. The tiles that are now visible in higher detail need to load their data in higher resolution
• If the user zooms out, visible tiles can reduce their memory load trimming data to the amount needed for the current LOD. New tiles become visible, they need to load and cache some data

Don't always "new" and "delete" items. Instead, make a tile displaying no data very lightweight. That way, you can keep your scene structure fully intact, and just load or unload data as the user scroll and zooms.

[Buckwheat]

@j_omega , @Asperamanca gives a good algorithm. QGraphicsItemGroup can be used for "ALL" the tiles you wish to display but I would only do that if it was one piece of multiple display items. In my case it was the background map. There were waypoints and text and range lines as the primary pieces.

You would still need to manage the matrix. I created a class called TileItem that was derived from QSharedData and stored QSharedDataPointer (nice reference counted pointer). When No data could be found for the item I just had an empty QSharedDataPointer.

The reason to use them is to only have one item loaded but to be able to share it as needed without making copies. A lot of the underlying data items in Qt are done this way so it conserves memory. I used my own QGraphicsItem derived class to contain the pointer and it overrode paint and boundingRect. Positions were then set by the map item data.

My LOD maxed out at 360nm radius and started at 1.5nm radius so zooming was never too much of an issue.

[j_omega]

@Asperamanca said in Overcoming the memory limitation of QGraphicsView:

The question is: Are you ever in a situation where the user wants an overview over the whole data ("zoom out"), then see every detail in a certain spot ("zoom in")? In that case, a combination of tiles, level of detail (LOD) and smart data caching should work.

Yes, the map needs to be able to handle a wide range of zoom levels.

• For tiling, I would create a custom QGraphicsItem that can draw a part of your scene.
• For level of detail, see the 40000 chips demo.
• Smart caching: Make the single items a lightweight shell that can draw a tile at different level of details, without the need to have all data stored in memory. Add a cache so the currently visible tiles have the data they need for the current level of detail.
• If the user zooms in, the tiles that are no longer visible can empty their caches. The tiles that are now visible in higher detail need to load their data in higher resolution
• If the user zooms out, visible tiles can reduce their memory load trimming data to the amount needed for the current LOD. New tiles become visible, they need to load and cache some data

Don't always "new" and "delete" items. Instead, make a tile displaying no data very lightweight. That way, you can keep your scene structure fully intact, and just load or unload data as the user scroll and zooms.

This approach seems very good except for one part. I do not know how the graphics item tile would know when it had gone out-of-view.

[j_omega]

@Buckwheat said in Overcoming the memory limitation of QGraphicsView:

@j_omega , @Asperamanca gives a good algorithm. QGraphicsItemGroup can be used for "ALL" the tiles you wish to display but I would only do that if it was one piece of multiple display items. In my case it was the background map. There were waypoints and text and range lines as the primary pieces.

When you talk about "multiple display items", are you referring to different layers/overlays of the map?

You would still need to manage the matrix. I created a class called TileItem that was derived from QSharedData and stored QSharedDataPointer (nice reference counted pointer). When No data could be found for the item I just had an empty QSharedDataPointer.

The reason to use them is to only have one item loaded but to be able to share it as needed without making copies. A lot of the underlying data items in Qt are done this way so it conserves memory. I used my own QGraphicsItem derived class to contain the pointer and it overrode paint and boundingRect. Positions were then set by the map item data.

My LOD maxed out at 360nm radius and started at 1.5nm radius so zooming was never too much of an issue.

Are you saying that each tile shared the same dataset (TileItem) and decided which part of that data to render?

[Buckwheat]

@j_omega said in Overcoming the memory limitation of QGraphicsView:

you saying that each tile shared the same dataset (TileItem) and decided which part of that data to render?

Hi @j_omega

For the QDisplayItemGroup: If I had a complex graphics item. say a tiled-map, that would be in a QGraphicsItemGroup made up of QGraphicsPixmapItems so I could treat it as a single item and show/hide it as required easily as the bottom layer item. Waypoints, waypoint text, radio and airport symbols, battle area symbology, and custom zones would be part of their own QGraphicsItemGroup(s) as well.
For QSharedDataItem: I kept a tile manager around that loaded/unloaded tiles as I moved. The manager stored TileItems. These TIleItems could also be shared. I did not want to copy them so the QSharedDataPointer gave me nice reference counted copies. When an item was finally displayed, because it was shared, the manager could delete the item from its structure and it would not effect the drawing cycle until the render pass and then the tile would be updated in the display. Hence, only one copy of the item (which contains, row, column, pixmap [which could be a subset of a larger pixmap or a complete pixmap based on resolution]). These TileItems could be EMITed as the management thread loaded new areas. Very low overhead for signal/slot handling.