how to GUI show data in fraction of seconds for large data sets

SGaist · 11 Dec 2023, 20:45

@Qt-Enthusiast said in how to GUI show data in fraction of seconds for large data sets:

I have a graph with 20 millon nodes and I have to show the same as squares in QgraphicsView and QgraphicsScence in fraction on seconds . Could you please help

Hi,

As my fellow already wrote, trying to render that many point is at best useless.

You should use the same technique as video games do: use level of details. Group your points together when showing the full extend of the dataset. Then when people start to zoom in, show them a subset of the points for that region, again keep them grouped together in a meaningful way until they are at a sufficient level of details to show the actual points.

mpergand · 12 Dec 2023, 00:34

Google Maps could be a source a inspiration.

Qt Enthusiast · wrote on 12 Dec 2023, 00:34

@mpergand Do we have algorithms for Google maps and Incremental rendering . Any example will help

jeremy_k · 12 Dec 2023, 14:01

The 40000 chips demo might help.

https://doc.qt.io/qt-6/qtwidgets-graphicsview-chip-example.html

Chip::paint() is where the level of detail is handled.
https://code.qt.io/cgit/qt/qtbase.git/tree/examples/widgets/graphicsview/chip/chip.cpp?h=6.6#n33

Qt Enthusiast · 12 Dec 2023, 14:05

@jeremy_k

Can we discuss PyQt over here , I have made a prototype

Qt Enthusiast · 12 Dec 2023, 14:06

@Qt-Enthusiast said in how to GUI show data in fraction of seconds for large data sets:

made

import sys
from PyQt5.QtWidgets import QGraphicsView, QGraphicsScene, QGraphicsRectItem, QApplication, QMainWindow, QVBoxLayout, QWidget, QPushButton
from PyQt5.QtCore import Qt, QTimer
from PyQt5.QtGui import QPainter

Tile class representing a tile in the scene

class Tile(QGraphicsRectItem):
def init(self, x, y, size):
super(Tile, self).init(x, y, size, size)
self.setBrush(Qt.NoBrush) # Transparent fill for demonstration
self.macros = []

def add_macro(self, macro):
    self.macros.append(macro)
    self.addItem(macro)

Base class for both macro types

class BaseMacroNode(QGraphicsRectItem):
def init(self, name, x, y, size):
super(BaseMacroNode, self).init(x, y, size, size)
self.name = name

AdaptiveMacroNode class representing a macro with an adaptive level of detail

class AdaptiveMacroNode(BaseMacroNode):
def init(self, name, x, y, size, lod):
super(AdaptiveMacroNode, self).init(name, x, y, size)
self.lod = lod # Initial level of detail

def update_lod(self, lod):
    self.lod = lod
    self.update()  # Trigger a repaint when LOD changes

def paint(self, painter, option, widget=None):
    # Custom paint method based on LOD
    painter.setBrush(Qt.blue if self.lod > 0.5 else Qt.red)
    painter.drawRect(self.rect())
    painter.drawText(self.rect(), Qt.AlignCenter, self.name)

AsynchronousMacroNode class representing a macro with asynchronous loading

class AsynchronousMacroNode(BaseMacroNode):
def init(self, name, x, y, size):
super(AsynchronousMacroNode, self).init(name, x, y, size)
# Simulate asynchronous loading with a timer
QTimer.singleShot(1000, self.on_loaded)

def on_loaded(self):
    # Set color after loading
    self.setBrush(Qt.green)

ChipDesignScene class with tile-based rendering and dynamic addition of macros

class ChipDesignScene(QGraphicsScene):
def init(self):
super(ChipDesignScene, self).init()
self.tile_size = 10000
self.tiles = {}

def add_tile(self, x, y):
    tile = Tile(x, y, self.tile_size)
    self.addItem(tile)
    self.tiles[(x, y)] = tile

def remove_tile(self, x, y):
    if (x, y) in self.tiles:
        tile = self.tiles[(x, y)]
        self.removeItem(tile)
        del self.tiles[(x, y)]

def add_macro_to_tile(self, x, y, macro):
    tile = self.tiles.get((x, y))
    if tile:
        tile.add_macro(macro)

def update_tiles(self, lod):
    # For simplicity, consider a fixed-size scene
    scene_width = 1000
    scene_height = 1000

    # Calculate visible tiles based on LOD
    visible_tiles = [
        (x, y)
        for x in range(0, scene_width, self.tile_size)
        for y in range(0, scene_height, self.tile_size)
    ]

    # Add new tiles and macros
    for tile_pos in visible_tiles:
        if tile_pos not in self.tiles:
            self.add_tile(*tile_pos)

            # Add AdaptiveMacroNodes to the newly visible tile during zoom-out
            for i in range(3):
                adaptive_macro = AdaptiveMacroNode(f"AdaptiveMacro{i}", tile_pos[0] + i * 50, tile_pos[1] + i * 50, 30, 0.8)
                self.add_macro_to_tile(*tile_pos, adaptive_macro)

            # Add AsynchronousMacroNodes to the newly visible tile during zoom-out
            asynchronous_macro = AsynchronousMacroNode("AsynchronousMacro", tile_pos[0] + 150, tile_pos[1] + 150, 30)
            self.add_macro_to_tile(*tile_pos, asynchronous_macro)

    # Remove tiles that are no longer visible
    for tile_pos in list(self.tiles.keys()):
        if tile_pos not in visible_tiles:
            self.remove_tile(*tile_pos)

ChipDesignView class with zoom features

class ChipDesignView(QGraphicsView):
def init(self, scene):
super(ChipDesignView, self).init(scene)
self.setRenderHint(QPainter.Antialiasing, True)
self.setSceneRect(0, 0, 1000, 1000)
self.setRenderHint(QPainter.Antialiasing)

def zoom_in(self):
    zoom_factor = 1.2
    self.scale(zoom_factor, zoom_factor)
    self.update_zoom()

def zoom_out(self):
    zoom_factor = 1 / 1.2
    self.scale(zoom_factor, zoom_factor)
    self.update_zoom()

def update_zoom(self):
    zoom_factor = self.transform().m11()
    lod_value = zoom_factor
    self.scene().update_lod(lod_value)
    self.scene().update_tiles(lod=lod_value)

MainWindow class with zoom buttons

class MainWindow(QMainWindow):
def init(self):
super(MainWindow, self).init()

    self.scene = ChipDesignScene()
    self.view = ChipDesignView(self.scene)

    self.zoom_in_button = QPushButton("Zoom In")
    self.zoom_in_button.clicked.connect(self.view.zoom_in)

    self.zoom_out_button = QPushButton("Zoom Out")
    self.zoom_out_button.clicked.connect(self.view.zoom_out)

    layout = QVBoxLayout()
    layout.addWidget(self.view)
    layout.addWidget(self.zoom_in_button)
    layout.addWidget(self.zoom_out_button)

    widget = QWidget()
    widget.setLayout(layout)
    self.setCentralWidget(widget)

Run the application

def main():
app = QApplication(sys.argv)
window = MainWindow()

# Add an initial AdaptiveMacroNode and AsynchronousMacroNode
initial_adaptive_macro = AdaptiveMacroNode("InitialAdaptiveMacro", 300, 300, 30, 1.0)
initial_asynchronous_macro = AsynchronousMacroNode("InitialAsynchronousMacro", 450, 450, 30)
window.scene.add_macro_to_tile(300, 300, initial_adaptive_macro)
window.scene.add_macro_to_tile(450, 450, initial_asynchronous_macro)

window.setGeometry(100, 100, 800, 600)
window.show()
sys.exit(app.exec_())

if name == "main":
main()

Qt Enthusiast · wrote on 12 Dec 2023, 14:06

@Qt-Enthusiast

Can we review the code and let me know if it solves the purpose

JonB · 12 Dec 2023, 14:10

@Qt-Enthusiast
This is a large listing of code. Is there a question?

Qt Enthusiast · 12 Dec 2023, 14:27

i have made a prototype code for how to make GUI to show large data sets in fraction of second . if someone can review and suggest improvements. Can some one review the code if the changes helps in performance improvement

jsulm · wrote on 12 Dec 2023, 15:26

@Qt-Enthusiast You should at least format your code properly...

Qt Enthusiast · 13 Dec 2023, 08:43

https://codeshare.io/ZJYk4o

I have put the formatted code in the link . PLease review

SimonSchroeder · wrote on 13 Dec 2023, 08:16

@Qt-Enthusiast said in how to GUI show data in fraction of seconds for large data sets:

Can we discuss PyQt over here

If you really want to have good performance for millions of nodes, Python is not the best choice for that. It does help that Qt will handle most of the heavy lifting and is written in C++. You might get lucky with you paint function and it will quickly be optimized by the interpreter, but there is no guarantee. Performance-critical code should not be written in an interpreted language. (That being said: Measure first before optimizing prematurely.)

jeremy_k · wrote on 13 Dec 2023, 08:43

@Qt-Enthusiast said in how to GUI show data in fraction of seconds for large data sets:

https://codeshare.io/ZJYk4o

I have put the formatted code in the link . PLease review

Tip 1: Use the forum code button instead of an external website:

or format manually:
```
code goes here
```

Tip 2: For python3, avoid super(Class, object) when super() will work. Can you spot the error? This is valid code, but probably not what is intended.

class Base:
    def __init__(self, *args, **kwargs):
        print("Base.__init__")

class Mid(Base):
    def __init__(self, *args, **kwargs):
        super(Mid, self).__init__(*args, **kwargs)
        print("Mid.__init__")

class Derived(Mid):
    def __init__(self, *args, **kwargs):
        super(Mid, self).__init__(*args, **kwargs)
        print("Derived.__init__")

obj = Derived()

As for the general strategy, displaying or hiding visible items based on what is visible is the job of the view. Unless the overhead of non-viewable items in the scene is significant, don't remove them from the scene. One of the points of the chips demo is to alter the details of the visible items as appropriate, rather than removing or adding them.