how to GUI show data in fraction of seconds for large data sets

[Qt Enthusiast]

I have a graph with 20 millon nodes and I have to show the same as squares in QgraphicsView and QgraphicsScence in fraction on seconds . Could you please help

[JonB]

For that many I would draw them (on background or foreground), not create QGraphicItems for them. You can use clipping to ensure you only bother to draw those actually in view, screen is not going to show 20,000,000 rectangles at once.

[Christian Ehrlicher]

20 million nodes on a 4k display = 5 nodes per pixel...
Your requirement is just stupid.

[J.Hilk]

@Qt-Enthusiast 20 million nodes, for fractions of a second ?

You're going for subconscious programming of some kind ?

If not, do it like everyone else and fake it.

[mpergand]

@J-Hilk said in how to GUI show data in fraction of seconds for large data sets:

20 million nodes, for fractions of a second ?

Quantum computers are coming.
:)

[JonB]

@mpergand Yeah, but not quantum humans who can see 20,000,000 squares at a time on the screen ;)

[JoeCFD]

@Qt-Enthusiast Create A region of interest(ROI) and display the nodes only in this region. And this region can be dynamically moved. That is the usual way to show a large set of data.

[MyNameIsQt]

@Christian-Ehrlicher Is there a problem with my question? link text

[mpergand]

@JonB said in how to GUI show data in fraction of seconds for large data sets:

Yeah, but not quantum humans who can see 20,000,000 squares at a time on the screen ;)

Cybertronic eyes are coming.

[SGaist]

@MyNameIsQt said in how to GUI show data in fraction of seconds for large data sets:

@Christian-Ehrlicher Is there a problem with my question? link text

Please don't highjack other people threads like that. It's a lack of respect for their own issues.

[SGaist]

@Qt-Enthusiast said in how to GUI show data in fraction of seconds for large data sets:

I have a graph with 20 millon nodes and I have to show the same as squares in QgraphicsView and QgraphicsScence in fraction on seconds . Could you please help

Hi,

As my fellow already wrote, trying to render that many point is at best useless.

You should use the same technique as video games do: use level of details. Group your points together when showing the full extend of the dataset. Then when people start to zoom in, show them a subset of the points for that region, again keep them grouped together in a meaningful way until they are at a sufficient level of details to show the actual points.

[mpergand]

Google Maps could be a source a inspiration.

[Qt Enthusiast]

@mpergand Do we have algorithms for Google maps and Incremental rendering . Any example will help

[jeremy_k]

The 40000 chips demo might help.

https://doc.qt.io/qt-6/qtwidgets-graphicsview-chip-example.html

Chip::paint() is where the level of detail is handled.
https://code.qt.io/cgit/qt/qtbase.git/tree/examples/widgets/graphicsview/chip/chip.cpp?h=6.6#n33

[Qt Enthusiast]

@jeremy_k

Can we discuss PyQt over here , I have made a prototype

[Qt Enthusiast]

@Qt-Enthusiast said in how to GUI show data in fraction of seconds for large data sets:

made

import sys
from PyQt5.QtWidgets import QGraphicsView, QGraphicsScene, QGraphicsRectItem, QApplication, QMainWindow, QVBoxLayout, QWidget, QPushButton
from PyQt5.QtCore import Qt, QTimer
from PyQt5.QtGui import QPainter

Tile class representing a tile in the scene

class Tile(QGraphicsRectItem):
def init(self, x, y, size):
super(Tile, self).init(x, y, size, size)
self.setBrush(Qt.NoBrush) # Transparent fill for demonstration
self.macros = []

def add_macro(self, macro):
    self.macros.append(macro)
    self.addItem(macro)

Base class for both macro types

class BaseMacroNode(QGraphicsRectItem):
def init(self, name, x, y, size):
super(BaseMacroNode, self).init(x, y, size, size)
self.name = name

AdaptiveMacroNode class representing a macro with an adaptive level of detail

class AdaptiveMacroNode(BaseMacroNode):
def init(self, name, x, y, size, lod):
super(AdaptiveMacroNode, self).init(name, x, y, size)
self.lod = lod # Initial level of detail

def update_lod(self, lod):
    self.lod = lod
    self.update()  # Trigger a repaint when LOD changes

def paint(self, painter, option, widget=None):
    # Custom paint method based on LOD
    painter.setBrush(Qt.blue if self.lod > 0.5 else Qt.red)
    painter.drawRect(self.rect())
    painter.drawText(self.rect(), Qt.AlignCenter, self.name)

AsynchronousMacroNode class representing a macro with asynchronous loading

class AsynchronousMacroNode(BaseMacroNode):
def init(self, name, x, y, size):
super(AsynchronousMacroNode, self).init(name, x, y, size)
# Simulate asynchronous loading with a timer
QTimer.singleShot(1000, self.on_loaded)

def on_loaded(self):
    # Set color after loading
    self.setBrush(Qt.green)

ChipDesignScene class with tile-based rendering and dynamic addition of macros

class ChipDesignScene(QGraphicsScene):
def init(self):
super(ChipDesignScene, self).init()
self.tile_size = 10000
self.tiles = {}

def add_tile(self, x, y):
    tile = Tile(x, y, self.tile_size)
    self.addItem(tile)
    self.tiles[(x, y)] = tile

def remove_tile(self, x, y):
    if (x, y) in self.tiles:
        tile = self.tiles[(x, y)]
        self.removeItem(tile)
        del self.tiles[(x, y)]

def add_macro_to_tile(self, x, y, macro):
    tile = self.tiles.get((x, y))
    if tile:
        tile.add_macro(macro)

def update_tiles(self, lod):
    # For simplicity, consider a fixed-size scene
    scene_width = 1000
    scene_height = 1000

    # Calculate visible tiles based on LOD
    visible_tiles = [
        (x, y)
        for x in range(0, scene_width, self.tile_size)
        for y in range(0, scene_height, self.tile_size)
    ]

    # Add new tiles and macros
    for tile_pos in visible_tiles:
        if tile_pos not in self.tiles:
            self.add_tile(*tile_pos)

            # Add AdaptiveMacroNodes to the newly visible tile during zoom-out
            for i in range(3):
                adaptive_macro = AdaptiveMacroNode(f"AdaptiveMacro{i}", tile_pos[0] + i * 50, tile_pos[1] + i * 50, 30, 0.8)
                self.add_macro_to_tile(*tile_pos, adaptive_macro)

            # Add AsynchronousMacroNodes to the newly visible tile during zoom-out
            asynchronous_macro = AsynchronousMacroNode("AsynchronousMacro", tile_pos[0] + 150, tile_pos[1] + 150, 30)
            self.add_macro_to_tile(*tile_pos, asynchronous_macro)

    # Remove tiles that are no longer visible
    for tile_pos in list(self.tiles.keys()):
        if tile_pos not in visible_tiles:
            self.remove_tile(*tile_pos)

ChipDesignView class with zoom features

class ChipDesignView(QGraphicsView):
def init(self, scene):
super(ChipDesignView, self).init(scene)
self.setRenderHint(QPainter.Antialiasing, True)
self.setSceneRect(0, 0, 1000, 1000)
self.setRenderHint(QPainter.Antialiasing)

def zoom_in(self):
    zoom_factor = 1.2
    self.scale(zoom_factor, zoom_factor)
    self.update_zoom()

def zoom_out(self):
    zoom_factor = 1 / 1.2
    self.scale(zoom_factor, zoom_factor)
    self.update_zoom()

def update_zoom(self):
    zoom_factor = self.transform().m11()
    lod_value = zoom_factor
    self.scene().update_lod(lod_value)
    self.scene().update_tiles(lod=lod_value)

MainWindow class with zoom buttons

class MainWindow(QMainWindow):
def init(self):
super(MainWindow, self).init()

    self.scene = ChipDesignScene()
    self.view = ChipDesignView(self.scene)

    self.zoom_in_button = QPushButton("Zoom In")
    self.zoom_in_button.clicked.connect(self.view.zoom_in)

    self.zoom_out_button = QPushButton("Zoom Out")
    self.zoom_out_button.clicked.connect(self.view.zoom_out)

    layout = QVBoxLayout()
    layout.addWidget(self.view)
    layout.addWidget(self.zoom_in_button)
    layout.addWidget(self.zoom_out_button)

    widget = QWidget()
    widget.setLayout(layout)
    self.setCentralWidget(widget)

Run the application

def main():
app = QApplication(sys.argv)
window = MainWindow()

# Add an initial AdaptiveMacroNode and AsynchronousMacroNode
initial_adaptive_macro = AdaptiveMacroNode("InitialAdaptiveMacro", 300, 300, 30, 1.0)
initial_asynchronous_macro = AsynchronousMacroNode("InitialAsynchronousMacro", 450, 450, 30)
window.scene.add_macro_to_tile(300, 300, initial_adaptive_macro)
window.scene.add_macro_to_tile(450, 450, initial_asynchronous_macro)

window.setGeometry(100, 100, 800, 600)
window.show()
sys.exit(app.exec_())

if name == "main":
main()

[Qt Enthusiast]

@Qt-Enthusiast

Can we review the code and let me know if it solves the purpose

[JonB]

@Qt-Enthusiast
This is a large listing of code. Is there a question?

[Qt Enthusiast]

i have made a prototype code for how to make GUI to show large data sets in fraction of second . if someone can review and suggest improvements. Can some one review the code if the changes helps in performance improvement

[jsulm]

@Qt-Enthusiast You should at least format your code properly...