how to GUI show data in fraction of seconds for large data sets
-
I have a graph with 20 millon nodes and I have to show the same as squares in QgraphicsView and QgraphicsScence in fraction on seconds . Could you please help
@Qt-Enthusiast Create A region of interest(ROI) and display the nodes only in this region. And this region can be dynamically moved. That is the usual way to show a large set of data.
-
20 million nodes on a 4k display = 5 nodes per pixel...
Your requirement is just stupid.@Christian-Ehrlicher Is there a problem with my question? link text
-
@mpergand Yeah, but not quantum humans who can see 20,000,000 squares at a time on the screen ;)
-
@Christian-Ehrlicher Is there a problem with my question? link text
@MyNameIsQt said in how to GUI show data in fraction of seconds for large data sets:
@Christian-Ehrlicher Is there a problem with my question? link text
Please don't highjack other people threads like that. It's a lack of respect for their own issues.
-
I have a graph with 20 millon nodes and I have to show the same as squares in QgraphicsView and QgraphicsScence in fraction on seconds . Could you please help
@Qt-Enthusiast said in how to GUI show data in fraction of seconds for large data sets:
I have a graph with 20 millon nodes and I have to show the same as squares in QgraphicsView and QgraphicsScence in fraction on seconds . Could you please help
Hi,
As my fellow already wrote, trying to render that many point is at best useless.
You should use the same technique as video games do: use level of details. Group your points together when showing the full extend of the dataset. Then when people start to zoom in, show them a subset of the points for that region, again keep them grouped together in a meaningful way until they are at a sufficient level of details to show the actual points.
-
@Qt-Enthusiast said in how to GUI show data in fraction of seconds for large data sets:
I have a graph with 20 millon nodes and I have to show the same as squares in QgraphicsView and QgraphicsScence in fraction on seconds . Could you please help
Hi,
As my fellow already wrote, trying to render that many point is at best useless.
You should use the same technique as video games do: use level of details. Group your points together when showing the full extend of the dataset. Then when people start to zoom in, show them a subset of the points for that region, again keep them grouped together in a meaningful way until they are at a sufficient level of details to show the actual points.
-
@mpergand Do we have algorithms for Google maps and Incremental rendering . Any example will help
-
The 40000 chips demo might help.
https://doc.qt.io/qt-6/qtwidgets-graphicsview-chip-example.html
Chip::paint() is where the level of detail is handled.
https://code.qt.io/cgit/qt/qtbase.git/tree/examples/widgets/graphicsview/chip/chip.cpp?h=6.6#n33 -
The 40000 chips demo might help.
https://doc.qt.io/qt-6/qtwidgets-graphicsview-chip-example.html
Chip::paint() is where the level of detail is handled.
https://code.qt.io/cgit/qt/qtbase.git/tree/examples/widgets/graphicsview/chip/chip.cpp?h=6.6#n33Can we discuss PyQt over here , I have made a prototype
-
Can we discuss PyQt over here , I have made a prototype
@Qt-Enthusiast said in how to GUI show data in fraction of seconds for large data sets:
made
import sys
from PyQt5.QtWidgets import QGraphicsView, QGraphicsScene, QGraphicsRectItem, QApplication, QMainWindow, QVBoxLayout, QWidget, QPushButton
from PyQt5.QtCore import Qt, QTimer
from PyQt5.QtGui import QPainterTile class representing a tile in the scene
class Tile(QGraphicsRectItem):
def init(self, x, y, size):
super(Tile, self).init(x, y, size, size)
self.setBrush(Qt.NoBrush) # Transparent fill for demonstration
self.macros = []def add_macro(self, macro): self.macros.append(macro) self.addItem(macro)Base class for both macro types
class BaseMacroNode(QGraphicsRectItem):
def init(self, name, x, y, size):
super(BaseMacroNode, self).init(x, y, size, size)
self.name = nameAdaptiveMacroNode class representing a macro with an adaptive level of detail
class AdaptiveMacroNode(BaseMacroNode):
def init(self, name, x, y, size, lod):
super(AdaptiveMacroNode, self).init(name, x, y, size)
self.lod = lod # Initial level of detaildef update_lod(self, lod): self.lod = lod self.update() # Trigger a repaint when LOD changes def paint(self, painter, option, widget=None): # Custom paint method based on LOD painter.setBrush(Qt.blue if self.lod > 0.5 else Qt.red) painter.drawRect(self.rect()) painter.drawText(self.rect(), Qt.AlignCenter, self.name)AsynchronousMacroNode class representing a macro with asynchronous loading
class AsynchronousMacroNode(BaseMacroNode):
def init(self, name, x, y, size):
super(AsynchronousMacroNode, self).init(name, x, y, size)
# Simulate asynchronous loading with a timer
QTimer.singleShot(1000, self.on_loaded)def on_loaded(self): # Set color after loading self.setBrush(Qt.green)ChipDesignScene class with tile-based rendering and dynamic addition of macros
class ChipDesignScene(QGraphicsScene):
def init(self):
super(ChipDesignScene, self).init()
self.tile_size = 10000
self.tiles = {}def add_tile(self, x, y): tile = Tile(x, y, self.tile_size) self.addItem(tile) self.tiles[(x, y)] = tile def remove_tile(self, x, y): if (x, y) in self.tiles: tile = self.tiles[(x, y)] self.removeItem(tile) del self.tiles[(x, y)] def add_macro_to_tile(self, x, y, macro): tile = self.tiles.get((x, y)) if tile: tile.add_macro(macro) def update_tiles(self, lod): # For simplicity, consider a fixed-size scene scene_width = 1000 scene_height = 1000 # Calculate visible tiles based on LOD visible_tiles = [ (x, y) for x in range(0, scene_width, self.tile_size) for y in range(0, scene_height, self.tile_size) ] # Add new tiles and macros for tile_pos in visible_tiles: if tile_pos not in self.tiles: self.add_tile(*tile_pos) # Add AdaptiveMacroNodes to the newly visible tile during zoom-out for i in range(3): adaptive_macro = AdaptiveMacroNode(f"AdaptiveMacro{i}", tile_pos[0] + i * 50, tile_pos[1] + i * 50, 30, 0.8) self.add_macro_to_tile(*tile_pos, adaptive_macro) # Add AsynchronousMacroNodes to the newly visible tile during zoom-out asynchronous_macro = AsynchronousMacroNode("AsynchronousMacro", tile_pos[0] + 150, tile_pos[1] + 150, 30) self.add_macro_to_tile(*tile_pos, asynchronous_macro) # Remove tiles that are no longer visible for tile_pos in list(self.tiles.keys()): if tile_pos not in visible_tiles: self.remove_tile(*tile_pos)ChipDesignView class with zoom features
class ChipDesignView(QGraphicsView):
def init(self, scene):
super(ChipDesignView, self).init(scene)
self.setRenderHint(QPainter.Antialiasing, True)
self.setSceneRect(0, 0, 1000, 1000)
self.setRenderHint(QPainter.Antialiasing)def zoom_in(self): zoom_factor = 1.2 self.scale(zoom_factor, zoom_factor) self.update_zoom() def zoom_out(self): zoom_factor = 1 / 1.2 self.scale(zoom_factor, zoom_factor) self.update_zoom() def update_zoom(self): zoom_factor = self.transform().m11() lod_value = zoom_factor self.scene().update_lod(lod_value) self.scene().update_tiles(lod=lod_value)MainWindow class with zoom buttons
class MainWindow(QMainWindow):
def init(self):
super(MainWindow, self).init()self.scene = ChipDesignScene() self.view = ChipDesignView(self.scene) self.zoom_in_button = QPushButton("Zoom In") self.zoom_in_button.clicked.connect(self.view.zoom_in) self.zoom_out_button = QPushButton("Zoom Out") self.zoom_out_button.clicked.connect(self.view.zoom_out) layout = QVBoxLayout() layout.addWidget(self.view) layout.addWidget(self.zoom_in_button) layout.addWidget(self.zoom_out_button) widget = QWidget() widget.setLayout(layout) self.setCentralWidget(widget)Run the application
def main():
app = QApplication(sys.argv)
window = MainWindow()# Add an initial AdaptiveMacroNode and AsynchronousMacroNode initial_adaptive_macro = AdaptiveMacroNode("InitialAdaptiveMacro", 300, 300, 30, 1.0) initial_asynchronous_macro = AsynchronousMacroNode("InitialAsynchronousMacro", 450, 450, 30) window.scene.add_macro_to_tile(300, 300, initial_adaptive_macro) window.scene.add_macro_to_tile(450, 450, initial_asynchronous_macro) window.setGeometry(100, 100, 800, 600) window.show() sys.exit(app.exec_())if name == "main":
main() -
@Qt-Enthusiast said in how to GUI show data in fraction of seconds for large data sets:
made
import sys
from PyQt5.QtWidgets import QGraphicsView, QGraphicsScene, QGraphicsRectItem, QApplication, QMainWindow, QVBoxLayout, QWidget, QPushButton
from PyQt5.QtCore import Qt, QTimer
from PyQt5.QtGui import QPainterTile class representing a tile in the scene
class Tile(QGraphicsRectItem):
def init(self, x, y, size):
super(Tile, self).init(x, y, size, size)
self.setBrush(Qt.NoBrush) # Transparent fill for demonstration
self.macros = []def add_macro(self, macro): self.macros.append(macro) self.addItem(macro)Base class for both macro types
class BaseMacroNode(QGraphicsRectItem):
def init(self, name, x, y, size):
super(BaseMacroNode, self).init(x, y, size, size)
self.name = nameAdaptiveMacroNode class representing a macro with an adaptive level of detail
class AdaptiveMacroNode(BaseMacroNode):
def init(self, name, x, y, size, lod):
super(AdaptiveMacroNode, self).init(name, x, y, size)
self.lod = lod # Initial level of detaildef update_lod(self, lod): self.lod = lod self.update() # Trigger a repaint when LOD changes def paint(self, painter, option, widget=None): # Custom paint method based on LOD painter.setBrush(Qt.blue if self.lod > 0.5 else Qt.red) painter.drawRect(self.rect()) painter.drawText(self.rect(), Qt.AlignCenter, self.name)AsynchronousMacroNode class representing a macro with asynchronous loading
class AsynchronousMacroNode(BaseMacroNode):
def init(self, name, x, y, size):
super(AsynchronousMacroNode, self).init(name, x, y, size)
# Simulate asynchronous loading with a timer
QTimer.singleShot(1000, self.on_loaded)def on_loaded(self): # Set color after loading self.setBrush(Qt.green)ChipDesignScene class with tile-based rendering and dynamic addition of macros
class ChipDesignScene(QGraphicsScene):
def init(self):
super(ChipDesignScene, self).init()
self.tile_size = 10000
self.tiles = {}def add_tile(self, x, y): tile = Tile(x, y, self.tile_size) self.addItem(tile) self.tiles[(x, y)] = tile def remove_tile(self, x, y): if (x, y) in self.tiles: tile = self.tiles[(x, y)] self.removeItem(tile) del self.tiles[(x, y)] def add_macro_to_tile(self, x, y, macro): tile = self.tiles.get((x, y)) if tile: tile.add_macro(macro) def update_tiles(self, lod): # For simplicity, consider a fixed-size scene scene_width = 1000 scene_height = 1000 # Calculate visible tiles based on LOD visible_tiles = [ (x, y) for x in range(0, scene_width, self.tile_size) for y in range(0, scene_height, self.tile_size) ] # Add new tiles and macros for tile_pos in visible_tiles: if tile_pos not in self.tiles: self.add_tile(*tile_pos) # Add AdaptiveMacroNodes to the newly visible tile during zoom-out for i in range(3): adaptive_macro = AdaptiveMacroNode(f"AdaptiveMacro{i}", tile_pos[0] + i * 50, tile_pos[1] + i * 50, 30, 0.8) self.add_macro_to_tile(*tile_pos, adaptive_macro) # Add AsynchronousMacroNodes to the newly visible tile during zoom-out asynchronous_macro = AsynchronousMacroNode("AsynchronousMacro", tile_pos[0] + 150, tile_pos[1] + 150, 30) self.add_macro_to_tile(*tile_pos, asynchronous_macro) # Remove tiles that are no longer visible for tile_pos in list(self.tiles.keys()): if tile_pos not in visible_tiles: self.remove_tile(*tile_pos)ChipDesignView class with zoom features
class ChipDesignView(QGraphicsView):
def init(self, scene):
super(ChipDesignView, self).init(scene)
self.setRenderHint(QPainter.Antialiasing, True)
self.setSceneRect(0, 0, 1000, 1000)
self.setRenderHint(QPainter.Antialiasing)def zoom_in(self): zoom_factor = 1.2 self.scale(zoom_factor, zoom_factor) self.update_zoom() def zoom_out(self): zoom_factor = 1 / 1.2 self.scale(zoom_factor, zoom_factor) self.update_zoom() def update_zoom(self): zoom_factor = self.transform().m11() lod_value = zoom_factor self.scene().update_lod(lod_value) self.scene().update_tiles(lod=lod_value)MainWindow class with zoom buttons
class MainWindow(QMainWindow):
def init(self):
super(MainWindow, self).init()self.scene = ChipDesignScene() self.view = ChipDesignView(self.scene) self.zoom_in_button = QPushButton("Zoom In") self.zoom_in_button.clicked.connect(self.view.zoom_in) self.zoom_out_button = QPushButton("Zoom Out") self.zoom_out_button.clicked.connect(self.view.zoom_out) layout = QVBoxLayout() layout.addWidget(self.view) layout.addWidget(self.zoom_in_button) layout.addWidget(self.zoom_out_button) widget = QWidget() widget.setLayout(layout) self.setCentralWidget(widget)Run the application
def main():
app = QApplication(sys.argv)
window = MainWindow()# Add an initial AdaptiveMacroNode and AsynchronousMacroNode initial_adaptive_macro = AdaptiveMacroNode("InitialAdaptiveMacro", 300, 300, 30, 1.0) initial_asynchronous_macro = AsynchronousMacroNode("InitialAsynchronousMacro", 450, 450, 30) window.scene.add_macro_to_tile(300, 300, initial_adaptive_macro) window.scene.add_macro_to_tile(450, 450, initial_asynchronous_macro) window.setGeometry(100, 100, 800, 600) window.show() sys.exit(app.exec_())if name == "main":
main()Can we review the code and let me know if it solves the purpose
-
@Qt-Enthusiast said in how to GUI show data in fraction of seconds for large data sets:
made
import sys
from PyQt5.QtWidgets import QGraphicsView, QGraphicsScene, QGraphicsRectItem, QApplication, QMainWindow, QVBoxLayout, QWidget, QPushButton
from PyQt5.QtCore import Qt, QTimer
from PyQt5.QtGui import QPainterTile class representing a tile in the scene
class Tile(QGraphicsRectItem):
def init(self, x, y, size):
super(Tile, self).init(x, y, size, size)
self.setBrush(Qt.NoBrush) # Transparent fill for demonstration
self.macros = []def add_macro(self, macro): self.macros.append(macro) self.addItem(macro)Base class for both macro types
class BaseMacroNode(QGraphicsRectItem):
def init(self, name, x, y, size):
super(BaseMacroNode, self).init(x, y, size, size)
self.name = nameAdaptiveMacroNode class representing a macro with an adaptive level of detail
class AdaptiveMacroNode(BaseMacroNode):
def init(self, name, x, y, size, lod):
super(AdaptiveMacroNode, self).init(name, x, y, size)
self.lod = lod # Initial level of detaildef update_lod(self, lod): self.lod = lod self.update() # Trigger a repaint when LOD changes def paint(self, painter, option, widget=None): # Custom paint method based on LOD painter.setBrush(Qt.blue if self.lod > 0.5 else Qt.red) painter.drawRect(self.rect()) painter.drawText(self.rect(), Qt.AlignCenter, self.name)AsynchronousMacroNode class representing a macro with asynchronous loading
class AsynchronousMacroNode(BaseMacroNode):
def init(self, name, x, y, size):
super(AsynchronousMacroNode, self).init(name, x, y, size)
# Simulate asynchronous loading with a timer
QTimer.singleShot(1000, self.on_loaded)def on_loaded(self): # Set color after loading self.setBrush(Qt.green)ChipDesignScene class with tile-based rendering and dynamic addition of macros
class ChipDesignScene(QGraphicsScene):
def init(self):
super(ChipDesignScene, self).init()
self.tile_size = 10000
self.tiles = {}def add_tile(self, x, y): tile = Tile(x, y, self.tile_size) self.addItem(tile) self.tiles[(x, y)] = tile def remove_tile(self, x, y): if (x, y) in self.tiles: tile = self.tiles[(x, y)] self.removeItem(tile) del self.tiles[(x, y)] def add_macro_to_tile(self, x, y, macro): tile = self.tiles.get((x, y)) if tile: tile.add_macro(macro) def update_tiles(self, lod): # For simplicity, consider a fixed-size scene scene_width = 1000 scene_height = 1000 # Calculate visible tiles based on LOD visible_tiles = [ (x, y) for x in range(0, scene_width, self.tile_size) for y in range(0, scene_height, self.tile_size) ] # Add new tiles and macros for tile_pos in visible_tiles: if tile_pos not in self.tiles: self.add_tile(*tile_pos) # Add AdaptiveMacroNodes to the newly visible tile during zoom-out for i in range(3): adaptive_macro = AdaptiveMacroNode(f"AdaptiveMacro{i}", tile_pos[0] + i * 50, tile_pos[1] + i * 50, 30, 0.8) self.add_macro_to_tile(*tile_pos, adaptive_macro) # Add AsynchronousMacroNodes to the newly visible tile during zoom-out asynchronous_macro = AsynchronousMacroNode("AsynchronousMacro", tile_pos[0] + 150, tile_pos[1] + 150, 30) self.add_macro_to_tile(*tile_pos, asynchronous_macro) # Remove tiles that are no longer visible for tile_pos in list(self.tiles.keys()): if tile_pos not in visible_tiles: self.remove_tile(*tile_pos)ChipDesignView class with zoom features
class ChipDesignView(QGraphicsView):
def init(self, scene):
super(ChipDesignView, self).init(scene)
self.setRenderHint(QPainter.Antialiasing, True)
self.setSceneRect(0, 0, 1000, 1000)
self.setRenderHint(QPainter.Antialiasing)def zoom_in(self): zoom_factor = 1.2 self.scale(zoom_factor, zoom_factor) self.update_zoom() def zoom_out(self): zoom_factor = 1 / 1.2 self.scale(zoom_factor, zoom_factor) self.update_zoom() def update_zoom(self): zoom_factor = self.transform().m11() lod_value = zoom_factor self.scene().update_lod(lod_value) self.scene().update_tiles(lod=lod_value)MainWindow class with zoom buttons
class MainWindow(QMainWindow):
def init(self):
super(MainWindow, self).init()self.scene = ChipDesignScene() self.view = ChipDesignView(self.scene) self.zoom_in_button = QPushButton("Zoom In") self.zoom_in_button.clicked.connect(self.view.zoom_in) self.zoom_out_button = QPushButton("Zoom Out") self.zoom_out_button.clicked.connect(self.view.zoom_out) layout = QVBoxLayout() layout.addWidget(self.view) layout.addWidget(self.zoom_in_button) layout.addWidget(self.zoom_out_button) widget = QWidget() widget.setLayout(layout) self.setCentralWidget(widget)Run the application
def main():
app = QApplication(sys.argv)
window = MainWindow()# Add an initial AdaptiveMacroNode and AsynchronousMacroNode initial_adaptive_macro = AdaptiveMacroNode("InitialAdaptiveMacro", 300, 300, 30, 1.0) initial_asynchronous_macro = AsynchronousMacroNode("InitialAsynchronousMacro", 450, 450, 30) window.scene.add_macro_to_tile(300, 300, initial_adaptive_macro) window.scene.add_macro_to_tile(450, 450, initial_asynchronous_macro) window.setGeometry(100, 100, 800, 600) window.show() sys.exit(app.exec_())if name == "main":
main()@Qt-Enthusiast
This is a large listing of code. Is there a question? -
@Qt-Enthusiast
This is a large listing of code. Is there a question?i have made a prototype code for how to make GUI to show large data sets in fraction of second . if someone can review and suggest improvements. Can some one review the code if the changes helps in performance improvement
-
i have made a prototype code for how to make GUI to show large data sets in fraction of second . if someone can review and suggest improvements. Can some one review the code if the changes helps in performance improvement
@Qt-Enthusiast You should at least format your code properly...
-
I have put the formatted code in the link . PLease review
-
Can we discuss PyQt over here , I have made a prototype
@Qt-Enthusiast said in how to GUI show data in fraction of seconds for large data sets:
Can we discuss PyQt over here
If you really want to have good performance for millions of nodes, Python is not the best choice for that. It does help that Qt will handle most of the heavy lifting and is written in C++. You might get lucky with you
paintfunction and it will quickly be optimized by the interpreter, but there is no guarantee. Performance-critical code should not be written in an interpreted language. (That being said: Measure first before optimizing prematurely.) -
I have put the formatted code in the link . PLease review
@Qt-Enthusiast said in how to GUI show data in fraction of seconds for large data sets:
I have put the formatted code in the link . PLease review
Tip 1: Use the forum code button instead of an external website:
or format manually:
```
code goes here
```Tip 2: For python3, avoid super(Class, object) when super() will work. Can you spot the error? This is valid code, but probably not what is intended.
class Base: def __init__(self, *args, **kwargs): print("Base.__init__") class Mid(Base): def __init__(self, *args, **kwargs): super(Mid, self).__init__(*args, **kwargs) print("Mid.__init__") class Derived(Mid): def __init__(self, *args, **kwargs): super(Mid, self).__init__(*args, **kwargs) print("Derived.__init__") obj = Derived()As for the general strategy, displaying or hiding visible items based on what is visible is the job of the view. Unless the overhead of non-viewable items in the scene is significant, don't remove them from the scene. One of the points of the chips demo is to alter the details of the visible items as appropriate, rather than removing or adding them.
