Loading tree into QStandardItemModel from QThreadPool

[Ashen]

I am trying to learn how to keep my UI responsive while loading large models. So far I have experimented with staggering the loading with QTimers, and pushing loading to a single separate thread with QThread both of which work fine. However I feel the holy grail for quickly loading a tree into qstandarditemmodel would be to split the loading into multiple threads based on heirarchy.

Below I was attempting to split it up into a thread per level. So the top level would be one thread and then each item's group of immediate children would be a separate thread. However I am obviously doing something wrong as I get random crashes when I run it. I tried to fix the problem by mutex locking the parent item whenever I was trying to add a row to it, but that doesn't seem to help.

I'm rather new to asynchronous operations in general, but have been working through the documentation to try and better understand it. Any advice would be amazing.

class DataFactory(object):
    
    def create(self, row, column, parent):
        return ('Row = {}, Level = {}, Parent = {}'.format(row, column, parent))

def generate_data(instructions, data_factory, parent=None, column=0):
    for row in range(instructions[column]):
        data = data_factory.create(row, column, parent)
        if len(instructions) > (column+1):
            yield [
                data,
                generate_data(
                    instructions,
                    data_factory,
                    row, column+1
                )
            ]
        else:
            yield [data]

class RunnableLoader(QtCore.QRunnable):
    
    def __init__(self, parent, data):
        super(RunnableLoader, self).__init__()
        self.mutex = QtCore.QMutex()
        
        self.mutex.lock()
        if isinstance(parent, QtGui.QStandardItemModel):
            parent = parent.invisibleRootItem()
        self.mutex.unlock()
        self.parent = parent
        
        self.data = data
        
    def run(self):
        for sub_data in data:
            curr_data = sub_data[0]
            new_item = QtGui.QStandardItem(curr_data)
            self.mutex.lock()
            self.parent.appendRow(new_item)
            self.mutex.unlock()
            loader = RunnableLoader(new_item, sub_data[1:])
            QtCore.QThreadPool.globalInstance().start(loader)

w = QtWidgets.QWidget()
l = QtWidgets.QVBoxLayout()
w.setLayout(l)

view = QtWidgets.QTreeView()
model = QtGui.QStandardItemModel()
view.setModel(model)
l.addWidget(view)

w.show()

data = generate_data([200,200,200], DataFactory())
runnable = RunnableLoader(model, data)
QtCore.QThreadPool.globalInstance().start(runnable)

[SGaist]

Hi and welcome to devnet,

First error: you are trying to modify GUI elements from a different thread than the GUI thread. You can't do that.

Then if since you want to handle large amounts of data through threads, you should rather implement your own model so you can really separate data handling and GUI. QFileSystemModel does that for example.

[Ashen]

I see, I guess I never thought of the model as being a GUI element. It would be a bit of a shame to have to construct my own model to do this since QStandardItemModel does most of what I need, perhaps it can't be avoided though. So if I understand, I would probably subclass AbstractItemModel and internally have it thread the operation for loading the data.

I want to be able to update the tree view as it loads though, so one thing that confuses me is how I would emit update signals from the threads to signify once a new piece of data has become available for the view to display, since from your advice it sounds like I am not supposed to use QObjects in a separate thread.

[SGaist]

I just realised that I misread your code, and thought you where using a convenience class like QTreeWidget.

Anyway, that doesn't mean you can't use QStandardItemModel, far from it. However I would rework the code to not manipulate the items like you do but only generate them / update their content once your processing is done.

You can use QObjects in other threads but you have to do that carefully, see the QThread.