Qtablewidget max data?

Q139

App is x64 and compiled on msvc.
What is maximal ammount of sortable data that Qtablwidget can hold?
Is there any way to make sorting multithreaded?

kshegunov

@Q139

What is maximal ammount of sortable data that Qtablwidget can hold?

The maximum value you can store in an int theoretically (about 2 bilion items for a 32bit integer), but it'll give away way before that.

Is there any way to make sorting multithreaded?

I haven't tried it, but possibly if you provide your own model implementation. However, QTableWidget is not for that case, you should then use QTableView instead.

Kind regards.

Q139

Tested number sorting with float data data of ~20 million rows and 5 columns, it took ~5 sec on 4.3 ghz.
What would be best way to implement fast sorting of very long struct vector of 5 floating numbers?

kshegunov

@Q139 said:

Tested number sorting with float data data of ~20 million rows and 5 columns, it took ~5 sec on 4.3 ghz.

With that widget and the standard model, this is not a number sorting, but a string sorting. The model will sort based on the Qt::DisplayRole data which is interpreted as a string and then sorted accordingly.

What would be best way to implement fast sorting of very long struct vector of 5 floating numbers?

My first instict would be to use a custom model, which treats the data as numbers and doesn't go through the internal QVariants. 20M structs is a relatively small dataset and should be sorted quite fast if you do it internally for the model. Displaying these 20 million rows is another matter altogether.

Chris Kawa

There's no way you'll fit 20M of numbers on a screen in any reasonable way, so there's no point sorting them all.
I'd look into creating a model that fetches subsets of the data (see canFetchMore()) and do only partial sorting of the exposed subset, for example compose std::nth_element and std::partial_sort together to get a sorted sub-range. That should make a big difference to start with. There are also experimental parallel versions of these algorithms. You could look into that for another boost.

kshegunov

@Chris-Kawa
Well, I personally would consider BSP-like (i.e. a k-d tree) organization of the data, so one can easily fetch small subsets of data from the proper place, but as you said there certainly isn't a good way to display that data in a table ... :)

Q139

Thanks for all info will try dif methods of sorting and displaying data.

Chris Kawa

@kshegunov
Might be a good idea, but this route requires special care. OP didn't mention what kind of data we are talking about. It's worth mentioning that BSPs are not well suited for a dynamic dataset, as constant balancing of the structure may proove quite an overhead. For a static set of data the implementation becomes a delicate problem. Any kind of node based approach is out of question with data this size and a naive implementation with a vector is gonna suffer from cache misses a lot. It can get hairy fast.

I'd stick to a simple, non-structured vector and sort incrementally i.e. first X of elements, then next X as the user scrolls through the view and fetchMore() is called. This way that 5s needed to sort all the data are distributed in time and the user won't notice it.

Q139

@Chris-Kawa
Using fetchMore() method does it cause overall max/min values to be fully sorted at their place only after full 20M list is scrolled troug?

Chris Kawa

@Q139 Yes, that's the idea. The model returns a small number (lets say 100) as rowCountand only that number of element need to be at their sorted position (hence std::partial_sort comes to mind). The model also returns true from canFetchMore, which causes the view to call fetchMore() when user scrolls to the bottom. Inside fetch more you sort elements 100-200 and from now on report rowCount as 200 . This goes on until you run out of elements at which point you return false from canFetchMore(). This way fully sorted is only the range the user actually scrolls through.