Qtablewidget max data?
-
What is maximal ammount of sortable data that Qtablwidget can hold?
The maximum value you can store in an
int
theoretically (about 2 bilion items for a 32bit integer), but it'll give away way before that.Is there any way to make sorting multithreaded?
I haven't tried it, but possibly if you provide your own model implementation. However,
QTableWidget
is not for that case, you should then useQTableView
instead.Kind regards.
-
@Q139 said:
Tested number sorting with float data data of ~20 million rows and 5 columns, it took ~5 sec on 4.3 ghz.
With that widget and the standard model, this is not a number sorting, but a string sorting. The model will sort based on the
Qt::DisplayRole
data which is interpreted as a string and then sorted accordingly.What would be best way to implement fast sorting of very long struct vector of 5 floating numbers?
My first instict would be to use a custom model, which treats the data as numbers and doesn't go through the internal
QVariant
s. 20M structs is a relatively small dataset and should be sorted quite fast if you do it internally for the model. Displaying these 20 million rows is another matter altogether. -
There's no way you'll fit 20M of numbers on a screen in any reasonable way, so there's no point sorting them all.
I'd look into creating a model that fetches subsets of the data (see canFetchMore()) and do only partial sorting of the exposed subset, for example compose std::nth_element and std::partial_sort together to get a sorted sub-range. That should make a big difference to start with. There are also experimental parallel versions of these algorithms. You could look into that for another boost. -
@Chris-Kawa
Well, I personally would consider BSP-like (i.e. a k-d tree) organization of the data, so one can easily fetch small subsets of data from the proper place, but as you said there certainly isn't a good way to display that data in a table ... :) -
@kshegunov
Might be a good idea, but this route requires special care. OP didn't mention what kind of data we are talking about. It's worth mentioning that BSPs are not well suited for a dynamic dataset, as constant balancing of the structure may proove quite an overhead. For a static set of data the implementation becomes a delicate problem. Any kind of node based approach is out of question with data this size and a naive implementation with a vector is gonna suffer from cache misses a lot. It can get hairy fast.I'd stick to a simple, non-structured vector and sort incrementally i.e. first X of elements, then next X as the user scrolls through the view and
fetchMore()
is called. This way that 5s needed to sort all the data are distributed in time and the user won't notice it. -
@Chris-Kawa
Using fetchMore() method does it cause overall max/min values to be fully sorted at their place only after full 20M list is scrolled troug? -
@Q139 Yes, that's the idea. The model returns a small number (lets say 100) as
rowCount
and only that number of element need to be at their sorted position (hence std::partial_sort comes to mind). The model also returns true fromcanFetchMore
, which causes the view to callfetchMore()
when user scrolls to the bottom. Inside fetch more you sort elements 100-200 and from now on reportrowCount
as 200 . This goes on until you run out of elements at which point you return false fromcanFetchMore()
. This way fully sorted is only the range the user actually scrolls through.