Model with large data from file

dridk2

Hi,

I wonder how I can manage a QAbstractListModel which get data from a huge file ( more than 3 Gb) .
Obviously I will not load all data into memory..
But do I have to read directly from the file using seek() in the QAbstractListModel::data() ? I don't think so..
Is there a solution to load some data into cache, and reload when it is necessary ? How ? Could you get me any solution ?

mrjj

Hi
What format is that huge file?
You are free to do what ever u like in data().

update:
there is
http://doc.qt.io/qt-5/qcache.html#details

Update 2:
Maybe
http://doc.qt.io/qt-5/qfiledevice.html#map
could be used. to only map a segment of data.
but really depends on format.

dridk2

It's a tabular data, contaning genom data !

dridk

Damn.. I have 2 Qt account !
So, yes I will look on qCache! It looks greate!

mrjj

@dridk2 said:

tabular data

Ok, so is each row is same size or will it be very different ?

dridk

@mrjj Same size !
Something like :

chromosom start end value
1 435345 234234234 A
1 4234234 23423424 C
1 13232 5534534 T

dridk

So I would like to do something like :

QCache<int, MyItem > datas; 

model::data(QModelIndex index) 
   {
   return datas[index.row()]
   }

mrjj

@dridk
Ok so binary or text format ?
3 GB of such lines. wow :)
must be many millions.

dridk

@mrjj Yes text format ! It's your DNA ! 3 billons of letter A,C,G,T !

I m starting a new application : a genom browser ! Alternative of this web application : UCSC

mrjj

Im not sure cache will make you happy.
But yes something like that.

But I worry about filling the cache.
as file seek into such file is expensive too.

How big section of the data do u need to have loaded at any given time?

dridk

@mrjj This the main difficulty ! The view is a painted widget ! Where you can zoom in/out, drag left and right !
Imagine a very large horizontal scrollable plot ! This is it ! So, that's mean, I have to fit the cache with the current view zoom level ! ... Looks difficults!

mrjj

@dridk

Yes it sounds somewhat/very complex.
Normally plot widgets die around some millions plots.
Each "DNA" is that just a dot or does it have lines etc?

Is this file u fool around with , available from
https://genome.ucsc.edu

dridk

Briefly, the DNA is a string of 3 billions of A,C,G,T .
They are many extra data containing feature of dna. For exemple :
between positionA and positionB, they are a gene. Those features are saved into different file format , and always have this fields avaible : posA, posB, value1, value2...

So, I want to make a genome browser, which show "track" aligned with DNA. It can have many track. For exemple, this picture is a genome browser showing :

https://www.youtube.com/watch?v=HnVrjj0RmOg

And this screenshot which show different tracks :
http://www.cureffi.org/wp-content/uploads/2013/03/ucsc-genome-browser-mapability-gc-percent-highlighted-1024x440.png

All data are avaible from UCSC table browser :
https://genome-euro.ucsc.edu/cgi-bin/hgTables?hgsid=212129308_3BqNFprOMlddL1eZgG2seuarNOqp

mrjj

@dridk
Ok. thank you for the info.
I see what you try to make. Interesting.

Since you are viewing a segment at a time, maybe a memory mapped file
approach is suitable?

is the platform windows or linux?

dridk

@mrjj Should be avaible everywhere. Why this question ?

dridk

@dridk I see one segment at a time.. But I want to have the feeling that you can move smoothy without any load

mrjj

@dridk
well both platforms supports mem files.
I have not tried Qt version so was just to know.
I have used it on windows with native API to fool around in 23 GB image.

But it seems Qt support this crossplatform.