Qt Forum

    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Unsolved

    Update: Forum Guidelines & Code of Conduct

    Unsolved Model with large data from file

    General and Desktop
    3
    17
    4308
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      dridk2 last edited by dridk2

      Hi,

      I wonder how I can manage a QAbstractListModel which get data from a huge file ( more than 3 Gb) .
      Obviously I will not load all data into memory..
      But do I have to read directly from the file using seek() in the QAbstractListModel::data() ? I don't think so..
      Is there a solution to load some data into cache, and reload when it is necessary ? How ? Could you get me any solution ?

      1 Reply Last reply Reply Quote 0
      • mrjj
        mrjj Lifetime Qt Champion last edited by mrjj

        Hi
        What format is that huge file?
        You are free to do what ever u like in data().

        update:
        there is
        http://doc.qt.io/qt-5/qcache.html#details

        Update 2:
        Maybe
        http://doc.qt.io/qt-5/qfiledevice.html#map
        could be used. to only map a segment of data.
        but really depends on format.

        1 Reply Last reply Reply Quote 0
        • D
          dridk2 last edited by

          It's a tabular data, contaning genom data !

          1 Reply Last reply Reply Quote 0
          • D
            dridk last edited by

            Damn.. I have 2 Qt account !
            So, yes I will look on qCache! It looks greate!

            Nothing in Biology Makes Sense Except in the Light of Evolution

            mrjj 1 Reply Last reply Reply Quote 0
            • mrjj
              mrjj Lifetime Qt Champion @dridk last edited by

              @dridk2 said:

              tabular data

              Ok, so is each row is same size or will it be very different ?

              D 1 Reply Last reply Reply Quote 0
              • D
                dridk @mrjj last edited by

                @mrjj Same size !
                Something like :

                chromosom start end value
                1 435345 234234234 A
                1 4234234 23423424 C
                1 13232 5534534 T

                Nothing in Biology Makes Sense Except in the Light of Evolution

                mrjj 1 Reply Last reply Reply Quote 0
                • D
                  dridk last edited by dridk

                  So I would like to do something like :

                  QCache<int, MyItem > datas; 
                  
                  model::data(QModelIndex index) 
                     {
                     return datas[index.row()]
                     }
                  

                  Nothing in Biology Makes Sense Except in the Light of Evolution

                  1 Reply Last reply Reply Quote 0
                  • mrjj
                    mrjj Lifetime Qt Champion @dridk last edited by

                    @dridk
                    Ok so binary or text format ?
                    3 GB of such lines. wow :)
                    must be many millions.

                    D 1 Reply Last reply Reply Quote 0
                    • D
                      dridk @mrjj last edited by dridk

                      @mrjj Yes text format ! It's your DNA ! 3 billons of letter A,C,G,T !

                      I m starting a new application : a genom browser ! Alternative of this web application : UCSC

                      Nothing in Biology Makes Sense Except in the Light of Evolution

                      1 Reply Last reply Reply Quote 0
                      • mrjj
                        mrjj Lifetime Qt Champion last edited by

                        Im not sure cache will make you happy.
                        But yes something like that.

                        But I worry about filling the cache.
                        as file seek into such file is expensive too.

                        How big section of the data do u need to have loaded at any given time?

                        D 1 Reply Last reply Reply Quote 0
                        • D
                          dridk @mrjj last edited by

                          @mrjj This the main difficulty ! The view is a painted widget ! Where you can zoom in/out, drag left and right !
                          Imagine a very large horizontal scrollable plot ! This is it ! So, that's mean, I have to fit the cache with the current view zoom level ! ... Looks difficults!

                          Nothing in Biology Makes Sense Except in the Light of Evolution

                          mrjj 1 Reply Last reply Reply Quote 0
                          • mrjj
                            mrjj Lifetime Qt Champion @dridk last edited by

                            @dridk

                            Yes it sounds somewhat/very complex.
                            Normally plot widgets die around some millions plots.
                            Each "DNA" is that just a dot or does it have lines etc?

                            Is this file u fool around with , available from
                            https://genome.ucsc.edu

                            1 Reply Last reply Reply Quote 0
                            • D
                              dridk last edited by

                              Briefly, the DNA is a string of 3 billions of A,C,G,T .
                              They are many extra data containing feature of dna. For exemple :
                              between positionA and positionB, they are a gene. Those features are saved into different file format , and always have this fields avaible : posA, posB, value1, value2...

                              So, I want to make a genome browser, which show "track" aligned with DNA. It can have many track. For exemple, this picture is a genome browser showing :

                              https://www.youtube.com/watch?v=HnVrjj0RmOg

                              And this screenshot which show different tracks :
                              http://www.cureffi.org/wp-content/uploads/2013/03/ucsc-genome-browser-mapability-gc-percent-highlighted-1024x440.png

                              All data are avaible from UCSC table browser :
                              https://genome-euro.ucsc.edu/cgi-bin/hgTables?hgsid=212129308_3BqNFprOMlddL1eZgG2seuarNOqp

                              Nothing in Biology Makes Sense Except in the Light of Evolution

                              mrjj 1 Reply Last reply Reply Quote 0
                              • mrjj
                                mrjj Lifetime Qt Champion @dridk last edited by

                                @dridk
                                Ok. thank you for the info.
                                I see what you try to make. Interesting.

                                Since you are viewing a segment at a time, maybe a memory mapped file
                                approach is suitable?

                                is the platform windows or linux?

                                D 1 Reply Last reply Reply Quote 0
                                • D
                                  dridk @mrjj last edited by

                                  @mrjj Should be avaible everywhere. Why this question ?

                                  Nothing in Biology Makes Sense Except in the Light of Evolution

                                  D mrjj 2 Replies Last reply Reply Quote 0
                                  • D
                                    dridk @dridk last edited by

                                    @dridk I see one segment at a time.. But I want to have the feeling that you can move smoothy without any load

                                    Nothing in Biology Makes Sense Except in the Light of Evolution

                                    1 Reply Last reply Reply Quote 0
                                    • mrjj
                                      mrjj Lifetime Qt Champion @dridk last edited by

                                      @dridk
                                      well both platforms supports mem files.
                                      I have not tried Qt version so was just to know.
                                      I have used it on windows with native API to fool around in 23 GB image.

                                      But it seems Qt support this crossplatform.

                                      1 Reply Last reply Reply Quote 0
                                      • First post
                                        Last post