Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. Model with large data from file
QtWS25 Last Chance

Model with large data from file

Scheduled Pinned Locked Moved Unsolved General and Desktop
17 Posts 3 Posters 5.3k Views
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • D Offline
    D Offline
    dridk2
    wrote on 28 Jan 2016, 13:42 last edited by dridk2
    #1

    Hi,

    I wonder how I can manage a QAbstractListModel which get data from a huge file ( more than 3 Gb) .
    Obviously I will not load all data into memory..
    But do I have to read directly from the file using seek() in the QAbstractListModel::data() ? I don't think so..
    Is there a solution to load some data into cache, and reload when it is necessary ? How ? Could you get me any solution ?

    1 Reply Last reply
    0
    • M Offline
      M Offline
      mrjj
      Lifetime Qt Champion
      wrote on 28 Jan 2016, 13:46 last edited by mrjj
      #2

      Hi
      What format is that huge file?
      You are free to do what ever u like in data().

      update:
      there is
      http://doc.qt.io/qt-5/qcache.html#details

      Update 2:
      Maybe
      http://doc.qt.io/qt-5/qfiledevice.html#map
      could be used. to only map a segment of data.
      but really depends on format.

      1 Reply Last reply
      0
      • D Offline
        D Offline
        dridk2
        wrote on 28 Jan 2016, 13:47 last edited by
        #3

        It's a tabular data, contaning genom data !

        1 Reply Last reply
        0
        • D Offline
          D Offline
          dridk
          wrote on 28 Jan 2016, 13:51 last edited by
          #4

          Damn.. I have 2 Qt account !
          So, yes I will look on qCache! It looks greate!

          Nothing in Biology Makes Sense Except in the Light of Evolution

          M 1 Reply Last reply 28 Jan 2016, 13:53
          0
          • D dridk
            28 Jan 2016, 13:51

            Damn.. I have 2 Qt account !
            So, yes I will look on qCache! It looks greate!

            M Offline
            M Offline
            mrjj
            Lifetime Qt Champion
            wrote on 28 Jan 2016, 13:53 last edited by
            #5

            @dridk2 said:

            tabular data

            Ok, so is each row is same size or will it be very different ?

            D 1 Reply Last reply 28 Jan 2016, 13:55
            0
            • M mrjj
              28 Jan 2016, 13:53

              @dridk2 said:

              tabular data

              Ok, so is each row is same size or will it be very different ?

              D Offline
              D Offline
              dridk
              wrote on 28 Jan 2016, 13:55 last edited by
              #6

              @mrjj Same size !
              Something like :

              chromosom start end value
              1 435345 234234234 A
              1 4234234 23423424 C
              1 13232 5534534 T

              Nothing in Biology Makes Sense Except in the Light of Evolution

              M 1 Reply Last reply 28 Jan 2016, 13:57
              0
              • D Offline
                D Offline
                dridk
                wrote on 28 Jan 2016, 13:56 last edited by dridk
                #7

                So I would like to do something like :

                QCache<int, MyItem > datas; 
                
                model::data(QModelIndex index) 
                   {
                   return datas[index.row()]
                   }
                

                Nothing in Biology Makes Sense Except in the Light of Evolution

                1 Reply Last reply
                0
                • D dridk
                  28 Jan 2016, 13:55

                  @mrjj Same size !
                  Something like :

                  chromosom start end value
                  1 435345 234234234 A
                  1 4234234 23423424 C
                  1 13232 5534534 T

                  M Offline
                  M Offline
                  mrjj
                  Lifetime Qt Champion
                  wrote on 28 Jan 2016, 13:57 last edited by
                  #8

                  @dridk
                  Ok so binary or text format ?
                  3 GB of such lines. wow :)
                  must be many millions.

                  D 1 Reply Last reply 28 Jan 2016, 13:58
                  0
                  • M mrjj
                    28 Jan 2016, 13:57

                    @dridk
                    Ok so binary or text format ?
                    3 GB of such lines. wow :)
                    must be many millions.

                    D Offline
                    D Offline
                    dridk
                    wrote on 28 Jan 2016, 13:58 last edited by dridk
                    #9

                    @mrjj Yes text format ! It's your DNA ! 3 billons of letter A,C,G,T !

                    I m starting a new application : a genom browser ! Alternative of this web application : UCSC

                    Nothing in Biology Makes Sense Except in the Light of Evolution

                    1 Reply Last reply
                    0
                    • M Offline
                      M Offline
                      mrjj
                      Lifetime Qt Champion
                      wrote on 28 Jan 2016, 14:02 last edited by
                      #10

                      Im not sure cache will make you happy.
                      But yes something like that.

                      But I worry about filling the cache.
                      as file seek into such file is expensive too.

                      How big section of the data do u need to have loaded at any given time?

                      D 1 Reply Last reply 28 Jan 2016, 14:04
                      0
                      • M mrjj
                        28 Jan 2016, 14:02

                        Im not sure cache will make you happy.
                        But yes something like that.

                        But I worry about filling the cache.
                        as file seek into such file is expensive too.

                        How big section of the data do u need to have loaded at any given time?

                        D Offline
                        D Offline
                        dridk
                        wrote on 28 Jan 2016, 14:04 last edited by
                        #11

                        @mrjj This the main difficulty ! The view is a painted widget ! Where you can zoom in/out, drag left and right !
                        Imagine a very large horizontal scrollable plot ! This is it ! So, that's mean, I have to fit the cache with the current view zoom level ! ... Looks difficults!

                        Nothing in Biology Makes Sense Except in the Light of Evolution

                        M 1 Reply Last reply 28 Jan 2016, 14:10
                        0
                        • D dridk
                          28 Jan 2016, 14:04

                          @mrjj This the main difficulty ! The view is a painted widget ! Where you can zoom in/out, drag left and right !
                          Imagine a very large horizontal scrollable plot ! This is it ! So, that's mean, I have to fit the cache with the current view zoom level ! ... Looks difficults!

                          M Offline
                          M Offline
                          mrjj
                          Lifetime Qt Champion
                          wrote on 28 Jan 2016, 14:10 last edited by
                          #12

                          @dridk

                          Yes it sounds somewhat/very complex.
                          Normally plot widgets die around some millions plots.
                          Each "DNA" is that just a dot or does it have lines etc?

                          Is this file u fool around with , available from
                          https://genome.ucsc.edu

                          1 Reply Last reply
                          0
                          • D Offline
                            D Offline
                            dridk
                            wrote on 28 Jan 2016, 14:34 last edited by
                            #13

                            Briefly, the DNA is a string of 3 billions of A,C,G,T .
                            They are many extra data containing feature of dna. For exemple :
                            between positionA and positionB, they are a gene. Those features are saved into different file format , and always have this fields avaible : posA, posB, value1, value2...

                            So, I want to make a genome browser, which show "track" aligned with DNA. It can have many track. For exemple, this picture is a genome browser showing :

                            https://www.youtube.com/watch?v=HnVrjj0RmOg

                            And this screenshot which show different tracks :
                            http://www.cureffi.org/wp-content/uploads/2013/03/ucsc-genome-browser-mapability-gc-percent-highlighted-1024x440.png

                            All data are avaible from UCSC table browser :
                            https://genome-euro.ucsc.edu/cgi-bin/hgTables?hgsid=212129308_3BqNFprOMlddL1eZgG2seuarNOqp

                            Nothing in Biology Makes Sense Except in the Light of Evolution

                            M 1 Reply Last reply 28 Jan 2016, 14:41
                            0
                            • D dridk
                              28 Jan 2016, 14:34

                              Briefly, the DNA is a string of 3 billions of A,C,G,T .
                              They are many extra data containing feature of dna. For exemple :
                              between positionA and positionB, they are a gene. Those features are saved into different file format , and always have this fields avaible : posA, posB, value1, value2...

                              So, I want to make a genome browser, which show "track" aligned with DNA. It can have many track. For exemple, this picture is a genome browser showing :

                              https://www.youtube.com/watch?v=HnVrjj0RmOg

                              And this screenshot which show different tracks :
                              http://www.cureffi.org/wp-content/uploads/2013/03/ucsc-genome-browser-mapability-gc-percent-highlighted-1024x440.png

                              All data are avaible from UCSC table browser :
                              https://genome-euro.ucsc.edu/cgi-bin/hgTables?hgsid=212129308_3BqNFprOMlddL1eZgG2seuarNOqp

                              M Offline
                              M Offline
                              mrjj
                              Lifetime Qt Champion
                              wrote on 28 Jan 2016, 14:41 last edited by
                              #14

                              @dridk
                              Ok. thank you for the info.
                              I see what you try to make. Interesting.

                              Since you are viewing a segment at a time, maybe a memory mapped file
                              approach is suitable?

                              is the platform windows or linux?

                              D 1 Reply Last reply 28 Jan 2016, 14:43
                              0
                              • M mrjj
                                28 Jan 2016, 14:41

                                @dridk
                                Ok. thank you for the info.
                                I see what you try to make. Interesting.

                                Since you are viewing a segment at a time, maybe a memory mapped file
                                approach is suitable?

                                is the platform windows or linux?

                                D Offline
                                D Offline
                                dridk
                                wrote on 28 Jan 2016, 14:43 last edited by
                                #15

                                @mrjj Should be avaible everywhere. Why this question ?

                                Nothing in Biology Makes Sense Except in the Light of Evolution

                                D M 2 Replies Last reply 28 Jan 2016, 14:45
                                0
                                • D dridk
                                  28 Jan 2016, 14:43

                                  @mrjj Should be avaible everywhere. Why this question ?

                                  D Offline
                                  D Offline
                                  dridk
                                  wrote on 28 Jan 2016, 14:45 last edited by
                                  #16

                                  @dridk I see one segment at a time.. But I want to have the feeling that you can move smoothy without any load

                                  Nothing in Biology Makes Sense Except in the Light of Evolution

                                  1 Reply Last reply
                                  0
                                  • D dridk
                                    28 Jan 2016, 14:43

                                    @mrjj Should be avaible everywhere. Why this question ?

                                    M Offline
                                    M Offline
                                    mrjj
                                    Lifetime Qt Champion
                                    wrote on 28 Jan 2016, 14:45 last edited by
                                    #17

                                    @dridk
                                    well both platforms supports mem files.
                                    I have not tried Qt version so was just to know.
                                    I have used it on windows with native API to fool around in 23 GB image.

                                    But it seems Qt support this crossplatform.

                                    1 Reply Last reply
                                    0

                                    10/17

                                    28 Jan 2016, 14:02

                                    • Login

                                    • Login or register to search.
                                    10 out of 17
                                    • First post
                                      10/17
                                      Last post
                                    0
                                    • Categories
                                    • Recent
                                    • Tags
                                    • Popular
                                    • Users
                                    • Groups
                                    • Search
                                    • Get Qt Extensions
                                    • Unsolved