Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. Model with large data from file
Forum Updated to NodeBB v4.3 + New Features

Model with large data from file

Scheduled Pinned Locked Moved Unsolved General and Desktop
17 Posts 3 Posters 5.5k Views 1 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • D dridk

    Damn.. I have 2 Qt account !
    So, yes I will look on qCache! It looks greate!

    mrjjM Offline
    mrjjM Offline
    mrjj
    Lifetime Qt Champion
    wrote on last edited by
    #5

    @dridk2 said:

    tabular data

    Ok, so is each row is same size or will it be very different ?

    D 1 Reply Last reply
    0
    • mrjjM mrjj

      @dridk2 said:

      tabular data

      Ok, so is each row is same size or will it be very different ?

      D Offline
      D Offline
      dridk
      wrote on last edited by
      #6

      @mrjj Same size !
      Something like :

      chromosom start end value
      1 435345 234234234 A
      1 4234234 23423424 C
      1 13232 5534534 T

      Nothing in Biology Makes Sense Except in the Light of Evolution

      mrjjM 1 Reply Last reply
      0
      • D Offline
        D Offline
        dridk
        wrote on last edited by dridk
        #7

        So I would like to do something like :

        QCache<int, MyItem > datas; 
        
        model::data(QModelIndex index) 
           {
           return datas[index.row()]
           }
        

        Nothing in Biology Makes Sense Except in the Light of Evolution

        1 Reply Last reply
        0
        • D dridk

          @mrjj Same size !
          Something like :

          chromosom start end value
          1 435345 234234234 A
          1 4234234 23423424 C
          1 13232 5534534 T

          mrjjM Offline
          mrjjM Offline
          mrjj
          Lifetime Qt Champion
          wrote on last edited by
          #8

          @dridk
          Ok so binary or text format ?
          3 GB of such lines. wow :)
          must be many millions.

          D 1 Reply Last reply
          0
          • mrjjM mrjj

            @dridk
            Ok so binary or text format ?
            3 GB of such lines. wow :)
            must be many millions.

            D Offline
            D Offline
            dridk
            wrote on last edited by dridk
            #9

            @mrjj Yes text format ! It's your DNA ! 3 billons of letter A,C,G,T !

            I m starting a new application : a genom browser ! Alternative of this web application : UCSC

            Nothing in Biology Makes Sense Except in the Light of Evolution

            1 Reply Last reply
            0
            • mrjjM Offline
              mrjjM Offline
              mrjj
              Lifetime Qt Champion
              wrote on last edited by
              #10

              Im not sure cache will make you happy.
              But yes something like that.

              But I worry about filling the cache.
              as file seek into such file is expensive too.

              How big section of the data do u need to have loaded at any given time?

              D 1 Reply Last reply
              0
              • mrjjM mrjj

                Im not sure cache will make you happy.
                But yes something like that.

                But I worry about filling the cache.
                as file seek into such file is expensive too.

                How big section of the data do u need to have loaded at any given time?

                D Offline
                D Offline
                dridk
                wrote on last edited by
                #11

                @mrjj This the main difficulty ! The view is a painted widget ! Where you can zoom in/out, drag left and right !
                Imagine a very large horizontal scrollable plot ! This is it ! So, that's mean, I have to fit the cache with the current view zoom level ! ... Looks difficults!

                Nothing in Biology Makes Sense Except in the Light of Evolution

                mrjjM 1 Reply Last reply
                0
                • D dridk

                  @mrjj This the main difficulty ! The view is a painted widget ! Where you can zoom in/out, drag left and right !
                  Imagine a very large horizontal scrollable plot ! This is it ! So, that's mean, I have to fit the cache with the current view zoom level ! ... Looks difficults!

                  mrjjM Offline
                  mrjjM Offline
                  mrjj
                  Lifetime Qt Champion
                  wrote on last edited by
                  #12

                  @dridk

                  Yes it sounds somewhat/very complex.
                  Normally plot widgets die around some millions plots.
                  Each "DNA" is that just a dot or does it have lines etc?

                  Is this file u fool around with , available from
                  https://genome.ucsc.edu

                  1 Reply Last reply
                  0
                  • D Offline
                    D Offline
                    dridk
                    wrote on last edited by
                    #13

                    Briefly, the DNA is a string of 3 billions of A,C,G,T .
                    They are many extra data containing feature of dna. For exemple :
                    between positionA and positionB, they are a gene. Those features are saved into different file format , and always have this fields avaible : posA, posB, value1, value2...

                    So, I want to make a genome browser, which show "track" aligned with DNA. It can have many track. For exemple, this picture is a genome browser showing :

                    https://www.youtube.com/watch?v=HnVrjj0RmOg

                    And this screenshot which show different tracks :
                    http://www.cureffi.org/wp-content/uploads/2013/03/ucsc-genome-browser-mapability-gc-percent-highlighted-1024x440.png

                    All data are avaible from UCSC table browser :
                    https://genome-euro.ucsc.edu/cgi-bin/hgTables?hgsid=212129308_3BqNFprOMlddL1eZgG2seuarNOqp

                    Nothing in Biology Makes Sense Except in the Light of Evolution

                    mrjjM 1 Reply Last reply
                    0
                    • D dridk

                      Briefly, the DNA is a string of 3 billions of A,C,G,T .
                      They are many extra data containing feature of dna. For exemple :
                      between positionA and positionB, they are a gene. Those features are saved into different file format , and always have this fields avaible : posA, posB, value1, value2...

                      So, I want to make a genome browser, which show "track" aligned with DNA. It can have many track. For exemple, this picture is a genome browser showing :

                      https://www.youtube.com/watch?v=HnVrjj0RmOg

                      And this screenshot which show different tracks :
                      http://www.cureffi.org/wp-content/uploads/2013/03/ucsc-genome-browser-mapability-gc-percent-highlighted-1024x440.png

                      All data are avaible from UCSC table browser :
                      https://genome-euro.ucsc.edu/cgi-bin/hgTables?hgsid=212129308_3BqNFprOMlddL1eZgG2seuarNOqp

                      mrjjM Offline
                      mrjjM Offline
                      mrjj
                      Lifetime Qt Champion
                      wrote on last edited by
                      #14

                      @dridk
                      Ok. thank you for the info.
                      I see what you try to make. Interesting.

                      Since you are viewing a segment at a time, maybe a memory mapped file
                      approach is suitable?

                      is the platform windows or linux?

                      D 1 Reply Last reply
                      0
                      • mrjjM mrjj

                        @dridk
                        Ok. thank you for the info.
                        I see what you try to make. Interesting.

                        Since you are viewing a segment at a time, maybe a memory mapped file
                        approach is suitable?

                        is the platform windows or linux?

                        D Offline
                        D Offline
                        dridk
                        wrote on last edited by
                        #15

                        @mrjj Should be avaible everywhere. Why this question ?

                        Nothing in Biology Makes Sense Except in the Light of Evolution

                        D mrjjM 2 Replies Last reply
                        0
                        • D dridk

                          @mrjj Should be avaible everywhere. Why this question ?

                          D Offline
                          D Offline
                          dridk
                          wrote on last edited by
                          #16

                          @dridk I see one segment at a time.. But I want to have the feeling that you can move smoothy without any load

                          Nothing in Biology Makes Sense Except in the Light of Evolution

                          1 Reply Last reply
                          0
                          • D dridk

                            @mrjj Should be avaible everywhere. Why this question ?

                            mrjjM Offline
                            mrjjM Offline
                            mrjj
                            Lifetime Qt Champion
                            wrote on last edited by
                            #17

                            @dridk
                            well both platforms supports mem files.
                            I have not tried Qt version so was just to know.
                            I have used it on windows with native API to fool around in 23 GB image.

                            But it seems Qt support this crossplatform.

                            1 Reply Last reply
                            0

                            • Login

                            • Login or register to search.
                            • First post
                              Last post
                            0
                            • Categories
                            • Recent
                            • Tags
                            • Popular
                            • Users
                            • Groups
                            • Search
                            • Get Qt Extensions
                            • Unsolved