Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. Insertings rows into QSQlite database eats up all RAM Memory
Forum Updated to NodeBB v4.3 + New Features

Insertings rows into QSQlite database eats up all RAM Memory

Scheduled Pinned Locked Moved Unsolved General and Desktop
5 Posts 3 Posters 236 Views
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • G Offline
    G Offline
    guru007
    wrote on last edited by
    #1

    I am using QSQlite database for analyzer application. Goal, is to parse traffic data of some protocol (say PCIe) and show it to User details of each packet. For Dataset of 400 MB, row count reach 60 million.
    Currently, there are 8 columns and each column is of type TEXT.
    When Application starts insertions into database, RAM Memory usage increases significantly.
    Using below Insert function:

    void database::insertData(const QString &a_offset,const QString &a_byte3,const QString &a_byte2,
                              const QString &a_byte1,const QString &a_byte0,const QString &a_dataType,
                              const QString &a_status,const QString &a_location)
    {
    static QSqlQuery insertQuery;
        bool ok = false;
    
      QueryString = "insert into arincData values('"+a_offset+"'," \
                                                   "'"+a_byte3+"'," \
                                                   "'"+a_byte2+"'," \
                                                   "'"+a_byte1+"'," \
                                                   "'"+a_byte0+"'," \
                                                   "'"+a_dataType+"'," \
                                                   "'"+a_status+"'," \
                                                   "'"+a_location+"')";
    
        ok = insertQuery.exec(QueryString);
    
    }
    
    JonBJ 1 Reply Last reply
    0
    • G guru007

      I am using QSQlite database for analyzer application. Goal, is to parse traffic data of some protocol (say PCIe) and show it to User details of each packet. For Dataset of 400 MB, row count reach 60 million.
      Currently, there are 8 columns and each column is of type TEXT.
      When Application starts insertions into database, RAM Memory usage increases significantly.
      Using below Insert function:

      void database::insertData(const QString &a_offset,const QString &a_byte3,const QString &a_byte2,
                                const QString &a_byte1,const QString &a_byte0,const QString &a_dataType,
                                const QString &a_status,const QString &a_location)
      {
      static QSqlQuery insertQuery;
          bool ok = false;
      
        QueryString = "insert into arincData values('"+a_offset+"'," \
                                                     "'"+a_byte3+"'," \
                                                     "'"+a_byte2+"'," \
                                                     "'"+a_byte1+"'," \
                                                     "'"+a_byte0+"'," \
                                                     "'"+a_dataType+"'," \
                                                     "'"+a_status+"'," \
                                                     "'"+a_location+"')";
      
          ok = insertQuery.exec(QueryString);
      
      }
      
      JonBJ Offline
      JonBJ Offline
      JonB
      wrote on last edited by JonB
      #2

      @guru007
      A (Q)SQLite operation will consume a certain, unspecified amount of memory. Your description states "increases significantly" while your title states "eats up all RAM Memory", these are not the same thing.

      Because your QSqlQuery is static it will hang onto whatever resources. Start by removing the static. Even then it is likely/possible that some memory will be consumed which may or may not be returned to the OS for re-use, so how are you measuring memory usage and when?

      After that test, if you insist on maintaining a static then at least call QSqlSquery::clear(), though I don't know how much difference that will make.

      Christian EhrlicherC 1 Reply Last reply
      1
      • JonBJ JonB

        @guru007
        A (Q)SQLite operation will consume a certain, unspecified amount of memory. Your description states "increases significantly" while your title states "eats up all RAM Memory", these are not the same thing.

        Because your QSqlQuery is static it will hang onto whatever resources. Start by removing the static. Even then it is likely/possible that some memory will be consumed which may or may not be returned to the OS for re-use, so how are you measuring memory usage and when?

        After that test, if you insist on maintaining a static then at least call QSqlSquery::clear(), though I don't know how much difference that will make.

        Christian EhrlicherC Offline
        Christian EhrlicherC Offline
        Christian Ehrlicher
        Lifetime Qt Champion
        wrote on last edited by
        #3

        Clear() is not needed and static is plain wrong. Also use a prepared query and bind the values instead creating a sql out of your strings to avoid sql injection problems.

        Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
        Visit the Qt Academy at https://academy.qt.io/catalog

        JonBJ 1 Reply Last reply
        3
        • Christian EhrlicherC Christian Ehrlicher

          Clear() is not needed and static is plain wrong. Also use a prepared query and bind the values instead creating a sql out of your strings to avoid sql injection problems.

          JonBJ Offline
          JonBJ Offline
          JonB
          wrote on last edited by JonB
          #4

          @Christian-Ehrlicher
          Agreed static is wrong (apart from anything else it will hold onto the QSqlDatabase, e.g. at close down). I only suggested clear() if user insists because of

          Clears the result set and releases any resources held by the query. Sets the query state to inactive. You should rarely if ever need to call this function.

          Christian EhrlicherC 1 Reply Last reply
          0
          • JonBJ JonB

            @Christian-Ehrlicher
            Agreed static is wrong (apart from anything else it will hold onto the QSqlDatabase, e.g. at close down). I only suggested clear() if user insists because of

            Clears the result set and releases any resources held by the query. Sets the query state to inactive. You should rarely if ever need to call this function.

            Christian EhrlicherC Offline
            Christian EhrlicherC Offline
            Christian Ehrlicher
            Lifetime Qt Champion
            wrote on last edited by
            #5

            @JonB The QSqlQuery is on the stack when it's not static so... And the clear also happens on each new exec/prepare.

            Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
            Visit the Qt Academy at https://academy.qt.io/catalog

            1 Reply Last reply
            0

            • Login

            • Login or register to search.
            • First post
              Last post
            0
            • Categories
            • Recent
            • Tags
            • Popular
            • Users
            • Groups
            • Search
            • Get Qt Extensions
            • Unsolved