Insertings rows into QSQlite database eats up all RAM Memory
-
I am using QSQlite database for analyzer application. Goal, is to parse traffic data of some protocol (say PCIe) and show it to User details of each packet. For Dataset of 400 MB, row count reach 60 million.
Currently, there are 8 columns and each column is of type TEXT.
When Application starts insertions into database, RAM Memory usage increases significantly.
Using below Insert function:void database::insertData(const QString &a_offset,const QString &a_byte3,const QString &a_byte2, const QString &a_byte1,const QString &a_byte0,const QString &a_dataType, const QString &a_status,const QString &a_location) { static QSqlQuery insertQuery; bool ok = false; QueryString = "insert into arincData values('"+a_offset+"'," \ "'"+a_byte3+"'," \ "'"+a_byte2+"'," \ "'"+a_byte1+"'," \ "'"+a_byte0+"'," \ "'"+a_dataType+"'," \ "'"+a_status+"'," \ "'"+a_location+"')"; ok = insertQuery.exec(QueryString); }
-
I am using QSQlite database for analyzer application. Goal, is to parse traffic data of some protocol (say PCIe) and show it to User details of each packet. For Dataset of 400 MB, row count reach 60 million.
Currently, there are 8 columns and each column is of type TEXT.
When Application starts insertions into database, RAM Memory usage increases significantly.
Using below Insert function:void database::insertData(const QString &a_offset,const QString &a_byte3,const QString &a_byte2, const QString &a_byte1,const QString &a_byte0,const QString &a_dataType, const QString &a_status,const QString &a_location) { static QSqlQuery insertQuery; bool ok = false; QueryString = "insert into arincData values('"+a_offset+"'," \ "'"+a_byte3+"'," \ "'"+a_byte2+"'," \ "'"+a_byte1+"'," \ "'"+a_byte0+"'," \ "'"+a_dataType+"'," \ "'"+a_status+"'," \ "'"+a_location+"')"; ok = insertQuery.exec(QueryString); }
@guru007
A (Q)SQLite operation will consume a certain, unspecified amount of memory. Your description states "increases significantly" while your title states "eats up all RAM Memory", these are not the same thing.Because your
QSqlQuery
isstatic
it will hang onto whatever resources. Start by removing thestatic
. Even then it is likely/possible that some memory will be consumed which may or may not be returned to the OS for re-use, so how are you measuring memory usage and when?After that test, if you insist on maintaining a
static
then at least call QSqlSquery::clear(), though I don't know how much difference that will make. -
@guru007
A (Q)SQLite operation will consume a certain, unspecified amount of memory. Your description states "increases significantly" while your title states "eats up all RAM Memory", these are not the same thing.Because your
QSqlQuery
isstatic
it will hang onto whatever resources. Start by removing thestatic
. Even then it is likely/possible that some memory will be consumed which may or may not be returned to the OS for re-use, so how are you measuring memory usage and when?After that test, if you insist on maintaining a
static
then at least call QSqlSquery::clear(), though I don't know how much difference that will make.Clear() is not needed and static is plain wrong. Also use a prepared query and bind the values instead creating a sql out of your strings to avoid sql injection problems.
-
Clear() is not needed and static is plain wrong. Also use a prepared query and bind the values instead creating a sql out of your strings to avoid sql injection problems.
@Christian-Ehrlicher
Agreedstatic
is wrong (apart from anything else it will hold onto theQSqlDatabase
, e.g. at close down). I only suggestedclear()
if user insists because ofClears the result set and releases any resources held by the query. Sets the query state to inactive. You should rarely if ever need to call this function.
-
@Christian-Ehrlicher
Agreedstatic
is wrong (apart from anything else it will hold onto theQSqlDatabase
, e.g. at close down). I only suggestedclear()
if user insists because ofClears the result set and releases any resources held by the query. Sets the query state to inactive. You should rarely if ever need to call this function.
@JonB The QSqlQuery is on the stack when it's not static so... And the clear also happens on each new exec/prepare.