How to save image (about 36 mb) to db via QSqlQuery?

[August Wiener]

Hello there!
I have a problem with QSqlQuery, when trying to save relatively large (~36 mb) image. When I call QSqlQuery::exec(), I get std::bad_alloc exeption. When I work with smaller images, everything is ok. Need to be sayed - this code executes in other instance of QThread in order to prevent GUI freezing, all code compiled in MinGW-32 compiler, there are 8 GB RAM on PC. And some times (very rarely) I've got this code running without this exception even with images this size (36 mb)

here is the code:

......
QByteArray byte_array_image, byte_array_icon;
QBuffer buffer_image(&byte_array_image), buffer_icon(&byte_array_icon);

buffer_image.open(QIODevice::WriteOnly);
buffer_icon.open(QIODevice::WriteOnly);

image->save(&buffer_image, format.toLocal8Bit().data());
icon->save(&buffer_icon, "JPEG");

query.prepare("INSERT INTO pictures (id, id_act, date_time, name, picture, icon) VALUES (:id, :id_act, :date_time, :name, :picture, :icon)");
......
query.bindValue(":picture", byte_array_image);
query.bindValue(":icon", byte_array_icon);
byte_array_image.clear();
byte_array_icon.clear();

save_succeed = query.exec();

if (!save_succeed)
    emit database_failed(QUERY_ERROR_HEADER, QUERY_ERROR_SAVE_IMAGE + query.lastError().text());

emit query_finished();

return save_succeed;

As far as I can see, problem can be related with lack of heap space for QApplication to create instance of QString included all bytecode of image. In order to fix that, I ve tried to split this image up to ten parts, but it doesn't help.

Here is ten-parts-splitted query code:

............
QByteArray byte_array_image, byte_array_icon;
QBuffer buffer_image(&byte_array_image), buffer_icon(&byte_array_icon);

buffer_image.open(QIODevice::WriteOnly);
buffer_icon.open(QIODevice::WriteOnly);

image->save(&buffer_image, format.toLocal8Bit().data());
icon->save(&buffer_icon, "JPEG");

int partlen, parts_count = 0;
save_succeed = false;

while (!save_succeed && ++parts_count < 10)
{
     partlen = byte_array_image.size();
     partlen = byte_array_image.size() / parts_count;

     query.clear();
     query.prepare("INSERT INTO pictures (id, id_act, date_time, name, part_one, parts_count, icon, src, src_act) "
                      "VALUES (:id, :id_act, :date_time, :name, :part_one, :parts_count, :icon, :src, :src_act)");
     ........
     query.bindValue(":part_one", byte_array_image.mid(0, partlen));
     query.bindValue(":parts_count", parts_count);
     query.bindValue(":icon", byte_array_icon);

     try
     {
         save_succeed = query.exec();
     }
     catch (std::bad_alloc &error)
     {
         if (parts_count < 10)
             continue;

         emit database_failed(QUERY_HEAP_ERROR_HEADER, QUERY_HEAP_ERROR_TEXT);

         return false;
     }

 }

 if (save_succeed)
 {
    QStringList part_names = { "part_one", "part_two", "part_three", "part_four", "part_five",
                                   "part_six", "part_seven", "part_eight", "part_nine", "part_ten"};

    for (int current_part = 1; current_part < parts_count; ++current_part)
    {
        query.clear();

        query.prepare("UPDATE pictures SET "+part_names[current_part]+" = :current_part WHERE id = :id AND src = :src");
        query.bindValue(":current_part", current_part == parts_count - 1 ? byte_array_image.mid(current_part*partlen) : byte_array_image.mid(current_part*partlen, partlen));
        ........
        save_succeed = query.exec();
    }
 }

 byte_array_image.clear();
 byte_array_icon.clear();

Am I correct with the problem source? Is there anything I can do to fix it? 36 mb for image is not so much.

[SGaist]

Hi,

You should check the current memory consumption of your application. Since you are building a 32bit application, you are limited to 2GB of RAM available to your process.

[August Wiener]

@sgaist
Hello and thank you. I checked in task manager, On start my application consumes around 400 mb, when I trying to take snap, this amount grows to 900 mb, but then drops to~ 600. So with every snap I loose about 200 mb.
I guess that is the problem. Can provided code leads to mem leak?

[SGaist]

What do you mean by snap ?

What kind of image are you manipulating ?
From the looks of it, you are making at list one copy of your image in a format that may or may not be compressed.
I don't know about the icon.

You should profile the application to see what is going on.

[August Wiener]

@sgaist
By snap I mean image, which I'm pulling from camera
I've noticed when I'm using try-catch block on qsqlquery::exec() it leads to memory leak. I mean is there a possability that when bad_alloc occurs inside qsqlquery class, program exits from exec() function without propper freeing memory up ?

[Christian Ehrlicher]

Why do you use a QImage pointer - where do you free it?

[August Wiener]

@christian-ehrlicher
I call this function from another one where QImage declared as a static variable.

[Christian Ehrlicher]

And you're sure you don't allocate it again on every call?

/edit: And why needs it to be a pointer at all?

[wrosecrans]

@august-wiener said in How to save image (about 36 mb) to db via QSqlQuery?:

36 mb for image is not so much.

For a file, 36 MB is not too much. For a single field in a relational database, 36 MB is extremely unusual. You will probably have a lot more luck saving the image to a file, and just retaining the filename in the database as a short string. This is the much more common way to do it. The SQL database isn't really designed around storing large binary blobs, and neither are the associated libraries.