Important: Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

How to save image (about 36 mb) to db via QSqlQuery?



  • Hello there!
    I have a problem with QSqlQuery, when trying to save relatively large (~36 mb) image. When I call QSqlQuery::exec(), I get std::bad_alloc exeption. When I work with smaller images, everything is ok. Need to be sayed - this code executes in other instance of QThread in order to prevent GUI freezing, all code compiled in MinGW-32 compiler, there are 8 GB RAM on PC. And some times (very rarely) I've got this code running without this exception even with images this size (36 mb)

    here is the code:

    ......
    QByteArray byte_array_image, byte_array_icon;
    QBuffer buffer_image(&byte_array_image), buffer_icon(&byte_array_icon);
    
    buffer_image.open(QIODevice::WriteOnly);
    buffer_icon.open(QIODevice::WriteOnly);
    
    image->save(&buffer_image, format.toLocal8Bit().data());
    icon->save(&buffer_icon, "JPEG");
    
    query.prepare("INSERT INTO pictures (id, id_act, date_time, name, picture, icon) VALUES (:id, :id_act, :date_time, :name, :picture, :icon)");
    ......
    query.bindValue(":picture", byte_array_image);
    query.bindValue(":icon", byte_array_icon);
    byte_array_image.clear();
    byte_array_icon.clear();
    
    save_succeed = query.exec();
    
    if (!save_succeed)
        emit database_failed(QUERY_ERROR_HEADER, QUERY_ERROR_SAVE_IMAGE + query.lastError().text());
    
    emit query_finished();
    
    return save_succeed;
    

    As far as I can see, problem can be related with lack of heap space for QApplication to create instance of QString included all bytecode of image. In order to fix that, I ve tried to split this image up to ten parts, but it doesn't help.

    Here is ten-parts-splitted query code:

    ............
    QByteArray byte_array_image, byte_array_icon;
    QBuffer buffer_image(&byte_array_image), buffer_icon(&byte_array_icon);
    
    buffer_image.open(QIODevice::WriteOnly);
    buffer_icon.open(QIODevice::WriteOnly);
    
    image->save(&buffer_image, format.toLocal8Bit().data());
    icon->save(&buffer_icon, "JPEG");
    
    int partlen, parts_count = 0;
    save_succeed = false;
    
    while (!save_succeed && ++parts_count < 10)
    {
         partlen = byte_array_image.size();
         partlen = byte_array_image.size() / parts_count;
    
         query.clear();
         query.prepare("INSERT INTO pictures (id, id_act, date_time, name, part_one, parts_count, icon, src, src_act) "
                          "VALUES (:id, :id_act, :date_time, :name, :part_one, :parts_count, :icon, :src, :src_act)");
         ........
         query.bindValue(":part_one", byte_array_image.mid(0, partlen));
         query.bindValue(":parts_count", parts_count);
         query.bindValue(":icon", byte_array_icon);
    
         try
         {
             save_succeed = query.exec();
         }
         catch (std::bad_alloc &error)
         {
             if (parts_count < 10)
                 continue;
    
             emit database_failed(QUERY_HEAP_ERROR_HEADER, QUERY_HEAP_ERROR_TEXT);
    
             return false;
         }
    
     }
    
     if (save_succeed)
     {
        QStringList part_names = { "part_one", "part_two", "part_three", "part_four", "part_five",
                                       "part_six", "part_seven", "part_eight", "part_nine", "part_ten"};
    
        for (int current_part = 1; current_part < parts_count; ++current_part)
        {
            query.clear();
    
            query.prepare("UPDATE pictures SET "+part_names[current_part]+" = :current_part WHERE id = :id AND src = :src");
            query.bindValue(":current_part", current_part == parts_count - 1 ? byte_array_image.mid(current_part*partlen) : byte_array_image.mid(current_part*partlen, partlen));
            ........
            save_succeed = query.exec();
        }
     }
    
     byte_array_image.clear();
     byte_array_icon.clear();
    

    Am I correct with the problem source? Is there anything I can do to fix it? 36 mb for image is not so much.


  • Lifetime Qt Champion

    Hi,

    You should check the current memory consumption of your application. Since you are building a 32bit application, you are limited to 2GB of RAM available to your process.



  • @sgaist
    Hello and thank you. I checked in task manager, On start my application consumes around 400 mb, when I trying to take snap, this amount grows to 900 mb, but then drops to~ 600. So with every snap I loose about 200 mb.
    I guess that is the problem. Can provided code leads to mem leak?


  • Lifetime Qt Champion

    What do you mean by snap ?

    What kind of image are you manipulating ?
    From the looks of it, you are making at list one copy of your image in a format that may or may not be compressed.
    I don't know about the icon.

    You should profile the application to see what is going on.



  • @sgaist
    By snap I mean image, which I'm pulling from camera
    I've noticed when I'm using try-catch block on qsqlquery::exec() it leads to memory leak. I mean is there a possability that when bad_alloc occurs inside qsqlquery class, program exits from exec() function without propper freeing memory up ?


  • Qt Champions 2019

    Why do you use a QImage pointer - where do you free it?



  • @christian-ehrlicher
    I call this function from another one where QImage declared as a static variable.


  • Qt Champions 2019

    And you're sure you don't allocate it again on every call?

    /edit: And why needs it to be a pointer at all?



  • @august-wiener said in How to save image (about 36 mb) to db via QSqlQuery?:

    36 mb for image is not so much.

    For a file, 36 MB is not too much. For a single field in a relational database, 36 MB is extremely unusual. You will probably have a lot more luck saving the image to a file, and just retaining the filename in the database as a short string. This is the much more common way to do it. The SQL database isn't really designed around storing large binary blobs, and neither are the associated libraries.


Log in to reply