QMap gone nuts



  • I am using multiple QMap objects (an array of 30 QMap objects) for storing data. These QMap objects are filled with about 35.000 integer key/value pairs each. Later the maps are cleared to discard those data.

    Filling and using the maps works fine. However when clearing the maps, the program needs up to 10 minutes (!) to discard all data. This is simply not acceptable. So I did a lot of work to find the cause. The result is a small example program (see below) that leaves me baffled.

    There are two alternative nested loops for filling the QMaps (40000 iterations result in about 23000 unique random keys; values are all "1"). The last clear-loop discards all data. When using loop 1), everything is ok. When using loop 2), the following clear-loop needs 2 minutes instead of 2 seconds to finish. As you can see, there shouldn't be a big runtime difference since the only change between loops 1) and 2) is the nesting order!

    @
    qDebug() << "Create" << QDateTime::currentDateTime();
    QMap<int,int> maps[30];

    // 1) Using this insert-loop, the clear-loop needs about 0:02 minutes to complete
    // -> Clear cycles 0..29 run very fast
    for (int i=0; i<30; i++)
    {
    for (int j=0; j<40000; j++)
    {
    int key = qrand();
    maps[i].insert(key, 1);
    }
    }

    // 2) Using this insert-loop, the clear-loop needs about 2:15 minutes to complete
    // -> Clear cycles 0..12 run very fast, 13..29 increasingly slow
    /*
    for (int j=0; j<40000; j++)
    {
    for (int i=0; i<30; i++)
    {
    int key = qrand();
    maps[i].insert(key, 1);
    }
    }
    */

    // Clear-loop
    for (int i=0; i<30; i++)
    {
    qDebug() << i << maps[i].size() << "Clear" << QDateTime::currentDateTime();
    maps[i].clear();
    }

    qDebug() << "Complete" << QDateTime::currentDateTime();
    @

    Does anyone have an idea why 1) and 2) make a difference when clearing the inserted data?

    P.S. I am using Qt4.8.2 and VC++2010 at Win7


  • Moderators

    Thanks for posting this.

    I believe that the reason is not QMap but the memory management (maybe of VC).

    I have an application showing something similar at the end. The destruction of all memory allocation takes ages. The application does not use QMap, but std::map.

    I am using vc2005 on Win7 64 bit. I am going to check your example right away and come back with info.



  • Thank you in advance! I appreciate your help very much!

    Actually this is a total showstopper for me. In my realworld software I can not simply change the nesting order because the inner and outer loop are separated by an API.


  • Moderators

    OK, I have done the following.
    Copied your little example into a small core application created in vc2005.

    I have compiled in debug and release modes. Happened to start with the release modes, but started both modes with Ctrl+F5
    Here are the results:
    @
    First method (debug started with Ctrl+F5)

    Create QDateTime("Mi 22. Aug 16:35:20 2012")
    0 23143 Clear QDateTime("Mi 22. Aug 16:35:21 2012")
    1 23044 Clear QDateTime("Mi 22. Aug 16:35:21 2012")
    2 23100 Clear QDateTime("Mi 22. Aug 16:35:21 2012")

    28 23098 Clear QDateTime("Mi 22. Aug 16:35:21 2012")
    29 23218 Clear QDateTime("Mi 22. Aug 16:35:21 2012")
    Complete QDateTime("Mi 22. Aug 16:35:21 2012")

    Second method (debug started with Ctrl+F5)

    Create QDateTime("Mi 22. Aug 16:37:02 2012")
    0 23117 Clear QDateTime("Mi 22. Aug 16:37:04 2012")
    1 23171 Clear QDateTime("Mi 22. Aug 16:37:04 2012")
    2 22963 Clear QDateTime("Mi 22. Aug 16:37:04 2012")

    28 23104 Clear QDateTime("Mi 22. Aug 16:37:05 2012")
    29 23081 Clear QDateTime("Mi 22. Aug 16:37:05 2012")
    Complete QDateTime("Mi 22. Aug 16:37:05 2012")

    First method (release)

    Create QDateTime("Mi 22. Aug 16:32:18 2012")
    0 23143 Clear QDateTime("Mi 22. Aug 16:32:18 2012")
    1 23044 Clear QDateTime("Mi 22. Aug 16:32:18 2012")
    2 23100 Clear QDateTime("Mi 22. Aug 16:32:18 2012")
    28 23098 Clear QDateTime("Mi 22. Aug 16:32:18 2012")
    29 23218 Clear QDateTime("Mi 22. Aug 16:32:18 2012")
    Complete QDateTime("Mi 22. Aug 16:32:18 2012")

    second method (release)

    Create QDateTime("Mi 22. Aug 16:30:55 2012")
    0 23117 Clear QDateTime("Mi 22. Aug 16:30:57 2012")
    1 23171 Clear QDateTime("Mi 22. Aug 16:30:57 2012")
    2 22963 Clear QDateTime("Mi 22. Aug 16:30:57 2012")

    28 23104 Clear QDateTime("Mi 22. Aug 16:30:57 2012")
    29 23081 Clear QDateTime("Mi 22. Aug 16:30:57 2012")
    Complete QDateTime("Mi 22. Aug 16:30:57 2012")
    @

    As you see, it does not confirm your results. However, when you start the debug compile in the debugger just with F5, I could confirm:
    !http://db.tt/sXbCuZKX(Method1)!
    and
    !http://db.tt/YoRD5RMu(Method2)!

    Since method 2 takes so long, you have a chance to start with F5 and detach the program. It remains slow even so it is no longer attach to the debugger.


  • Moderators

    So as a summary it depends certainly on the compilation, but also on the run mode during start.
    Again it has nothing to do with QMap, but should be the memory management in combination with the debugger. This at least with msvc.

    I have a real-time application which runs fine. I typically start in the debugger and let it run for quite some time. When I want to stop, I am sending a command via TCP/IP telling the application stop in an orderly mode (closing all stuff and writing the buffers).
    I have noticed that at the end it takes ages to close down. I found already that the issue is freeing the memory, because every time I paused in the debugger it was somewhere in the destructors freeing memory. I know for sure that the memory is fragmented.



  • With GCC the code completes fast, independent of creation method or compile mode. The first method completes in 360 milliseconds, the second method in 1100 milliseconds. (in debug mode, add ~200 milliseconds or so.)

    Maybe you could just use a proper compiler? Why do you use VC with Qt in the first place, isn't it a pain to get running compared to mingw/gcc?


  • Moderators

    [quote author="DerManu" date="1345671811"]With GCC the code completes fast, independent of creation method or compile mode. The first method completes in 360 milliseconds, the second method in 1100 milliseconds. (in debug mode, add ~200 milliseconds or so.)
    [/quote]
    Thanks for information.

    [quote author="DerManu" date="1345671811"]Maybe you could just use a proper compiler?[/quote]
    Everybody considers his own tools as the only proper ones ;-)
    BTW it is not the compiler as you will see when reading the details provided. The compile mode itself does not cause the ridiculous long execution time. It is only when the debug compile is started in the debugger when the whole overhead of comms with the debugger kicks in.
    Possibly there are debuggers handling that faster, but do they provide then the same convenience while debugging. I do not know.

    [quote author="DerManu" date="1345671811"]Why do you use VC with Qt in the first place, isn't it a pain to get running compared to mingw/gcc?[/quote]
    I am using VC with Qt since 2006. There are always issues with any tool chain. For me vc2005 with Qt runs smoothly. So why changing a running system especially when used to the IDE.
    Nevertheless, I am not as ignorant for not looking across the rim of the plate. Qt creator with its tool chains becomes for me personally an alternative. However, executable sizes of a factor of 5 or something in comparable compile modes were a bit shocking to say the least. There have been other things I have not been happy about.

    There are other posts detailing the reasons for choosing a particular IDE/compiler combo. I do not want to repeat here.



  • Very interesting. Thanks again.

    I tried starting the process detached... and yes, the application runs very fast when not attached to the debugger. It does not make a difference if a debug or rease build is used which is confirmed and explained in the following link:
    http://preshing.com/20110717/the-windows-heap-is-slow-when-launched-from-the-debugger
    The link also explains how to "solve" the problem.

    Now I know the reason for the problem but not how to deal with it. On one hand I can't wait 10 minutes during every debugging session for the data to be cleared. On the other hand I don't want to disable the debug heap manager including its error checking functions.

    I would gladly switch to GCC. I used that compiler before and it is very good. Especially it provides a lot of up-to-date C++11 features! But this issue is about the debugger... and the VC++ debugger is (appart from that performance issue) the best windows debugger I have seen and I don't want to miss it.

    Surely Ctrl+F5 will get more interesting for me in the future.


  • Moderators

    I knew about differences between starting debug-compiled programs in VC. However, I could not believe the significance.
    In my application I do heavy number crunching. The amount is considerably dependent on the input data. Sometimes it does haywire because of weak conditions. However, I have noticed that apparently the clean-up takes considerably longer. I had it on my list to do some performance analysis and try to find the bottle-neck. Especially I started to get worried about undetected memory corruption.

    Anyhow, C++11 features are also the reason for me to look for alternatives. The debugging within Qt creator was not as comfortable for me so far. Nevertheless, it should be noted that it has improved significantly since the beginning. I am wondering if it is even possible to "copy-cat" somehow the possibilities as available with a closely bundled IDE as with VC.

    Also the testing done by DerManu may be still comparing apples and pears. It leaves open, if the debug runs are done under full use of the debugger and under which.



  • What exactly was the problem with Qt Creator for you (especially debugging)?

    I couldn't test Qt Creator yet but I planned to do. I was hoping that it could be the VC++ alternative I am looking for.


  • Moderators

    [quote author="silicomancer" date="1345709956"]What exactly was the problem with Qt Creator for you (especially debugging)?

    I couldn't test Qt Creator yet but I planned to do. I was hoping that it could be the VC++ alternative I am looking for.[/quote]

    It is mainly the look and feel in the first place. Tools, you are not used to, are typically a bit weird in handling. Your routine handling is simply interrupted. So, you are not comfortable. If you have things to be done, you are not enjoying to try find out how to do what.

    I am mainly a windows guy. I had to use windows and vc in the past. Now basically I could switch to linux, but I am not doing because it has advantages to live in both worlds.

    Qt creator is the tool available on both linux as well as on windows. So it is a natural choice in my opinion. However, to use Qt creator under linux in a VirtualBox I found is not really fun. It is workable for getting the stuff compiled, but the delays of the mouse stress a bit my patience too much. Therefore, the decision is to do the main stuff in windows and vc.

    I have compiled my applications lately under Qt creator with MinGW, but that experience was not just fun. Most of it probably because seen with the eyes of a vc user. The compile messages were hard to understand. The compilation of a template using a template was no issue with the old vc 2005 not even a warning. Suddenly the mingw delivered with the recent creator complained with an error. What knocked me off at the end were the sizes of executables. Fairly small exe (debug compile with vc) were suddenly 5 to 10 times larger. At the moment I am not really prepared to accept exe sizes of 50MB and more ;-)

    So at the moment it is less a decision based on weighting all the pros and cons of the tools properly.



  • Ok. I see.


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.