Memory (?) woes
-
Whenever I run my application along with memory hogs like Opera and Firefox, which are eating plenty of RAM (over 1Gb each), my app starts freezing from time to time showing the busy cursor. After a few seconds it continues running. Both Opera and Firefox remain fully responsive at that time. I primarily use heap for object creation, but allocating more heap (via adding QMAKE_CXXFLAGS += -Wl, --heap=100000000 line to .pro file) doesn't solve this problem (in fact, it doesn't change anything). There are no memory leaks, and the program usually stays below 100M by memory use (but these hangs can happen even if it stays well below 50 Mb) . If I close Firefox and Opera, my app runs smoothly thereafter. There are no warnings or anything in the Application Output console
I'm running a virtual Windows 7 machine under VMware, and I have 4 Gb allocated for it with memory trimming option off (i.e. the machine gets allocated all 4 Gb of memory at once and in full), so that shouldn't be the cause of my problem. I have 16 Gb of physical memory in total (and it is free most of the time). I have over 1Gb shown as free in Windows Task Manager as well
How can I prevent these sudden hangs? After some time they become really annoying
-
Hi
What does your app do ?
When you app lags. if you start some other program, does that run as expected? -
It sends and receives some data to and from network (mostly receives and shows). It doesn't work with disk, and there are no disk operations going on when my app freezes (I can see that in the VMware console) if that was your point
@deisik
Well i'm just fishing for any reasonable explanation on how two browsers could have any influence your other app.
Since you app seems low mem ( < 100 MB ) its not from virtual mem paging directly.
You could try disable virtual memory in guest to check its not related to paging.Also, is it possible to provoke this effect with non network apps ?
Like using consume.exe
https://www.microsoft.com/en-us/download/details.aspx?displaylang=en&id=17657
or
https://blogs.msdn.microsoft.com/vijaysk/2012/10/26/tools-to-simulate-cpu-memory-disk-load/ -
@deisik
Well i'm just fishing for any reasonable explanation on how two browsers could have any influence your other app.
Since you app seems low mem ( < 100 MB ) its not from virtual mem paging directly.
You could try disable virtual memory in guest to check its not related to paging.Also, is it possible to provoke this effect with non network apps ?
Like using consume.exe
https://www.microsoft.com/en-us/download/details.aspx?displaylang=en&id=17657
or
https://blogs.msdn.microsoft.com/vijaysk/2012/10/26/tools-to-simulate-cpu-memory-disk-load/It seems to be related to memory allocation since my app constantly creates small objects as it receives data from the network (they are to stay in memory till the program is closed). I don't know while allocating more heap doesn't work though
-
It seems to be related to memory allocation since my app constantly creates small objects as it receives data from the network (they are to stay in memory till the program is closed). I don't know while allocating more heap doesn't work though
-
@deisik
well if you don't new/delete all the time it rules out fragmentation.what happens if u decrease the memory for the guest. Just Like when the hogs are doing it?
I've reduced memory to 2Gb and started only Firefox now. Now I have problems with browsing but my app works fine so far. I guess memory fragmentation is the main cause behind the lags since I do create and destroy objects all the time. And with more data received from the network the temporary data structures used for processing this data become bigger and bigger (though they are certainly not big in absolute terms). I thought increasing the heap size would help but it didn't (or it didn't get increased)
Is there a simple way to resolve this issue (and check how much heap the app actually uses)?
Update: I'm back to 4GB
-
I've reduced memory to 2Gb and started only Firefox now. Now I have problems with browsing but my app works fine so far. I guess memory fragmentation is the main cause behind the lags since I do create and destroy objects all the time. And with more data received from the network the temporary data structures used for processing this data become bigger and bigger (though they are certainly not big in absolute terms). I thought increasing the heap size would help but it didn't (or it didn't get increased)
Is there a simple way to resolve this issue (and check how much heap the app actually uses)?
Update: I'm back to 4GB
@deisik Are you sure it is related to RAM and not CPU load? Firefox can consume a lot of CPU depending on websites loaded.
-
@deisik Are you sure it is related to RAM and not CPU load? Firefox can consume a lot of CPU depending on websites loaded.
No, this has nothing to do with CPU load. Anyway, I have 4 cores in the guest and 8 cores physical, so this is not an issue. When my app freezes there are no hikes in CPU load, in any program. The system is fully responsive as well as everything else apart from my app. In these moments it looks as if covered by a white veil with a round "busy" Windows cursor over it. Sometimes there is no white veil at all, just a "busy" cursor
-
No, this has nothing to do with CPU load. Anyway, I have 4 cores in the guest and 8 cores physical, so this is not an issue. When my app freezes there are no hikes in CPU load, in any program. The system is fully responsive as well as everything else apart from my app. In these moments it looks as if covered by a white veil with a round "busy" Windows cursor over it. Sometimes there is no white veil at all, just a "busy" cursor
@deisik You said your app uses network - this could be another thing to check. Depending on how you implemented the networking in your app it could be blocked while waiting fore response for example.
Did you test your app on a real hardware? -
@deisik You said your app uses network - this could be another thing to check. Depending on how you implemented the networking in your app it could be blocked while waiting fore response for example.
Did you test your app on a real hardware?I'm running network in asynchronous mode so there can hardly be such issues. If it were network, then closing browsers (and freeing memory) wouldn't change anything, right? It is obviously related to memory allocation or memory fragmentation due to a lot of new/delete operations. Yes, I run the same code on Linux (which is the host), but not as extensively, and I didn't see anything like that there
-
I'm running network in asynchronous mode so there can hardly be such issues. If it were network, then closing browsers (and freeing memory) wouldn't change anything, right? It is obviously related to memory allocation or memory fragmentation due to a lot of new/delete operations. Yes, I run the same code on Linux (which is the host), but not as extensively, and I didn't see anything like that there
Have you disabled paging on your guest? Try doing that and see if anything pops up. Be aware that doing this will cause exhausting the available memory to wreak all kinds of havoc on the running application(s). But at least should give you some idea whether memory access speed is the culprit.
Read and abide by the Qt Code of Conduct
-
Have you disabled paging on your guest? Try doing that and see if anything pops up. Be aware that doing this will cause exhausting the available memory to wreak all kinds of havoc on the running application(s). But at least should give you some idea whether memory access speed is the culprit.
If it were swapping I would see that in the Virtual Machine console, but there is nothing. Moreover, I still have about 1Gb of RAM free as shown by the Windows Task Manager. Just in case, in VMware preferences I set the option to fit all virtual machine memory into reserved host RAM. Host RAM for virtual machines is set to maximum and equals almost 14 Gb. There are no other virtual machines running
-
If it were swapping I would see that in the Virtual Machine console, but there is nothing. Moreover, I still have about 1Gb of RAM free as shown by the Windows Task Manager. Just in case, in VMware preferences I set the option to fit all virtual machine memory into reserved host RAM. Host RAM for virtual machines is set to maximum and equals almost 14 Gb. There are no other virtual machines running
@deisik said in Memory (?) woes:
If it were swapping I would see that in the Virtual Machine console,
Why? You mean to say that the virtual machine console will tell you all the HDD read/writes occurring with their sources, or that the VM console tracks windows' internals to know when the OS decides to swap?
The guest system is oblivious that it's running on a virtual machine, additionally windows can decide to swap before exhausting all the memory (which is actually the case), it's not something you can control.Read and abide by the Qt Code of Conduct
-
@deisik said in Memory (?) woes:
If it were swapping I would see that in the Virtual Machine console,
Why? You mean to say that the virtual machine console will tell you all the HDD read/writes occurring with their sources, or that the VM console tracks windows' internals to know when the OS decides to swap?
The guest system is oblivious that it's running on a virtual machine, additionally windows can decide to swap before exhausting all the memory (which is actually the case), it's not something you can control.I can easily see disk operations in the VM Console (when disks are accessed the disk icons blink and it is obvious that disks are rarely accessed), irrespective of whether Windows uses swap or not. Anyway, this would affect all programs, especially those browsers which are eating over 1Gb each but they are working fine. In short, this is not a memory swap issue, I certainly know how it feels when Windows starts swapping heavily. I've checked that the first thing
-
I can easily see disk operations in the VM Console (when disks are accessed the disk icons blink and it is obvious that disks are rarely accessed), irrespective of whether Windows uses swap or not. Anyway, this would affect all programs, especially those browsers which are eating over 1Gb each but they are working fine. In short, this is not a memory swap issue, I certainly know how it feels when Windows starts swapping heavily. I've checked that the first thing
Fine.
Do you use threading in your application and if you do, how do you do it? Are you making sure you're not blocking the GUI thread's event loop?PS.
You should really attach a debugger to your application while running in the guest system and interrupt the process, then extract a stack trace to see what's going on.Read and abide by the Qt Code of Conduct
-
Fine.
Do you use threading in your application and if you do, how do you do it? Are you making sure you're not blocking the GUI thread's event loop?PS.
You should really attach a debugger to your application while running in the guest system and interrupt the process, then extract a stack trace to see what's going on. -
Obviously, networking uses threads but if something would be blocking GUI internally then how Windows would know that? What would I interrupt if it is system related?
@deisik said in Memory (?) woes:
but if something would be blocking GUI internally then how Windows would know that?
It won't so it will show a cursor that's spinning around. And if the GUI event loop is blocked, no repainting will be done, so you might see (but not always) a white box instead of your application window. My other advice stands: compile in debug mode (and don't mess with the compilation flags), then run your app through the debugger and see what happens and if the problem manifests itself.
Read and abide by the Qt Code of Conduct
-
@deisik said in Memory (?) woes:
but if something would be blocking GUI internally then how Windows would know that?
It won't so it will show a cursor that's spinning around. And if the GUI event loop is blocked, no repainting will be done, so you might see (but not always) a white box instead of your application window. My other advice stands: compile in debug mode (and don't mess with the compilation flags), then run your app through the debugger and see what happens and if the problem manifests itself.
-
And how that would explain that closing browsers fixes this issue? Lags don't happen all the time, they happen completely at random but seem to come in packs
@deisik said in Memory (?) woes:
And how that would explain that closing browsers fixes this issue?
I'm not selling explanations here, and I'm not about to venture on a speculation expedition. I'm suggesting ways for you to track down your problem, you can take them or not, it's your choice. If you don't want to run everything through the debugger, then start your debug application and wait for the problem, then attach the debugger to the running process and then interrupt it to see what's going on inside.
Read and abide by the Qt Code of Conduct
