Stability Issues and Troubleshooting Ghost Bugs
-
I have been put in charge of maintaining an app that has been in development for quite a few years. It is based on QtQuick in some portions and QtWidgets in other portions (we are moving everything to QtQuick). The application is compiled for Windows and Linux (mingw and gcc). The version of Qt is 5.9.7.
The problem is that the Windows version seems to have stability issues. I originally thought it was a code issue somewhere. So I ran some analysis software over the code and turned up some style issues, but not really any smoking gun problems. I have walked through a lot of the code, but I am not finding anything that stands out. It seems like the problem moves around. I ran heob, but it crashes and refuses to run. When I do find ways to cause segfaults by running the app they are hard to reproduce. When I can cause issues repeatedly I try to capture what is going on with the debugger and it does not flag the issue anywhere near the code I think is responsible. When I walk through the code I do flag it does not turn up any solid problems. Its like chasing a ghost. I have found conditions where certain functions cause crashes. So I take those functions and test them in a separate test project and I run debuggers on them and find there are no issues with those functions/objects. I have tested quite a few objects this way. Still no smoking gun.
So I decided to compile under Linux and use the Qt Creator analysis tools to see if I can find the issues there. I ran the Clang Tidy and Clazy analysis, but it turned up style issues mostly. I also noticed that when running in Linux it does not have stability issues. It is like they all magically vanished. It runs exactly like it should for everything I test make when running in Linux. It is like the Linux runtime is better than the Windows runtime. I have started looking at places where we have Windows only versus Linux only code. I am not seeing any major issues there either.
I am going to run valgrind under Linux to see if there are memory issues. However, if the problem is related to Windows and I cannot run Heob without it crashing how do I analyze the code in Windows? I am looking for strategies for finding hard to find bugs/stability issues.
Thanks for any suggestions.
C++ is a perfectly valid school of magic.
-
I have been put in charge of maintaining an app that has been in development for quite a few years. It is based on QtQuick in some portions and QtWidgets in other portions (we are moving everything to QtQuick). The application is compiled for Windows and Linux (mingw and gcc). The version of Qt is 5.9.7.
The problem is that the Windows version seems to have stability issues. I originally thought it was a code issue somewhere. So I ran some analysis software over the code and turned up some style issues, but not really any smoking gun problems. I have walked through a lot of the code, but I am not finding anything that stands out. It seems like the problem moves around. I ran heob, but it crashes and refuses to run. When I do find ways to cause segfaults by running the app they are hard to reproduce. When I can cause issues repeatedly I try to capture what is going on with the debugger and it does not flag the issue anywhere near the code I think is responsible. When I walk through the code I do flag it does not turn up any solid problems. Its like chasing a ghost. I have found conditions where certain functions cause crashes. So I take those functions and test them in a separate test project and I run debuggers on them and find there are no issues with those functions/objects. I have tested quite a few objects this way. Still no smoking gun.
So I decided to compile under Linux and use the Qt Creator analysis tools to see if I can find the issues there. I ran the Clang Tidy and Clazy analysis, but it turned up style issues mostly. I also noticed that when running in Linux it does not have stability issues. It is like they all magically vanished. It runs exactly like it should for everything I test make when running in Linux. It is like the Linux runtime is better than the Windows runtime. I have started looking at places where we have Windows only versus Linux only code. I am not seeing any major issues there either.
I am going to run valgrind under Linux to see if there are memory issues. However, if the problem is related to Windows and I cannot run Heob without it crashing how do I analyze the code in Windows? I am looking for strategies for finding hard to find bugs/stability issues.
Thanks for any suggestions.
-
The biggest issue I have is when it crashes it is generally not in an area where the issue appears. I found a segfault in one section, but when I ran the debugger it finds problems in a completely different location. As I step through the other area I am not finding the issue in the ram. Then I take that code and put it into a test project and test it in isolation and it finds zero issues. It keeps moving around.
I did run valgrind and it turned up one issue so far:
union { float fvalue; unsigned long lvalue=0; // 8 bytes on 64 bit system, created on stack high bytes would be uninitialized };The fvalue was being set, but that means only bottom 4 bytes were getting initialized. So I set the lvalue explicitly. Previously the system was compiled in 32 bit so the sizes used to match. The union was created on the stack so was not initialized.
Hmmm, now valgrind is crashing... I think it needs more ram. I am running it in a virtual machine.
-
Valgrind output is telling me to submit a bug report:
11:14:55: valgrind --child-silent-after-fork=yes --xml-socket=127.0.0.1:44329 --log-socket=127.0.0.1:40375 --xml=yes --smc-check=stack --tool=memcheck --gen-suppressions=all --track-origins=yes --leak-check=summary --num-callers=25 --vgdb=yes --vgdb-error=0 /home/amet/git/xm-controller/build-xmedit-Desktop_Qt_5_9_7_GCC_64bit-Debug/xmedit 11:14:55: Debugging starts ==2208== (action at startup) vgdb me ... ==2208== ==2208== TO DEBUG THIS PROCESS USING GDB: start GDB like this ==2208== /path/to/gdb /home/amet/git/xm-controller/build-xmedit-Desktop_Qt_5_9_7_GCC_64bit-Debug/xmedit ==2208== and then give GDB the following command ==2208== target remote | /usr/lib/valgrind/../../bin/vgdb --pid=2208 ==2208== --pid is optional if only one valgrind process is running ==2208== remote target does not support file transfer, attempting to access files from local filesystem. 11:14:58: Unexpected run control state RunControlState::Running when worker DebuggerRunTool started. QML debugging is enabled. Only use this in a safe environment. VEX temporary storage exhausted. Pool = TEMP, start 0x38f8e668 c urr 0x3940ca58 end 0x394531a7 (size 5000000) vex: the `impossible' happened: VEX temporary storage exhausted. Increase N_{TEMPORARY,PERMANENT}_BYTES and recompile. vex storage: T total 26236458736 bytes allocated vex storage: P total 640 bytes allocated valgrind: the 'impossible' happened: LibVEX called failure_exit(). host stacktrace: <frame> <ip>0x81FD360</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>pthread_cond_wait@@GLIBC_2.3.2</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl/../sysdeps/unix/sysv/linux/x86_64</dir> <file>pthread_cond_wait.S</file> <line>185</line> </frame> <frame> <ip>0x22A1C5EA</ip> <obj>/usr/lib/x86_64-linux-gnu/dri/swrast_dri.so</obj> </frame> <frame> <ip>0x22A2764F</ip> <obj>/usr/lib/x86_64-linux-gnu/dri/swrast_dri.so</obj> </frame> <frame> <ip>0x22A28182</ip> <obj>/usr/lib/x86_64-linux-gnu/dri/swrast_dri.so</obj> </frame> <frame> <ip>0x2249F260</ip> <obj>/usr/lib/x86_64-linux-gnu/dri/swrast_dri.so</obj> </frame> <frame> <ip>0x90BABF6</ip> <obj>/usr/lib/x86_64-linux-gnu/mesa/libGL.so.1.2.0</obj> </frame> <frame> <ip>0x11F50801</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/plugins/xcbglintegrations/libqxcb-glx-integration.so</obj> </frame> <frame> <ip>0x607CA0C</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Gui.so.5.9.7</obj> <fn>QOpenGLContext::swapBuffers(QSurface*)</fn> </frame> <frame> <ip>0x4FBF46D</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Quick.so.5.9.7</obj> </frame> <frame> <ip>0x4FC007C</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Quick.so.5.9.7</obj> </frame> <frame> <ip>0x60509CA</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Gui.so.5.9.7</obj> <fn>QWindow::event(QEvent*)</fn> </frame> <frame> <ip>0x502DAB4</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Quick.so.5.9.7</obj> <fn>QQuickWindow::event(QEvent*)</fn> </frame> <frame> <ip>0x583570B</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Widgets.so.5.9.7</obj> <fn>QApplicationPrivate::notify_helper(QObject*, QEvent*)</fn> </frame> <frame> <ip>0x583CB36</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Widgets.so.5.9.7</obj> <fn>QApplication::notify(QObject*, QEvent*)</fn> </frame> <frame> <ip>0x7D2AED7</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> <fn>QCoreApplication::notifyInternal2(QObject*, QEvent*)</fn> </frame> <frame> <ip>0x6046006</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Gui.so.5.9.7</obj> <fn>QGuiApplicationPrivate::processExposeEvent(QWindowSystemInterfacePrivate::ExposeEvent*)</fn> </frame> <frame> <ip>0x6046B4C</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Gui.so.5.9.7</obj> <fn>QGuiApplicationPrivate::processWindowSystemEvent(QWindowSystemInterfacePrivate::WindowSystemEvent*)</fn> </frame> <frame> <ip>0x60215BA</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Gui.so.5.9.7</obj> <fn>QWindowSystemInterface::sendWindowSystemEvents(QFlags<QEventLoop::ProcessEventsFlag>)</fn> </frame> <frame> <ip>0xF95CF1F</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5XcbQpa.so.5.9.7</obj> </frame> <frame> <ip>0xBD92196</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> <fn>g_main_context_dispatch</fn> </frame> <frame> <ip>0xBD923EF</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> </frame> <frame> <ip>0xBD9249B</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> <fn>g_main_context_iteration</fn> </frame> <frame> <ip>0x7D7D7DE</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> <fn>QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>)</fn> </frame> <frame> <ip>0x7D294C9</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> <fn>QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>)</fn> </frame> <frame> <ip>0x7D31DA3</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> <fn>QCoreApplication::exec()</fn> </frame> <frame> <ip>0x54F7A2</ip> <obj>/home/amet/git/xm-controller/build-xmedit-Desktop_Qt_5_9_7_GCC_64bit-Debug/xmedit</obj> <fn>main</fn> <dir>/home/amet/git/xm-controller/build-xmedit-Desktop_Qt_5_9_7_GCC_64bit-Debug/../xmedit</dir> <file>main.cpp</file> <line>68</line> </frame> </stack> <stack> <frame> <ip>0x8DA974D</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <dir>/build/glibc-Cl5G7W/glibc-2.23/io/../sysdeps/unix</dir> <file>syscall-template.S</file> <line>84</line> </frame> <frame> <ip>0xDC41C61</ip> <obj>/usr/lib/x86_64-linux-gnu/libxcb.so.1.1.0</obj> </frame> <frame> <ip>0xDC438D6</ip> <obj>/usr/lib/x86_64-linux-gnu/libxcb.so.1.1.0</obj> <fn>xcb_wait_for_event</fn> </frame> <frame> <ip>0xF8E99F8</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5XcbQpa.so.5.9.7</obj> </frame> <frame> <ip>0x7B5B979</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> </frame> <frame> <ip>0x81F76B9</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>start_thread</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl</dir> <file>pthread_create.c</file> <line>333</line> </frame> <frame> <ip>0x8DB541C</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <fn>clone</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64</dir> <file>clone.S</file> <line>109</line> </frame> </stack> <stack> <frame> <ip>0x8DA974D</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <dir>/build/glibc-Cl5G7W/glibc-2.23/io/../sysdeps/unix</dir> <file>syscall-template.S</file> <line>84</line> </frame> <frame> <ip>0xBD9238B</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> </frame> <frame> <ip>0xBD9249B</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> <fn>g_main_context_iteration</fn> </frame> <frame> <ip>0x1891828C</ip> <obj>/usr/lib/x86_64-linux-gnu/gio/modules/libdconfsettings.so</obj> </frame> <frame> <ip>0xBDB8C54</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> </frame> <frame> <ip>0x81F76B9</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>start_thread</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl</dir> <file>pthread_create.c</file> <line>333</line> </frame> <frame> <ip>0x8DB541C</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <fn>clone</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64</dir> <file>clone.S</file> <line>109</line> </frame> </stack> <stack> <frame> <ip>0x8DA974D</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <dir>/build/glibc-Cl5G7W/glibc-2.23/io/../sysdeps/unix</dir> <file>syscall-template.S</file> <line>84</line> </frame> <frame> <ip>0xBD9238B</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> </frame> <frame> <ip>0xBD9249B</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> <fn>g_main_context_iteration</fn> </frame> <frame> <ip>0xBD924D8</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> </frame> <frame> <ip>0xBDB8C54</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> </frame> <frame> <ip>0x81F76B9</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>start_thread</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl</dir> <file>pthread_create.c</file> <line>333</line> </frame> <frame> <ip>0x8DB541C</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <fn>clone</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64</dir> <file>clone.S</file> <line>109</line> </frame> </stack> <stack> <frame> <ip>0x8DA974D</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <dir>/build/glibc-Cl5G7W/glibc-2.23/io/../sysdeps/unix</dir> <file>syscall-template.S</file> <line>84</line> </frame> <frame> <ip>0xBD9238B</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> </frame> <frame> <ip>0xBD92711</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> <fn>g_main_loop_run</fn> </frame> <frame> <ip>0x132BF9D5</ip> <obj>/usr/lib/x86_64-linux-gnu/libgio-2.0.so.0.4800.2</obj> </frame> <frame> <ip>0xBDB8C54</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> </frame> <frame> <ip>0x81F76B9</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>start_thread</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl</dir> <file>pthread_create.c</file> <line>333</line> </frame> <frame> <ip>0x8DB541C</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <fn>clone</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64</dir> <file>clone.S</file> <line>109</line> </frame> </stack> <stack> <frame> <ip>0x8DA974D</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <dir>/build/glibc-Cl5G7W/glibc-2.23/io/../sysdeps/unix</dir> <file>syscall-template.S</file> <line>84</line> </frame> <frame> <ip>0xBD9238B</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> </frame> <frame> <ip>0xBD9249B</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> <fn>g_main_context_iteration</fn> </frame> <frame> <ip>0x7D7D7FA</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> <fn>QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>)</fn> </frame> <frame> <ip>0x7D294C9</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> <fn>QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>)</fn> </frame> <frame> <ip>0x7B56DBB</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> <fn>QThread::exec()</fn> </frame> <frame> <ip>0x10106664</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5DBus.so.5.9.7</obj> </frame> <frame> <ip>0x7B5B979</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> </frame> <frame> <ip>0x81F76B9</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>start_thread</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl</dir> <file>pthread_create.c</file> <line>333</line> </frame> <frame> <ip>0x8DB541C</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <fn>clone</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64</dir> <file>clone.S</file> <line>109</line> </frame> </stack> <stack> <frame> <ip>0x8DA974D</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <dir>/build/glibc-Cl5G7W/glibc-2.23/io/../sysdeps/unix</dir> <file>syscall-template.S</file> <line>84</line> </frame> <frame> <ip>0xBD9238B</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> </frame> <frame> <ip>0xBD9249B</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> <fn>g_main_context_iteration</fn> </frame> <frame> <ip>0x7D7D7DE</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> <fn>QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>)</fn> </frame> <frame> <ip>0x7D294C9</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> <fn>QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>)</fn> </frame> <frame> <ip>0x7B56DBB</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> <fn>QThread::exec()</fn> </frame> <frame> <ip>0x6999904</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Qml.so.5.9.7</obj> </frame> <frame> <ip>0x7B5B979</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> </frame> <frame> <ip>0x81F76B9</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>start_thread</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl</dir> <file>pthread_create.c</file> <line>333</line> </frame> <frame> <ip>0x8DB541C</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <fn>clone</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64</dir> <file>clone.S</file> <line>109</line> </frame> </stack> <stack> <frame> <ip>0x8DA974D</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <dir>/build/glibc-Cl5G7W/glibc-2.23/io/../sysdeps/unix</dir> <file>syscall-template.S</file> <line>84</line> </frame> <frame> <ip>0xBD9238B</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> </frame> <frame> <ip>0xBD9249B</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> <fn>g_main_context_iteration</fn> </frame> <frame> <ip>0x7D7D7DE</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> <fn>QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>)</fn> </frame> <frame> <ip>0x7D294C9</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> <fn>QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>)</fn> </frame> <frame> <ip>0x7B56DBB</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> <fn>QThread::exec()</fn> </frame> <frame> <ip>0x7B5B979</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> </frame> <frame> <ip>0x81F76B9</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>start_thread</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl</dir> <file>pthread_create.c</file> <line>333</line> </frame> <frame> <ip>0x8DB541C</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <fn>clone</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64</dir> <file>clone.S</file> <line>109</line> </frame> </stack> <stack> <frame> <ip>0x305434B9</ip> </frame> <frame> <ip>0x22A1C4BC</ip> <obj>/usr/lib/x86_64-linux-gnu/dri/swrast_dri.so</obj> </frame> <frame> <ip>0x22A1CF2F</ip> <obj>/usr/lib/x86_64-linux-gnu/dri/swrast_dri.so</obj> </frame> <frame> <ip>0x22A1BB38</ip> <obj>/usr/lib/x86_64-linux-gnu/dri/swrast_dri.so</obj> </frame> <frame> <ip>0x22A1C213</ip> <obj>/usr/lib/x86_64-linux-gnu/dri/swrast_dri.so</obj> </frame> <frame> <ip>0x22A1C036</ip> <obj>/usr/lib/x86_64-linux-gnu/dri/swrast_dri.so</obj> </frame> ==2208== at 0x38083F48: ??? (in /usr/lib/valgrind/memcheck-amd64-linux) ==2208== by 0x38084064: ??? (in /usr/lib/valgrind/memcheck-amd64-linux) ==2208== by 0x380842A1: ??? (in /usr/lib/valgrind/memcheck-amd64-linux) ==2208== by 0x380842CA: ??? (in /usr/lib/valgrind/memcheck-amd64-linux) ==2208== by 0x3809F682: ??? (in /usr/lib/valgrind/memcheck-amd64-linux) ==2208== by 0x38145428: ??? (in /usr/lib/valgrind/memcheck-amd64-linux) ==2208== by 0x38145494: ??? (in /usr/lib/valgrind/memcheck-amd64-linux) ==2208== by 0x3816A8A7: ??? (in /usr/lib/valgrind/memcheck-amd64-linux) ==2208== by 0x3814342F: ??? (in /usr/lib/valgrind/memcheck-amd64-linux) ==2208== by 0x380A1C0B: ??? (in /usr/lib/valgrind/memcheck-amd64-linux) ==2208== by 0x380D296B: ??? (in /usr/lib/valgrind/memcheck-amd64-linux) ==2208== by 0x380D45CF: ??? (in /usr/lib/valgrind/memcheck-amd64-linux) ==2208== by 0x380E3946: ??? (in /usr/lib/valgrind/memcheck-amd64-linux) ==2208== by 0x380E3E1A: ??? (in /usr/lib/valgrind/memcheck-amd64-linux) ==2208== by 0x3810C62D: ??? (in /usr/lib/valgrind/memcheck-amd64-linux) ==2208== by 0xDEADBEEFDEADBEEE: ??? ==2208== by 0xDEADBEEFDEADBEEE: ??? ==2208== by 0xDEADBEEFDEADBEEE: ??? sched status: running_tid=9 Thread 1: status = VgTs_WaitSys (lwpid 2208) Thread 2: status = VgTs_WaitSys (lwpid 2229) Thread 3: status = VgTs_WaitSys (lwpid 2236) Thread 4: status = VgTs_WaitSys (lwpid 2238) <frame> <ip>0x81F76B9</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>start_thread</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl</dir> <file>pthread_create.c</file> <line>333</line> </frame> <frame> <ip>0x8DB541C</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <fn>clone</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64</dir> <file>clone.S</file> <line>109</line> </frame> </stack> <stack> <frame> <ip>0x81FE154</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>futex_wait</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl/../sysdeps/unix/sysv/linux</dir> <file>futex-internal.h</file> <line>61</line> </frame> <frame> <ip>0x81FE154</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>futex_wait_simple</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl/../sysdeps/nptl</dir> <file>futex-internal.h</file> <line>135</line> </frame> <frame> <ip>0x81FE154</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>pthread_barrier_wait</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl</dir> <file>pthread_barrier_wait.c</file> <line>184</line> </frame> <frame> <ip>0x22A1C207</ip> <obj>/usr/lib/x86_64-linux-gnu/dri/swrast_dri.so</obj> </frame> <frame> <ip>0x22A1C036</ip> <obj>/usr/lib/x86_64-linux-gnu/dri/swrast_dri.so</obj> </frame> <frame> <ip>0x81F76B9</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>start_thread</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl</dir> <file>pthread_create.c</file> <line>333</line> </frame> <frame> <ip>0x8DB541C</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <fn>clone</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64</dir> <file>clone.S</file> <line>109</line> </frame> </stack> <stack> <frame> <ip>0x81FE154</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>futex_wait</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl/../sysdeps/unix/sysv/linux</dir> <file>futex-internal.h</file> <line>61</line> </frame> <frame> <ip>0x81FE154</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>futex_wait_simple</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl/../sysdeps/nptl</dir> <file>futex-internal.h</file> <line>135</line> </frame> <frame> <ip>0x81FE154</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>pthread_barrier_wait</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl</dir> <file>pthread_barrier_wait.c</file> <line>184</line> </frame> <frame> <ip>0x22A1C207</ip> <obj>/usr/lib/x86_64-linux-gnu/dri/swrast_dri.so</obj> </frame> <frame> <ip>0x22A1C036</ip> <obj>/usr/lib/x86_64-linux-gnu/dri/swrast_dri.so</obj> </frame> <frame> <ip>0x81F76B9</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>start_thread</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl</dir> <file>pthread_create.c</file> <line>333</line> </frame> <frame> <ip>0x8DB541C</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <fn>clone</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64</dir> <file>clone.S</file> <line>109</line> </frame> </stack> <stack> <frame> <ip>0x81FE154</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>futex_wait</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl/../sysdeps/unix/sysv/linux</dir> <file>futex-internal.h</file> <line>61</line> </frame> <frame> <ip>0x81FE154</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>futex_wait_simple</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl/../sysdeps/nptl</dir> <file>futex-internal.h</file> <line>135</line> </frame> <frame> <ip>0x81FE154</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>pthread_barrier_wait</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl</dir> <file>pthread_barrier_wait.c</file> <line>184</line> </frame> <frame> <ip>0x22A1C207</ip> <obj>/usr/lib/x86_64-linux-gnu/dri/swrast_dri.so</obj> </frame> <frame> <ip>0x22A1C036</ip> <obj>/usr/lib/x86_64-linux-gnu/dri/swrast_dri.so</obj> </frame> <frame> <ip>0x81F76B9</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>start_thread</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl</dir> <file>pthread_create.c</file> <line>333</line> </frame> <frame> <ip>0x8DB541C</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <fn>clone</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64</dir> <file>clone.S</file> <line>109</line> </frame> </stack> <stack> <frame> <ip>0x8DA974D</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <dir>/build/glibc-Cl5G7W/glibc-2.23/io/../sysdeps/unix</dir> <file>syscall-template.S</file> <line>84</line> </frame> <frame> <ip>0xBD9238B</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> </frame> <frame> <ip>0xBD9249B</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> <fn>g_main_context_iteration</fn> </frame> <frame> <ip>0x7D7D7DE</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> <fn>QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>)</fn> </frame> <frame> <ip>0x7D294C9</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> <fn>QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>)</fn> </frame> <frame> <ip>0x7B56DBB</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> <fn>QThread::exec()</fn> </frame> <frame> <ip>0x7B5B979</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> </frame> <frame> <ip>0x81F76B9</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>start_thread</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl</dir> <file>pthread_create.c</file> <line>333</line> </frame> <frame> <ip>0x8DB541C</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <fn>clone</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64</dir> <file>clone.S</file> <line>109</line> </frame> </stack> <stack> <frame> <ip>0x8DA974D</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <dir>/build/glibc-Cl5G7W/glibc-2.23/io/../sysdeps/unix</dir> <file>syscall-template.S</file> <line>84</line> </frame> <frame> <ip>0xBD9238B</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> </frame> <frame> <ip>0xBD9249B</ip> <obj>/lib/x86_64-linux-gnu/libglib-2.0.so.0.4800.2</obj> <fn>g_main_context_iteration</fn> </frame> <frame> <ip>0x7D7D7DE</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> <fn>QEventDispatcherGlib::processEvents(QFlags<QEventLoop::ProcessEventsFlag>)</fn> </frame> <frame> <ip>0x7D294C9</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> <fn>QEventLoop::exec(QFlags<QEventLoop::ProcessEventsFlag>)</fn> </frame> <frame> <ip>0x7B56DBB</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> <fn>QThread::exec()</fn> </frame> <frame> <ip>0x7B5B979</ip> <obj>/home/amet/apps/Qt/5.9.7/gcc_64/lib/libQt5Core.so.5.9.7</obj> </frame> <frame> <ip>0x81F76B9</ip> <obj>/lib/x86_64-linux-gnu/libpthread-2.23.so</obj> <fn>start_thread</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/nptl</dir> <file>pthread_create.c</file> <line>333</line> </frame> <frame> <ip>0x8DB541C</ip> <obj>/lib/x86_64-linux-gnu/libc-2.23.so</obj> <fn>clone</fn> <dir>/build/glibc-Cl5G7W/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64</dir> <file>clone.S</file> <line>109</line> </frame> </stack> Thread 5: status = VgTs_WaitSys (lwpid 2239) Thread 6: status = VgTs_WaitSys (lwpid 2255) Thread 7: status = VgTs_WaitSys (lwpid 2264) Thread 8: status = VgTs_WaitSys (lwpid 2265) Thread 9: status = VgTs_Runnable (lwpid 2268) Thread 10: status = VgTs_WaitSys (lwpid 2269) Thread 11: status = VgTs_WaitSys (lwpid 2270) Thread 12: status = VgTs_WaitSys (lwpid 2271) Thread 14: status = VgTs_WaitSys (lwpid 2317) Thread 15: status = VgTs_WaitSys (lwpid 2319) Note: see also the FAQ in the source distribution. It contains workarounds to several common problems. In particular, if Valgrind aborted or crashed after identifying problems in your program, there's a good chance that fixing those problems will prevent Valgrind aborting or crashing, especially if it happened in m_mallocfree.c. If that doesn't help, please report this bug to: www.valgrind.org In the bug report, send all the above text, the valgrind version, and what OS and version you are using. Thanks. 11:16:08: Analyzing finished.I found this info on valgrind and it says that temporary storage exhausted might be related to a valgrind setting? or a jit setting? I know QML uses JIT. Is that what is causing valgrind to burp?
-
@fcarney you could also play with the various sanitize option of modern compilers, like address sanitizer. static code analisys is also possible.
if that does not help, you'll have to modify code (exclude parts until it runs stable).
-
When compiling the code using 5.9.7 under Linux all the stability and crash problems went away from the application. I could not get the app to repeat the problems experienced with the Windows compile. I ran some analyzers and did not find any memory issues that could cause a crash.
So, I decided to try Qt 5.12 as some of our other projects are moving to 5.12 because it is the new LTS version. After compiling with Qt 5.12 with the mingw 7.3 64 bit compiler I ran the application in Windows. I can no longer duplicate the crashing issue. The program now behaves the way it does in Linux. No weird issues, no crashing.
When I was able to capture a crash when running the 5.9.7 version I would get crashes deep into the system calls of the OS or somewhere in the Qt code. I could never trace it back to any errors in my own code. Honestly I am not sure what to think.
We are now using 5.12 as our default version of Qt so I will continue developing and testing with that version.
Thanks for everyone's help. I learned how to use a few more analyzer tools. I really like the sanitizers as they are almost free to use. I plan on keeping them enabled during development.
