System files get corrupted when compiling with QtCreator+clang on Linux

Nianoniaa

Hi there,

It's the third time now that my system gets corrupted while compiling a C++ program on ArchLinux using QtCreator and clang. Otherwise my system runs fine. I use many demanding programs without any issues (to do video and image editing, 3D rendering, etc). I installed ArchLinux 3 years ago on a new laptop (quad i7 with Intel graphics). I've been using QtCreator for 2 years without issues. These corruptions started happening about a month ago.

I'm using clang 5.0.1, QtCreator 4.5.0, all system packages up to date. My project uses OpenFrameworks (a FOSS C++ framework). In some cases while compiling the free RAM goes to 0. I can see this in my status bar with a simple script that runs every second: mem=$( awk '/MemFree/ {mem=$2}; END { printf("%.1f Gib", mem/1000000)}' /proc/meminfo ) I know it's not precise, but it lets me see the RAM go down when compiling. If things go well (normally) it goes down to about 0.4 Gib, the compiled program launches, and RAM is freed. If things do not go well, it goes down to 0.1, the system becomes very slow and then the troubles start.

I suspect that when the memory goes so low somehow random data gets written on the wrong places in my SSD. I know this because programs stop working after such an event and I see corrupted text files full of binary data. I have no idea how this happens.

To fix the corrupted files I have written a bash script to detect binary data in files that are supposed to be text files, basically commands in this fashion: find ./ -iname "*.inl" -o -iname "*.hpp" -o -iname "*.cpp" -o -iname "*.h" | xargs -I '%' sh -c 'file "%"' | grep -v text I do the same with other extensions, like "*.conf", "*.py" etc. Once I detect corrupted files I find out which package owns those broken files with pacman -Qo /the/broken/file and finally reinstall the package. Thanks to this I've been able to repair my system now 3 times.

My question is, has anyone run into similar problems when using QtCreator?

ps. I've had the same issue with two SSD drives, one actually new from the shop. I've run memtest on my 2x8 Gib RAM, no errors.

SGaist

Hi and welcome to devnet,

I don't see any reason that it would be Qt Creator. You can check that by starting to build in the shadow build folder from the command line.

How much RAM do you have at hand ?
What build parameters are you using to eat all your RAM like that ?
Do you have swap activated ?
Are you doing anything else on your machine when that happens ?

aha_1980

Hi @Nianoniaa,

just for clarification: With system files you mean the headers below /usr/include ?

These files are usually owned by root and normal users can't write them. Are you compiling as root?

I've not seen such issues so far, but I don't use clang (and I'm compiling Qt and Qt Creator itself).

For me that rather sound like a problem with your SSD, maybe it goes to end of life... you should check the SMART data of your SSDs.

Edit: now I see you have the same problem with a new one... anyway, check the SMART data.

Nianoniaa

@SGaist Thank you :)

A difficulty in debugging this is that it does not happen every time. It happened 3 times in one month. I can switch to compiling from the command line, and see how it goes.

I have 16 Gib RAM, 2 Gib swap. I think OpenFrameworks is quite big, with lots of templates that take up RAM.
I was not doing other things when it happened. Maybe it was playing music in the background with cmus.

@aha_1980 With system files I mean files owned by root in various folders like /usr/include and others. Yesterday X would not start because it got corrupted. I'm not compiling as root. The SSD works fine and SMART looks ok too. I use the computer all day and this only happens when building this project, not editing video or other heavy disk+CPU tasks.

I wonder what happens when the swap gets full. I'm going to read about that. I just reduced the building from 8 to 4 threads and I'm going to increase the swap size to see if that helps.

jsulm

@Nianoniaa If those files are owned by root and the user you use does not have write access to these files, then how could QtCreator/compiler/... corrupt these files?
I don't think your problem is related to compiling, it looks like a problem with your machine.

SGaist

See this interesting article about choosing the swap size.

JonB

@Nianoniaa
As @jsulm says, files owned by root cannot be written to by a non-root user process like Qt Creator or clang or whatever.

If you really want to verify that these files are not being written to, why don't you use strace against whatever process you run and grep the (large) output to see where files are opened?

Nianoniaa

I didn't want to imply that clang or QtCreator would attempt to write random data on random root files on my system (nor that it could), only that it happens sometimes when compiling with QtCreator and in no other circumstances. But I see that it's probably the wrong place to ask about this issue, since it may not be QtCreator or clang specific. I was only hoping to find someone who has heard of something like this before.

@JonB Thanks for the tip. I could try to provoke the issue on purpose and strace it, but I really want to avoid this happening again. I'm afraid I break it beyond repair and it no longer boots. I have only detected corrupted text files so far. I have no idea if there are lots of broken binary files on my computer but it seems to behave normal except when this crash happens...

JonB

@Nianoniaa
I doubt the files' content is getting zapped by a "legitimate" write to the file, more that something nasty is causing the underlying disk area to get corrupted.

In itself I can't see that Qt would be the cause. However you probably do a lot more reading/writing of files etc. when you compiling or clanging than in other situations, so the problem may be more likely to show up then.