5.12.11 binaries ignore RPATH and LD_LIBRARY_PATH settings

qa_test

I'm trying to build 5.12.11 LTS source on Ubuntu 18.04 (in a VM) such that the result of the make install should be the installation lives under an NFS mount point that does not correspond to any of the standard linker search paths. Or binaries in this NFS mounted 5.12.11 distribution should never use Qt libraries from a local machine under /usr/lib. Prior to building, I removed the Qt packages on the build VM such that there are no Qt libs under /usr/lib and I locally created the same path for the install prefix that will subsequently be duplicated with an NFS mount. The result of the source build and install is the RPATH of $ORIGIN/../lib in a binary like <install prefix>/bin/uic does in fact appear to work in the build VM (no NFS mount). Or at least I see the ../ notation in ldd output referencing Qt libraries from the ../lib subdir parallel to the bin subdir. But when I subsequently mount the distribution over NFS on another machine running Ubuntu 18.04, neither RPATH or LD_LIBRARY_PATH work to find the libraries. I've read some threads about the "not found" output from ldd for Qt libs, but it seems that setting LD_LIBRARY_PATH fxed that - which is not the case here. Do I need to edit a LIBDIR setting in a Qt configuration file before compiling? Appending the absolute NFS path to lib subdir to RPATH also did not work.

jeremy_k

Is the .so reported as a valid library on the second system using ldd?
ld.so has several options that might help. Setting the environment LD_DEBUG to "all" should produce lots of output

Also: Please post text as text, not as screenshots.

qa_test

The example below shows the same binary built in a container with our build of Qt 5.11 and then QT 5.12.11. The link to libQt5core gets resolved via RPATH with the 5.11 build, but always finds the system copy instead of the copy pointed to by RPATH when linked against the 5.12.11 build. The output from LD_DEBUG is also shown where the link map gets created. I'm not sure what else I can try to defeat finding the system copy of libQt5core when using the 5.12.11 build. I note that of all the Qt libs, only libQt5core itself has no RPATH entry compared to the other QT libs, though I don't see how this would affect the binary RPATH referencing libQt5core.

$ cat /etc/os-release | head -6
NAME="Ubuntu"
VERSION="18.04.6 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.6 LTS"
VERSION_ID="18.04"

==================================================================
PATH TO libQt5Core CORRECT WHEN myviewer BUILT WITH QT 5.11 BUILD
==================================================================

$ pwd
/usr/local/myviewer_beta/bin
$ ldd myviewer | grep -i libqt
	libQt5Widgets.so.5 => /usr/local/myviewer_beta/bin/./../lib/qt/lib/libQt5Widgets.so.5 (0x00007f51eb080000)
	libQt5X11Extras.so.5 => /usr/local/myviewer_beta/bin/./../lib/qt/lib/libQt5X11Extras.so.5 (0x00007f51ec4c9000)
	libQt5Gui.so.5 => /usr/local/myviewer_beta/bin/./../lib/qt/lib/libQt5Gui.so.5 (0x00007f51e43e6000)
	libQt5Core.so.5 => /usr/local/myviewer_beta/bin/./../lib/qt/lib/libQt5Core.so.5 (0x00007f51e3e06000)  <--- CORRECT COPY OF libQt5Core.so.5

$ patchelf --print-rpath myviewer
$ORIGIN/../lib/vtk:$ORIGIN/../lib/qt/lib:
$ chrpath -l myviewer
myviewer: RUNPATH=$ORIGIN/../lib/vtk:$ORIGIN/../lib/qt/lib:

$ export LD_DEBUG=all
$ ldd myviewer > ld_qt_5.11.out 2>&1
--- identify where link map is created ---
     16044:	file=libQt5Core.so.5 [0];  needed by ./myviewer [0]
     16044:	find library=libQt5Core.so.5 [0]; searching
     16044:	 search path=/usr/local/myviewer_beta/bin/./../lib/vtk:/usr/local/myviewer_beta/bin/./../lib/qt/lib:tls/x86_64/x86_64:tls/x86_64:tls/x86_64:tls:x86_64/x86_64:x86_64:x86_64:(RUNPATH from file ./myviewer)
     16044:	  trying file=/usr/local/myviewer_beta/bin/./../lib/vtk/libQt5Core.so.5
     16044:	  trying file=/usr/local/myviewer_beta/bin/./../lib/qt/lib/libQt5Core.so.5  <--- CORRECT COPY OF libQt5Core.so.5
     16044:	
     16044:	file=libQt5Core.so.5 [0];  generating link map
     16044:	  dynamic: 0x00007fa694060000  base: 0x00007fa693a95000   size: 0x00000000005df678
     16044:	    entry: 0x00007fa693b30850  phdr: 0x00007fa693a95040  phnum:                  9

$ cd ../lib/qt/lib
$ ls -l libQt5Core.so.5
lrwxrwxrwx 1 qatest qatest 20 Apr 19 23:13 libQt5Core.so.5 -> libQt5Core.so.5.11.3
$ patchelf --print-rpath libQt5Core.so.5

$ chrpath -l libQt5Core.so.5
libQt5Core.so.5: no rpath or runpath tag found.

=======================================================================
PATH TO libQt5Core INCORRECT WHEN myviewer BUILT WITH QT 5.12.11 BUILD
=======================================================================

$ pwd
/usr/local/myviewer_beta/bin
$ ldd myviewer | grep -i libqt
	libQt5Widgets.so.5 => /usr/local/myviewer_beta/bin/./../lib/qt/lib/libQt5Widgets.so.5 (0x00007f8ac8d76000)
	libQt5X11Extras.so.5 => /usr/local/myviewer_beta/bin/./../lib/qt/lib/libQt5X11Extras.so.5 (0x00007f8ac8b71000)
	libQt5Gui.so.5 => /usr/local/myviewer_beta/bin/./../lib/qt/lib/libQt5Gui.so.5 (0x00007f8ac1cbf000)
	libQt5Core.so.5 => /usr/lib/x86_64-linux-gnu/libQt5Core.so.5 (0x00007f8ac1574000)  <--- WRONG COPY OF libQt5Core.so.5

$ patchelf --print-rpath myviewer
$ORIGIN/../lib/vtk:$ORIGIN/../lib/qt/lib:
$ chrpath -l myviewer
myviewer: RUNPATH=$ORIGIN/../lib/vtk:$ORIGIN/../lib/qt/lib:

$ export LD_DEBUG=all
$ ldd myviewer > ld_qt_5.12.11.out 2>&1
--- identify where link map is created ---
     16144:     file=libQt5Core.so.5 [0];  needed by ./myviewer [0]
     16144:     find library=libQt5Core.so.5 [0]; searching
     16144:      search path=/usr/local/myviewer_beta/bin/./../lib/vtk:/usr/local/myviewer_beta/bin/./../lib/qt/lib:tls/x86_64/x86_64:tls/x86_64:tls/x86_64:tls:x86_64/x86_64:x86_64:x86_64:(RUNPATH from file ./myviewer)
     16144:       trying file=/usr/local/myviewer_beta/bin/./../lib/vtk/libQt5Core.so.5
     16144:       trying file=/usr/local/myviewer_beta/bin/./../lib/qt/lib/libQt5Core.so.5 <--- CORRECT COPY OF libQt5Core.so.5
     16144:       trying file=tls/x86_64/x86_64/libQt5Core.so.5
     16144:       trying file=tls/x86_64/libQt5Core.so.5
     16144:       trying file=tls/x86_64/libQt5Core.so.5
     16144:       trying file=tls/libQt5Core.so.5
     16144:       trying file=x86_64/x86_64/libQt5Core.so.5
     16144:       trying file=x86_64/libQt5Core.so.5
     16144:       trying file=x86_64/libQt5Core.so.5
     16144:       trying file=libQt5Core.so.5
     16144:      search cache=/etc/ld.so.cache
     16144:       trying file=/usr/lib/x86_64-linux-gnu/libQt5Core.so.5  <--- WRONG COPY OF libQt5Core.so.5
     16144:
     16144:     file=libQt5Core.so.5 [0];  generating link map
     16144:       dynamic: 0x00007ff85234f400  base: 0x00007ff851c0a000   size: 0x000000000074a728
     16144:         entry: 0x00007ff851c9dd20  phdr: 0x00007ff851c0a040  phnum:                 10

$ cd ../lib/qt/lib
$ ls -l libQt5Core.so.5
lrwxrwxrwx 1 qatest qatest 21 Apr 20 01:21 libQt5Core.so.5 -> libQt5Core.so.5.12.11
$ patchelf --print-rpath libQt5Core.so.5

$ chrpath -l libQt5Core.so.5
libQt5Core.so.5: no rpath or runpath tag found.

--- try override with LD_LIBRARY_PATH ---

$ export LD_LIBRARY_PATH=/usr/local/myviewer_beta/lib/qt/lib
$ ls -l $LD_LIBRARY_PATH/libQt5Core.so.5
lrwxrwxrwx 1 qatest qatest 21 Apr 20 01:21 /usr/local/myviewer_beta/lib/qt/lib/libQt5Core.so.5 -> libQt5Core.so.5.12.11
$ ldd myviewer | grep -i libqt
	libQt5Widgets.so.5 => /usr/local/myviewer_beta/lib/qt/lib/libQt5Widgets.so.5 (0x00007f15a5f18000)
	libQt5X11Extras.so.5 => /usr/local/myviewer_beta/lib/qt/lib/libQt5X11Extras.so.5 (0x00007f15a5d13000)
	libQt5Gui.so.5 => /usr/local/myviewer_beta/lib/qt/lib/libQt5Gui.so.5 (0x00007f159ee61000)
	libQt5Core.so.5 => /usr/lib/x86_64-linux-gnu/libQt5Core.so.5 (0x00007f159e716000) <--- WRONG COPY OF libQt5Core.so.5

qa_test

This post is deleted!

hskoglund

Hi, interesting bug! For some reason ldd finds but then abandons your copy of libQt5Core.so.5. for 5.12.11 but then selects the system one anyway :-(

...
     16144:       trying file=/usr/local/myviewer_beta/bin/./../lib/qt/lib/libQt5Core.so.5 <--- CORRECT COPY OF libQt5Core.so.5
     16144:       trying file=tls/x86_64/x86_64/libQt5Core.so.5
...

Just guessing but what happens if you swap them?
i.e. make a backup of your VM first, then replace your version of libQt5Core.so.5 with the one from /usr/lib/x86_64-linux-gnu/ and the other way around. Does ldd still prefer the .so from /usr/lib/x86_64-linux-gnu?

qa_test

Thank you @hskoglund for that suggestion. When I swap the locations of the system libQTlibraries and my Qt build, ldd shows the system libQT5Core.so is found through RPATH from the location where my Qt build lived - which means RPATH is working. That got me thinking and starting to search on when does the linker ignore a library, as in there must be something about the build of my libQtcore that causing that (despite RPATH, LD_LIBRARY_PATH, settings etc...)

I eventually came across this posting where someone had the same issue with libQt5Core in WSL (Windows Subsystem for Linux). This post shows there is apparently an ABI tag compiled into libQt5Core that will not allow linking to occur if the kernel version is "wrong",
https://github.com/Microsoft/WSL/issues/3023

In the context of that post, the ABI note cannot work in WSL because of how the kernel version is reported. Further along in that post, someone notes, ... If you were to compile Qt5 yourself on Ubuntu, you'd still run into this problem, albeit not in the statx(2) case, since it took until glibc 2.28 for that to be implemented in the C library. On the upcoming Ubuntu 19.04, this is going to be the case, though ... In my case I compiled Qt 5.12.11 LTS on Ubuntu 18.04. I did the same build on CentOS7 and CentOS8 and did not have this linking issue.

There is link in that post to this bug report,
https://sourceware.org/bugzilla/show_bug.cgi?id=23072

The workaround is to strip out the tag with,
$ strip --remove-section=.note.ABI-tag libQt5Core.so.5.12.11

Then RPATH works as expected with my Qt 5.12.11 build back in it's normal non-system path location.

hskoglund

Thanks for posting about your research!
It makes sense that it was the "ELF stripping" command that was the magic recipe (you did everything else correct). Now I remember another forum post discussing problems when starting Qt Creator on WSL where I found the same solution via the same link you posted :-)