"libQt6Core.so.6: cannot open shared object file" even though it exists, seems to depend on Docker host OS

[Stefan Scherfke]

[Disclaimer: This is a duplicate of https://unix.stackexchange.com/questions/700290/existing-so-file-cannot-be-loaded-even-though-it-exists-seems-to-depend-on-doc]

I have built Qt6 in an Alma8 based Docker container, with the Docker host being Fedora 35.

Under some circumstances (described below), all Qt libs cannot load libQt6Core.so[.6[.2.4]]. But that file exists and the correct directory is searched for that file. Other Qt libs (e.g., libQt6Dbus.so) are found and loaded.

Extensive debugging, re-building, seaching-the-web did not yield any clues what the underlying cause is and how I could fix it.

Locating the problem

I have narrowed down the problem to the following scenario:

• I created two minimal VMs, one with centos7 and one with alma8.
• I installed Docker from the official repos into both of them.
• I ran the same Docker image in both VMs and installed the same qt6 package.
• It breaks when the Docker host is centos7.
• It works when the Docker host is alma8.

Theory and Question

• Qt6 was built on Alma8 and links to some system libraries newer than what Centos7 provides, so Qt6 cannot run unter Centos7 (this is totally expected and okay). But it should run anywhere in the Alma8 Docker container.
• Container images should be able to run anywhere, but in this case "something" from the host OS sneaks into the container and causes the Issue – even though both containers use the exact same image!

The question is: What is this "something" and how/why does it break the build?

What I tried

I inspected libQt6Gui.so to see whether or not it can load libQt6Core.so and I inspected libQt6Core.so to see if something looks bogus using:

• ldd and LD_DEBUG=libs ldd which indeed showed some differences (see below)
• libtree which showed no differences (but a nice tree :))
• pyldd (from conda-build)
• readelf -d

What I also tried:

• Setting LD_LIBRARY_PATH (did not change anything – no surprise since I know that the correct path is always searched)
• Building Qt6 in an alma8 container with a centos7 host (build failed with "libQt6Core.so.6: cannot open file", same error as with the built lib)
• Building Qt6 in a centos7 container (build failed due to other problems I could not yet fix)

Differences from ldd

In the screenshots below, you see a the Alma8-Docker-Container on a Centos7 host on the left and the Alma8-Docker-Container on an Alma8 host on the *right.

The first two images show the results for ldd /opt/emsconda/lib/libQt6Gui.so. libQt6Core can not be found on the left but is found on the right.

This second screenshot shows that other Qt libs are found and loaded. The ICU libs are also missing on the left - maybe they are only loaded when libQt6Core was also loaded?

This screenshot shows the results of LD_DEBUG=libs ldd .... You can see that in both cases, libQt6Core is search in the correct location (/opt/emsconda/lib). But it is only loaded in the right container. The left one additionally looks in `/opt/emsconda/lib/./ (haha)) and then silently walks on to the next lib ...

I could not find any error messages. This file is just not opened/loaded.

Inspecting the libQt6Core.so itself might give us a clue. It links to a linux-vdso.so.1.

According to this SO question, that file is a virtual lib injected into the userspace by the OS kernel.

Since Docker containers do not run their own kernel, I suspect that that file comes from the host OS. Maybe, libQt6Core relies on some functionality in the linux-vdso.so.1 that the centos7 kernel cannot provide? I have no idea ...

---

Since nothing I tried so far yields an error message, I have no clue what the acutal problem might be or how to proceeded with debugging. I'd be greatful for any kind of hints, tips or help.

[hskoglund]

Hi, this is just a wild guess, but when I read "... and then silently walks away..." I think of a section in the .elf file called ABI-tag, when it's present it causes exactly that "silently walk away-syndrome". Most Qt users on WSL stumbles on this: post here
Recent forums posts: here and here

If this happens to be your problem also, then the fix is to strip that section, i.e.
strip --remove-section=.note.ABI-tag libQt6Core.so.6

[SGaist]

Hi,

Would it be possible to take a look at your Dockerfile ?

[Stefan Scherfke]

The build image acually consists of three images, but here is a merged version (some internal stuff is removed :)):

FROM almalinux:8

#
# Base image
#

ENV \
    LANG=en_US.UTF-8 \
    LC_ALL=en_US.UTF-8 \
    LC_CTYPE=en_US.UTF-8

RUN echo "Create .ssh dir for root" \
 && mkdir -p -m 0700 /root/.ssh \
 && echo "machine-id is empty in the Docker image" \
 && systemd-machine-id-setup \
 && echo "Install required system packages" \
 && dnf install -y glibc-langpack-en epel-release wget \
 && echo "Install updates" \
 && dnf upgrade -y \
 && echo "Install required system libraries (qt6 libs are only avialable in the develop image)" \
 && dnf install -y glibc glib2 libgcc libstdc++ \
    curl \
    expat \
    fftw-libs-double \
    jasper-libs \
    keyutils \
    libgfortran \
    libgomp \
    libnsl \
    libxcb \
    nspr \
    nss \
    udunits2 \
 && echo "Install \"tini\" as init script for containers" \
 && dnf install -y tini \
 && dnf clean all

#
# Runtime image
#

ENV PATH="/opt/emsconda/bin:${PATH}"

RUN find /root -type f -delete
COPY root/* /root/

# Emsconda Python Distribution
RUN echo "Download and install emsconda" \
 && dnf install -y bzip2 \
 && wget --quiet $dist_url -O emsconda.sh \
 && bash ./emsconda.sh -b -p /opt/emsconda \
 && conda install --quiet --yes neovim \
 && rm emsconda.sh \
 && dnf autoremove -y bzip2 \
 && dnf clean all \
 && echo "Update installation" \
 && conda update --all --quiet --yes \
 && conda clean --all --yes \
 && rm -rf /opt/emsconda/pkgs

STOPSIGNAL SIGINT

#
# Develop image
#
# Build dependencies
#
# The addional toolsets which we add to "/opt/rh/enable" are sourced in "root/.bashrc".
RUN echo "Enable additional repositories" \
 && sed -i '0,/enabled=0/{s/enabled=0/enabled=1/}' /etc/yum.repos.d/almalinux-powertools.repo \
 && echo "Install updates" \
 && dnf upgrade -y \
 && echo "Basic tools" \
 && dnf install -y \
    autoconf automake byacc cmake make ninja-build \
    gcc gcc-c++ gcc-gfortran gcc-toolset-11 libtool \
    perl python39 rust cargo \
    git mercurial  \
    gperf patch patchutils sudo which \
 && echo "Merge envs scripts" \
 && touch /opt/rh/enable \
 && echo ". /opt/rh/gcc-toolset-11/enable" >> /opt/rh/enable \
 && echo "Basic libs" \
 && dnf install -y glibc-devel glibc-headers glib2-devel libstdc++-devel \
 && echo "Xvfb for running GUI tests in headless mode" \
 && dnf install -y xorg-x11-server-Xvfb mesa-dri-drivers \
 && echo "cdo" \
 && dnf install -y libcurl-devel fftw-devel udunits2-devel \
 && echo "eccodes" \
 && dnf install -y jasper-devel \
 && echo "krb5" \
 && dnf install -y keyutils-libs-devel \
 && echo "libgdal" \
 && dnf install -y expat-devel \
 && echo "libnetcdf" \
 && dnf install -y libcurl-devel \
 && echo "libxmlsec1" \
 && dnf install -y nspr-devel nss-devel nss-util-devel \
 && echo "mariadb-connector-c" \
 && dnf install -y libcurl-devel \
 && echo "qt5/6" \
 && dnf install -y bison flex \
    alsa-lib alsa-lib-devel cups-devel cups-libs dbus-devel dbus-libs \
    expat expat-devel jasper-libs jasper-devel libICE libICE-devel \
    libSM libSM-devel libX11 libX11-devel libXScrnSaver libXScrnSaver-devel \
    libXcomposite libXcomposite-devel libXcursor libXcursor-devel \
    libXdamage libXdamage-devel libXext libXext-devel libXfixes libXfixes-devel \
    libXi libXi-devel libXrandr libXrandr-devel libXrender libXrender-devel \
    libXtst libXtst-devel libdrm libdrm-devel libxcb libxcb-devel \
    libxkbcommon libxkbcommon-devel libxkbcommon-x11 libxkbcommon-x11-devel \
    libxkbfile-devel libxshmfence-devel \
    mesa-libEGL mesa-libEGL-devel mesa-libGL mesa-libGL-devel \
    nspr nspr-devel nss nss-devel nss-util-devel pciutils-devel pciutils-libs \
    pulseaudio-libs pulseaudio-libs-devel pulseaudio-libs-glib2 \
    wayland-devel xcb-util-devel xcb-util-image-devel xcb-util-keysyms-devel \
    xcb-util-renderutil-devel xcb-util-wm-devel \
 && echo "uwsgi" \
 && dnf install -y expat-devel \
 && echo "Javac (needed for swagger)" \
 && dnf install -y java-latest-openjdk-devel \
 && dnf clean all

# Enable the devtoolset-11 and git (but not perl526) for all bash scripts
# That script is also sourced from ".bashrc"
ENV BASH_ENV=/opt/rh/enable \
    ENV=/opt/rh/enable

[Stefan Scherfke]

We use some libraries from the OS (glibc, libstdc++, ... and x-libs) and install a few other libraries via Conda.

[hskoglund]

Hi, this is just a wild guess, but when I read "... and then silently walks away..." I think of a section in the .elf file called ABI-tag, when it's present it causes exactly that "silently walk away-syndrome". Most Qt users on WSL stumbles on this: post here
Recent forums posts: here and here

If this happens to be your problem also, then the fix is to strip that section, i.e.
strip --remove-section=.note.ABI-tag libQt6Core.so.6

[Stefan Scherfke]

Wow! That did it! The WSL post was a very interesting and 🤯-inducing read. Thank you very much!

Summary:

• The .so contains an ABI tag that denotes the minimum kernel version required. You can see this via objdump -s -j .note.ABI-tag libQt6Core.so.6.2.4. The result is in the last three blocks (0x03 0x11 0x00 -> 3.17.0 in my case).
• This information is placed there on purpose since QT uses a few system calls that are only available with newer kernels.
• Glibc reads this information when it loads a shared object and compares it to the current kernel's version. If it doesn't match, the file is not loaded.
• Since Docker has no own kernel, the Docker host’s kernel version is used for that comparison. So even if the Docker image is Alma8, the kernel is still the old v3.10.0 from the Centos7 host in my case.
• You can use strip --remove-section=.note.ABI-tag libQt6Core.so.6.2.4. Qt seems to have fallback code, so nothing breaks.

Source: https://github.com/Microsoft/WSL/issues/3023