ARM toolchain looks in the wrong directory for libraries

Darkmalex

For my project I need to compile Qt 5.9.1 for a generic ARM device. I've set up linaro's toolchain. But Qt configure halts at this:

/usr/bin/arm-linux-gnueabihf-g++ -mfloat-abi=hard --sysroot=/mnt/elvees-rootfs/ -Wl,-O1 -fuse-ld=gold -o verifyspec verifyspec.o
/usr/bin/arm-linux-gnueabihf-ld.gold: error: /lib/libc.so.6: incompatible target
/usr/bin/arm-linux-gnueabihf-ld.gold: error: cannot open /lib/ld-linux-armhf.so.3: No such file or directory
/usr/bin/arm-linux-gnueabihf-ld.gold: error: /lib/libc_nonshared.a(elf-init.oS): incompatible target 
../sysdeps/arm/start.S:124: error: undefined reference to '__libc_start_main' 
../sysdeps/arm/start.S:128: error: undefined reference to 'abort' 
../sysdeps/arm/start.S:113: error: undefined reference to '__libc_csu_fini' 
../sysdeps/arm/start.S:120: error: undefined reference to '__libc_csu_init' collect2: 
error: ld returned 1 exit status

From the looks of it, arm-linux-gnueabihf-g++ looks for libs in host's /lib/ directory, instead of the sysroot's. Any reason for such a behavior and how to fix it? My best bet is the broken symlinks in the mounted sysroot, but I've manualy changed the "libc.so.6" one, and the output is still the same. To clarify, host and target are running Arch.

SGaist

Hi,

What parameters did you pass to configure ?

Darkmalex

@SGaist said in ARM toolchain looks in the wrong directory for libraries:

Hi,
What parameters did you pass to configure ?

These params

./configure \
-opengl es2 \
-device arch-elvees-g++ \
-device-option CROSS_COMPILE=/usr/bin/arm-linux-gnueabihf- \
-sysroot /mnt/elvees-rootfs/ \
-opensource \
-confirm-license \
-optimized-qmake \
-release \
-make libs \
-prefix /opt/qt-elvees\
-no-pch \
-nomake examples \
-nomake tests \
-no-xcb \
-eglfs \
-v

SGaist

Did you create your own mkspec ?

Darkmalex

@SGaist said in ARM toolchain looks in the wrong directory for libraries:

Did you create your own mkspec ?

Yes I did. Just edited generic-arm one to suit my needs.
Here my qmake.conf, if it matters:

include(../common/linux_device_pre.conf)

ELVEES_CFLAGS             = -march=armv7-a -mtune=cortex-a9 -mfpu=neon-fp16 -DLINUX=1
QMAKE_CFLAGS           += $$ELVEES_CFLAGS
QMAKE_CXXFLAGS         += $$ELVEES_CFLAGS 

DISTRO_OPTS += hard-float

QT_QPA_DEFAULT_PLATFORM = eglfs

include(../common/linux_arm_device_post.conf)

QMAKE_INCDIR_EGL        = /mnt/elvees-rootfs/usr/include/EGL
QMAKE_LIBDIR_EGL        = /mnt/elvees-rootfs/usr/lib
QMAKE_INCDIR_OPENGL_ES2 = /mnt/elvees-rootfs/include/GLES2
QMAKE_LIBDIR_OPENGL_ES2 = /mnt/elvees-rootfs/usr/lib

QMAKE_LIBS_EGL          = -lEGL
QMAKE_LIBS_OPENGL_ES2   = -lGLESv2 $${QMAKE_LIBS_EGL}

load(qt_config)

ambershark

@Darkmalex It looks like it isn't picking up the sysroot stuff properly.

Since I can see it passes --sysroot to g++ I'm guessing your sysroot/lib directory doesn't have the proper libraries in it.

What is the result of file /mnt/elvees-rootfs/lib/libc.so.6? And can you show ls -lh /mnt/elvees-rootfs/lib?

Edit: also maybe try dropping the trailing / on your sysroot path (shouldn't matter but sometimes does). And maybe add -gcc-sysroot to your configure line (should be defaulted to yes, but can't hurt to force it).

Darkmalex

@ambershark
I literaly triple checked that all the symlinks point exactly where they should, but just in case I ran "file ..." as a sanity check:

/mnt/elvees-rootfs/lib/libc.so.6: symbolic link to /mnt/elvees-rootfs/usr/lib/libc-2.23.so

Which is precisely what it should be linking to.
Same with the others

Edit

Here is "ls -lh" output:

lrwxrwxrwx 1 root root 7 окт  5  2015 /mnt/elvees-rootfs/lib -> usr/lib

A symlink to a relative path, nothing suspicious

I will try your suggestions, and report on my progress

ambershark

@Darkmalex Thanks, sorry I was looking for the actual binary information... can you do file whatever-the-actual-non-symlink-binary-is?

Edit: If this is the actual binary file, then file /mnt/elvees-rootfs/usr/lib/libc-2.23.so

Example of the output I'm looking for on your libc, although this one is a host libc:

libc-2.25.so: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /usr/lib/ld-linux-x86-64.so.2, BuildID[sha1]=599894f097519996b776d6daef9dd26756e02a95, for GNU/Linux 2.6.32, not stripped

Darkmalex

@ambershark said in ARM toolchain looks in the wrong directory for libraries:
Sure thing, here it is:

/mnt/elvees-rootfs/usr/lib/libc-2.23.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /usr/lib/ld-linux-armhf.so.3, BuildID[sha1]=f2686e40e0d4b73dc9092119a6c27be143fcf462, for GNU/Linux 2.6.32, not stripped

I just thought you were checking for broken symlinks, which was my first thought, when I first encountered this problem.

ambershark

@Darkmalex I was actually making sure it was compiled for the ARM platform properly. :) I've helped a lot of people with this type of issue on these forums and you'd be surprised how many have sysroots with elf 64-bit x86-64 objects in them. ;)

So this goes back to it seeming like it's not picking up the sysroot properly.

You can try hacking it as a test and pass CXXFLAGS=-L/path/to/your/sysroot/lib. See if that fixes it. If it does I do not recommend using that but instead figuring out where in your mkspec or configure things are going wrong. Chances are you'll have issues if you just use CXXFLAGS or LDFLAGS. :) Also you may need to pass that same thing to LDFLAGS.

Darkmalex

@ambershark
Well, I'll be... -no-gcc-sysroot did the trick. Thank you so much.
I'll close the question, but first

Why? This seem like a counter-intuitive param, why must it be added? I actualy never had such a problem on older versions of Qt when compiling for arm generics. And what exactly does it do, and why it fixed things?
It says OpenGLES support is now broken. Can it be "no-gcc-sysroot" fault, or sould I look somewere else? Here is the output:

ERROR: Feature 'opengles2' was enabled, but the pre-condition 'config.win32 || (!config.watchos && !features.opengl-desktop && libs.opengl_es2)' failed.

ERROR: Feature 'eglfs' was enabled, but the pre-condition '!config.android && !config.darwin && !config.win32 && features.egl' failed.

ambershark

@Darkmalex Lol well that is weird. I expected you needed to pass -gcc-sysroot, not -no-gcc-sysroot. So I definitely agree with you that is a backwards parameter. Something feels off with your setup for sure.

As for the next problem it sounds like eglfs doesn't support anything that isn't mac/windows/android. I haven't used it so I can't be sure on that but maybe something to look into since you seem to be building for a linux embedded device (non-android).

Darkmalex

@ambershark
Well, I definitively tried -gcc-sysroot but it gave me pretty much the same output, so I tried -no-gcc-sysroot out of desperation and curiosity, and it worked. From what I can find, it's merely a hack, that works buy passing "sysroot" to everything but gcc. So the behavior is not quite unexpected, but still puzzles me, as to why this was a problem in the first place.

About eglfs: It doesn't make any sence, no matter how I think about it.
First:

@ambershark said in ARM toolchain looks in the wrong directory for libraries:

@Darkmalex
As for the next problem it sounds like eglfs doesn't support anything that isn't mac/windows/android. I haven't used it so I can't be sure on that but maybe something to look into since you seem to be building for a linux embedded device (non-android).

But why?
Firstly, in Qt docs EGLFS is clearly described as a supported option for linux embedded targets.
Secondly, the whole point of supporting EGLFS, afaik, is to abstract Qt from a windowing system, so making it mac/windows/android only sounds weird. Why abstract from windowing systems, in operating systems that offer no choice in window composition in the first place.
Thirdly, EGLSFS supported build for RasPi exists. Ofc, I could just look there, and try to repeat, but I cant for a few reasons:

RasPi by default uses debian - my options: ether Arch, ether buildroot. So the whole "move libs from /opt..." just adds confusion (Arch by default, has an empty opt, unlike Rasbian)
RasPi tutorial heavily depends on tools and packages build with RasPi in mind. I have to reverse engineer things, like raspi-tools.

The worst part is that I don't even know what exactly Qt wants from me. It just kinda says: "can't find opengles, do smth" instead of giving a verbose list of things, I should point it to.

ambershark

@Darkmalex Yea I agree with you on the sysroot thing. It is confusing for sure and seems backwards.

I wish I knew more about eglfs in order to help you but I just don't. I've never tried to do what you're doing so I'm not sure how to guide you here. Perhaps someone who has done this or at least knows eglfs can chime in here?

Edit: I was reading about eglfs and found this on the Qt page:

QPA backends:
EGLFS ................................ yes
EGLFS details:
  EGLFS i.Mx6 ........................ no
  EGLFS i.Mx6 Wayland ................ no
  EGLFS EGLDevice .................... no
  EGLFS GBM .......................... no
  EGLFS Mali ......................... no
  EGLFS Rasberry Pi .................. yes
  EGL on X11 ......................... no

So that shows it explicitly working with a Pi. So it should work, that makes that assertion you are getting even more curious.

Are you working in clean directory for the build? If you configured in source then I would extract a fresh copy of the qt source to make sure things are clean. You can use out of source builds too so you can test things without re-extracting qt all the time. Qt is highly susceptible to dirty build environments. You could just be seeing that as an issue.

Darkmalex

@ambershark said in ARM toolchain looks in the wrong directory for libraries:

@Darkmalex
Are you working in clean directory for the build? If you configured in source then I would extract a fresh copy of the qt source to make sure things are clean. You can use out of source builds too so you can test things without re-extracting qt all the time. Qt is highly susceptible to dirty build environments. You could just be seeing that as an issue.

Every build I do, is a clean one. Every failure I delete/re-extract everything, because I'm not only trying to get Qt to work, but also writing a bash script that automates the setup process. Plus my workstation is pretty powerfull, so the whole process takes no more than 10 min. so it's no big deal.
And just to clarify, board definately supports opengles. GPU is mali 300 and the board itself is running es2gears, just as I type.
But since my original problem is solved, I guess I'll mark the problem as solved, and try to figure it out on my own for now. Maybe I'll settle for non-accelerated fb rendering.
Thanks for help