QT Creator does not stop on breakpoints of sub project, but is able to step manually into it

JoVi

I have a main executable which does have a dependency to a library.
Both projects are configured as CMake-Projects and are locally available. Each project builds without errors.

I did not mark the library specifically as "sub project" in the CMakeLists.txt or something like that.
I just to the linking and including the main CMakeLists.txt and the library does have a separate CMakeLists.txt.

When I debug the main project and set a breakpoint somewhere, it does hit the breakpoint and I can step through my program.
I also set some breakpoints in my library but they are not getting hit.

Now comes the weird part:

I can step from the main executable into the library method. I even see the breakpoint on the line while stepping through. Once I "entered" the sub project with "stepping into it", the other breakpoints suddenly are getting hit (I have multiple threads)
QT-Creator does "grey" some parts that are in an #ifdef _DEBUG ... . My breakpoints inside this ifdef are not getting hit when I am stepping through it.

Details:
The library seems to be a real debug build (Judging by the size and that I can step into it). Changing build settings to release produces a smaller library.
Both projects are configured as Debug-Build and show CMAKE_BUILD_TYPE=Debug on the "Build Settings" page

I am using QT Creator 4.13.1 on Ubuntu 20.04. Using system GDB 9.2 and system CMake 3.16.3