Compiled qml sources visible inside executable

SGaist

Hi,

Not knowing the content of your .pro file nor the Qt version you are using it's really hard to check what might be happening.

FalsinSoft

Hi
At first thank you for your reply. Qt version 5.15.2, Qt Creator version 4.13.2. Content of .pro file what you would to know? Based to the official procedure here:

The integration into the build system happens at the level of the Qt Resource System. In order to use the Qt Quick Compiler, your application needs to be prepared:

- Create a .qrc file for your application and include all the files your project needs, including .qml , .js and qmldir files.
Add the file name of your .qrc file to RESOURCES in your .pro file.
- Change your application to use qrc: URLs instead of file: in order to become file system independent.
- This way all dependent files are bundled into your application binary and you should verify that your application works without the .qml source code in the file system.
- When your application has been converted to use the Qt Resource System, you can activate the integration with the Qt Quick Compiler by passing CONFIG += qtquickcompiler to qmake, for example on the command line or in the build settings of your project in Qt Creator, as additional argument in the qmake build steps.

After activating the Qt Quick Compiler in the build integration, the build system will automatically execute the tool to compile your .qml and .js files into the binary and avoid the inclusion of their source code in the resource system.

I created a resource file called Source.qrc with inside the list of all my qml files and added to the RESOURCES label inside .pro file. Based to the last note the compiler should avoid to include the compiled qml files inside the resource system but it seem it's not happening in my case...

SGaist

Then do you have a minimal compilable example that shows that behaviour ?

FalsinSoft

@SGaist Just added in the original post, If I try to compile I can see in the generated file folder the source main_qml.cpp but inside the executable I can find the source of main.qml also.

SGaist

From the documentation you linked, aren't you missing

CONFIG += qtquickcompiler

in your .pro file ?

FalsinSoft

@SGaist Just for verify I tried to add manually but same result. However Qt Creator already have in the project settings the "Qt Quick compiler" setting that is enabled by default.

fcarney

Just to be sure after you added qtquickcompiler did you:

clean project
run qmake
build again

KH-219Design

@FalsinSoft This thread is very intriguing.

I have been using qrc files and the RESOURCES directive in my pro files for several years, and I never stopped to think deeply about what was happening under the hood.

Windows Disclaimer: I have not compiled anything (Qt or otherwise) on Windows in many years. So I certainly won't have anything interesting to say that is specific to Windows.

But you made me curious about whether my qml files are getting pre-compiled, or whether their human-readable lines of source code are simply "pasted" into the binary and processed as QML text at runtime.

I did a small investigation just now using a "hello world" QML project I contribute to on GitHub.

Here is my "libresources.qrc" file: https://github.com/219-design/qt-qml-project-template-with-ci/blob/8f5511d425/src/lib_app/libresources.qrc

Here is the generated cpp file: https://gist.github.com/pestophagous/bf30369f824a7344d047496002afdc52

Just looking at the start of the generated cpp file:

static const unsigned char qt_resource_data[] = {
  // /opt/repositories/client/219/qt-qml-cpp-proj-template/src/lib_app/qml/ImageSvgHelper.qml
  0x0,0x0,0x5,0x23,
  0x69,
  0x6d,0x70,0x6f,0x72,0x74,0x20,0x51,0x74,0x51,0x75,0x69,0x63,0x6b,0x20,0x32,0x2e,
  0x31,0x32,0xa,0x69,0x6d,0x70,0x6f,0x72,0x74,0x20,0x51,0x74,0x51,0x75,0x69,0x63,
  0x6b,0x2e,0x43,0x6f,0x6e,0x74,0x72,0x6f,0x6c,0x73,0x20,0x32,0x2e,0x31,0x32,0xa,
  0xa,0x49,0x74,0x65,0x6d,0x20,0x7b,0xa,0x20,0x20,0x69,0x64,0x3a,0x20,0x72,0x6f,

Using python to decode those bytes:

# python3
In [5]: bytearray.fromhex("6d 70 6f 72 74 20 51 74 51 75 69 63 6b 20 32 2e 31 32 0a 69 6d 70 6f 72 74 20 51 74 51 75 69 63 6b 2e 43 6f 6e 74 72 6f 6c 73 20 32 2e 31 32 0a 0a 49 74 65 6d 20 7b 0a 20 20 69 
   ...: 64 3a 20 72 6f 6f 74 0a 0a 20 20 2f 2f 20 57 65 20 63 61 6e 6e 6f 74 20 75 73 65 20 27 61 6c 69 61 73 27 2c 20 73 69 6e 63 65 20 74 68 65 73 65 20 6d 75 73 74 20 62 65 20 61 76 61 69 6c 61 62 6c 6
   ...: 5 20 74 6f 20 74 68 65 20 49 6d 61 67 65 20 41 4e 44 20 74 68 65 20 41 6e 69 6d 61 74 65 64 49 6d 61 67 65 0a 20 20 70 72 6f 70 65 72 74 79 20 76 61 72 20 66 69 6c 6c 4d 6f 64 65 3a 20 6e 69 6c 0a
   ...:  20 20 70 72 6f 70 65 72 74 79 20 75 72 6c 20 73 6f 75 72 63 65 3a 20 22 22 0a 0a 20 20 49 6d 61 67 65 20 7b 0a 20 20 20 20 69 64 3a 20 6e 6f 6e 47 69 66 0a 0a 20 20 20 20 76 69 73 69 62 6c 65 3a 
   ...: 20 21 28 73 6f 75 72 63 65 20 2b 20 22 22 29 2e 65 6e 64 73 57 69 74 68 28 22 67 69 66 22 29 0a 20 20 20 20 61 6e 63 68 6f 72 73 2e 66 69 6c 6c 3a 20 70 61 72 65 6e 74 0a 20 20 20 20 73 6f 75 72 6
   ...: 3 65 3a 20 72 6f 6f 74 2e 73 6f 75 72 63 65 0a 20 20 20 20 66 69 6c 6c 4d 6f 64 65 3a 20 7b 0a 20 20 20 20 20 20 69 66 20 28 21 21 72 6f 6f 74 2e 66 69 6c 6c 4d 6f 64 65 29 0a 20 20 20 20 20 20 20
   ...:  20 72 6f 6f 74 2e 66 69 6c 6c 4d 6f 64 65 0a 20 20 20 20 20 20 65 6c 73 65 0a 20 20 20 20 20 20 20 20 49 6d 61 67 65 2e 53 74 72 65 74 63").decode()

Out[5]: 'mport QtQuick 2.12\nimport QtQuick.Controls 2.12\n\nItem {\n  id: root\n\n  // We cannot use \'alias\', since these must be available to the Image AND the AnimatedImage\n  property var fillMode: nil\n  property url source: ""\n\n  Image {\n    id: nonGif\n\n    visible: !(source + "").endsWith("gif")\n    anchors.fill: parent\n    source: root.source\n    fillMode: {\n      if (!!root.fillMode)\n        root.fillMode\n      else\n        Image.Stretc'

There is my hand-written QML code, comments and all!

So here's what I can observe so far:

This happens during my build:

rcc -name libresources qt-qml-cpp-proj-template/src/lib_app/libresources.qrc -o qrc_libresources.cpp

The resulting cpp file (qrc_libresources.cpp) contains the raw bytes of all my hand-written QML code (even my comments).

Given that qrc_libresources.cpp then gets compiled and linked into my larger executable, it is "obvious" that these static const unsigned char qt_resource_data[] byte arrays are part of the executable, and therefore the raw (untransformed, uncompiled) QML code that I wrote is part of the executable, and would be visible in Notepad++ or similar.

Whether or not this is what a given individual or team wants or expects is another matter. I can't make any comment on that.

Locating these byte arrays in the generated cpp does also not tell me with any certainty whether there is or is not another byte representation getting compiled into my app somewhere that represents a more optimized intermediate version of the QML (pre-just-in-time compiled for the QML/JS interpreter).

It sounds to me like you (you, @FalsinSoft ) were:

expecting that the generated CPP code would not just contain verbatim the text of your hand-written QML code, and
also expecting that the generated CPP code would contain some other, transformed format of the QML, something like QML bytecode/assembly

Both of those are interesting propositions. I can already see that on my project, the verbatim hand-written QML is "pasted" into the generated cpp file. I cannot yet see whether there is also a QML bytecode/assembly structure in there anywhere.

I'm curious to hear about other intricacies anyone can share on these compilation steps.

KH-219Design

I see the giant omission in my prior post.

I did not run with qmake CONFIG+=qtquickcompiler !!

Oops. Doing that now, and will report the result.

fcarney

@KH-219Design I like how you are doing this for "science". 1 point 21 jigawatts!

FalsinSoft

@KH-219Design If you didn't set the qtquickcompiler option your qml file has been integrated as normal resource and compressed and this could be the reasons you don't see any readable source text inside executable (but this is only an hypothesis)

KH-219Design

@FalsinSoft said in Compiled qml sources visible inside executable:

@KH-219Design If you didn't set the qtquickcompiler option your qml file has been integrated as normal resource and compressed and this could be the reasons you don't see any readable source text inside executable (but this is only an hypothesis)

I explicitly do see readable QML in my executable.

So here is a comparison of what I see without CONFIG+=qtquickcompiler and what I see with CONFIG+=qtquickcompiler:

Seen after building without qtquickcompiler:

strings libapp.so.1.0.0 | grep DebugRect

  // In this way, you can insert 'DebugRectangle' items wherever you want in your QML.
  // When you wish for the DebugRectangle(s) to become visible (in a production binary executable
        DebugRectangle {
          // To make the DebugRectangle show up in the GUI, change your environment:
          // (See comments in DebugRectangle.qml for more information.)

Seen after building with qtquickcompiler:

strings libapp.so.1.0.0 | grep DebugRect

_ZN21QmlCacheGeneratedCode23_qml_DebugRectangle_qml7qmlDataE
  // In this way, you can insert 'DebugRectangle' items wherever you want in your QML.
  // When you wish for the DebugRectangle(s) to become visible (in a production binary executable
        DebugRectangle {
          // To make the DebugRectangle show up in the GUI, change your environment:
          // (See comments in DebugRectangle.qml for more information.)
qml_DebugRectangle_qml.cpp
_qml_DebugRectangle_qml
_ZN21QmlCacheGeneratedCode23_qml_DebugRectangle_qml7qmlDataE
qml_DebugRectangle_qml.cpp
qml_DebugRectangle_qml.cpp
_ZN21QmlCacheGeneratedCode23_qml_DebugRectangle_qmlL4unitE
_ZN21QmlCacheGeneratedCode23_qml_DebugRectangle_qml7qmlDataE

Correct me if I am misreading the initial inquiry of this thread, but I think we are all interested in two things:

whether or not the generated CPP code should or should not contain verbatim the text of your hand-written QML code, and
whether or not the generated CPP code should also contain some other, transformed format of the QML, something like QML bytecode/assembly

Regarding (1), I can observe that in both kinds of build, I do see my verbatim QML in the binary.

Regarding (2), I can now observe that using "CONFIG+=qtquickcompiler" makes a difference on that front.

But I seem to have confirmed one thing that @FalsinSoft noticed, which is that the verbatim QML is "pasted" into the binary either way.

So I am still curious about why that is. Maybe there could be some other flag or makefile/pro-file tactic that would (in the case of using CONFIG+=qtquickcompiler), essentially keep this in the binary:

_ZN21QmlCacheGeneratedCode23_qml_DebugRectangle_qml7qmlDataE
qml_DebugRectangle_qml.cpp
_qml_DebugRectangle_qml
_ZN21QmlCacheGeneratedCode23_qml_DebugRectangle_qml7qmlDataE
qml_DebugRectangle_qml.cpp
qml_DebugRectangle_qml.cpp
_ZN21QmlCacheGeneratedCode23_qml_DebugRectangle_qmlL4unitE
_ZN21QmlCacheGeneratedCode23_qml_DebugRectangle_qml7qmlDataE

... but strip this out of the binary:

  // In this way, you can insert 'DebugRectangle' items wherever you want in your QML.
  // When you wish for the DebugRectangle(s) to become visible (in a production binary executable
        DebugRectangle {
          // To make the DebugRectangle show up in the GUI, change your environment:
          // (See comments in DebugRectangle.qml for more information.)

fcarney

@KH-219Design
Is this a debug binary? I wonder if that makes a difference.

FalsinSoft

@fcarney In my case it's a release

FalsinSoft

Created bug report here

jobusch

@KH-219Design Hey dude, you nearly saved my day. I was trying to use your method to get back some lost source .qml files, because I lost data of my source files but still got the directory it was compiled.

I have exactly the same files as you have and I did it the same way.
Using your example file I get clear readable text, but using my file it is not working, so maybe I have another encoding? Any idea on how I could try to find out my encoding? In Qt Editor it says Windows CRLF (which is only about line endings, right?) Looking into my settings it says utf-8, which didnt work.

Any help would really be appreciated!

This is the output I get:

ê  
VÄo|n2ôZcf¨xäÂTq;õjk§µs’¦J
B.1°1W*½J2Ld{û•äæî\ºÄÌ.À9>½ ¼(Þ±`d¦€	-Ë
V\#|
ë5ýsïTÞÉ{zz½Fýü-êîPŸ‚ÎðÞ?3¯Åºû¯¬ŸEºû_H÷^@ú‹ÔôÎT*ÓÐ¯‡»÷ª¸{Oã^^5¤ÌŠ~^ßìÖ!×"1Ö…hÒ?.t½Öá‚Û‰ÔEâÔü¨×I<Ë;xå^vô2Rq÷~Û„:U&•ß9õËuÕ$òw\Sææ‘° úáqm‘nŠ˜9°+ñ~ñEÑ R-¸•ˆ£RØ•þèûØë£Á

from:

  b = bytearray.fromhex("00 00 02 ea \
  00 \
  00 0a 1b 78 9c bd 95 6d 4f db 30 10 c7 df 23 f5 \
  3b 58 d9 0b 40 db d2 07 b1 09 55 94 09 ba 49 a0 \
  8d 17 ac 6c 7b ed 38 d7 c6 c2 b5 83 7d a1 14 c4 \
  77 df 39 49 97 a4 4f 0c a6 e1 b6 6a 72 ff f3 f9 \
  fc f3 25 27 a7 a9 b1 c8 2e f1 32 93 e2 9a f5 c2 \
  ee 87 d6 8e 6c 18 c3 5f 52 c7 66 b6 5e 1b 1a 8d \
  d6 28 f7 84 da 0d 0f 56 c4 6f 7c 6e 32 f4 5a 63 \
  66 a8 78 e4 c2 54 71 1c 1b 3b f5 6a 6b a7 b5 73 \
  92 a6 4a 0a 8e d2 e8 32 9d 87 d6 0e a3 21 e3 3e \
  9b 72 a9 67 b9 b5 b0 dd 4a 27 23 05 7d 86 36 83 \
  c2 34 93 31 26 7d f6 f1 a0 53 dc 27 20 27 09 f6 \
  d9 c1 61 69 40 89 7e c6 8d bb b2 7b c1 e7 9f ec \
  c2 44 52 01 1b 65 12 81 9d 19 87 c1 be cf c3 bb \
  1a 3d 54 c6 49 3d e9 2f 92 f0 43 90 0d 42 2e 04 \
  a4 08 31 1b b0 31 57 0e 2a bd 4a 32 4c 64 0c 7b \
  fb 95 e4 e6 ee 5c 18 1d ba c4 cc 2e c0 39 3e 81 \
  bd a0 bc 28 12 0b de b1 60 64 a6 80 09 2d cb 0a \
  56 5c 23 13 7c 0a 2c 4b 43 36 54 fe f8 48 76 0c \
  0d bb d6 04 68 6a 2c 84 41 b9 cc e3 22 f9 d1 dc \
  21 4c af 2c 9f fb 25 eb 1b f0 24 cb 4c 2a e3 1a \
  94 b9 6f 9e ae c9 ac 20 29 b8 b1 a2 df b6 e0 e8 \
  37 a6 bf 24 4c f5 24 a8 9c d1 18 85 32 dd 06 97 \
  8d c0 de 82 0d 6a 50 8c 3e 11 28 6f 39 c1 6c 70 \
  5e 62 e9 99 d5 59 2e c9 96 4b 07 db 74 b8 c9 c0 \
  e1 62 a9 ba 67 8e ac 46 af b8 bc e2 d1 29 b7 cb \
  dc 22 6e 2b 43 59 6b 29 b7 a0 31 cc ef 2a 31 e2 \
  e2 7a 62 4d a6 69 d6 77 10 74 8a 13 22 b1 b4 3f \
  3f da 6d 61 94 b1 84 f7 4d f7 d0 7f 82 a6 cf 42 \
  45 cb b5 2b d6 5a f2 78 ac 55 58 ca 45 5e b0 9d \
  ca e6 77 92 21 36 8b 60 b1 21 e4 51 a7 69 45 b8 \
  a3 e7 25 38 1a 1b 5f 77 7e f1 c1 ee 2c a1 03 dc \
  3d 0e d8 db f2 6c bf 52 3d 5a ee 82 7d b2 04 47 \
  6d ef 7b 1c 2c d0 3d 1b c1 d2 56 09 72 28 32 eb \
  77 7a ae 63 b8 63 83 01 eb b0 4f 15 20 46 f9 45 \
  8a a2 6f 06 e1 47 ca e3 38 87 d1 eb 2c a7 d6 ce \
  53 0e 23 a3 e2 0d eb 35 fd 73 ef 54 de 81 1a c9 \
  7b 7a 14 7a bd 46 fd fc 2d ea ee 0b 50 9f 82 ce \
  f0 de 3f 33 af c5 ba fb af ac 9f 45 ba fb 5f 48 \
  f7 5e 40 fa 8b d4 f4 ce 54 2a d3 13 d0 af 87 bb \
  f7 aa b8 7b 4f e3 5e 01 5e 35 15 a4 cc 8a 7e 5e \
  df ec d6 17 21 d7 22 31 d6 85 68 d2 3f 2e 74 bd \
  d6 e1 82 db 89 d4 45 e2 d4 fc a8 d7 49 3c cb 3b \
  78 e5 5e 76 f4 32 52 71 f7 7e db 84 3a 81 55 26 \
  95 df 39 f5 cb 75 d5 24 f2 77 1d 15 5c 53 1a e6 \
  e6 91 b0 00 fa e1 71 6d 91 6e 8a 98 39 b0 2b f1 \
  7e 90 f1 45 d1 1c 20 52 2d b8 95 88 a3 52 d8 18 \
  95 fe e8 fb 1b d8 eb a3 c1")

KH-219Design

@jobusch two questions for you:

What operating system are you on? (I'm guessing windows because you mentioned CRLF, but best to be explicit.)
Are you using CONFIG+=qtquickcompiler in your project build, yes or no?

What we determined in this thread previously is that without CONFIG+=qtquickcompiler, the raw QML text is present, and likewise with CONFIG+=qtquickcompiler the plain QML text is still present, but a bunch of other stuff could be inserted around it.

So if qtquickcompiler was part of the configuration when your binary was produced, it could complicate your efforts (but should not thwart them entirely).

KH-219Design

@jobusch As far as I was aware several years ago, Microsoft still had a penchant for preferring UTF-16/UCS-2 over utf8. So that's something to maybe consider.

KH-219Design

@jobusch another thought:

the qt resource compiler can be used to "bake in" more than just QML text. resources can be of arbitrary files types. (e.g. pdf files, graphical assets, other documents).

Maybe you are mistakenly focusing on the wrong batch of hex bytes? Maybe the bytes you grabbed for examination via python are not, in fact, bytes from a QML text file?

If you look at my gist again: https://gist.github.com/pestophagous/bf30369f824a7344d047496002afdc52

You'll see that qt_resource_data is one giant static array, but the resource compiler inserted various comments lines throughout:

// /opt/repositories/client/219/qt-qml-cpp-proj-template/src/lib_app/qml/ImageSvgHelper.qml
...
 // /opt/repositories/client/219/qt-qml-cpp-proj-template/src/lib_app/qml/VersionLabel.qml
...
// /opt/repositories/client/219/qt-qml-cpp-proj-template/src/lib_app/qml/qmldir

Does your file have such comment lines?

Another factual question that I should have asked earlier, for completeness:

What version of Qt are you using?

KH-219Design

@jobusch GOT IT.

today is my most hacking-est day in a while.

So I ended up here: https://en.wikipedia.org/wiki/List_of_file_signatures

Noticed that 78 9C is a marker for zlib.

So I took the python that you shared, chopped off everything that was in front of 78 9C.

That left me with something like:

bb = bytearray.fromhex("78 9c bd 95 6d 4f db 30 10 c7 df 23 f5 \
  3b 58 d9 0b 40 db d2 07 b1 09 55 94 09 ba 49 a0 \
  8d 17 ac 6c 7b ed 38 d7 c6 c2 b5 83 7d a1 14 c4 \

...

I wrote that to a file named abc.zlib

Then ran this (in bash terminal):

zlib-flate -uncompress < abc.zlib  > out.qml

And I now see your code:

...

        TabButton {
            id: tab1
            text: "<font color='white'>" + qsTr("Benutzer") + "</font>"

            background: Rectangle {
                        color: bar.currentIndex == 1 ? "#181818" : "black"
            }
            padding: 20
            //font.bold: bar.currentIndex == 1
            font.pixelSize: 22
        }
        TabButton {
            id: tab2
            text: "<font color='white'>" + qsTr("Einstellungen") + "</font>"

            background: Rectangle {
                        color: bar.currentIndex == 2 ? "#181818" : "black"
            }
            padding: 20
            //font.bold: bar.currentIndex == 2
            font.pixelSize: 22

        }

...