Compiled qml sources visible inside executable

fcarney

Just to be sure after you added qtquickcompiler did you:

clean project
run qmake
build again

KH-219Design

@FalsinSoft This thread is very intriguing.

I have been using qrc files and the RESOURCES directive in my pro files for several years, and I never stopped to think deeply about what was happening under the hood.

Windows Disclaimer: I have not compiled anything (Qt or otherwise) on Windows in many years. So I certainly won't have anything interesting to say that is specific to Windows.

But you made me curious about whether my qml files are getting pre-compiled, or whether their human-readable lines of source code are simply "pasted" into the binary and processed as QML text at runtime.

I did a small investigation just now using a "hello world" QML project I contribute to on GitHub.

Here is my "libresources.qrc" file: https://github.com/219-design/qt-qml-project-template-with-ci/blob/8f5511d425/src/lib_app/libresources.qrc

Here is the generated cpp file: https://gist.github.com/pestophagous/bf30369f824a7344d047496002afdc52

Just looking at the start of the generated cpp file:

static const unsigned char qt_resource_data[] = {
  // /opt/repositories/client/219/qt-qml-cpp-proj-template/src/lib_app/qml/ImageSvgHelper.qml
  0x0,0x0,0x5,0x23,
  0x69,
  0x6d,0x70,0x6f,0x72,0x74,0x20,0x51,0x74,0x51,0x75,0x69,0x63,0x6b,0x20,0x32,0x2e,
  0x31,0x32,0xa,0x69,0x6d,0x70,0x6f,0x72,0x74,0x20,0x51,0x74,0x51,0x75,0x69,0x63,
  0x6b,0x2e,0x43,0x6f,0x6e,0x74,0x72,0x6f,0x6c,0x73,0x20,0x32,0x2e,0x31,0x32,0xa,
  0xa,0x49,0x74,0x65,0x6d,0x20,0x7b,0xa,0x20,0x20,0x69,0x64,0x3a,0x20,0x72,0x6f,

Using python to decode those bytes:

# python3
In [5]: bytearray.fromhex("6d 70 6f 72 74 20 51 74 51 75 69 63 6b 20 32 2e 31 32 0a 69 6d 70 6f 72 74 20 51 74 51 75 69 63 6b 2e 43 6f 6e 74 72 6f 6c 73 20 32 2e 31 32 0a 0a 49 74 65 6d 20 7b 0a 20 20 69 
   ...: 64 3a 20 72 6f 6f 74 0a 0a 20 20 2f 2f 20 57 65 20 63 61 6e 6e 6f 74 20 75 73 65 20 27 61 6c 69 61 73 27 2c 20 73 69 6e 63 65 20 74 68 65 73 65 20 6d 75 73 74 20 62 65 20 61 76 61 69 6c 61 62 6c 6
   ...: 5 20 74 6f 20 74 68 65 20 49 6d 61 67 65 20 41 4e 44 20 74 68 65 20 41 6e 69 6d 61 74 65 64 49 6d 61 67 65 0a 20 20 70 72 6f 70 65 72 74 79 20 76 61 72 20 66 69 6c 6c 4d 6f 64 65 3a 20 6e 69 6c 0a
   ...:  20 20 70 72 6f 70 65 72 74 79 20 75 72 6c 20 73 6f 75 72 63 65 3a 20 22 22 0a 0a 20 20 49 6d 61 67 65 20 7b 0a 20 20 20 20 69 64 3a 20 6e 6f 6e 47 69 66 0a 0a 20 20 20 20 76 69 73 69 62 6c 65 3a 
   ...: 20 21 28 73 6f 75 72 63 65 20 2b 20 22 22 29 2e 65 6e 64 73 57 69 74 68 28 22 67 69 66 22 29 0a 20 20 20 20 61 6e 63 68 6f 72 73 2e 66 69 6c 6c 3a 20 70 61 72 65 6e 74 0a 20 20 20 20 73 6f 75 72 6
   ...: 3 65 3a 20 72 6f 6f 74 2e 73 6f 75 72 63 65 0a 20 20 20 20 66 69 6c 6c 4d 6f 64 65 3a 20 7b 0a 20 20 20 20 20 20 69 66 20 28 21 21 72 6f 6f 74 2e 66 69 6c 6c 4d 6f 64 65 29 0a 20 20 20 20 20 20 20
   ...:  20 72 6f 6f 74 2e 66 69 6c 6c 4d 6f 64 65 0a 20 20 20 20 20 20 65 6c 73 65 0a 20 20 20 20 20 20 20 20 49 6d 61 67 65 2e 53 74 72 65 74 63").decode()

Out[5]: 'mport QtQuick 2.12\nimport QtQuick.Controls 2.12\n\nItem {\n  id: root\n\n  // We cannot use \'alias\', since these must be available to the Image AND the AnimatedImage\n  property var fillMode: nil\n  property url source: ""\n\n  Image {\n    id: nonGif\n\n    visible: !(source + "").endsWith("gif")\n    anchors.fill: parent\n    source: root.source\n    fillMode: {\n      if (!!root.fillMode)\n        root.fillMode\n      else\n        Image.Stretc'

There is my hand-written QML code, comments and all!

So here's what I can observe so far:

This happens during my build:

rcc -name libresources qt-qml-cpp-proj-template/src/lib_app/libresources.qrc -o qrc_libresources.cpp

The resulting cpp file (qrc_libresources.cpp) contains the raw bytes of all my hand-written QML code (even my comments).

Given that qrc_libresources.cpp then gets compiled and linked into my larger executable, it is "obvious" that these static const unsigned char qt_resource_data[] byte arrays are part of the executable, and therefore the raw (untransformed, uncompiled) QML code that I wrote is part of the executable, and would be visible in Notepad++ or similar.

Whether or not this is what a given individual or team wants or expects is another matter. I can't make any comment on that.

Locating these byte arrays in the generated cpp does also not tell me with any certainty whether there is or is not another byte representation getting compiled into my app somewhere that represents a more optimized intermediate version of the QML (pre-just-in-time compiled for the QML/JS interpreter).

It sounds to me like you (you, @FalsinSoft ) were:

expecting that the generated CPP code would not just contain verbatim the text of your hand-written QML code, and
also expecting that the generated CPP code would contain some other, transformed format of the QML, something like QML bytecode/assembly

Both of those are interesting propositions. I can already see that on my project, the verbatim hand-written QML is "pasted" into the generated cpp file. I cannot yet see whether there is also a QML bytecode/assembly structure in there anywhere.

I'm curious to hear about other intricacies anyone can share on these compilation steps.

KH-219Design

I see the giant omission in my prior post.

I did not run with qmake CONFIG+=qtquickcompiler !!

Oops. Doing that now, and will report the result.

fcarney

@KH-219Design I like how you are doing this for "science". 1 point 21 jigawatts!

FalsinSoft

@KH-219Design If you didn't set the qtquickcompiler option your qml file has been integrated as normal resource and compressed and this could be the reasons you don't see any readable source text inside executable (but this is only an hypothesis)

KH-219Design

@FalsinSoft said in Compiled qml sources visible inside executable:

@KH-219Design If you didn't set the qtquickcompiler option your qml file has been integrated as normal resource and compressed and this could be the reasons you don't see any readable source text inside executable (but this is only an hypothesis)

I explicitly do see readable QML in my executable.

So here is a comparison of what I see without CONFIG+=qtquickcompiler and what I see with CONFIG+=qtquickcompiler:

Seen after building without qtquickcompiler:

strings libapp.so.1.0.0 | grep DebugRect

  // In this way, you can insert 'DebugRectangle' items wherever you want in your QML.
  // When you wish for the DebugRectangle(s) to become visible (in a production binary executable
        DebugRectangle {
          // To make the DebugRectangle show up in the GUI, change your environment:
          // (See comments in DebugRectangle.qml for more information.)

Seen after building with qtquickcompiler:

strings libapp.so.1.0.0 | grep DebugRect

_ZN21QmlCacheGeneratedCode23_qml_DebugRectangle_qml7qmlDataE
  // In this way, you can insert 'DebugRectangle' items wherever you want in your QML.
  // When you wish for the DebugRectangle(s) to become visible (in a production binary executable
        DebugRectangle {
          // To make the DebugRectangle show up in the GUI, change your environment:
          // (See comments in DebugRectangle.qml for more information.)
qml_DebugRectangle_qml.cpp
_qml_DebugRectangle_qml
_ZN21QmlCacheGeneratedCode23_qml_DebugRectangle_qml7qmlDataE
qml_DebugRectangle_qml.cpp
qml_DebugRectangle_qml.cpp
_ZN21QmlCacheGeneratedCode23_qml_DebugRectangle_qmlL4unitE
_ZN21QmlCacheGeneratedCode23_qml_DebugRectangle_qml7qmlDataE

Correct me if I am misreading the initial inquiry of this thread, but I think we are all interested in two things:

whether or not the generated CPP code should or should not contain verbatim the text of your hand-written QML code, and
whether or not the generated CPP code should also contain some other, transformed format of the QML, something like QML bytecode/assembly

Regarding (1), I can observe that in both kinds of build, I do see my verbatim QML in the binary.

Regarding (2), I can now observe that using "CONFIG+=qtquickcompiler" makes a difference on that front.

But I seem to have confirmed one thing that @FalsinSoft noticed, which is that the verbatim QML is "pasted" into the binary either way.

So I am still curious about why that is. Maybe there could be some other flag or makefile/pro-file tactic that would (in the case of using CONFIG+=qtquickcompiler), essentially keep this in the binary:

_ZN21QmlCacheGeneratedCode23_qml_DebugRectangle_qml7qmlDataE
qml_DebugRectangle_qml.cpp
_qml_DebugRectangle_qml
_ZN21QmlCacheGeneratedCode23_qml_DebugRectangle_qml7qmlDataE
qml_DebugRectangle_qml.cpp
qml_DebugRectangle_qml.cpp
_ZN21QmlCacheGeneratedCode23_qml_DebugRectangle_qmlL4unitE
_ZN21QmlCacheGeneratedCode23_qml_DebugRectangle_qml7qmlDataE

... but strip this out of the binary:

  // In this way, you can insert 'DebugRectangle' items wherever you want in your QML.
  // When you wish for the DebugRectangle(s) to become visible (in a production binary executable
        DebugRectangle {
          // To make the DebugRectangle show up in the GUI, change your environment:
          // (See comments in DebugRectangle.qml for more information.)

fcarney

@KH-219Design
Is this a debug binary? I wonder if that makes a difference.

FalsinSoft

@fcarney In my case it's a release

FalsinSoft

Created bug report here

jobusch

@KH-219Design Hey dude, you nearly saved my day. I was trying to use your method to get back some lost source .qml files, because I lost data of my source files but still got the directory it was compiled.

I have exactly the same files as you have and I did it the same way.
Using your example file I get clear readable text, but using my file it is not working, so maybe I have another encoding? Any idea on how I could try to find out my encoding? In Qt Editor it says Windows CRLF (which is only about line endings, right?) Looking into my settings it says utf-8, which didnt work.

Any help would really be appreciated!

This is the output I get:

ê  
VÄo|n2ôZcf¨xäÂTq;õjk§µs’¦J
B.1°1W*½J2Ld{û•äæî\ºÄÌ.À9>½ ¼(Þ±`d¦€	-Ë
V\#|
ë5ýsïTÞÉ{zz½Fýü-êîPŸ‚ÎðÞ?3¯Åºû¯¬ŸEºû_H÷^@ú‹ÔôÎT*ÓÐ¯‡»÷ª¸{Oã^^5¤ÌŠ~^ßìÖ!×"1Ö…hÒ?.t½Öá‚Û‰ÔEâÔü¨×I<Ë;xå^vô2Rq÷~Û„:U&•ß9õËuÕ$òw\Sææ‘° úáqm‘nŠ˜9°+ñ~ñEÑ R-¸•ˆ£RØ•þèûØë£Á

from:

  b = bytearray.fromhex("00 00 02 ea \
  00 \
  00 0a 1b 78 9c bd 95 6d 4f db 30 10 c7 df 23 f5 \
  3b 58 d9 0b 40 db d2 07 b1 09 55 94 09 ba 49 a0 \
  8d 17 ac 6c 7b ed 38 d7 c6 c2 b5 83 7d a1 14 c4 \
  77 df 39 49 97 a4 4f 0c a6 e1 b6 6a 72 ff f3 f9 \
  fc f3 25 27 a7 a9 b1 c8 2e f1 32 93 e2 9a f5 c2 \
  ee 87 d6 8e 6c 18 c3 5f 52 c7 66 b6 5e 1b 1a 8d \
  d6 28 f7 84 da 0d 0f 56 c4 6f 7c 6e 32 f4 5a 63 \
  66 a8 78 e4 c2 54 71 1c 1b 3b f5 6a 6b a7 b5 73 \
  92 a6 4a 0a 8e d2 e8 32 9d 87 d6 0e a3 21 e3 3e \
  9b 72 a9 67 b9 b5 b0 dd 4a 27 23 05 7d 86 36 83 \
  c2 34 93 31 26 7d f6 f1 a0 53 dc 27 20 27 09 f6 \
  d9 c1 61 69 40 89 7e c6 8d bb b2 7b c1 e7 9f ec \
  c2 44 52 01 1b 65 12 81 9d 19 87 c1 be cf c3 bb \
  1a 3d 54 c6 49 3d e9 2f 92 f0 43 90 0d 42 2e 04 \
  a4 08 31 1b b0 31 57 0e 2a bd 4a 32 4c 64 0c 7b \
  fb 95 e4 e6 ee 5c 18 1d ba c4 cc 2e c0 39 3e 81 \
  bd a0 bc 28 12 0b de b1 60 64 a6 80 09 2d cb 0a \
  56 5c 23 13 7c 0a 2c 4b 43 36 54 fe f8 48 76 0c \
  0d bb d6 04 68 6a 2c 84 41 b9 cc e3 22 f9 d1 dc \
  21 4c af 2c 9f fb 25 eb 1b f0 24 cb 4c 2a e3 1a \
  94 b9 6f 9e ae c9 ac 20 29 b8 b1 a2 df b6 e0 e8 \
  37 a6 bf 24 4c f5 24 a8 9c d1 18 85 32 dd 06 97 \
  8d c0 de 82 0d 6a 50 8c 3e 11 28 6f 39 c1 6c 70 \
  5e 62 e9 99 d5 59 2e c9 96 4b 07 db 74 b8 c9 c0 \
  e1 62 a9 ba 67 8e ac 46 af b8 bc e2 d1 29 b7 cb \
  dc 22 6e 2b 43 59 6b 29 b7 a0 31 cc ef 2a 31 e2 \
  e2 7a 62 4d a6 69 d6 77 10 74 8a 13 22 b1 b4 3f \
  3f da 6d 61 94 b1 84 f7 4d f7 d0 7f 82 a6 cf 42 \
  45 cb b5 2b d6 5a f2 78 ac 55 58 ca 45 5e b0 9d \
  ca e6 77 92 21 36 8b 60 b1 21 e4 51 a7 69 45 b8 \
  a3 e7 25 38 1a 1b 5f 77 7e f1 c1 ee 2c a1 03 dc \
  3d 0e d8 db f2 6c bf 52 3d 5a ee 82 7d b2 04 47 \
  6d ef 7b 1c 2c d0 3d 1b c1 d2 56 09 72 28 32 eb \
  77 7a ae 63 b8 63 83 01 eb b0 4f 15 20 46 f9 45 \
  8a a2 6f 06 e1 47 ca e3 38 87 d1 eb 2c a7 d6 ce \
  53 0e 23 a3 e2 0d eb 35 fd 73 ef 54 de 81 1a c9 \
  7b 7a 14 7a bd 46 fd fc 2d ea ee 0b 50 9f 82 ce \
  f0 de 3f 33 af c5 ba fb af ac 9f 45 ba fb 5f 48 \
  f7 5e 40 fa 8b d4 f4 ce 54 2a d3 13 d0 af 87 bb \
  f7 aa b8 7b 4f e3 5e 01 5e 35 15 a4 cc 8a 7e 5e \
  df ec d6 17 21 d7 22 31 d6 85 68 d2 3f 2e 74 bd \
  d6 e1 82 db 89 d4 45 e2 d4 fc a8 d7 49 3c cb 3b \
  78 e5 5e 76 f4 32 52 71 f7 7e db 84 3a 81 55 26 \
  95 df 39 f5 cb 75 d5 24 f2 77 1d 15 5c 53 1a e6 \
  e6 91 b0 00 fa e1 71 6d 91 6e 8a 98 39 b0 2b f1 \
  7e 90 f1 45 d1 1c 20 52 2d b8 95 88 a3 52 d8 18 \
  95 fe e8 fb 1b d8 eb a3 c1")

KH-219Design

@jobusch two questions for you:

What operating system are you on? (I'm guessing windows because you mentioned CRLF, but best to be explicit.)
Are you using CONFIG+=qtquickcompiler in your project build, yes or no?

What we determined in this thread previously is that without CONFIG+=qtquickcompiler, the raw QML text is present, and likewise with CONFIG+=qtquickcompiler the plain QML text is still present, but a bunch of other stuff could be inserted around it.

So if qtquickcompiler was part of the configuration when your binary was produced, it could complicate your efforts (but should not thwart them entirely).

KH-219Design

@jobusch As far as I was aware several years ago, Microsoft still had a penchant for preferring UTF-16/UCS-2 over utf8. So that's something to maybe consider.

KH-219Design

@jobusch another thought:

the qt resource compiler can be used to "bake in" more than just QML text. resources can be of arbitrary files types. (e.g. pdf files, graphical assets, other documents).

Maybe you are mistakenly focusing on the wrong batch of hex bytes? Maybe the bytes you grabbed for examination via python are not, in fact, bytes from a QML text file?

If you look at my gist again: https://gist.github.com/pestophagous/bf30369f824a7344d047496002afdc52

You'll see that qt_resource_data is one giant static array, but the resource compiler inserted various comments lines throughout:

// /opt/repositories/client/219/qt-qml-cpp-proj-template/src/lib_app/qml/ImageSvgHelper.qml
...
 // /opt/repositories/client/219/qt-qml-cpp-proj-template/src/lib_app/qml/VersionLabel.qml
...
// /opt/repositories/client/219/qt-qml-cpp-proj-template/src/lib_app/qml/qmldir

Does your file have such comment lines?

Another factual question that I should have asked earlier, for completeness:

What version of Qt are you using?

KH-219Design

@jobusch GOT IT.

today is my most hacking-est day in a while.

So I ended up here: https://en.wikipedia.org/wiki/List_of_file_signatures

Noticed that 78 9C is a marker for zlib.

So I took the python that you shared, chopped off everything that was in front of 78 9C.

That left me with something like:

bb = bytearray.fromhex("78 9c bd 95 6d 4f db 30 10 c7 df 23 f5 \
  3b 58 d9 0b 40 db d2 07 b1 09 55 94 09 ba 49 a0 \
  8d 17 ac 6c 7b ed 38 d7 c6 c2 b5 83 7d a1 14 c4 \

...

I wrote that to a file named abc.zlib

Then ran this (in bash terminal):

zlib-flate -uncompress < abc.zlib  > out.qml

And I now see your code:

...

        TabButton {
            id: tab1
            text: "<font color='white'>" + qsTr("Benutzer") + "</font>"

            background: Rectangle {
                        color: bar.currentIndex == 1 ? "#181818" : "black"
            }
            padding: 20
            //font.bold: bar.currentIndex == 1
            font.pixelSize: 22
        }
        TabButton {
            id: tab2
            text: "<font color='white'>" + qsTr("Einstellungen") + "</font>"

            background: Rectangle {
                        color: bar.currentIndex == 2 ? "#181818" : "black"
            }
            padding: 20
            //font.bold: bar.currentIndex == 2
            font.pixelSize: 22

        }

...

kibsoft

As a workaround we can use

QMAKE_RESOURCE_FLAGS += -threshold 0

for QMake projects. All the files from resources will be compressed and most part of the QML code will be hidden.
The reason why all QML files are included to the final executable when QtQuickCompiler is turned on is described here.

sidsukana

This post is deleted!