Sandbox app for the Mac App Store with Qt 5.8 and QtWebEngineProcess

Wolosocu · wrote on 22 Apr 2017, 03:57

I have run into the same problem that a few others have run into however their fixes are either ambiguous or did not help me.

My application uses QtWebEngine. I run macdeployqt on my app and all is well. However, once I sign the .app bundel, the app loads and everything EXCEPT the QWebEngine portions look fine. The logs say what the problem is pretty clearly:

QtWebEngineProce(1818) deny forbidden-sandbox-reinit

And the process never starts. So obviously this has something to do with how I am signing the everything.

My build process is pretty simple:

I build my Release build (MyProgram.app) through Xcode, with no signing.
I run the resulting bundle through macdeployqt. This puts the QtWebEngineProcess into my app's bundle, as well as many other files.
Next, I sign that QtWebEngineProcess specifically with these QtWebEngineProcess-specific entitlements:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>com.apple.security.app-sandbox</key>
	<true/>
	<key>com.apple.security.inherit</key>
	<true/>
</dict>
</plist>

My command looks like this (Python code)

subprocess.call([r'/usr/bin/codesign', 
    r'--deep',
    r'--verify',
    r'--verbose',
    r'--sign',
    appSignature,	
    r'-i',
    r'com.yourcompany.QtWebEngineProcess',
    r'--entitlements',
    webProcessEntitlementsFile,
    bundleFolder + '/MyProgram.app/Contents/Frameworks/QtWebEngineCore.framework/Versions/5/Helpers/QtWebEngineProcess.app/Contents/MacOS/QtWebEngineProcess'

Then I the only other thing I sign is my app's main bundle:

subprocess.call([r'/usr/bin/codesign', 
    r'--deep',
    r'--force',
    r'--verify',
    r'--verbose',
    r'--sign',
    appSignature,	
    r'--entitlements',
    entitlements,
    bundleFolder + '/MyProgram.app'
])

Signing goes through without an error. Yet, when I run the program my C++ and Qml widgets all render perfectly. The QWebEngine based control never renders anything. Presumably because QtWebEngineProcess fails to run. Which I can see in Console as:

4/21/17 23:55:06.020 sandboxd[135]: ([4543]) QtWebEngineProce(4543) deny forbidden-sandbox-reinit
4/21/17 23:55:06.023 sandboxd[135]: ([4544]) QtWebEngineProce(4544) deny forbidden-sandbox-reinit

How exactly should everything in the package be signed? Is doing one signing of MyProgram.app ok? Do I need to sign everything individually? Maybe certain frameworks need different entitlements? I don't know. Any help would be appreciated!

Is there any documentation or psuedo code for writing these build/signing procedures?

Thank you!

SGaist · wrote on 22 Apr 2017, 22:02

Hi,

The QtWebEngine macOS platform notes contains the key to your trouble.

Wolosocu · wrote on 23 Apr 2017, 02:32

@SGaist said in Sandbox app for the Mac App Store with Qt 5.8 and QtWebEngineProcess:

Hi,

The QtWebEngine macOS platform notes contains the key to your trouble.

This looks promising. My only question is, how do I specify that option when building Qt? If I look here (http://doc.qt.io/qt-5/osx-building.html) I see no mention of running qmake. Should I just run that qmake-command after configure and before make?

SGaist · 23 Apr 2017, 12:38

You don't need to rebuild the whole Qt, just the module itself. That's why qmake is mentioned in the notes.

Wolosocu · S SGaist 23 Apr 2017, 07:28

@SGaist said in Sandbox app for the Mac App Store with Qt 5.8 and QtWebEngineProcess:

You don't need to rebuild the whole Qt, just the module itself. That's why qmake is mentioned in the notes.

Ok, so then:

./configure
./qtbase/bin/qmake WEBENGINE_CONFIG+=use_appstore_compliant_code
make module-qtwebengine

Sound about right?

Wolosocu · wrote on 23 Apr 2017, 13:05

I am still unable to get it running. Here's what I did:

Last night I downloaded the Qt source and run the following commands in order:

./configure (and accepted all the license stuff)
./qtbase/bin/qmake WEBENGINE_CONFIG+=use_appstore_compliant_code (using the qmake that ./configure must've built)
make --jobs=16 (this was before I saw that I only need to rebuild the QtWebEngine module)

This ran overnight. I woke up this morning and this completed overnight. Great, so then I:

sudo make install

And now I have Qt in /usr/local/Qt5.8.0. Perfect. So I reset my CMake paths (I'm using CMake), generate the Xcode project and verify through Xcode's settings that I'm linking this new Qt build (which I am). Excellent., so I build the .app.

It gets built, no problem. Then I run my packaging script which does the following:

chdir into macdeployqt's folder, which may or not may not be necessary some people on Stackoverflow mentioned they were having trouble without doing this, so I make sure to do it.
Run macdeployqt <fullpath>/MyApp.app -appstore-compliant -qmldir=<my-projects-qml-folder> (Making sure to use the macdeployqt that I just built.)
chdir back to where I was before
Run /usr/bin/codesign --deep --sign <signature> --entitlements <entitlements-file> MyApp.app

Code signing reports no errors and runs fine. I launch the app and I see the same results. And Console reports the same thing:

4/23/17 09:00:49.103 sandboxd[135]: ([14028]) QtWebEngineProce(14028) deny forbidden-sandbox-reinit
4/23/17 09:00:49.122 sandboxd[135]: ([14029]) QtWebEngineProce(14029) deny forbidden-sandbox-reinit

So I tried inserting a step before the last one:

/usr/bin/codesign --deep --sign <signature> --entitlements <web-process-entitlements-file> <path-to>/QtWebEngineProcess.app

But still, the same problem persists. Suggestions?

Wolosocu · wrote on 23 Apr 2017, 13:20

I think my problem is that I'm not setting up the config right. If I run the following:

./qtbase/bin/qmake WEBENGINE_CONFIG+=no_spellcheck QtWebEngine

I see:

Info: creating cache file /Users/Me/Downloads/qt-everywhere-opensource-src-5.8.0/.qmake.cache

Running configuration tests...
Done running configuration tests.

Configure summary:

Qt WebEngine:
  Proprietary Codecs ..................... no
  Spellchecker ........................... yes
  ALSA ................................... no
  PulseAudio ............................. no

Running configure tests
Checking for alsa... no
Checking for khr... no
Checking for libvpx... no
Checking for snappy... no
Checking for srtp... no
Configurable features:
  Proprietary codecs (H264, MP3) ..... Not enabled         (Default, enable with WEBENGINE_CONFIG+=use_proprietary_codecs)
  Geolocation ........................ Enabled
  Mac App Store Compliant ............ Not enabled         (Default, enable with WEBENGINE_CONFIG+=use_appstore_compliant_code)
Native Spellchecker .............. Not enabled         (Default, enable with WEBENGINE_CONFIG+=use_native_spellchecker)

And I see the spellchecker is still enabled. So yeah, I need to figure out how to set this config item and then build.

Wolosocu · wrote on 23 Apr 2017, 14:15

I deleted the .qmake.cache file in my Qt source folder and then ran:

./qtbase/bin/qmake WEBENGINE_CONFIG+=use_appstore_compliant_code QtWebEngine

And I saw the output I was hoping to see:

Info: creating cache file /Users/Me/Downloads/qt-everywhere-opensource-src-5.8.0/.qmake.cache

Running configuration tests...
Done running configuration tests.

Configure summary:

Qt WebEngine:
  Proprietary Codecs ..................... no
  Spellchecker ........................... yes
  ALSA ................................... no
  PulseAudio ............................. no

Running configure tests
Checking for alsa... no
Checking for khr... no
Checking for libvpx... no
Checking for snappy... no
Checking for srtp... no
Configurable features:
  Proprietary codecs (H264, MP3) ..... Not enabled         (Default, enable with WEBENGINE_CONFIG+=use_proprietary_codecs)
  Geolocation ........................ Enabled
  Mac App Store Compliant ............ Enabled
Native Spellchecker .............. Not enabled         (Default, enable with WEBENGINE_CONFIG+=use_native_spellchecker)

With the most important part being Mac App Store Compliant ............ Enabled. I tried then running make module-qtwebengine but it returned an error saying there is no such make target. So, I went ahead and ran make and it looks like it's just rebuiling QtWebEngine. Once it's done I'll try it out. (Keeping my fingers crossed).

Wolosocu · wrote on 23 Apr 2017, 14:48

And now I got the following compiler error:

../../../qtwebengine/src/webengine/plugin/plugin.cpp:140:10: fatal error: 'plugin.moc' file not found
#include "plugin.moc"
         ^
1 error generated.
make[3]: *** [.obj/debug/plugin.o] Error 1
make[2]: *** [debug-all] Error 2
make[1]: *** [sub-webengine-plugin-make_first] Error 2
make: *** [sub-src-make_first] Error 2

For giggles I've make clean and am trying make again. Guess I'll know in a couple hours.

Wolosocu · wrote on 23 Apr 2017, 15:08

make finished quicker than I thought. As far as I can tell QtWebEngine we rebuilt and I see the config option being passed around all over the build process. For example a snippet:

( test -e Makefile.gyp_configure_host || /Users/Me/Downloads/qt-everywhere-opensource-src-5.8.0/qtbase/bin/qmake -o Makefile.gyp_configure_host /Users/Me/Downloads/qt-everywhere-opensource-src-5.8.0/qtwebengine/src/core/gyp_configure_host.pro WEBENGINE_CONFIG+=use_appstore_compliant_code ) && /Applications/Xcode.app/Contents/Developer/usr/bin/make -f Makefile.gyp_configure_host 
make[4]: Nothing to be done for `first'.

But yet the problem persists:

4/23/17 11:03:34.000 kernel[0]: Sandbox: QtWebEngineProce(30098) deny(1) forbidden-sandbox-reinit

I'm still not convinced I'm building QtWebEngine how I should be Google'ing is not being helpful.

What am I doing wrong?

Wolosocu · wrote on 23 Apr 2017, 16:10

Ok, I think I figured out that qmake QtWebEngine will create makefiles for the QtWebEngine module only. But even when I build the makefiles using qmake WEBENGINE_CONFIG+=use_appstore_compliant_code QtWebEngine and then build the module, I still see the QtWebEngineProcess.app in the QtWebEngineCode.framework. Is this expected? If so, then how it should be signed? How should my bundle in general be signed?

SGaist · 23 Apr 2017, 20:21

There's no need to configure Qt, you can use the qmake from your pre-built package if you'd rather use that one.

macdeployqt has two options that will be of interest:

codesign=<ident> : Run codesign with the given identity on all executables
appstore-compliant: Skip deployment of components that use private API

On a side, you're likely in possession of a multi-core machine, to speed things up when building you can do make -jX where X is (2 * number_fo_core) + 1,

Wolosocu · S SGaist 23 Apr 2017, 20:14

@SGaist So I should use macdeployqt to sign the bundle instead of doing it manually? I will try that.

Also, will QtWebEngineProcess.app still be included in the version of QtWebEngineCore built with the use_appstore_compliant_code?

Wolosocu · wrote on 23 Apr 2017, 20:26

And if I'm using macdeployqt to sign the bundle, then how do I specify the entitlements file?

SGaist · 23 Apr 2017, 20:35

The switch doesn't remove the build of the helper application, it disable some code that is using APIs not authorised in App Store applications.

Wolosocu · S SGaist 23 Apr 2017, 20:31

@SGaist Thank you.

I still do not understand how using macdeployqt to sign the bundle will apply an entitlements file for sandboxing, though.

Wolosocu · wrote on 23 Apr 2017, 21:33

I am still unable to get this to work. I am seeing a new error on Console, though:

4/23/17 17:20:31.148 QtWebEngineProcess[2551]: [0423/172031:ERROR:mach_port_broker.mm(43)] bootstrap_look_up: Permission denied (1100)

I have three steps I am toying with:

Run macdeployqt with -codesign as suggested.
Manually codesign QtWebProcessEngine.app, with the entitlements file from above.
Manually codesign MyApp.app bundle, with my app's entitlements file.

I have tried various combinations, such as just #1, or #1 and #3. I've tried switching the order of #2 and #3. I've even tried signing QtWebProcessEngine.app with my bundle identifier.

I am running out of ideas. Please help!

Wolosocu · S SGaist 23 Apr 2017, 20:31

@SGaist said in Sandbox app for the Mac App Store with Qt 5.8 and QtWebEngineProcess:

The switch doesn't remove the build of the helper application, it disable some code that is using APIs not authorised in App Store applications.

Do you know if these API were present in 5.7? My next thought is to try this with 5.7 since I am having so many problems with 5.8.

Wolosocu · wrote on 24 Apr 2017, 20:43

I was finally able to get my app to run in sandbox mode with Qt 5.6.2. I have not tried 5.7 yet, but I did have to reduce the version of QtQuick I was using from 2.7 to 2.6. Luckily I am not using any 2.7 specific features.

I was also able to get my app past the Application Loader portion of the app submission. It hasn't been approved yet but we'll see.

I wanted to mention a few things for anyone else that might have a problem with this.

macdeployqt's -codesign option did NOT work. The only way I was able to sign my app correctly was to write a script that manually walked through my app bundle and called codesign on every binary with the entitlements I wanted/needed.
macdeployqt also seems to create a lot of *.dSYM files. This is something I noticed in with 5.8 as well and Apple's Application Loader will reject these. I didn't pay much attention to the error but I just made sure my packaging script deleted them (before signing).
QtWebProcessEngine.app 's Info.plist file needs to be changed before you sign it. The CFBundleIdentifier needs to match that of your signature.

Like I said, my app has only passed Application Loader's loading process. I've had problems in the past with Qt and the App Store system (see http://stackoverflow.com/questions/25694066/submitting-qt-5-3-app-for-review-on-mac-app-store) before I could even submit the app for review, so I'm skeptical that I'm out of the woods yet.

I'll keep this thread updated.

Wolosocu · wrote on 25 Apr 2017, 03:25

Attempting Qt 5.7.1 brought its own problem. Namely that on startup the application gives the error:

This application failed to start because it could not find or load the Qt platform plugin "cocoa"
in "".

Available platform plugins are: cocoa.

Reinstalling the application may fix this problem.
Abort trap: 6

I did a little research and tried a couple minor things but nothing worked.

Meanwhile my app was rejected because of an unused sandbox permission. So I've corrected that and resubmitted a new build still using Qt 5.6.2.

Really wish I had been able to get 5.8 to work or even 5.7.