What's the maximum rcc size (or total rcc size) on Android 7.0?

[timday]

I have a Qt app with a couple of crazy big rcc files (there's a mass of images and videos baked into them): one is 457MByte and the other is 738MByte (and the signed release .apk is 1.2Gbyte). The app registers these with QResource::registerResource pretty much first thing in its main.

When my Samsung Tab S2 (the only device the app has really been tested with) was on whatever version of Android it came with new last year (6.0.?) and this did actually work fine (I also do desktop builds of it on 64bit Linux and OSX and those work fine too).

Having just updated the device to Android 7.0 (became available from Samsung back in mid-April) the app is now failing with a terminate called after throwing an instance of 'std::bad_alloc' exception from the QResource::registerResource call on the larger rcc file (if I do the smaller one first, it can succeed for the smaller one and then fail for the bigger one; but registering the bigger one always fails whether it's first or second).

Anyone got any advice on how to proceed or insight into what's going on here? My understanding is that rcc files are memory mapped into the app's address space (and presumably the bad_alloc arises from attempting to memory map such a huge amount of stuff)...

• I could break the rcc files into more smaller ones. This might help if there's a limit on the size of any individual rcc file, or if the failure happens because of a lack of contiguous address space (surely not in these 64bit days though?) but not if the total is the problem.

• I could move some of the rcc content out to "ordinary" files... but that maybe a bit of a pain because previous experience with assets: is that it's a half-baked excuse for a filesystem, doesn't really do directory hierarchy and isn't really up to dealing with anything but dropping a few .rcc files in it.

(Tried building the app with all of Qt 5.7, 5.8 and 5.9 and that makes no difference, it seems to be the device's Android version which causes problems for my large rcc files).

The app is at https://www.amazon.com/European-Space-Agency-Climate-from/dp/B01NBKKHYK (oh dear... looks like its being punished by reviewers for this issue too).

[SGaist]

Hi,

I'd recommend bringing this to the Qt Android development mailing list. You'll find there the platform developers/maintainers.

As for the reviews, not wanting to sound dark but better get used to it. Application developers get shot down pretty easily even for "new OS beta released yesterday at midnight and the application doesn't run on it yet properly".

[ekkescorner]

@timday your app is running on my Google Pixel C Tablet (Android 7.1.2) without crashing

but I never would deliver a APP with 1.2 GB size
APPs of this size can easy cause problems on many devices

I only would deliver small Teaser videos and let the user download on demand
then user can decide if and where Videos should be stored

I immediately deleted your APP after testing and I can imagine some users won't wait until 1.2 GB are downloaded. download took some time even on my very fast connection

[timday]

Thanks for the feedback! Indeed I'm rather dubious about the app size myself... but what the customer wants the customer gets... and they really really didn't want to be reliant on any access to online resources to use the app. Bear in mind the app is a derivative of a desktop version where big installs are more tolerated. Now I've been to enough talks on mobile development to hear about "mobile first" being a good idea... unfortunately in this case that's not the order it happened in. I suppose to some extent you could think of it as the eBook equivalent of one of those enormous unwieldy world atlas books... but then how practical are those in these days of Google Earth etc?

Anyway, app practicality aside... I'm relieved to report that breaking up the huge 738MByte rcc file into several smaller rccs (largest ones just over 100MByte) seems to result in the app running fine. So guessing the problem wasn't a limit on the total amount of memory-mapped resources, but some sort of issue with fragmentation of address space and maximum contiguous allocation. A little surprised given I thought Android was all 64-bit these days (this is more the sort of problem I associate with 32bit Windows development a decade ago), but then I understand very little about what wizardry is going on to get Qt QML&C++ apps to work on such a Java-ey platform.

Hopefully an update to the appstore soon... if my collaborators have attended to the licensing obligations!

[timday]

FWIW: the iOS build was also exhibiting very similar issues to the Android one with two huge multi-hundreds-of-megabytes rcc files... and breaking those up into smaller chunks helped the iOS build too.

(We didn't run into the issue on iOS originally because we simply bundled the files rather than baking them into rccs... but the purely file-based approach seemed to have problems with Androids "assets" system so we moved to rccs instead, and to rccs on iOS too to keep the code and the approach the same across platforms).

[timday]

Apparently still having startup problems on some Android devices (unfortunately no information from Amazon's testing as to what they are!), so bit the bullet and implemented a register-on-demand scheme in the app. App logic is such that it can actually can only ever use resources from 2 of the bulk of the rccs simultaneously (and the app knows what it's about to use), so used a simple LRU-eviction scheme to unregister recently unused rccs once more than a few have been loaded. Turned out to be easier than expected and seems to work rather well (although not knowing what devices I'm trying to fix it's hard to know whether it's a real solution).

Also made some manifest changes per https://developer.android.com/guide/practices/screens-distribution.html & https://developer.android.com/guide/topics/manifest/compatible-screens-element.html to limit it to tablet-size devices, on the assumption those are more likely to be beefy enough to handle this behemoth.

[timday]

May finally have achieved some sort of victory in the ongoing battle with Android...

Started using AWS' DeviceFarm to try out more feeble/older devices than the Galaxy Tab S2 I've been testing on (where the app always Just Works).

Fails all seemed to be implicated with the resource-file loader's mmap-ing of my biggest rcc file, which remained above 300MByte. Couple of interesting SO posts https://stackoverflow.com/q/33897711/24283 and https://stackoverflow.com/q/30180268/24283 seem to confirm that actually there really isn't as much contiguous address space available as you might hope. Since this large rcc was simply occupied by a handful of video files, I changed those back to be simply assets:- bundled files; that left me with (at least at startup) a largest rcc of 80MByte, and an initial total of little over 100MByte of rcc mapped. Some more will get loaded later (by the previously mentioned LRU-caching scheme) but there's nothing much bigger than 100MByte and hopefully if it was the contiguous address space was the fundamental problem rather than total mapped it should be ok. Amazon's review claims this latest version passed on all 14 of the Amazon Fire devices they test on (no 3rd party devices actually tested it seems!), so should be in their appstore soon.

DeviceFarm is quite amazing; I could see it becoming an expensive habit once the free "starter" time allocation is used up. Must look into how best to test Qt/QML apps on it though... so far "fuzz testing" seems to be the only useful option (all the others seem very Java-world oriented), but it's a complete lottery whether it gets into any bits of the app beyond the "front page".