Maximum amount of resource files on iOS?
-
(Bit of a "thinking out loud" thread here...)
I've just converted an application from using a bunch of data (~1GByte total) in a filesystem hierarchy to using Qt's resource system.
A few megabytes of it (QML and some icons etc) I compile into the executable, but the bulk of it I build into a couple of standalone resource files (.rcc files seems to be naming convention in Qt docs) and then have the application load them with QResource::registerResource.
This is all working good on Linux and OSX (where I get the rcc files into the app bundle with QMAKE_BUNDLE_DATA). But on iOS the first register of a ~600MByte resource seems to succeed, but then the second one of ~450MByte fails with some
myapp(23333,0x3adef9dc) malloc: *** mach_vm_map(size=478924800) failed (error code=3) *** error: can't allocate regionerror.
A bit of googling finds me this interesting thread: http://stackoverflow.com/questions/13425558/why-does-mmap-fail-on-ios which suggests iOS has a 750MByte address space limit.
That's a pain. Any suggestions for how to work round it? I don't really want to go back to the "naked" files as I want to get this to work on Android too and Android doesn't seem to be nearly as cooperative as OSX/iOS about allowing arbitrary files to be copied into the bundle, so far as I can tell. Breaking up the resources a bit more and trying to select only the needed subset with Q_INIT_RESOURCE/Q_CLEANUP_RESOURCE might just be an option I suppose. Bit messy though.
(There isn't some secret mode of the Qt resource system where it doesn't memory-map the resource files but just accesses them using filesystem semantics - seeks and reads - instead, is there?)
But I should check if this rcc-based approach is actually going to help for android before I pursue it too far; more tomorrow.
-
(Bit of a "thinking out loud" thread here...)
I've just converted an application from using a bunch of data (~1GByte total) in a filesystem hierarchy to using Qt's resource system.
A few megabytes of it (QML and some icons etc) I compile into the executable, but the bulk of it I build into a couple of standalone resource files (.rcc files seems to be naming convention in Qt docs) and then have the application load them with QResource::registerResource.
This is all working good on Linux and OSX (where I get the rcc files into the app bundle with QMAKE_BUNDLE_DATA). But on iOS the first register of a ~600MByte resource seems to succeed, but then the second one of ~450MByte fails with some
myapp(23333,0x3adef9dc) malloc: *** mach_vm_map(size=478924800) failed (error code=3) *** error: can't allocate regionerror.
A bit of googling finds me this interesting thread: http://stackoverflow.com/questions/13425558/why-does-mmap-fail-on-ios which suggests iOS has a 750MByte address space limit.
That's a pain. Any suggestions for how to work round it? I don't really want to go back to the "naked" files as I want to get this to work on Android too and Android doesn't seem to be nearly as cooperative as OSX/iOS about allowing arbitrary files to be copied into the bundle, so far as I can tell. Breaking up the resources a bit more and trying to select only the needed subset with Q_INIT_RESOURCE/Q_CLEANUP_RESOURCE might just be an option I suppose. Bit messy though.
(There isn't some secret mode of the Qt resource system where it doesn't memory-map the resource files but just accesses them using filesystem semantics - seeks and reads - instead, is there?)
But I should check if this rcc-based approach is actually going to help for android before I pursue it too far; more tomorrow.
-
@jsulm : unfortunately the app isn't really "modal" enough that it'd be easy to retrofit selective INIT/CLEANUP of resource subsets. Most of the access is from QML and theres a lot of constructed URLs accessing stuff at things like "qrc:/stuff/"+someProperty+"/"+someOtherProperty+"-"+andAnotherProperty+".png" which can jump around anywhere. Also: I don't think it's true to say "all these resources" are "in memory". With memory-mapped files (which is what Qt appears to be doing with resources on these OSs) the disk blocks are mapped into the address space, but then actually loaded into RAM (the OS's LRU-replacement disk-block cache) on demand by page faulting (and generally some prefetch I think; on Linux you can control that with
madvise). The LRU behaviour would be expected to keep the "current working set" of "hot" resources from the rcc files in RAM. Assuming these mobile OS have inherited enough of their Unix roots to behave to what I'm used to on the big iron anyway.At least some good news: just tried it with the two big rcc files (~1GByte total) on Android and that seems to work fine there (definitely doesn't feel quite as performant or smooth on this S2 Galaxy Tab as the same thing on an iPad though; was seeing that before the rcc files when I was manually file-transferring the "naked" files to the Android file system; I'm slightly suspicious QML
Images withasynchronous: trueproperty set aren't actually asynchronous on Android, because there it's behaving very much like it was on iOS before I put theasynchronous: trues in!... needs more investigation. But that's another story.).Going back to the memory-mapped resources: I just looked at how much RAM the app is using on Android and it's just ~100MByte, despite the ~1GByte of rcc files being live and in use.
What I'll probably do is keep the iOS (and maybe OSX too) builds using the old raw files bundling for one of the rcc files (one of them had a relatively compact list of files and kept the .pro reasonably clean) as those platforms support bundled files better, and use all-resources on Android. Always a bit annoying to have to have things diverge like that between platforms though... more stuff to go wrong on another platform when you change something on one platform.
