Good strategy to handle multiple GB of application data for an iPad app? [SOLVED]

timday

I have a Qt 5.2 application running nicely on Linux and MacOS built using QQuickView and some QML (it also makes use of QQuickFramebufferObject plugins and OpenGL).

The app is basically a "player" for browsing a directory structure of several GByte of scientific data; I'll call this the app's "content".

On the desktop platforms, I can just supply users with the application (i.e as a .app bundle or .deb), and they can get hold of the accompanying content as a .zip and unpack it at some prescribed location the application looks in. The content and the application evolve on different timescales, and there have been application updates put out without needing content changes, and updates to the content which haven't needed a new application.

I'm now looking into an iPad port of this. So far so good: I have a trivial app using the same bits of Qt on the iPad simulator (main issue encountered being converting some old-school OpenGL to OpenGLES). But what to do with the mass of "content" worries me. From reading the Apple's "info on the iOS file system":https://developer.apple.com/library/ios/documentation/FileManagement/Conceptual/FileSystemProgrammingGuide/FileSystemOverview/FileSystemOverview.html, it's clear the "content" data will have to live somewhere in the app sandbox, and my understanding is that general purpose file copying tools just don't/can't exist for the platform. So the application will have to be responsible for populating the "content" area somehow.

The most obvious approach is to move the "content" into the app bundle ( QMAKE_BUNDLE_DATA ) but I'm worried about the scalability of this, not least what it will do to development cycle times, because it will mean copying the massive amount of content onto the device every time a new application build is done, and distributing enormous multi-GB .app bundles to users who actually just needed a few MBs of new application installing.

Alternatively, having all the content out on the net at some URLs and loaded on demand isn't going to work very well either: the application can jump around chewing through 100MBs and frequent waits to download that will be annoying: the content really has to be "pre loaded" on the device before it's used in anger.

How do "real" apps tackle this sort of problem? (Although note that at this point the aim just to make a limited number iPads available to get the content dataset in front of a wider audience; this doesn't aspire to be a mass market app store thing).

Random thoughts:

Does Qt provide anything to help? (e.g QNetworkDiskCache looks like it might be useful, but what I'm looking for probably looks more like a file sync I think).
If the iPad had an sdcard slot (it doesn't) and that could be read from then occasionally updating a copy of content from that could work well (e.g this seems to be how many car satnav systems expect to get new mapping).
iOS obviously has some "file sharing" support http://support.apple.com/kb/ht4094 which can transfer from/to iTunes. What documentation I've seen seemed to imply only a limited number of media types are supported but I need to look at this one more closely; I'm unclear whether the app needs to cooperate beyond ticking some UIFileSharingEnabled box in the xcodeproj settings, or whether it actually needs to do things like respond to some incoming signal and do something (in which case, any helpful/corresponding bits of Qt?)

Any pointers/suggestions gratefully considered; I've been Qt-ing on the desktop for years but this mobile thing is quite fresh...

Xander84

Hi, as far as I know 2GB is the max size of iOS apps (60mb for the executable), so you might need to access the data over the network, since space is limited on mobile devices anyway it should be the better option.
If you are on the local network maybe you can provide the data from a local computer or other network device, if the speed of the WiFi is not fast enough for your app might think about optimizing the data stream to your app or whatever you transfer there?

I have not that much experience with native iOS apps, but to my knowledge you might be way better of with Android apps and SD card access and there are other ways on android, just saying :D

timday

OK I'm now registered as an iOS developer so I can actually try things out (and I do indeed have a simple Qt app running on an iPad with scarcely more effort than it takes to build for desktop platforms. Amazing!)

My understanding is that I should be able enable file sharing for an app, and then use iTunes to move this bulk data to the app's sandbox's Documents folder...

If I add
@<key>UIFileSharingEnabled</key><string>YES</string>@
within the @<dict>...</dict>@ scope of the qmake generated Info.plist, then I see "Application supports iTunes file sharing" : "YES" in xcode's Info for the target. But then iTunes doesn't "see" the app (ie list it somewhere as something which can have files moved to it). Is there some extra magic needed? (For example, I note that under the "Custom iOS Target Properties" the file sharing thing appears in, there is an empty "Document types" list; do I need to fill that out with something too?)

Puzzled...

Update: Ooops, above was a bit premature. At the bottom of the iTunes screens for the iPad, on the Apps tab, I find "File sharing... The apps listed below can transfer documents between your iPad and this computer", and my app listed there with its Documents folder. Interestingly, I note I can't just transfer a whole folder of data to Documents as a "document", it has to be files (although I can multi-pick). Which means I'll have to consider either flattening things which used to be (for the desktop version) in e.g xyz/abc/foo.jpg to xyz-abc-foo.jpg, or cramming them into some single file virtual filesystem (.qrc style);. Hmmm...

timday

Well I have my data compiled by rcc into a .rcc file just under 2GByte big. My test app has been modified to read files from that and works great on Linux and Mac Desktop builds, and I can load the .rcc file to the iPad Documents/ for the app using iTunes.

However, on the iPad, the attempt to Resource::registerResource on the the .rcc file fails with

@test(150,0x3d03c18c) malloc: *** mach_vm_map(size=1938997248) failed (error code=3) *** error: can't allocate region@

I'm not sure whether this is because there's not enough RAM, or not enough address space (how do I tell for sure whether an iPad is 64bit OS or not anyway? I do know it's not an "A7" processor, but it is running iOS 7.1.1).

I'm a little surprised to see a problem as QResource's API explicitly uses qint64 as the return type for it's size() method.

timday

OK I have a workable strategy:

pack my Data folder with its GBytes of files into a tar file (easily done with tar commandline utility on Mac or Linux).
Upload Data.tar to iPad app's Documents/ folder
When the app starts, if it finds a Documents/Data.tar it unpacks it to app's Library/Data/... and deletes the tar file so it doesn't unpack it again next time it's started.
App runs using normal filesystem access to Library/Data/ files.
Updates to the app data can be distributed without reinstalling the app.

There is conveniently some public domain tarfile unpacking code at "http://www.opensource.apple.com/source/libarchive/libarchive-23/libarchive/contrib/untar.c":http://www.opensource.apple.com/source/libarchive/libarchive-23/libarchive/contrib/untar.c which was easy to modify to be a function call, added to the Qt project and worked fine on all of Linux/MacOSX and ipad. At some point I'll do a more polished version which can drive some sort of progress bar/spinner while it's unpacking (c.f current spew to stderr).

timday

One detail remained to be overcome: the iPad doesn't like it if apps take a long time to startup, and kills them. This meant I couldn't just force users to stare at a blank screen while the unpacking happened before the app started running an event loop. So I wrapped the tar unpacking code in a QQuickItem subclassed "QUntar", had it do the unpacking asynchronously (QtConcurrent::run, status reported back by signals; more on this at http://stackoverflow.com/questions/23438044/how-to-communicate-a-progresstext-from-a-qtconcurrentrun-function-or-similar ) and hosted it on the apps first QML page (a splashscreen of sorts, although nothing to do with any splashscreen support on the iOS platform). Just to give some flavour, here's the QML which logs the unpacking progress and switches to a new page when done ("host" is how my apps' QQuickView subclass exposes iteslf as a context property to my QML):

@
Text {
anchors.fill: parent
text: "Welcome to Content Explorer\n"
color: "#00ff00"
smooth: true
SequentialAnimation {
running: true;
PauseAnimation {duration: 1000} // You will admire the splash screen
ScriptAction {script: untar.go()}
}

function log(msg) {
  //console.log("Log: "+msg)
  text=text+msg+"\n"
  while (paintedHeight>height) {
    text=text.split("\n").slice(1).join("\n")
  }
}

QUntar {
  id: untar
  onStatusMessageChanged: parent.log(statusMessage)
  onFinished: {
    if (success) {
      host.load("play.qml");
    } else {
      parent.log("Sorry, something bad happened unpacking, cannot continue");
    }
  }
  function go () {
    var tarfilename=host.newContentPath+"Content.tar";
    if (host.fileExists(tarfilename)) {
      parent.log("Content tarfile found");
      unpack(tarfilename,host.contentPath,"Content",true);
    } else if (host.directoryExists(host.contentPath+"Content")) {
      parent.log("Unpacked content found");
      host.load("play.qml");
    } else {
      parent.log("No content found, you need to provide a content tar file");
    }
  }
}

}
@

so newContentPath is set to Documents/ and contentPath is Library/ and the app will, if it finds one, unpack a Documents/Content.tar to Library/Content/ and then erase the .tar, else it'll attempt to use an existing Library/Content/, else it'll complain.