Puzzled by QDir and Android filesystem

timday

I have my Qt android app running on a device (a Samsung S2). Hurrah! Except it's not doing much because it can't find its data...

I have a bunch of files (well, directories containing files) copied into the device storage (using the Mac filetransfer utility) and I can see them on the device's MyFiles browser in Device storage->Android->data->org.qtproject.example.myapp->files->...some folders with lots of stuff in them.

In my code I can set a QDir to "/data/user/0/org.qtproject.example.cci_tablet/files" (which is what I get from querying for QStandardPaths::standardLocations(QStandardPaths::DataLocation)[0]) and that returns true for the QDir's exists(). However when I query the dir's entryList() all I get is "." and ".."... no sign of the several folders I think should be there.

If I cdUp() one level, I can see "files" and "cache" (which the file browser on the tablet also shows me). But also a bunch of other stuff like "app_outdex", "code_cache", "qt-reserved-files" and "lib" and "shaders". I could understand the system hiding some stuff from the user though. My problem is more that files visible to the user aren't visible to Qt!

Not sure what's going on here.
Am I in completely the wrong place? I note QStandardPaths::standardLocations(QStandardPaths::CacheLocation)[0] returns (I assume) the adjacent /data/user/0/org.qtproject.example.cci_tablet/cache. QStandardPaths::DocumentsLocation points at a /storage/emulated/0/Documents.
Any pointers how to get QDir to navigate to where my data is greatly appreciated thanks.

Googling finds some stuff (old Jira issue QTBUG-36528 ) about QDir having some problems with files in the assets:/ space, but I'm not using that for these files.

raven-worx

@timday
please show the code for the traversal with QDir

timday

Here (function locateDir below) is code I'm using... the idea is it starts in directory initial looking for a folder named tgtdir. If it doesn't find it it moves up a level and looks there. Repeats until runs out of filesystem. This has its roots in the app's desktop origins where the data might be in various locations relative to the app binary (c.f the more constrained mobile app world) and this exact code has been in use on Linux, OSX, Windows and iOS extensively for quite a while now. In this android (and the iOS case) I wouldn't actually expect that searching to be needed because I should be able to call it with initial pointing straight at the right place using QStandardPaths...

On android I'm calling this with

locateDir("/data/user/0/org.qtproject.example.cci_tablet/files","CciResources")

where initial is obtained from QStandardPaths::standardLocations(QStandardPaths::DataLocation)[0].

I'm assuming this must surely be the same location the Samsung's file-browser shows me:

File browser screenshot

(If dropbox is being uncooperative with embedded image links: here's the URL: https://www.dropbox.com/s/u2wp52kbzmsyqri/Screenshot_samsung_files.png?dl=0 )

... which shows it populated with the directory I'm looking for, but as noted in original post, all it logs as finding for the contents of that folder is "." and ".." (I'd expect it to see the other stuff too of course) so it moves up a level (where it does see cache and some other things) but at that point it's ultimately doomed to fail to find what it's looking for.

QDir locateDir(const QDir& initial,const QString& tgtdir) {

  QDir dir(initial);
  if (!dir.exists()) {    
    std::cerr << "Fatal error in locateDir(" << initial.absolutePath().toLocal8Bit().data() << "," << tgtdir.toLocal8Bit().data() << "): initial directory doesn't exist" << std::endl;
    exit(1);
  }
  while (true) {
    std::cerr << "locateDir: In " << dir.absolutePath().toLocal8Bit().data() << " found: " << dir.entryList() << std::endl;
    if (dir.exists(tgtdir)) {
      if (!dir.cd(tgtdir)) {
        std::cerr << "Fatal error in locateDir(" << initial.absolutePath().toLocal8Bit().data() << "," << tgtdir.toLocal8Bit().data() << "): target exists but is not a directory" << std::endl;
	exit(1);
      }
      return dir;
    }
    if (!dir.cdUp()) {
      std::cerr << "Fatal error in locateDir(" << initial.absolutePath().toLocal8Bit().data() << "," << tgtdir.toLocal8Bit().data() << "): target couldn't be found at any level" << std::endl;
      exit(1);
    }
  }
}

Note that I have an override for operator<<(std::ostream&,const QStringList&). Also, I use the absolutely genius trick described here https://codelab.wordpress.com/2014/11/03/how-to-use-standard-output-streams-for-logging-in-android-apps/ to redirect native code's stdio logging to the android logging system so I can see it with adb logcat.

The output is (some additional spew replaced by ..., some interesting things like the QStandardPaths retained):

09-29 12:48:10.807 16735 16750 D cci-tablet: Android stdio logger redirection started
...
09-29 12:48:10.847 16735 16750 D cci-tablet: Application path: /data/app/org.qtproject.example.cci_tablet-1/lib/arm
09-29 12:48:10.857 16735 16750 D cci-tablet: Documents location (1st):/storage/emulated/0/Documents
09-29 12:48:10.857 16735 16750 D cci-tablet: Data location (1st):/data/user/0/org.qtproject.example.cci_tablet/files
09-29 12:48:10.857 16735 16750 D cci-tablet: Cache location (1st):/data/user/0/org.qtproject.example.cci_tablet/cache
09-29 12:48:10.857 16735 16750 D cci-tablet: locateDir: In /data/user/0/org.qtproject.example.cci_tablet/files found: ".",".."
09-29 12:48:10.857 16735 16750 D cci-tablet: locateDir: In /data/user/0/org.qtproject.example.cci_tablet found: ".","..","app_outdex","cache","code_cache","files","lib","qt-reserved-files","shaders"
09-29 12:48:10.857 16735 16750 D cci-tablet: locateDir: In /data/user/0 found: 
09-29 12:48:10.857 16735 16750 D cci-tablet: locateDir: In /data/user found: 
09-29 12:48:10.857 16735 16750 D cci-tablet: locateDir: In /data found: 
09-29 12:48:10.857 16735 16750 D cci-tablet: locateDir: In / found: ".","..","acct","bt_firmware","cache","carrier","config","d","data","default.prop","dev","dsp","efs","etc","file_contexts","firmware","firmware-modem","fstab.qcom","init","init.carrier.rc","init.class_main.sh","init.container.rc","init.environ.rc","init.mdm.sh","init.qcom.bms.sh","init.qcom.class_core.sh","init.qcom.early_boot.sh","init.qcom.factory.rc","init.qcom.rc","init.qcom.sh","init.qcom.syspart_fixup.sh","init.qcom.usb.rc","init.qcom.usb.sh","init.rc","init.recovery.qcom.rc","init.rilchip.rc","init.rilcommon.rc","init.target.rc","init.trace.rc","init.usb.configfs.rc","init.usb.rc","init.zygote32.rc","init.zygote64_32.rc","knox_data","mnt","oem","persdata","persist","postrecovery.do","preload","proc","property_contexts","publiccert.pem","root","sbin","sdcard","seapp_contexts","sepolicy","sepolicy_version","service_contexts","storage","sys","system","tombstones","ueventd.qcom.rc","ueventd.rc","vendor","verity_key"
09-29 12:48:10.857 16735 16750 D cci-tablet: Fatal error in locateDir(/data/user/0/org.qtproject.example.cci_tablet/files,CciTabletResources): target couldn't be found at any level

Investigations continue...

raven-worx

@timday said in Puzzled by QDir and Android filesystem:
you may want to try QDir::AllDirs as parameter of entryList(). Maybe it's a bug on Android.
Also make sure you have the permissions (READ_EXTERNAL_STORAGE mabye)

Note that I have an override for operator<<(std::ostream&,const QStringList&)

you could also simply use qDebug()/qWarning()/... which also get output to logcat

timday

The manifest XML (which I've not touched yet, just using whatever qmake or androiddeployqt or whatever creates it emits) certainly seems to have a WRITE_EXTERNAL_STORAGE, and that apparently implicitly includes read too.

The code has a long history and is littered with std::cerr << from its earlier days. If I was starting again from scratch I'd use 100% qDebug().

Interestingly, just tried putting my app's directories of files in ...org.qtproject.example.cci_tablet/files/Documents/ instead, but then trying to call my locateDir function with that as starting point fails immediately with the QDir initialized to /data/user/0/org.qtproject.example.cci_tablet/files/Documents failing its .exists() test... which seems like evidence the location I'm seeing in the Samsung file browser is nothing to do with the locations QStandardPaths is pointing me at.

Will try the QDir::AllDirs thing thanks.

raven-worx

@timday
check if it's a symlink
Also make sure to call QDir::exists() with an absolute path. Otherwise the current directory is used, which changes e.g. with the cd() call.

timday

Invoking entryList with QDir::AllDirs (or QDir::System for that matter) still doesn't find anything there but "." and "..".

Any paths in play should all be absolute in that they all start with "/" (although on these mobile devices with all their sandboxing I'm not convinced the notion of absolute is as absolute as it used to be on good old desktop OSs). For good measure I added a makeAbsolute() to the newly constructed QDir dir(initial) and checked it's not reporting an error.

timday

Hmmm finding this interesting reading: https://www.reddit.com/r/Android/comments/496sn3/lets_clear_up_the_confusion_regarding_storage_in/ , at least for some context of why the Android filesystem is the way it is (I haven't added an SD card to this device yet so most of what it describes is irrelevant).

timday

OK this looks promising: from an adb shell, I can ls /sdcard/Android/data/org.qtproject.example.cci_tablet/files and see the directories and files I've copied there. This is clearly somewhere completely different from any of the QStandardPaths I've looked at... hmmm it occurs to me I've only been looking at [0] the first thing returned by QStandardPaths (because that's all you get on all the other OSs I've used). But with Android's potentially multiple locations (see reddit link in previous post) I wouldn't be surprised if there's an additional one (we'll see :^)...

OK, result!

QStandardPaths does indeed return two data locations (also 2 cache locations):

09-29 16:09:54.987 23553 23568 D ClimateFromSpace: Data locations:"/data/user/0/org.qtproject.example.cci_tablet/files","/storage/emulated/0/Android/data/org.qtproject.example.cci_tablet/files"

and using the second one of those does get me a location with the files I'd copied to the device. With those now found, all the QML (including some video usage) and C++ plugins (OpenGL) seem to be pretty much Just Working as intended (maybe some resolution/scaling issues; this app has only really targeted iOS until now); now that's the reason I like Qt!

C++ plugin

Video