Display Gstreamer video on Android



  • Hi!
    I try to display video on Android using Gstreamer like on other platforms:

            GstElement *pipeline;
            GError *error = NULL;
            pipeline = gst_parse_launch("videotestsrc ! glimagesink", &error);
            if (!pipeline) {
                ui->label->setText("error");
                return;
            }
            if(error != NULL){
                qDebug("GST error: ");
                qDebug(error->message);
            }
            else{
                qDebug("GST without errors");
            }
    
            gst_element_set_state(pipeline, GST_STATE_READY);
    
            GstElement *video_sink = gst_bin_get_by_interface(GST_BIN(pipeline), GST_TYPE_VIDEO_OVERLAY);
            if (!video_sink) {
                qDebug ("Could not retrieve video sink");
                return;
            }
            gst_video_overlay_set_window_handle(GST_VIDEO_OVERLAY(video_sink), this->ui->playback_widget->winId());
    	//playback_widget - QOpenGLWidget
    
            gst_element_set_state(pipeline, GST_STATE_PLAYING);
    

    But this code doesn't works, after gst_video_overlay_set_window_handle() function I get:
    F libc : Fatal signal 11 (SIGSEGV), code 1, fault addr 0x5e in tid 3154 (gstglcontext)

    Maybe I need to get the pointer to ANativeWindow? But how can I get this with Qt on Android?
    Anyway I need to display video on Android using Gstreamer, so any help will be useful.
    Thanks.


  • Lifetime Qt Champion

    Hi and welcome to devnet,

    Did you build Qt's GStreamer backend for Android ?



  • @SGaist thanks for the response!
    I downloaded GStreamer for Android from https://gstreamer.freedesktop.org/data/pkg/android/, set INCLUDEPATH and LIBS flags for GStreamer libraries and got some working GStreamer samples with audio. All works fine, except display video.


  • Lifetime Qt Champion

    Then you should either try with their QtGstreamer module or try building Qt's GStreamer backend.



  • @SGaist are you sure? Because I just get only these issues with display video(audio playback and all other pipelines works fine). If yes, then where can I find how can I do this? And is QtGstreamer(not pure Gstreamer) cross-platform?


  • Lifetime Qt Champion

    Displaying a video is not a simple as playing a sound especially on mobile platforms where there are restrictions on the resources available for applications.

    Qt's GStreamer backend implements that part for you.

    QtGstreamer can run on platforms where its dependencies can be built.



  • @SGaist hm... interesting. I didn't know these details, many thanks! And how can I do this on Android?
    And some clarifying for our mutual understanding: Can I do this on Windows, Linux, OS X, iOS and Android? Because I only heard about QtGstreamer on Linux.
    Regards.


  • Lifetime Qt Champion

    Do you mean build Qt's GStreamer backend for Android ?

    You have to check the availability of the module's dependencies for these platform (or build them by hand)



  • @SGaist ok, and can you share some simple example how can I build Qt's GStreamer backend for Android? Because I even don't know how I can start.


  • Lifetime Qt Champion

    Sorry, I don't have an example by hand but based on the QtMultimedia sources, the first thing I'd do is get gstreamer for Android and then I'd modify the plugins.pro file and add the gstreamer subdir to the list in the android scope.



  • @SGaist ok, thanks. And what is the difference between QtGstreamer module and Qt's GStreamer backend?


  • Lifetime Qt Champion

    The QtGstreamer module is on a lower-level. It essentially provides you a nice Qt API to build your GStreamer pipelines and a set of video sync for QtWidgets and QtQuick.



  • @SGaist as I understand QtGstreamer will works for my issue too, so what is more preferable? How can I use QtGstreamer? And is it cross-platform? Thanks!


  • Lifetime Qt Champion

    It should run on the same platforms as Qt as long as you can satisfy its dependencies.

    I'd first try to build the backend. It will likely be less work.



  • @SGaist understood. Thanks for the explanations. About backend:
    I need download GStreamer for Android(this I already did, right?), create plugins.pro file in the my project(I don't know even what is it, but I will read) and add GStreamer dir to the list in the android scope(don't heard about this too). Is it all what I need?


  • Lifetime Qt Champion

    No, you have to build the plugin from the sources and then install them



  • @SGaist can you share the link which describes what I need to do? Or some additional reading? I'm sorry but I really don't understand a lot of things. Where I need to set a plugins.pro? What I need to set within it? About which plugin are you said? How and where I need to install it? How connect GStreamer to all these things?


  • Lifetime Qt Champion

    You don't need to create any file. You have to first get the sources of the QtMultimedia module.

    There you'll find the plugins folder with the matching plugins.pro file



  • Hi don-prog!

    I've had the same issue as you...displaying a GStreamer video on Android within a Qt application!
    It was a hard way to find out how to get it alltogether work and it took me a long time! I wasn't able to find really useful hints about this issue.

    First I also tried it with QtGStreamer, but I didn't get it work for Android. After trying and trying and trying (...) I decided to look for another way to go. By the way - for Linux the QtGStreamer worked like a charm.

    As you can see at this link (https://wiki.qt.io/Qt_5.5.0_Multimedia_Backends) the multimedia backend of Qt uses different multimedia frameworks on different platforms --> GStreamer is only used for Linux and I think this decision wasn't made causeless! I don't think it will be so easy to get the Qt's GStreamer-Backend to work for Android since a Qt application for Android comes with partial other characteristics than a Qt application for Linux. That's based in the differences between the dissimilar windowing-systems of Android and Desktop-Linux.

    If you display some patience I'll show you how I did it?

    Greetz, Lukas


  • Lifetime Qt Champion

    Hi and welcome to devnet,

    @LuJoRi Indeed it wasn't mad causeless because Qt uses as much as possible native frameworks.

    If I understand you correctly you successfully used GStreamer on Android with Qt to do video rendering ?



  • @SGaist ok, I downloaded QtMultimedia source and found that block:

    android {
       SUBDIRS += android opensles
    }
    

    What I should to do next? Just add /home/user/Downloads/gstreamer-1.0-android-armv7-1.8.0 in the end?



  • @SGaist, yes exactly! It was one of my core topics of my master thesis which I'll finally release after the weekend :)
    @don-prog, @SGaist, are you interested in my solution? It will take some time for me to explain you in english since it isn't my first language (it's german).

    A few quick hints:

    • you should look for a surface of the Java/Android-API and get the pointer on it via ANativeWindow_fromSurface(...) --> so you can handle the surface's display area within C++/Qt

    • with this pointer you are able to use the standard GStreamer way to render a video in a specific display area.
      Your function call was:
      gst_video_overlay_set_window_handle(GST_VIDEO_OVERLAY(video_sink), this->ui->playback_widget->winId());
      --> you have to use the pointer on the native window instead of winId()...the winId() function returns useful values on Linux but not on Android!


  • Lifetime Qt Champion

    @don-prog Sure thing !

    That may avoid @don-prog some pain.

    @don-prog you would need to add gstreamer to SUBDIRS. It's the name of the plugin dir. However, AFAIK, that plugin uses pkg-config to find gstreamer's library/includes so it may be problematic. You would have to put the the paths yourself in there.



  • @LuJoRi many thanks for your support and experience! I am very interested in your solution because I already thought about some way like yours, but I have some misunderstandings:
    How can I use ANativeWindow_fromSurface() from Qt and with which arguments?



  • @don-prog you have to do function calls from C++/Qt to Java/Android-API via JNI. Look for QAndroidJniObject (http://doc.qt.io/qt-5/qandroidjniobject.html) --> that's a way to create instances of classes of the Java/Android-API. Furthermore you can call their methods - for example via

     QAndroidJniObject::callObjectMethod(const char *methodName, const char *signature, ... ) 
    

    That means that you first have to manage a Java surface object will be created. You do this via JNI. Then you can put this one as an argument to the ANativeWindow_fromSurface function.



  • @LuJoRi understood, via JNI :)
    But I still don't understand what I need to put in the ANativeWindow_fromSurface() function as first(JNIEnv * env) and second(jobject surface) argument?



  • @don-prog
    The first agrument requires something like a pointer on the Java environment which is needed because on Android even a Qt application is based on a JVM respectively ART (Android Runtime).
    The second one is the C++/Qt handle of the Java surface. Once you've retrieved it after your JNI calls, you can cast it to jobject type and then put it to the ANativeWindow_fromSurface function :)

    Unfortunately I'm short of time the next days (till sunday). Regardless I'd really like to help you! Feel free to ask if there are any questions :)



  • @LuJoRi excuse for troubling! And what is the second argument in Qt exactly? I think it's not qwidget->winId(), so what?



  • @don-prog
    Sorry for the late response!

    No, forget winId() respectively the returned WId on Android for your purposes :)

    The second argument for ANativeWindow_fromSurface is the pure JNI object of a Java surface which you'll retrieve from the Java/Android world! The ANativeWindow_fromSurface function will return a pointer to the display which is managed by the Java surface. With this pointer you are able to access this display area within C++/Qt...since Qt's winId() function doesn't return proper values on Android the pointer you retrieve via ANativeWindow_fromSurface will be something like a replacement for the "missing" WId.

    Are you familiar with the Android-API and how to get a surface in standard Android way? --> you need a SurfaceView and with its SurfaceHolder you can get access to the corresponding Surface. Since you are working with Qt, you have to try doing this stuff via JNI respectively QAndroidJNIObject :) When you get the JNI object of a Java surface this will be your second argument for the function ANativeWindow_fromSurface.

    For my thesis I've created a clear view of the whole procedure...I hope you understand I have first to go in conversation with my professor before I'll post things (diagrams, figures) which I created for my thesis. These days I'm doing the final cut for my thesis document which I'll deliver after the weekend. I'd like to do the delivery of the thesis first and then help you out with detailed description :) I hope it's okay for you?

    Nevertheless I'll try to answer your questions in the meantime :)

    Lukas



  • @LuJoRi of course, it's absolutely ok! I'm student too and I understand you fully :)
    I just have two questions:

    1. Can I display video in NOT full screen(display video in some child widget of my application)? Or I can display video ONLY in full screen? Of course I mean via Qt and GStreamer.
    2. How can I get SurfaceView of the widget(if I can display video in not full screen) or of the full window(if I can display video only in full screen)?
      Regards.


  • Okay, good to know :)

    @don-prog said:

    1. Can I display video in NOT full screen(display video in some child widget of my application)? Or I can display video ONLY in full screen? Of course I mean via Qt and GStreamer.
    2. How can I get SurfaceView of the widget(if I can display video in not full screen) or of the full window(if I can display video only in full screen)?
      Regards.
    1. Yes, you can display a video wherever you want and in every size you want
    2. In fact the surface isn't part of your Qt-GUI --> you'll get something like a "mixed GUI" with a Qt part and a Java part. All of your GUI will be drawn by Qt except the part where your video will be rendered because this will happen inside the display area which is hold by the Java surface. If you create a Java surface with size 100x100 px via JNI there will be an area with this size ontop of your Qt-GUI which has nothing to do with Qt itself. To describe how you can get it is a longer term which I'll show you after my delivery :))


  • @LuJoRi understood, that is a very good news! I will wait when you can share your information about this subject. By the way, good luck :)



  • @LuJoRi I also have exactly the same requirement that is to render video inside a Qt application on Android using gstreamer. It would be really helpful if you could explain the steps for building gstreamer for a qt application with Android as a target.



  • Hey guys ... my thesis is done :)

    I will explain you the procedure step by step this evening or tomorrow in the afternoon.

    Greetz Lukas



  • A rough overview to get an idea how it works:

    First you should take a general look on the dependencies between SurfaceView, SurfaceHolder, Surface in the documentation of the Android API. This is essential to understand what you have to do to get access to an Android Surface.

    If you know how it works you can do the creation of the needed objects (of SurfaceView, SurfaceHolder, ...) via JNI calls respective QAndroidJniObject! For all of this you should create a QRunnable --> in its run()-method you do the creation of the objects you need --> QAndroidJniObject is your friend here :)

    void QRunnable::run()
    {
         //create your needed objects of SurfaceView, SurfaceHolder, ... via QAndroidJniObject
    }
    

    Later you have to take care about executing this run()-method in context of the Android UI thread! The created instance of Surface is your second parameter for the ANativeWindow_fromSurface function :)



  • @LuJoRi hello!
    I'm very sorry for the so late reply, but I just had some urgent problems, and I had to leave this forum and work with another things.
    I see that you shared your advice and sample for this issue as you promised.
    So I want to thank you for this!
    Unfortunately, I had to do other things, but I think that you helped other users (including chaudhry) to solve this not simple task! :)
    Sorry and thank you one more time!