Carnival LiveCam, augmented reality made easy



  • !http://s7.postimage.org/mkrdntcsb/snowexample.png(Example of the snow plugin)!

    Intro

    After a long time to be developing in the background, finally present my most ambitious project "Carnival LiveCam":https://github.com/hipersayanX/Carnival-LiveCam .
    Carnival LiveCam can capture images from multiple sources (webcams, video files, image files), and apply various effects such as effects of rain, fire, masks, 3D effects and 2D of all kinds. And also can take pictures and record video.
    Carnival LiveCam is still in an early stage of development, but it shows many of the features that are able to offer. Is the ideal time to join the the development.
    Carnival LiveCam is licensed under GPLv3 and hosted on "github":https://github.com/.

    Basics

    The design of Carnival LiveCam is thought to be completely modular and is very easy to expand the program by using plugins based on QtPlugin system, so far the types of plugins available are:

    • Candy: This type of plugins are responsible for providing the graphics effects for Carnival LiveCam.
    • Drivers: These plugins are used to add new capture devices. These are not drivers such as the operating system, these drivers do not work at the hardware level, but rather working on a top layer above the operating system drivers, for example these drivers capture images from webcam using OpenCV, or V4L2, but if you have not installed the drivers for your webcam then you can not capture images from your webcam using this driver.
    • Shells: A shell is a graphical interface or not with which the user can interact with Carnival LiveCam, this interface may be based on Qt Widgets, QML, Web, command line, etc.. The concept of "shell" comes from the *nix-like environments such as GNU/Linux where the user can choose the graphical environment with which interact with the computer. Carnival LiveCam also offers the same freedom.

    A small tour

    At the bottom center of Carnival LiveCam can see the control bar, from left to right the controls are:

    • Select the capture device.
    • Take a picture.
    • Record video.
    • Open the Candy Bar.
    • Set the window to full screen.
    • Information about the program.

    !http://s10.postimage.org/82s7bnbe1/controlsbar.png(Controls bar)!

    On the right side of Carnival LiveCam can see the Candy Bar, here we can turn on the effects we want to apply to the video. The effects are activated or deactivated by simply clicking on the preview, the deactivated effects has a red box around, the activated has a green box. Each candy has 3 controls:

    • Upper right: This control displays if the candy requires or not 3D acceleration.
    • Bottom left: Displays information about the candy.
    • Bottom right corner: You can configure the candy. Only available in some candys.

    We can also filter by the type of candy using the combobox at the top.

    !http://s10.postimage.org/3otdanwix/cadybar.png(The Candy Bar)!

    Inside the candy bar we also have the Candy Stack, here we can change the order in which each effect is applied.

    !http://s8.postimage.org/igqbk7wzp/stackbar.png(The Candy Stack)!

    Here we see one of the configuration dialogs of a driver in this case the driver of images. With a simple design intended for an point&click experience.

    !http://s12.postimage.org/m4earfrlp/imageselect.png(This is the configuration dialog for the image driver)!

    Obtain and test Carnival LiveCam

    To obtain a copy of Carnival LiveCam and start working on the project, run the following command:

    @git clone https://github.com/hipersayanX/Carnival-LiveCam.git@

    To compile and run Carnival LiveCam run the following command:

    @
    cd Carnival-LiveCam
    ./build_all.sh
    ./Carnival-LiveCam
    @

    To compile and run Carnival LiveCam need the following software packages:

    • Qt >= 4.7.x
    • OpenSceneGraph >= 3.0.x
    • OpenCV >= 2.3.x
    • FFmpeg
    • ffmpegthumbnailer >= 2.xx
    • Doxygen >= 1.7.5.x
    • Graphviz >= 2.28.x
    • Texlive-bin >= 2011.x
    • Git >= 1.7.x
    • Mscgen >= 0.22

    May be you can compile and use Carnival LiveCam with an older versions of the recommended software, I don't know. The environment in which I am currently developing Carnival LiveCam is "Arch Linux":http://www.archlinux.org/ x86_64. The packages needed by Arch Linux are:

    @yaourt -S qt openscenegraph opencv ffmpeg ffmpegthumbnailer doxygen graphviz texlive-bin git mscgen@

    And finally...

    In general, if you know how to use Git then you know how to collaborate with the project. Just clone the repository and start working on your copy and when you have everything up and running do a push request.
    Here I leave some links on how to use Git:

    And if anyone is interested in participating in the project is welcome :D

    Questions?



  • This looks very cool hipersayan_x :)



  • Thanks :D



  • Why is the title of the last dialog you show "Dialog"? That is not very descriptive...



  • Because I forget to change that XD
    I did that driver at the last moment for test purposes.



  • Hi Hipersayan_x.
    I've found your application very usefull and suitable , enough novice and unigue for our times.
    Trying to build , run test and understand the application.
    I've got segmentation fault when run on my Ubuntu 84 bit:

    inside shellmanager.cpp

    this->activeShell->widget()

    returns null.

    Didn't get yet how do you implement ( realize )shell class? There is only header shell.h

    ...

    Thanks.



  • The most likely reason is that you was not built the shell. Build the shell with:

    @cd Carnival-LiveCam/share/shells/DefaultShell
    qmake && make@

    Or build DefaultShell.pro from Qt Creator.
    Or better yet, build all the proyect:

    @cd Carnival-LiveCam
    ./build_all.sh@

    If followed the steps correctly, running the program, you should see output similar to this:

    @Loading: "/home/hipersayan_x/Carnival-LiveCam/share/drivers/ImageDriver/libImageDriver.so"
    Loading: "/home/hipersayan_x/Carnival-LiveCam/share/drivers/VideoDriver/libVideoDriver.so"
    Loading: "/home/hipersayan_x/Carnival-LiveCam/share/drivers/WebcamLinux/libWebcamLinux.so"
    Loading: "/home/hipersayan_x/Carnival-LiveCam/share/shells/DefaultShell/libDefaultShell.so"
    Loading: "/home/hipersayan_x/Carnival-LiveCam/share/plugins/DefaultPlugin/libDefaultPlugin.so"
    Loading: "/home/hipersayan_x/Carnival-LiveCam/share/plugins/SnowFall/libSnowFall.so"
    Loading: "/home/hipersayan_x/Carnival-LiveCam/share/plugins/TheMask/libTheMask.so"@



  • Succeed to run. I built the parts from QtCreator but linked to opencv using package config :

    unix {
    CONFIG += link_pkgconfig
    PKGCONFIG += opencv
    }

    I am thinking how to prepare installer or shell script to install the application together with third party dependencies ( OpenCV ) on end-user machine using one-click ( one run ).

    Does the software work on ARM processors, 800-1000HZ, 1GB memory with Linux based platform?
    Have it been tested on Embedded Linux kernels? if yes what kernels tested?

    Thanks.



  • Wow, thank you! PKGCONFIG option is new for me, I will add this to devel branch.

    bq. I am thinking how to prepare installer or shell script to install the application together with third party dependencies ( OpenCV ) on end-user machine using one-click ( one run ).

    That would be great! That's what I want.

    bq. Does the software work on ARM processors, 800-1000HZ, 1GB memory with Linux based platform?

    No, only tested on x86-64 processors, I haven't a tablet PC. But I'm interested in make it work on tablet devices, the current design is intended for that.
    The current testing plataform is a Lenovo G550 (a laptop) with:

    • Pentium Dual-Core CPU T4400 @ 2.20GHz (64 bits)
    • 4 Gb of RAM
    • Intel GMA 4500M graphics card (256 MB).
    • 2 webcams with a resolution of 300k pixels. (U_u) Snif, snif.

    bq. Have it been tested on Embedded Linux kernels? if yes what kernels tested?

    No, only on normal kernels, kernel version = 3.1.7 for 64 bits



  • Ok
    I am interesting these days in the OpenCV - Qt integration.
    Maybe you can advice me as a developer of this software.
    I have one another question ( maybe interesting ):
    When I open webcam at my place using openCV by next code:

    cv::VideoCapture m_Camera = cv::VideoCapture();
    m_Camera.open(0);

    I observe a bit confusing unpleasant high delay between press button and actual start of displaying video stream ( suppose by button press above code instructions called ) that I research how to minimize.
    I observe the led of webcam ( embedded camera in laptop ) is lighted almost immidiately ( after 1 sec ) but I see frames of video on my widget where I render frames from web cam only after 2 - 4 sec and that is high perceptible delay. that should be removed

    --Why such delay ( what is source of the delay ). How to minimize it ?

    --What is a delay between call to m_Camera.open(0) to frames stream displayed at your platform ?

    --There is another option to get frames from camera using openCV by

    CvCapture *m_Camera = cvCreateCameraCapture(0);
    ...

    but then you should convert IplImage to QImage and I see yet only solutions with 2 loops for conversion while option you use with cv::Mat frame; - you convert in one row that is of course preferable than 2 long loops - reduces execution time ( maybe usefull note FYI to reduce execution times if will port in future on small resource/time critical systems ( embedded ) ). -

    Thank you in advance.



  • bq. --Why such delay ( what is source of the delay ). How to minimize it ?

    Perhaps it is because of the different stages through which must pass signals or frames to be processed.

    The calling trace for a device activation is:

    @
    // Connect the signal from device selection button to another slot...
    bool ShellManager::enableShell(QString id)
    {
    connect(this->activeShell, SIGNAL(deviceSelected(QString)), this, SLOT(onDeviceSelected(QString)));
    }

    // then emmit the signal. Is there possible to connect a signal directly to another signal?
    void ShellManager::onDeviceSelected(QString deviceId)
    {
    emit deviceSelected(deviceId);
    }

    // Connect the signal emmited from the ShellManager...
    Core::Core(QObject *parent): QObject(parent)
    {
    connect(&this->shellManager, SIGNAL(deviceSelected(QString)), this, SLOT(deviceSelected(QString)));
    }

    // to this slot and then setup the new device and update the viewport on PluginManager.
    void Core::deviceSelected(QString deviceId)
    {
    this->deviceManager.setDevice(deviceId);
    this->pluginManager.resize(this->deviceManager.frameSize());
    }

    // Setup de new device.
    bool DeviceManager::setDevice(QString id)
    {
    if (this->devicesInfo[this->activeDevice].driverId() != this->devicesInfo[id].driverId())
    {
    // If the current device and the new selected device uses a diferent driver,
    // disable the current device and unload the driver (unload the QtPlugin from memory),
    this->disableDevice(this->activeDevice);

        // and load the new driver and setup the new device.
        this->enableDevice(id);
    }
    else
    {
        // Both, the current device and the new selected device uses the same driver,
        // Just get a reference to the current loaded driver.
        Driver *driver = this->driverManager.driver(this->devicesInfo[id].driverId());
    
        if (driver)
        {
            // Disable the current device,
            driver->disableDevice(this->activeDevice);
    
            // and load the new device.
            driver->enableDevice(id);
        }
    }
    

    }

    // For example, the new device is a webcam.
    bool WebcamLinux::enableDevice(QString id)
    {
    cv::VideoCapture webcam;

    // Try to open webcam device,
    if (!webcam.open(QString(id).remove("/dev/video").toInt()))
        return false;
    
    // and append it to activeWebcams.
    this->activeWebcams[id] = webcam;
    
    return true;
    

    }
    @

    The calling trace for frames retrieve and show is:

    @
    // MediaStreamming emmit a new captureFrame every 1ms.
    Core::Core(QObject *parent): QObject(parent)
    {
    connect(&this->mediaStreaming, SIGNAL(captureFrame()), this, SLOT(captureFrame()));
    }

    void Core::captureFrame()
    {
    // Capture a frame from DeviceManager, send it to PluginManager to apply the effects,
    // and return the frame with the effects applied.
    QImage frame = this->pluginManager.getFrame(this->deviceManager.captureFrame());

    // Send the frame to MediaStreamming for save it to a video or image file.
    this->mediaStreaming.setFrame(frame);
    
    // And send it to the current shell.
    this->shellManager.setFrame(frame);
    

    }

    // Caspturing a frame from a device.
    QImage DeviceManager::captureFrame()
    {
    if (this->activeDevice != "")
    {
    // Get a reference to the current loaded driver.
    Driver *driver = this->driverManager.driver(this->devicesInfo[this->activeDevice].driverId());

        if (driver)
            // Capture a frame from the current device.
            return driver->captureFrame(this->activeDevice);
    }
    
    // Else return a 1x1 black image.
    QImage frame(1, 1, QImage::Format_RGB888);
    
    frame.fill(0);
    
    return frame;
    

    }

    // Caspturing a frame from a webcam.
    QImage WebcamLinux::captureFrame(QString id)
    {
    if (this->activeWebcams.contains(id))
    {
    cv::Mat matFrame;

        // Capture a frame in cvMat format...
        this->activeWebcams[id] >> matFrame;
    
        // and convert it to QImage.
        QImage qtFrame((const uchar *)matFrame.data, matFrame.cols, matFrame.rows, QImage::Format_RGB888);
    
        return qtFrame.rgbSwapped();
    }
    else
    {
        // Else return a 1x1 black image.
        QImage frame(1, 1, QImage::Format_RGB888);
    
        frame.fill(0);
    
        return frame;
    }
    

    }

    // Here we apply all the active candys (effects) to the frame.
    QImage PluginManager::getFrame(const QImage &image)
    {
    QImage frame = image;

    for (int currentPlugin = 0; currentPlugin < this->activePlugins.size(); currentPlugin++)
        frame = this->activePlugins[currentPlugin]->render(frame);
    
    return frame;
    

    }
    @

    BUT, still is necessary to add a new stage after processing the frames and before displaying it in the shell, the Spaces. The Spaces allow you to display several devices at same time. I'm working on that right now.

    bq. --What is a delay between call to m_Camera.open(0) to frames stream displayed at your place?

    I think just 1 or 2 seconds.

    bq. --There is another option to get frames from camera using openCV by ...

    Yes, before releasing Carnival LiveCam, I was using the C API of OpenCV, with the same results as the C++ API, no timing difference.



  • Ok, I was discovered the frame delay bug, disable the webcam, wait 2 or 3 seconds, and enable again. Apparently, it is not related to OpenCV. Working on that.


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.