Animated GIF : QImage & Libav API

xtingray

Hi,

I am working on a Qt class to convert a set of QImage objects into an animated GIF. Right now, I can create movies using other formats like AVI and MOV, taking advantage of the Libav API. Nevertheless, it seems that the GIF format is another story.

I already implemented the general procedure to create a GIF file, I mean the header, the color palette and the end of the file. Using an hex editor I could confirm that those parts are ok.

The point where I am stuck is when I have to get the content of the QImage object to process it using the Libav functions, as the image data for a GIF file requires an specific compression algorithm (LZW) to be applied. Just passing the pixel data using the bits() method is not enough.

This is the piece of code I currently use to get the data from one image to create a frame of a movie file (MOV, AVI, MP4, etc) and it works perfectly:

        int size = avpicture_get_size(PIX_FMT_YUV420P, width, height);
        uint8_t *pic_dat = (uint8_t *) av_malloc(size);
        RGBtoYUV420P(image.bits(), pic_dat, image.depth()/8, true, width, height);
        avpicture_fill((AVPicture *)frame, pic_dat, PIX_FMT_YUV420P, width, height); // Libav method

The RGBtoYUV420P() method transforms the pixel data coming from the bits() method into a YUV420P data structure that is required by the avpicture_fill() method to create one frame of the movie. The result is accurate.

Now, in the case of a GIF image, the pixel data coming from the bits() method requires some kind of treatment before calling the avpicture_fill() method. I have tried to implement the LZW algorithm for GIF images data in that point, but nothing seems to work.

I really appreciate any hint. Thanks!

SGaist

Hi,

What about using sws_scale to do the format conversion ? That might avoid some pain.

Hope it helps.

xtingray

Thank you for the advice! I started to play around with the sws_scale() function, but there is a parameter that I can't figure out how to set.

This is a piece of my code:

        QImage *image = new QImage("experiment.png", "PNG");
        struct SwsContext *sws_context = sws_getContext(w, h, PIX_FMT_RGB24, w, h,  PIX_FMT_YUV420P, 
                                                                              SWS_BICUBIC, NULL, NULL, NULL);
        sws_scale(sws_context, image.bits(), linesize, 0, h, frame->data, frame->linesize);

The third variable (linesize) is a mistery for me. I mean, how can I calculate that value from the QImage object?

I was checking the header of the function in the libav source code:

int sws_scale(struct SwsContext *c, const uint8_t *const srcSlice[],
              const int srcStride[], int srcSliceY, int srcSliceH,
              uint8_t *const dst[], const int dstStride[]);

In my case, the linesize variable should be an array with type const int srcStride[], but where the content for this variable come from?
Any suggestion? Thanks!

kshegunov

@xtingray said in Animated GIF : QImage & Libav API:

The third variable (linesize) is a mistery for me. I mean, how can I calculate that value from the QImage object?

You mean the strides? In principle you can calculate that based on the image depth and internal buffer alignment (which is always 32bits for QImage). In this particular case, however, you should get the same stride for each line and should be able to do it directly by calling QImage::bytesPerLine. E.g.:

QVector<int> strides(image.height(), image.bytesPerLine());

sws_scale(sws_context, image.bits(), strides.constData(), 0, h, frame->data, frame->linesize);

xtingray

I tried to follow your suggestion directly:

QVector<int> strides(image.height(), image.bytesPerLine());
sws_scale(sws_context, (const uint8_t * const *) image.bits(), strides.constData(), 0, h, frame->data, frame->linesize);

But the result wasn't so good. The animated GIF is a mess of gray lines.

Looking for an alternative solution, I found this approach:

        AVPicture picture;
        avpicture_alloc(&picture, PIX_FMT_RGB32, width, height);
        memcpy(picture.data[0], image.bits(), width*height*4);

        struct SwsContext *sws_context = sws_getContext(w, h, PIX_FMT_RGB32, width, height, 
                                               PIX_FMT_YUV420P, SWS_BICUBIC, NULL, NULL, NULL);
        sws_scale(sws_context, picture.data, picture.linesize, 0, height, frame->data, frame->linesize); 
        avpicture_free(&picture);

        AVPacket pkt;
        av_init_packet(&pkt);
        pkt.flags |= AV_PKT_FLAG_KEY;
        pkt.stream_index= video_st->index;
        pkt.data = (uint8_t *) frame;
        pkt.size = sizeof(AVFrame);
        ret = av_write_frame(oc, &pkt);

The result is still wrong, but you can recognize some parts of the animation. Here is an example: http://maefloresta.com/tmp/test.gif

Additionally, I am getting this warning message in the console every time I process a frame:

[swscaler @ 0x2280320] bad dst image pointers

Comments and suggestions are very welcome! ;)

xtingray

Looking for more information about the problem, I was studying the Libav source code. Here are the files related to the encoding/decoding GIF format process:

https://github.com/libav/libav/blob/master/libavformat/gif.c
https://github.com/libav/libav/blob/master/libavcodec/gif.c

If you check the code within the method gif_image_write_image() you will see how Libav makes the translation from a RGB bitmap into the GIF encoding calling routines related to the LZW algorithm.

static int gif_image_write_image(AVCodecContext *avctx,
                                 uint8_t **bytestream, uint8_t *end,
                                 const uint8_t *buf, int linesize)

The sws_scale() function is not making the required GIF encoding, so I guess I will have to try another trick.

SGaist

If I read your code correctly, you selected PIX_FMT_YUV420P as output format for sws_scale. The problem I see with this is that this format is not in the list of formats supported by the gif encoder.

You should likely change that for AV_PIX_FMT_RGB8.

xtingray

I set the format value AV_PIX_FMT_RGB8 instead PIX_FMT_YUV420P and the resulting GIF file is still wrong. Any way, running some tests I got an interesting debugging message:

#3 0x00007f082b8cc073 in gif_clut_index (b=<error reading variable: Cannot access memory at address 0x4039001>, g=<error reading variable: Cannot access memory at address 0x4039000>, r=0 '\000') at libavformat/gif.c:189
#4 gif_image_write_image (x1=0, y1=0, pix_fmt=2, linesize=<optimized out>, buf=0x4038b70 "", height=<optimized out>, width=<optimized out>, pb=0x3d90540) at libavformat/gif.c:227 
#5 gif_write_video (s=<optimized out>, size=<optimized out>, buf=<optimized out>, enc=<optimized out>) at libavformat/gif.c:329
#6 gif_write_packet (s=<optimized out>, pkt=<optimized out>) at libavformat/gif.c:341
#7 0x00007f082b916d65 in write_packet (pkt=0x7ffc79f4b4e0, s=0x3d8ff60) at libavformat/mux.c:334
#8 av_write_frame (s=0x3d8ff60, pkt=0x7ffc79f4b4e0) at libavformat/mux.c:384 <p></p>#9 0x00007f082bbd9bdb in TLibavMovieGenerator::Private::writeVideoFrame (this=0x3d9b700, movieFile=..., image=...) at tlibavmoviegenerator.cpp:385

Initially I was doubting if the Libav API was calling the GIF procedures to encode the image as that format require it. Now that I see a reference to the function gif_image_write_image (), I am pretty sure that it does it, the right encoding process is happening! :D

So, I feel that my problem is more specific now: In some way, I am failing to cast the data from the QImage object into the AVPicture variable or maybe I have to make the cast directly to an AVFrame structure. Not sure at all, I will have to try several ways until got the solution.

I would love to find previous reference about Qt and Libav procedures related specifically to the GIF format , but it seems there is not much art state about it.

SGaist

What format are you using for your QImage ?

xtingray

This is the way I initialize the QImage objects:

QSize size(width, height);
QImage image = QImage(size, QImage::Format_RGB32);

I want to make one point clear: I already can create MOV, AVI and MP4 videos from a QImage array using this code:

int size = avpicture_get_size(PIX_FMT_YUV420P, width, height);
uint8_t *pic_dat = (uint8_t *) av_malloc(size);
RGBtoYUV420P(image.bits(), pic_dat, image.depth()/8, true, width, height);
avpicture_fill((AVPicture *)frame, pic_dat, PIX_FMT_YUV420P, width, height)
ret = avcodec_encode_video2(c, &pkt, frame, &got_output);

My problem is ONLY related to the animated GIF format.

PS: In case you want to take a look to the whole class (without the GIF part) -> https://github.com/xtingray/tupi/blob/master/src/plugins/export/libavplugin/tlibavmoviegenerator.cpp

SGaist

That's because these encoders supports PIX_FMT_YUV420P as input which is not the case for gif.

One thing you could do to simplify your life a bit is to use Format_RGB888 for your QImage so you wouldn't even need the conversion.

xtingray

Every day I'm getting closer to the solution. This is the latest GIF file I could create -> http://maefloresta.com/tmp/test.gif
Using this code:

    QImage img = image.convertToFormat(Format_RGB888);

    AVPacket pkt;
    av_init_packet(&pkt);
    pkt.flags |= AV_PKT_FLAG_KEY;
    pkt.stream_index = video_st->index;
    pkt.data = (uint8_t *) img.bits();
    pkt.size = sizeof(AVFrame);

    av_write_frame(oc, &pkt);

As you can see it, the GIF file is not animated. Just the first frame is displayed, but the format is right. I was sneaking around inside the container using a hex editor and the 15 frames I created are there. I am missing some kind of flag or instruction to activate the animated format.

Any suggestion?

PS: I tried to use the function avcodec_encode_video2() to add the frames into the file, but it was unsuccessful. Creating my own packets (AVPacket) was the best approach.

SGaist

I think you have to set the loop private property on the muxer.

xtingray

Finally, I could create an animated GIF for the first time. Nevertheless, I need to adjust the FPS parameter in some point because the animation looks too slow. I am doing it from the AVCodecContext variable, like this:

    int fps = 24;
    AVCodecContext *c;
    c->time_base.den = fps;
    c->time_base.num = 1;

But anyway, it doesn't matter the value I set for the fps variable, the result is always the same (slow). On the other hand, this is the code I use to process every QImage object:

        int got_packet = 0;
        AVPacket pkt;
        av_init_packet(&pkt);
        pkt.data = NULL; // packet data will be allocated by the encoder
        pkt.size = 0;

        QImage img = image.convertToFormat(Format_RGB888);
        avpicture_fill((AVPicture *)frame, img.bits(), AV_PIX_FMT_RGB24, w, h);

        int ret = avcodec_encode_video2(c, &pkt, frame, &got_packet);
        if (ret < 0) {
            tError() << "Error encoding video frame!";
            return false;
        }

        if (got_packet) {
            pkt.stream_index = video_st->index;
            ret = av_interleaved_write_frame(oc, &pkt);
        } else {
            ret = 0;
        }

Any suggestion about how to fix the FPS issue?

SGaist

IIRC I used something along these lines:

st->time_base.num = 1000;
st->time_base.den = framerate * st->time_base.num;

st being the video AVStream for the output context and code.

The final result of the math should be the same but it gave me the correct output.

xtingray

I was wrong! The problem wasn't the FPS parameter. The problem was the size of the GIF files, they are giants! That's the reason the browser was playing them so slowly. I am talking about tens of MBs for just few seconds.

I need to reduce the frames size,. Maybe I have to retake the sws_scale() approach, the good news is that for the first time I have a working (not so efficient) implementation :P

I guess I will have to look for more documentation about either libswscale or the scale filter. Not so sure yet if the solution is in that direction, but I will give it a try.

PS: The lack of documentation about ffmpeg/libav is intimidating :/

SGaist

That's one good news ! :)

What are the size of your input images ?

xtingray

The size of the input images is actually small: 500 pixels x 500 pixels. The array contains only 20 images and the size of the GIF is around 10 MB (unacceptable!).
Some guys from the FFmpeg list told me that the problem is not related to the dimension, but about either the codec or the format of the images. Additionally, I have to play with some filters to reduce the size of the outcome following some recommendations from this handy link: http://blog.pkh.me/p/21-high-quality-gif-with-ffmpeg.html

It seems the challenge is bigger than me, but I will keep trying to implement a better approach. As animated GIF are hot stuff again, I think there is a great potential to create this kind of files from Qt directly.

SGaist

Thanks for sharing ! Looks pretty interesting !

I agree FFmpeg is not the easiest library to use but it is powerful. I'd recommend also digging in the implementation of the tools and examples they provide. You can get some useful hints in there and check the codecs code, you'll find also interesting stuff.

Otherwise, one thing I did once (but I've lost the code) was to write a "dumper" that gets all possible informations from the various data structure of the codecs like the possible image input formats. It gave me some insight that proved useful to tune the application and avoid e.g. useless conversions.

Animated GIF : QImage & Libav API

Qt Developer

Qt Developer

Qt Developer

Qt Developer

Qt Developer

Qt Developer

Qt Developer

Qt Developer

Qt Developer

Qt Developer