Audio glitch/skip with QAudioOuput in QOpenglWidget

[HSLee]

Hello guys,

Currently I am working on a test application allow to read YUV and PCM raw A/V data from buffer and output image and audio frame by frame just like video player. This is for embedded system (Intel NUC) it runs with Boot2Qt (Yocto). I used QOpenGLWidget/QOpenGLTexture to load and convert YUV frame data to RGB and paint every 40ms (25fps). In the meanwhile I use QAudioOuput to output linear PCM data at the same frequency. It's 32 bits 48k mono sample data so before every 'paintGL' I audioOutput->write 7680 bytes of audio sample store on RAM.

First I tried my test application on a i7 system it plays both video and audio smoothly. But when I test run on my embedded target system the audio sound glitch/skip (with same input sources). So I comment out 'update' to make the test application do audio only and no audio glitch/skip happens. Then I tried just 'glDrawArrays' same frame so there's no data copying involved, and the audio plays smoothly in this case too. I also found CPU usage is higher when data copying involved.

How to make the audio plays smoothly without interrupted by video data copying to GPU? Should I make another thread for the audio output? Is there any way to lower CPU usage for Opengl texture data update? I googled seems PBO DMA method is the way but it is not available in Opengl ES2. Does latest Boot2Qt supports Opengl ES3? (The one I am is v.5.7 seems supports Opengl ES2 only)

[HSLee]

Hi guys,

I found my audio issue is because I read and output both video and audio in the same thread. But my video is not able to be rendered fast enough so makes audio sound skipped because not enough audio sample provided throughout the time.

Currently I am still working on to boost up my video rendering rate. To avoid audio output is interfered by slow video, I create another thread to read and output audio at correct frequency. (by QThread::msleep after audioDev->write ) My setup is same as previous message. (48k, 32 bits, mono PCM) I subclass QThread as my audio thread. And used QAudioOutput/QIODevice for implementing audio out. For some reason, I still hear the clicks when audio is done in the thread. If I called the same audio output function in main thread at correct frequency (by timer signal/slot) it play flawlessly.

Here's run() of my thread:

void WorkerThread::run()
{
struct timeval clock;
int prev_clock, cur_clock, delta_clock;
prev_clock = 0;
delta_clock = -1;
cur_clock = 0;

while (1)
{
    if(threadstoper == true)
    {
        qDebug() << "break";
        break;
    }
    mutex.lock();
    //convert pcm data stream from 24 bits to 32 bits (pad zero to MSB). notice incoming data is little endian in this sample file
    int j = 0;
    int k = audSrcPtr*pcmSize;
    for(int i=0; i<pcmSize; i++)
    {
        if((i%3) == 0)
        {
            audio32Bits[j++] = 0;
        }
        audio32Bits[j++] = pcm[i+k];
    }
    pcmFrameLen = (pcmSize/3)*4;

    if(audioOutput->error() == QAudio::FatalError)
    {
        qDebug () << audioOutput->error();
        threadstoper = true;
        break;
    }

    int e = 0;
    if(audioOutput->error() == QAudio::UnderrunError)
    {
        qDebug () << audioOutput->error();
        e = 1;
    }

    if(delta_clock>0)
    {
        gettimeofday(&clock, NULL);
        cur_clock = clock.tv_usec;
        delta_clock = (cur_clock+1000000-prev_clock)%1000000;
        if(delta_clock>19900)//20000)
            qDebug() << "unexpected time out when doing audio and data DMA";
        else
            delta_clock = 19900-delta_clock;//20000-delta_clock;
        QThread::usleep(delta_clock);
    }

    int bytes_to_write = 3840;
    int offset = 0;
    while (bytes_to_write)
    {
        if(audioOutput && audioOutput->state() != QAudio::StoppedState && audioOutput->state() !=  QAudio::SuspendedState)
        {
            if(audioOutput->bytesFree()>4096)
            {
                audioDev->write((const char *)audio32Bits, 3840);
                debugFile->write((const char *)audio32Bits, 3840);
                bytes_to_write = 0;
            }
        }
    }

    int temp = prev_clock;
    gettimeofday(&clock, NULL);
    prev_clock = clock.tv_usec;
    delta_clock = 1;

    if(audioBuffered == false && audSrcPtr>2)
    {
        audioBuffered = true;
    }
    curPtr++;
    if(curPtr>=4)
    {
        curPtr = 0;
    }
    bufferDepth++;
    mutex.unlock();

    vidSrcPtr++;
    audSrcPtr++;
    if(vidSrcPtr == 10)
        vidSrcPtr = 0;
    if(audSrcPtr == 1000)
        audSrcPtr = 0;
}

}

Should I not use QThread::msleep as timing regulator for audio output? Does audio out has to be implemented in main thread?

[jsulm]

@HSLee Almost everything in your while loop is protected by a mutex. Is it really necessary? This way there is not much parallelism if you lock same mutex in other threads.

[HSLee]

Dear jsulm ,

Thank you for your reply. Yes you are right. It's not necessary to lock everything in the while. I did fix that in my latest to lock when convert 24 bits to 32 bits pcm and write to audio device. But it seems not help to solve the problem.

So I decide to change different way to deal with timing by QTimer instead of msleep. Turns out it works. Because QTimer can't be initiated inside 'run', I also change method to initiate thread like following:

GLWidget::GLWidget(QWidget *parent) : QOpenGLWidget(parent)
{
setFixedSize(1024, 768);
setAutoFillBackground(false);
..................................................

thread = new QThread; // First thread
thread->start();
audioOut->moveToThread(thread);

}

And I create QObject class "Audio" for the thread operations.
In "Audio", constructor, QTimer is create and setup:

timer = new QTimer(this);
connect(timer, SIGNAL(timeout()), this, SLOT(test()));
timer->start(20);

By this method, I can do both video thread (OpenGLWidget) and audio thread without audio glitch/skips. I guess I can't use msleep to regulate audio output timing. I wonder it's because msleep actually halt unfinished audio output process after "audioDev->write" has been called.