Important: Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

Realtime Audio Playback in QT: QAudioOutput seems to wait unnecessarily



  • Hello QT-Community,

    just in advance, this is my first time posting here, so I am really sorry for the wall of text and code that is following, because I don't really know which parts are important and which i should have left out...

    I am working on a realtime audio project in which I read audio data from the microphone with the QAudioInput class, process it and play it with the QAudioOutput class.
    I implemented my own QIODevice and its read- and write-functions. To mimimize the latency I set the input and output buffer sizes as low as audio playback still works (3000 Bytes for now, could be still lower I think) and set the QIODevice property to "Unbuffered". So far it's working really well.

    The problem is, when I start the Input and Output simultaneously, the Output Device reads a still empty buffer, which still works, if I catch that special case in the read function of my QIODevice, but it doesn't seem to me as a good solution overall.
    So I waited till the AudioInput collected as many data as the size of the QAudioOut Buffer Size and then started the AudioOutput. My expectation was, that as soon as I start the QAudioOutput, it will fill its internal buffer. But here are the strange things to occur: Wenn I start the QAudioOutput, it still waits like 5 to 9 cycles of AudioInput till it begins to reads data, even if there's already enough data available! Interestingly, if I start both simultaneously, the QAudioOutput immediatly begins to read 3000 (not existing) bytes. This behaviour seems quite illogicaly to me.
    Here is some code of my implementations:

    Audio Format

        QAudioFormat format;
        QAudioDeviceInfo deviceInfo_in = QAudioDeviceInfo::defaultInputDevice();
        QAudioDeviceInfo deviceInfo_out = QAudioDeviceInfo::defaultOutputDevice();
        format.setSampleRate(48000);
        format.setChannelCount(1);
        format.setSampleSize(16);
        format.setSampleType(QAudioFormat::SignedInt);
        format.setByteOrder(QAudioFormat::LittleEndian);
        format.setCodec("audio/pcm");
        
        format_in = format;
        format_out = format;
    
        audiobuffer.resize(16*3000);    // buffer to store data (QByteArray)
    

    Audio Devices: I pass the pointer to the audiobuffer and the position 'write_pos' to the constructor of the QIODevice (called "AudioDevice"). write_pos is the position in audiobuffer, where the next chunk of date has to be written (or read)

        audiodevice_in = new AudioDevice(this,format_in,0,&audiobuffer);
        audioinput = new QAudioInput(deviceInfo_in,format_in,this);
        audioinput->setBufferSize(2000);
        audiodevice_in->open(QIODevice::WriteOnly | QIODevice::Unbuffered);
    
        audiodevice_out = new AudioDevice(this,format_out,audiodevice_in->write_pos,&audiobuffer);
        audiooutput = new QAudioOutput(deviceInfo_out,format_out,this);
        audiooutput->setBufferSize(3000);
        audiodevice_out->open(QIODevice::ReadOnly | QIODevice::Unbuffered);
    
        audioinput->start(audiodevice_in);
        qWarning() << "Input Period Size =" << audioinput->periodSize();
    

    my AudioDevice.cpp:

    AudioDevice::AudioDevice(QObject* parent,QAudioFormat form,qint64 writepos,QByteArray* opb) : QIODevice(parent)
    {
        audiobuffer = opb;
        format = form;
        prefill = 8;
        x = 1;
        channel1.resize(1920);
        write_pos = writepos;
    }
    
    qint64 AudioDevice::writeData(const char* data, qint64 len) 
    {                                                          
          Q_ASSERT(format.sampleSize() % 8 == 0);
        const int channelBytes = format.sampleSize() / 8;
        const int sampleBytes = format.channelCount() * channelBytes;
        Q_ASSERT(len % sampleBytes == 0);
        const int numSamples = len / sampleBytes;
        const unsigned char *ptr = reinterpret_cast<const unsigned char *>(data);
        // pointer "data" points to the data which has to be written
    
        for (int samp=0; samp<numSamples; samp++)
        {
            for (int chan=0; chan<format.channelCount(); chan++)
            {
                qint16 value = qFromLittleEndian<qint16>(ptr);
                if (value>32760) value = 32760;
                if (value<-32760) value = -32760;
                channel1.replace(samp,value);// for channelCount>1 more have to be added
                ptr += channelBytes;
            }
        }
        // result: QVector "channel1" is filled with samples, now I could do my audio processing
    
        //------------write data in audiobuffer----------------    
        signed char* ptr2 = reinterpret_cast<signed char *>(audiobuffer->data()+write_pos);
       // ptr2 points to audiobuffer+data (where next data has to be written)
        for (int samp=0; samp<channel1.length(); samp++)
        {
            for (int chan=0; chan<format.channelCount(); chan++)
            {
                qToLittleEndian<qint16>(channel1.value(samp), ptr2);
                ptr2 += channelBytes;
            }
        }
    // result: chunk of data is added at position write_pos in audiobuffer
    
        write_pos += len;                 //both instances of AudioDevice have their own
        emit writeposchanged(write_pos);  //'write_pos' -> value is "shared" with signals and slots
    
        qWarning() << "Successfully written" << len << "Bytes, write_pos =" << write_pos;
    
        if (x == prefill) emit startoutput();  // after 'prefillI' write processes startoutput() is called
        if (x < prefill + 1) x++;    // do not endlessly increment x
    
        return len;
    }
    
    qint64 AudioDevice::readData(char *data, qint64 len) 
    {                          // *data = pointer to where the data has to be written
        qWarning() << "Reading Data";
    
        qint64 total = 0;
        qint64 m_pos = 0;
    
        if (!audiobuffer->isEmpty()) 
        {
            while (len - total > 0)
            {
                const qint64 chunk = qMin((audiobuffer->length() - m_pos), len - total);
                memcpy(data + total, audiobuffer->constData() + m_pos, chunk);
                m_pos = (m_pos + chunk);
                total += chunk;
            }
            // result: all bytes have been written: total==len
        }
        //Now I rotate all bytes in audiobuffer 'total' places to the left
        for (int i=0; i<(audiobuffer->size()-total); i++)
            memcpy(audiobuffer->data()+i,audiobuffer->constData()+i+total,1);
      
        write_pos -= total;
        if (write_pos < 0) {qWarning() << "Buffer Underrun occured"; write_pos = 0;}
        emit writeposchanged(write_pos);
       
        qWarning() << "Successfully read" << total << "Bytes, write_pos =" << write_pos;
        return total;
    }
    

    As you may have read from the code, I read always from the very beginning of audiobuffer and afterwards shift the bytes to the left, so the next-to-be-read Bytes are at the beginning again. "write_pos" is saving the position, where the Input has to write the next chunk of data. The value of write_pos is shared between the two instances of AudioDevice between signals and slots (yes i connected them, i just didn't post these lines here)

    At last, after 'prefill' times of writing input data, the slot on_startoutput() is called in the main function, which just starts the audiooutput:

    void AudioProcessing::on_startoutput()
    {
        audiooutput->start(audiodevice_out);
        qWarning() << "Output Period Size:" << audiooutput->periodSize();
        qWarning() << "Output Status:" << audiooutput->state();
    }
    

    I would imagine, that the Output now reads 3000 Bytes of data (its buffer size) from audiobuffer (which is already available because of the prefill mechanism), BUT the console prints the following:

    Input Period Size = 400
    Successfully written 400 Bytes, write_pos = 400
    Successfully written 400 Bytes, write_pos = 800
    Successfully written 400 Bytes, write_pos = 1200
    Successfully written 400 Bytes, write_pos = 1600
    Successfully written 400 Bytes, write_pos = 2000
    Successfully written 400 Bytes, write_pos = 2400
    Successfully written 400 Bytes, write_pos = 2800
    Successfully written 400 Bytes, write_pos = 3200
    Output Period Size: 600
    Output Status: ActiveState  *Why don't you read data?!?!*
    Successfully written 400 Bytes, write_pos = 3600
    Successfully written 400 Bytes, write_pos = 4000
    Successfully written 400 Bytes, write_pos = 4400
    Successfully written 400 Bytes, write_pos = 4800
    Successfully written 400 Bytes, write_pos = 5200
    Successfully written 400 Bytes, write_pos = 5600
    Successfully written 400 Bytes, write_pos = 6000
    Successfully written 400 Bytes, write_pos = 6400
    Successfully written 400 Bytes, write_pos = 6800
    Reading Data
    Successfully read 3000 Bytes, write_pos = 3800
    Successfully written 400 Bytes, write_pos = 4200
    Successfully written 400 Bytes, write_pos = 4600
    Successfully written 400 Bytes, write_pos = 5000
    Successfully written 400 Bytes, write_pos = 5400
    Successfully written 400 Bytes, write_pos = 5800
    Successfully written 400 Bytes, write_pos = 6200
    Successfully written 400 Bytes, write_pos = 6600
    Successfully written 400 Bytes, write_pos = 7000
    Successfully written 400 Bytes, write_pos = 7400
    Reading Data
    Successfully read 600 Bytes, write_pos = 6800
    Successfully written 400 Bytes, write_pos = 7200
    Successfully written 400 Bytes, write_pos = 7600
    Successfully written 400 Bytes, write_pos = 8000
    Reading Data
    Successfully read 1200 Bytes, write_pos = 6800
    Successfully written 400 Bytes, write_pos = 7200
    Successfully written 400 Bytes, write_pos = 7600
    Reading Data
    Successfully read 600 Bytes, write_pos = 7000
    Successfully written 400 Bytes, write_pos = 7400
    Successfully written 400 Bytes, write_pos = 7800
    Reading Data
    Successfully read 1200 Bytes, write_pos = 6600
    Successfully written 400 Bytes, write_pos = 7000
    

    As you can see, the Output starts after 400 bytes were written prefill=8 times (I choose 8 so that there are 3000 Bytes available at the start of the output), but the output still waits 9! times, till it reads for the first time.
    It's exactly this behaviour which ruins the latency.
    The value of write_pos is a good measure for the latency, because it represents the shift between where the current audio is written and where it is read (at write_pos=0 / beginning of array). With ~7000 Bytes = 3500 Samples it's about 73ms at 48kHz. Thats more than enough for a noticable annoying delay for realtime audio playback.

    Don't get me wrong, I am perfectly aware, that QT is not the best tool for realtime audio and that it was not designed with that in mind, but this seems to me like a such a simple fix! Just tell the Output to immediately start reading data when it's started and voila! write_pos will we at around 3000 Bytes, which translates to 31 ms, which would be perfectly fine for my application.

    I know, I know, 70ms or 30ms, it's kind of unnecessary trouble I bring myself (and you guys) into, but it just bothers me, because it seems so trivial.

    Maybe you guys could help me with this one, I would be really grateful.
    Feel free to critisize my overall concept of handling the data. I'm really new to QT and C++ in general. I'm really really happy, that my code works at all at this point.

    Thanks for your time,
    Kind Regards.


  • Lifetime Qt Champion

    Hi,

    That's something you should look into at the backend level. Depending on the OS you are on, it might not be possible.

    Since you want real-time, you should consider PortAudio which is likely better suited.


Log in to reply