Real-time Audio Processing with Qt

ryanmcgee

Hello, I am experienced with programming C/C++ sound synthesis and effects using PortAudio, RT Audio, and JUCE libraries. The Qt audio input and output code examples have shown me how to access sample buffers for audio input and output, but the methodology is quite different in Qt so I can not figure out how to simply route the incoming audio buffer straight to the output buffer i.e. a simple pass through that would allow me to then add real-time effects along the chain. Other libraries provide a callback that gives pointers to both input and output buffers at block rate, so achieving this is as simple as *out++ = *in++.

In Qt I have separate subclasses of QIODevice for my input and output. The output works great with a simple wavetable synthesizer that I have coded. The input is reading audio correctly and I see the value of the samples increase when tapping on the mic etc. But how to tie one to the other in sync? I gave my output a pointer to input's sample buffer and I do get audio from the mic but it is choppy and clips often. I set the buffersize of my input and output devices to match, but no luck. I'm assuming there has to be a better way to do this. I do not fully understand QIODevice methodology. I have read through the documentation and am guessing I need to use the readyRead() signal with a custom slot? Does anyone have experience with this or a simple input to output code example?

Thanks!
Ryan

vidar

The easiest way is to use one subclass of QIODevice where you override the writeData and readData methods. readData could do the processing of the incoming data and store the result in an "intermediate" buffer and writeData uses memcpy to copy this buffer to the *data buffer.

You can initiate the realtime processing by

@myProcessing->start(); // subclass of QIODevice
audioIn->start(myProcessing);
audioOut->start(myProcessing);@

ryanmcgee

That is what I originally tried, but it is not that simple.

So here are the read/writeData methods in my subclass of QIODevice:
(myInputData is the "intermediate buffer")

qint64 AudioMixer::readData(char *data, qint64 len){

memcpy(data, myInputData, len);

return len;

}

qint64 AudioMixer::writeData(const char *data, qint64 len){

memcpy(myInputData, data, len);

return len;

}

This does pass the incoming audio to the output as I was hoping to achieve, but it is very choppy. Can you suggest a solution to properly buffer everything? I tried a ring buffer so that the reading is delayed from the writing, but I get the same choppy audio.

Thanks!

ryanmcgee

Strange, also noticed that even though I'm calling:

m_audioOutput->setBufferSize(4096);
m_audioInput->setBufferSize(4096);

readData and writeData are not reading/writing the same lengths.

Maybe the solutions involves another way to synchronize the two?

Gilgrum

I am also working on real-time audio processing using 3rd party libraries and Qt, and have run into a snag. Perhaps if I share what I have done I will help you, and then you or someone else can point out what I am doing wrong and help us both.

First, note that I used the Spectrum Analyser discussed "here":http://labs.qt.nokia.com/2010/05/18/qtmultimedia-in-action-a-spectrum-analyser/ and found on your computer at InstallPath\Qt\Demos\4.7\spectrum to come up with most of what I know.

I am using QAudioInput and QAudioOutput for recording and playback. Much like you, I have setup a circular buffer to store the recorded audio (I use a QByteArray as the data structure for my buffer). In theory, as each element in the circular buffer fills up, I can pass it to my audio processing (in my case CMU Sphinx for voice recognition) and then clear that buffer element. For debug purposes, I do not clear the buffer and can choose to play or process any of the individual buffer elements on demand to make sure I have recorded what I expect and can double check the processed output for each buffer. This results in perfect audio recording and playback, however I get inconsistent results from processing the buffer that eventually lead to seg faults after enough processing attempts.

A summary of the code I use is as follows:

@// Setup QAudioInput and QAudioOutput, not shown for brevity. See the Spectrum example for details
QAudioInput input;
QAudioOutput output;
QIODevice *intermediateDevice; // This very temporarily stores recorded audio and then transfers it to the circular buffer

// Initialize the circular buffer, also shown in more detail in the Spectrum example
QByteArray byteArray[NUM_CIRCLE_BUFFER_ELEMENTS]; // Note that the spectrum example names this variable buffer despite it not being a QBuffer
qint64 byteArrayUsed[NUM_CIRCLE_BUFFER_ELEMENTS]; // This tracks how much of each buffer has been used

// Start recording audio
intermediateDevice = input->start();
connect(intermediateDevice, SIGNAL(readyRead()), this, SLOT(captureData())); // captureData is a function you define, and example is below

// Example captureData
void captureData(qint64 bytesToCapture)
{
qint64 bytesAvailable = byteArray[current].size() - byteArrayUsed[current];
qint64 bytesToRead = qMin(bytesToCapture, bytesAvailable);

// Transfer the audio from the intermediateDevice to the buffer
bytesActuallyRead = intermediateDevice->read(byteArray[current].data() + byteArrayUsed[current], bytesToRead);

byteArrayUsed[current] += bytesActuallyRead;

// Note that this is the simplified version, you would want to have the transferring in a loop and do this again in the next byteArray if bytesActuallyRead < bytesToRead
}

// How to play one of the byte arrays
QIODevice player;
player.close();
player.setBuffer(&(byteArray[current]));
player.open(QIODevice::ReadOnly);
output.play(&player;);
@

The last piece is handing each of the QByteArrays to your processing libraries. I've tried several approaches but haven't had any consistent results.

Let me know if this is helpful and if you can figure out how to do this last part.

vidar

Ryan: you have to take care of the "len" - parameters, they can change on every call. you could implement the buffer as some sort of circular buffer and check your write and read positions ...
this should work:

@qint64 myProcessing::writeData(const char *data, qint64 len) {

qint16* aVal;

qint64 numSamples = len / 2;
qint64 posSamples = 1;

aVal = (qint16*) data;

while (posSamples <= numSamples) {

mBuffer[mBufWritePos] = doSomething(*aVal);

mBufWritePos = (mBufWritePos+1) % bufSize;

aVal++;
posSamples++;

}

mBufFill += len;

return len;
}

qint64 myProcessing::readData(char *data, qint64 maxlen)
{
if (mBufFill<=0) return 0;

qint64 toDo = qMin(maxlen, (qint64) mBufFill);

if (toDo / 2 + mBufReadPos > bufSize) {

qint64 firstNum = (bufSize - mBufReadPos) * 2;

memcpy( data,
&mBuffer;[mBufReadPos],
firstNum );
memcpy( data + firstNum,
&mBuffer;[0],
toDo - firstNum);

} else {
memcpy(data, &mBuffer;[mBufReadPos], toDo);

}

mBufReadPos =  (mBufReadPos + toDo/2) % bufSize;
mBufFill -= toDo;

return toDo;

}@