Solved QAudioOutput becomes idle and stays idle when buffer underrun on MacOS
-
I have a program that streams audio from a socket to QAudioOutput. It buffers a little bit, but sometimes the buffer does run dry based on the connexion and needs to rebuffer. This program is working on Linux and Windows.
When I ported the program to MacOS I've noticed that if it runs out of buffer it turns into the idle state and won't return when I fill the buffer with more data. On Linux and Windows, when the buffer is empty, it also goes to idle but resumes when there is more data available. So I'm not sure if it's a bug on Windows and Linux and Mac is really how it's supposed to work, or the opposite.
I've noticed in the qtmultimedia framework src/plugins/coreaudio/coreaudiooutput.mm this section:
void CoreAudioOutput::audioDeviceIdle() { if (m_stateCode == QAudio::ActiveState) { QMutexLocker lock(&m_mutex); audioDeviceStop(); m_errorCode = QAudio::UnderrunError; m_stateCode = QAudio::IdleState; QMetaObject::invokeMethod(this, "deviceStopped", Qt::QueuedConnection); } }
This gets called seemingly when the buffer has run out of data. The deviceStopped function stops the timer. Which would explain why I'm having the "issue" I'm having.
However! I've created a minimal example of my issue and found that my minimal example works just fine! Even though the buffer runs out and the QAudioOutput goes idle, it does NOT stop the timer. I can see the continual calls to readData on my subclassed QIODevice.
So my question is, why in the world the different behaviours? Why on my program does the timer stop and there are no more calls to readData on the QIODevice but on my minimal example the timer keeps going despite numerous buffer empties?
Here is the best minimal example I can make. But this minimal example "works" and the timer never stops despite buffer underrun.
main.cpp#include <QCoreApplication> #include "HandleAudio.h" #include "moc_HandleAudio.cpp" int main( int argc, char *argv[] ) { QCoreApplication app( argc, argv ); // Create the audio handler HandleAudio *audio = new HandleAudio( &app ); // Start us audio->start(); return( app.exec() ); }
HandleAudio.h
#ifndef HANDLE_AUDIO_H #define HANDLE_AUDIO_H #include <QFile> #include <QTimer> #include <QAudio> #include <QAudioOutput> #include <QDateTime> #include "QIOConsumableBuffer.h" #include <QDebug> #define LOG_X( STREAM ) qDebug() << "(" << QDateTime::currentMSecsSinceEpoch() << ") " << __PRETTY_FUNCTION__ << ": " << STREAM; #define SAMPLERATE 48000 #define CHANNELS 1 #define SAMPLESIZE 16 #define FILL_MILLISECONDS 20 // How many milliseconds of audio we'll write and wait #define WAIT_START 10 #define WAIT_END 4000 #define WRITE_INTERVAL_BYTES ( SAMPLERATE*CHANNELS*( SAMPLESIZE/8 )*( FILL_MILLISECONDS/1000.0 ) ) #define FILENAME "input.raw" // Raw PCM data file class HandleAudio : public QObject { Q_OBJECT public: HandleAudio( QObject *parent=nullptr ) : QObject( parent ), file_( new QFile( FILENAME, this ) ), buffer_( new connect::QIOConsumableBuffer( this ) ) { qsrand( QDateTime::currentMSecsSinceEpoch() / 1000 ); // Init our device QAudioFormat format; if( !file_->open( QIODevice::ReadOnly) ) { LOG_X( "Couldn't open file '" << FILENAME << "'" ); exit( 1 ); } format.setSampleRate( SAMPLERATE ); format.setChannelCount( CHANNELS ); format.setSampleSize( SAMPLESIZE ); // 16-bit audio, 2 bytes each format.setCodec( "audio/pcm" ); format.setByteOrder( QAudioFormat::LittleEndian ); format.setSampleType( QAudioFormat::SignedInt ); // See if we support this QAudioDeviceInfo info( QAudioDeviceInfo::defaultOutputDevice() ); if( !info.isFormatSupported( format ) ) { LOG_X( "Format is not supported. Not even trying." ); // Big problem guys. exit( 1 ); } // Creat the audio object output_ = new QAudioOutput( format, this ); // Open our buffer buffer_->open( QIODevice::ReadWrite ); // Grab the callback for state changes connect( output_, SIGNAL( stateChanged(QAudio::State) ), this, SLOT( handleStateChanged(QAudio::State) ) ); } /** * Start us off */ void start() { if( audio_state_!=QAudio::ActiveState ) output_->start( buffer_ ); // Start us (again?) wait(); } private slots: /** * State changed slotifier */ void handleStateChanged( QAudio::State state ) { // Save the state audio_state_ = state; // Using the new state // Process the state switch( state ) { case QAudio::IdleState: { qint64 bytes_available = buffer_->bytesAvailable(); LOG_X( "Audio now in idle state. Bytes left=" << bytes_available ); checkAudioError(); // Are we done? if( bytes_available<=0 and done_ ) { // Exit gracefully QMetaObject::invokeMethod( QCoreApplication::instance(), "quit", Qt::QueuedConnection ); return; // Do NOT process anymore because the class is GONE } break; } case QAudio::ActiveState: LOG_X( "Audio now in active state." ); break; case QAudio::SuspendedState: LOG_X( "Audio now in suspended state." ); checkAudioError(); break; case QAudio::StoppedState: LOG_X( "Audio now in stopped state." ); break; default: LOG_X( "Unhandled audio state: " << state ); } } /** * We should write more data */ void writeMoreData() { writeData(); wait(); } private: /** * Write data to the buffer from the file */ void writeData() { // Start us off LOG_X( "Reading " << WRITE_INTERVAL_BYTES << " into buffer." ); QByteArray bytes = file_->read( WRITE_INTERVAL_BYTES ); if( bytes.size()>0 ) { // Write the audio data buffer_->write( bytes ); } // This will be our last if( bytes.size()!=WRITE_INTERVAL_BYTES ) done_ = true; } /** * Get the audio error */ inline static const char *audioError( QAudio::Error error ) { switch( error ) { case QAudio::NoError: return( "No errors have occurred." ); case QAudio::OpenError: return( "An error occurred opening the audio device." ); case QAudio::IOError: return( "An error occurred during read/write of audio device." ); case QAudio::UnderrunError: return( "Audio data is not being fed to the audio device at a fast enough rate." ); case QAudio::FatalError: return( "A non-recoverable error has occurred, the audio device is not usable at this time." ); } return( "Unknown Error." ); } /** * Schedule the wait timer */ void wait() { // Wait no more if we're already done if( done_ ) return; // Waiter qint64 wait_time = ( qrand()%(WAIT_END-WAIT_START) )+WAIT_START; LOG_X( "Waiting: " << wait_time ); // Wait QTimer::singleShot( wait_time, this, SLOT( writeMoreData() ) ); } /** * Check for an audio error */ inline void checkAudioError() { if( output_->error()!=QAudio::NoError ) LOG_X( "Audio error: " << audioError( output_->error() ) ); } QFile *file_; connect::QIOConsumableBuffer *buffer_=nullptr; QAudioOutput *output_=nullptr; bool done_=false; //!< Keep track of whether or not there is more data to read from the file QAudio::State audio_state_ = QAudio::StoppedState; //!< our audio state }; #endif
QIOConsumableBuffer.h
#ifndef QIO_CONSUMABLE_BUFFER_H #define QIO_CONSUMABLE_BUFFER_H #include <QIODevice> #include <QByteArray> #define QIO_CONSUMABLE_DEBUG #ifdef QIO_CONSUMABLE_DEBUG #include <QDebug> #include <QDateTime> #define QIO_LOG_X( STREAM ) { int _now=QDateTime::currentMSecsSinceEpoch();qDebug() << "(" << (_now-QIOConsumableBuffer::_last_millis()) << "ms) " << __PRETTY_FUNCTION__ << ": " << STREAM; QIOConsumableBuffer::_last_millis()=QDateTime::currentMSecsSinceEpoch(); } #else #define QIO_LOG_X( STREAM ) {} #endif namespace connect { /** * Interface for QIO buffers * @date 5/22/2018 * @author Michael A. Leonetti * @copyright Sonarcloud, 2018 */ class QIOConsumableBuffer : public QIODevice { public: #ifdef QIO_CONSUMABLE_DEBUG static int &_last_millis() { static int lm = QDateTime::currentMSecsSinceEpoch(); return( lm ); } #endif /** * C-tor */ QIOConsumableBuffer( QObject *parent=nullptr ) : QIODevice( parent ) { } /** * We are considered sequential */ bool isSequential() const override { QIO_LOG_X( "Check" ); return( true ); } /** * Whaa */ qint64 bytesAvailable() const override { qint64 available = byte_array_.size()+QIODevice::bytesAvailable(); QIO_LOG_X( "Enter function=" << available << ", errorString=" << errorString() ); return( available ); } /** * Our size */ qint64 size() const override { QIO_LOG_X( byte_array_.size() ); return( byte_array_.size() ); } /** * When we are reading from the buffer */ qint64 readData( char *data, qint64 maxlen ) override { QIO_LOG_X( "Requested: " << maxlen ); // Correct size maxlen = std::min( maxlen, qint64( byte_array_.size() ) ); QIO_LOG_X( "Have: " << maxlen ); if( maxlen>0 ) { // Copy memcpy( data, byte_array_.constData(), maxlen ); // Now remove byte_array_.remove( 0, maxlen ); } QIO_LOG_X( "Remaining: " << byte_array_.size() << ", available: " << bytesAvailable() << ", error=" << errorString() ); // Return how much we read return( maxlen ); } /** * When we are writing to the buffer */ qint64 writeData( const char *data, qint64 maxlen ) override { QIO_LOG_X( "Enter function" ); // Add it all to the end byte_array_.append( data, maxlen ); QIO_LOG_X( "Byte array is now " << byte_array_.size() ); // Max len that return( maxlen ); } private: QByteArray byte_array_; }; } #endif
-
Turns out I've wrongly misinterpreted how CoreAudioOutput is supposed to work. I see that in coreaudiooutput.mm even though the audio is stopped, it connects to the slot inputReady() of CoreAudioOutputBuffer and CoreAudioOutputBuffer's timer that periodically refills the circular buffer never stops. So it should work even if there is a buffer underrun, and my minimal example works fine.
So then the real issue comes with my program. I've noticed with my subclassed QIODevice that after I start the QAudioOutput and an underrun occurs, it will call bytesAvailable() again, but never call readData() ever again. So the audio just hangs on idle and never gets restarted because CoreAudioOutputBuffer never pulls any more data.
Even with calling suspend/resume it still remains stuck. It calls bytesAvailable() and never readData(). Even though bytesAvaillable() returns PLENTY of size.
Any ideas how I could be better debugging or looking? I'm guessing the next step would be to write a specialized audio output module myself for the Mac. The same code works both on Linux and Windows so it would only need to be for the Mac.
My actual code:
#ifndef AUDIO_QT_OUT_H #define AUDIO_QT_OUT_H // Include this so we can call it back #include "Audio/Queue.h" // Base class #include "Audio/Out.h" #include "QTHelper.h" #include "QIOConsumableBuffer.h" #include <exception> #include <memory> #include <QObject> #include <QAudioOutput> #include <QAudio> #include <QAudioFormat> #include <QAudioDeviceInfo> namespace connect { namespace audio { /** * QT Multimedia output module * @author Michael A. Leonetti * @date 5/15/2018 * @copyright Sonarcloud, 2018 */ class QTOut : public Out { Q_OBJECT public: /** * Our custom C-tor */ QTOut( int samplerate, int channels, const In::shared_ptr &in, Queue *queue ) : Out( samplerate, channels, in, queue ), samples_multiplier_( channels<<1 ), // Channels multiplied by 2 for the bitdepth buffer_( new QIOConsumableBuffer( this ) ) { } /** * Delete this later */ virtual ~QTOut() { //if( audio_output_ ) // audio_output_->deleteLater(); LOG_TRACE( logger() ) << __PRETTY_FUNCTION__; } /** * Create a shared pointer */ template<typename... Args> static std::shared_ptr<QTOut> create( Args&&... args ) { return( qthelper::make_shared<QTOut>( args... ) ); } /** * Write PCM data to the QT object */ void writeSamples( const int16_t *pcm, size_t samples ) override //void writePCM( const QByteArray &pcm ) { // Make sure we don't write after we've been done if( done_ ) throw std::runtime_error( "Audio out write after done." ); //LOG_TRACE( logger() ) << "Samples written " << samples; // The size we are writin const size_t bytes = samples*samples_multiplier_; // Write the audio data buffer_->write( (char*)pcm, bytes ); //const qint64 written = buffer_->write( (char*)pcm, bytes ); /* if( written<0 ) LOG_TRACE( logger() ) << "Error: " << buffer_->errorString(); */ // Do we have our out device? //if( audio_state_==QAudio::StoppedState ) if( audio_state_!=QAudio::ActiveState ) audio_output_->start( buffer_ ); // Start us (again?) //LOG_TRACE( logger() ) << "Bytes decoded: " << bytes << ", written: " << written << ", buffer size: " << buffer_->size(); } /** * Implement cancel */ void cancel() override { // Set the state audio_state_ = QAudio::StoppedState; // Stop the audio audio_output_->stop(); // Quit now reportDoneToQueue(); } /** * Our initializer */ void init() override { // Do NOT double init if( initted_ ) throw std::runtime_error( "Audio out already initialized." ); initted_ = true; // init up Out::init(); // Init our device QAudioFormat format; LOG_DEBUG( logger() ) << "Initializing " << samplerate_ << " Hz, " << channels_ << " Ch, 16-bit signed PCM"; format.setSampleRate( samplerate_ ); format.setChannelCount( channels_ ); format.setSampleSize( 16 ); // 16-bit audio, 2 bytes each format.setCodec( "audio/pcm" ); format.setByteOrder( QAudioFormat::LittleEndian ); format.setSampleType( QAudioFormat::SignedInt ); // See if we support this QAudioDeviceInfo info( QAudioDeviceInfo::defaultOutputDevice() ); if( !info.isFormatSupported( format ) ) throw std::runtime_error( "Audio format not supported by sound hardware: "+std::to_string( samplerate_ )+" Hz "+std::to_string( channels_ )+" Ch 16-bit Signed PCM" ); // Creat the audio object audio_output_ = new QAudioOutput( format, this ); // Open our buffer buffer_->open( QIODevice::ReadWrite ); // Grab the callback for state changes connect( audio_output_, SIGNAL( stateChanged(QAudio::State) ), this, SLOT( handleStateChanged(QAudio::State) ) ); //LOG_TRACE( logger() ) << "Initted audio."; } private slots: /** * State changed slotifier */ void handleStateChanged( QAudio::State state ) { // Save the state audio_state_ = state; // Using the new state // Process the state switch( state ) { case QAudio::IdleState: { const qint64 bytes_available = buffer_->bytesAvailable(); LOG_DEBUG( queue().logger() ) << "Audio now in idle state. Bytes left=" << bytes_available; const QAudio::Error error = checkAudioError(); // Are we done? if( done_ and ( (error!=QAudio::NoError and error!=QAudio::UnderrunError) or bytes_available<=0 ) ) { reportDoneToQueue(); return; // Do NOT process anymore because the class is GONE } else { LOG_DEBUG( queue().logger() ) << "Suspending..."; } // Free the IO device. We'll get another one anyway. //io_device_ = nullptr; break; } case QAudio::ActiveState: LOG_DEBUG( queue().logger() ) << "Audio now in active state."; break; case QAudio::SuspendedState: LOG_DEBUG( queue().logger() ) << "Audio now in suspended state."; checkAudioError(); break; case QAudio::StoppedState: LOG_DEBUG( queue().logger() ) << "Audio now in stopped state."; //io_device_ = nullptr; break; default: LOG_DEBUG( queue().logger() ) << "Unhandled audio state: " << state; /* case QAudio::InterruptedState: LOG_TRACE( queue().logger() ) << "Audio now in interrupted state."; break; */ } } public: /** * Check for an audio error */ inline QAudio::Error checkAudioError() { QAudio::Error error = audio_output_->error(); if( error!=QAudio::NoError ) LOG_ERROR( queue().logger() ) << "Audio error: " << audioError( error ); return( error ); } /** * Get the audio error */ inline static const char *audioError( QAudio::Error error ) { switch( error ) { case QAudio::NoError: return( "No errors have occurred." ); case QAudio::OpenError: return( "An error occurred opening the audio device." ); case QAudio::IOError: return( "An error occurred during read/write of audio device." ); case QAudio::UnderrunError: return( "Audio data is not being fed to the audio device at a fast enough rate." ); case QAudio::FatalError: return( "A non-recoverable error has occurred, the audio device is not usable at this time." ); } return( "Unknown Error." ); } /** * Override flush */ void flush() override { // Are we already done? if( done_ ) throw std::runtime_error( "Audio out already flushed." ); LOG_TRACE( logger() ) << "Flushing"; // Set the done value done_ = true; // Are we already in an inactive state? if( audio_state_!=QAudio::ActiveState ) { const QAudio::Error error = audio_output_->error(); if( error!=QAudio::NoError and error!=QAudio::UnderrunError and buffer_->bytesAvailable()>0 ) { LOG_ERROR( queue().logger() ) << "Still more left. Waiting until audio empties buffer."; /* audio_output_->stop(); audio_output_->start( buffer_ ); // Start us (again?) */ } else reportDoneToQueue(); } } private: QAudioOutput *audio_output_=nullptr; //!< Our QT audio output object int samples_multiplier_; //!< How much to multiply the sample count by bool initted_ = false; //!< Don't double init bool done_ = false; //!< Whether or not we've been flushed QAudio::State audio_state_ = QAudio::StoppedState; //!< our audio state //QIODevice *io_device_=nullptr; //!< What we'll be writing to QIOConsumableBuffer *buffer_; }; } } #endif
-
Hi,
What version of Qt are you using ?
On what version of macOS ? -
Oh wow. Can't believe I didn't post that. Sorry.
Qt 5.11.1 via homebrew on MacOS 10.13.6
-
@Michael-A.-Leonetti said in QAudioOutput becomes idle and stays idle when buffer underrun on MacOS:
if( audio_state_!=QAudio::ActiveState ) audio_output_->start( buffer_ ); // Start us (again?)
The problem in the code was here. Start being called on the same buffer twice caused some sort of a lock up. Changing the code to make sure start only get called ONCE fixed the issue.
So this issue is solved. Hopefully this saves somebody the days that it took me to figure out this simple problem.
-
Glad you found out and thanks for sharing !
Happy coding :)
-
@Michael-A-Leonetti Hi Michael, I am having a very similar issue with my code (which is a microphone routing audio to a speaker (I need access to the intermediary packets to forward them over an RTP network)). I was wondering where in your code you called 'audio_output_->start( buffer _); twice. I do not have any code setup to restart when I encounter the underruns (like you I assumed everything would recovoer nicely).
The work around I had for my issue (and its not very reliable) was to ensure that the output device (in my case the speaker) was started with a significant delay after starting the audio source.
I have a QRingBuffer in my sequential SinkDevice (a subclass of QIODevice opened for read/write that gets passed as the QIODevice to the QAudioSink::start(SinkDevice)). (this is how pull mode works). This gets filled regularly with captured microphone audio whenever the microphone captures a block of audio - the filling is done through hooking to the bytesAvailable signal from the QAudioSource. Without giving lots of code, does this sound familiar? In general do I need some mechanism to handle the idle state change and call resume somehow?
Thanks in advance -
@johnco3 I am not using a pull mode I don't believe. I am using a timer that fires every 20ms and writes more data as it becomes available. I'm sorry.