Important: Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

QAudioOutput becomes idle and stays idle when buffer underrun on MacOS



  • I have a program that streams audio from a socket to QAudioOutput. It buffers a little bit, but sometimes the buffer does run dry based on the connexion and needs to rebuffer. This program is working on Linux and Windows.

    When I ported the program to MacOS I've noticed that if it runs out of buffer it turns into the idle state and won't return when I fill the buffer with more data. On Linux and Windows, when the buffer is empty, it also goes to idle but resumes when there is more data available. So I'm not sure if it's a bug on Windows and Linux and Mac is really how it's supposed to work, or the opposite.

    I've noticed in the qtmultimedia framework src/plugins/coreaudio/coreaudiooutput.mm this section:

    void CoreAudioOutput::audioDeviceIdle()
    {
        if (m_stateCode == QAudio::ActiveState) {
            QMutexLocker lock(&m_mutex);
            audioDeviceStop();
    
            m_errorCode = QAudio::UnderrunError;
            m_stateCode = QAudio::IdleState;
            QMetaObject::invokeMethod(this, "deviceStopped", Qt::QueuedConnection);
        }
    }
    

    This gets called seemingly when the buffer has run out of data. The deviceStopped function stops the timer. Which would explain why I'm having the "issue" I'm having.

    However! I've created a minimal example of my issue and found that my minimal example works just fine! Even though the buffer runs out and the QAudioOutput goes idle, it does NOT stop the timer. I can see the continual calls to readData on my subclassed QIODevice.

    So my question is, why in the world the different behaviours? Why on my program does the timer stop and there are no more calls to readData on the QIODevice but on my minimal example the timer keeps going despite numerous buffer empties?

    Here is the best minimal example I can make. But this minimal example "works" and the timer never stops despite buffer underrun.
    main.cpp

    #include <QCoreApplication>
    
    #include "HandleAudio.h"
    
    #include "moc_HandleAudio.cpp"
    
    int main( int argc, char *argv[] )
    {
    	QCoreApplication app( argc, argv );
    
    	// Create the audio handler
    	HandleAudio *audio = new HandleAudio( &app );
    
    	// Start us
    	audio->start();
    
    	return( app.exec() );
    }
    

    HandleAudio.h

    #ifndef HANDLE_AUDIO_H
    #define HANDLE_AUDIO_H
    
    #include <QFile>
    #include <QTimer>
    #include <QAudio>
    #include <QAudioOutput>
    #include <QDateTime>
    
    #include "QIOConsumableBuffer.h"
    
    #include <QDebug>
    #define LOG_X( STREAM ) qDebug() << "(" << QDateTime::currentMSecsSinceEpoch() << ") " <<  __PRETTY_FUNCTION__ << ": " << STREAM;
    
    #define SAMPLERATE 48000
    #define CHANNELS 1
    #define SAMPLESIZE 16
    
    #define FILL_MILLISECONDS 20 // How many milliseconds of audio we'll write and wait
    #define WAIT_START 10
    #define WAIT_END 4000
    
    #define WRITE_INTERVAL_BYTES ( SAMPLERATE*CHANNELS*( SAMPLESIZE/8 )*( FILL_MILLISECONDS/1000.0 ) )
    
    #define FILENAME "input.raw" // Raw PCM data file
    
    class HandleAudio : public QObject
    {
    	Q_OBJECT
    	public:
    		HandleAudio( QObject *parent=nullptr ) :
    			QObject( parent ),
    			file_( new QFile( FILENAME, this ) ),
    			buffer_( new connect::QIOConsumableBuffer( this ) )
    		{
    			qsrand( QDateTime::currentMSecsSinceEpoch() / 1000 );
    
    			// Init our device
    			QAudioFormat format;
    
    			if( !file_->open( QIODevice::ReadOnly) )
    			{
    				LOG_X( "Couldn't open file '" << FILENAME << "'" );
    				exit( 1 );
    			}
    
    			format.setSampleRate( SAMPLERATE );
    			format.setChannelCount( CHANNELS );
    			format.setSampleSize( SAMPLESIZE ); // 16-bit audio, 2 bytes each
    			format.setCodec( "audio/pcm" );
    			format.setByteOrder( QAudioFormat::LittleEndian );
    			format.setSampleType( QAudioFormat::SignedInt );
    
    			// See if we support this
    			QAudioDeviceInfo info( QAudioDeviceInfo::defaultOutputDevice() );
    
    			if( !info.isFormatSupported( format ) )
    			{
    				LOG_X( "Format is not supported. Not even trying." );
    				// Big problem guys.
    				exit( 1 );
    			}
    
    			// Creat the audio object
    			output_ = new QAudioOutput( format, this );
    
    			// Open our buffer
    			buffer_->open( QIODevice::ReadWrite );
    
    			// Grab the callback for state changes
    			connect( output_, SIGNAL( stateChanged(QAudio::State) ), this, SLOT( handleStateChanged(QAudio::State) ) );
    		}
    
    		/**
    		 * Start us off
    		 */
    		void start()
    		{
    
    			if( audio_state_!=QAudio::ActiveState )
    				output_->start( buffer_ ); // Start us (again?)
    			
    
    			wait();
    		}
    
    
    	private slots:
    		/**
    		 * State changed slotifier
    		 */
    		void handleStateChanged( QAudio::State state )
    		{
    			// Save the state
    			audio_state_ = state; // Using the new state
    
    			// Process the state
    			switch( state )
    			{
    				case QAudio::IdleState:
    					{
    						qint64 bytes_available = buffer_->bytesAvailable();
    
    						LOG_X( "Audio now in idle state. Bytes left=" << bytes_available );
    
    						checkAudioError();
    
    						// Are we done?
    						if( bytes_available<=0 and done_ )
    						{
    							// Exit gracefully
    							QMetaObject::invokeMethod( QCoreApplication::instance(), "quit", Qt::QueuedConnection );
    							return; // Do NOT process anymore because the class is GONE
    						}
    
    						break;
    					}
    				case QAudio::ActiveState:
    					LOG_X( "Audio now in active state." );
    					break;
    				case QAudio::SuspendedState:
    					LOG_X( "Audio now in suspended state." );
    					checkAudioError();
    					break;
    				case QAudio::StoppedState:
    					LOG_X( "Audio now in stopped state." );
    					break;
    				default:
    					LOG_X( "Unhandled audio state: " << state );
    			}
    		}
    
    		/**
    		 * We should write more data
    		 */
    		void writeMoreData()
    		{
    			writeData();
    			wait();
    		}
    	private:
    		/**
    		 * Write data to the buffer from the file
    		 */
    		void writeData()
    		{
    			// Start us off
    			LOG_X( "Reading " << WRITE_INTERVAL_BYTES << " into buffer." );
    			QByteArray bytes = file_->read( WRITE_INTERVAL_BYTES );
    
    			if( bytes.size()>0 )
    			{
    				// Write the audio data
    				buffer_->write( bytes );
    			}
    
    			// This will be our last
    			if( bytes.size()!=WRITE_INTERVAL_BYTES )
    				done_ = true;
    		}
    		/**
    		 * Get the audio error
    		 */
    		inline static const char *audioError( QAudio::Error error )
    		{
    			switch( error )
    			{
    				case QAudio::NoError:
    					return( "No errors have occurred." );
    				case QAudio::OpenError:
    					return( "An error occurred opening the audio device." );
    				case QAudio::IOError:
    					return( "An error occurred during read/write of audio device." );
    				case QAudio::UnderrunError:
    					return( "Audio data is not being fed to the audio device at a fast enough rate." );
    				case QAudio::FatalError:
    					return( "A non-recoverable error has occurred, the audio device is not usable at this time." );
    			}
    
    			return( "Unknown Error." );
    		}
    
    		/**
    		 * Schedule the wait timer
    		 */
    		void wait()
    		{
    			// Wait no more if we're already done
    			if( done_ )
    				return;
    
    			// Waiter
    			qint64 wait_time = ( qrand()%(WAIT_END-WAIT_START) )+WAIT_START;
    
    			LOG_X( "Waiting: " << wait_time );
    
    			// Wait
    			QTimer::singleShot( wait_time, this, SLOT( writeMoreData() ) );
    		}
    
    		/**
    		 * Check for an audio error
    		 */
    		inline void checkAudioError()
    		{
    			if( output_->error()!=QAudio::NoError )
    				LOG_X( "Audio error:  " << audioError( output_->error() ) );
    		}
    
    		QFile *file_;
    		connect::QIOConsumableBuffer *buffer_=nullptr;
    		QAudioOutput *output_=nullptr;
    		bool done_=false; //!< Keep track of whether or not there is more data to read from the file
    		QAudio::State audio_state_ = QAudio::StoppedState; //!< our audio state
    };
    
    #endif
    

    QIOConsumableBuffer.h

    #ifndef QIO_CONSUMABLE_BUFFER_H
    #define QIO_CONSUMABLE_BUFFER_H
    
    #include <QIODevice>
    #include <QByteArray>
    
    #define QIO_CONSUMABLE_DEBUG
    #ifdef QIO_CONSUMABLE_DEBUG
    #include <QDebug>
    #include <QDateTime>
    
    #define QIO_LOG_X( STREAM ) { int _now=QDateTime::currentMSecsSinceEpoch();qDebug() << "(" << (_now-QIOConsumableBuffer::_last_millis()) << "ms) " << __PRETTY_FUNCTION__ << ": " << STREAM; QIOConsumableBuffer::_last_millis()=QDateTime::currentMSecsSinceEpoch(); }
    #else
    #define QIO_LOG_X( STREAM ) {}
    #endif
    
    namespace connect
    {
    	/**
    	 * Interface for QIO buffers
    	 * @date 5/22/2018
    	 * @author Michael A. Leonetti
    	 * @copyright Sonarcloud, 2018
    	 */
    	class QIOConsumableBuffer : public QIODevice
    	{
    		public:
    
    #ifdef QIO_CONSUMABLE_DEBUG
    			static int &_last_millis()
    			{
    				static int lm = QDateTime::currentMSecsSinceEpoch();
    
    				return( lm );
    			}
    #endif
    
    			/**
    			 * C-tor
    			 */
    			QIOConsumableBuffer( QObject *parent=nullptr ) :
    				QIODevice( parent )
    			{
    			}
    
    			/**
    			 * We are considered sequential
    			 */
    			bool isSequential() const override
    			{
    				QIO_LOG_X( "Check" );
    				return( true );
    			}
    
    			/**
    			 * Whaa
    			 */
    			qint64 bytesAvailable() const override
    			{
    				qint64 available = byte_array_.size()+QIODevice::bytesAvailable();
    				QIO_LOG_X( "Enter function=" << available <<  ", errorString=" << errorString() );
    
    				return( available );
    			}
    
    			/**
    			 * Our size
    			 */
    			qint64 size() const override
    			{
    				QIO_LOG_X( byte_array_.size() );
    
    				return( byte_array_.size() );
    			}
    
    			/**
    			 * When we are reading from the buffer
    			 */
    			qint64 readData( char *data, qint64 maxlen ) override
    			{
    				QIO_LOG_X( "Requested: " << maxlen );
    
    				// Correct size
    				maxlen = std::min( maxlen, qint64( byte_array_.size() ) );
    
    				QIO_LOG_X( "Have: " << maxlen );
    
    				if( maxlen>0 )
    				{
    					// Copy
    					memcpy( data, byte_array_.constData(), maxlen );
    
    					// Now remove
    					byte_array_.remove( 0, maxlen );
    				}
    
    				QIO_LOG_X( "Remaining: " << byte_array_.size() << ", available: " << bytesAvailable() << ", error=" << errorString() );
    
    				// Return how much we read
    				return( maxlen );
    			}
    
    			/**
    			 * When we are writing to the buffer
    			 */
    			qint64 writeData( const char *data, qint64 maxlen ) override
    			{
    				QIO_LOG_X( "Enter function" );
    
    				// Add it all to the end
    				byte_array_.append( data, maxlen );
    
    				QIO_LOG_X( "Byte array is now " << byte_array_.size() );
    
    				// Max len that
    				return( maxlen );
    			}
    		private:
    			QByteArray byte_array_;
    	};
    
    }
    
    #endif
    


  • Turns out I've wrongly misinterpreted how CoreAudioOutput is supposed to work. I see that in coreaudiooutput.mm even though the audio is stopped, it connects to the slot inputReady() of CoreAudioOutputBuffer and CoreAudioOutputBuffer's timer that periodically refills the circular buffer never stops. So it should work even if there is a buffer underrun, and my minimal example works fine.

    So then the real issue comes with my program. I've noticed with my subclassed QIODevice that after I start the QAudioOutput and an underrun occurs, it will call bytesAvailable() again, but never call readData() ever again. So the audio just hangs on idle and never gets restarted because CoreAudioOutputBuffer never pulls any more data.

    Even with calling suspend/resume it still remains stuck. It calls bytesAvailable() and never readData(). Even though bytesAvaillable() returns PLENTY of size.

    Any ideas how I could be better debugging or looking? I'm guessing the next step would be to write a specialized audio output module myself for the Mac. The same code works both on Linux and Windows so it would only need to be for the Mac.

    My actual code:

    #ifndef AUDIO_QT_OUT_H
    #define AUDIO_QT_OUT_H
    
    // Include this so we can call it back
    #include "Audio/Queue.h"
    
    // Base class
    #include "Audio/Out.h"
    #include "QTHelper.h"
    #include "QIOConsumableBuffer.h"
    
    #include <exception>
    #include <memory>
    
    #include <QObject>
    #include <QAudioOutput>
    #include <QAudio>
    #include <QAudioFormat>
    #include <QAudioDeviceInfo>
    
    namespace connect
    {
    	namespace audio
    	{
    		/**
    		 * QT Multimedia output module
    		 * @author Michael A. Leonetti
    		 * @date 5/15/2018
    		 * @copyright Sonarcloud, 2018
    		 */
    		class QTOut : public Out
    		{
    			Q_OBJECT
    
    			public:
    				/**
    				 * Our custom C-tor
    				 */
    				QTOut( int samplerate, int channels, const In::shared_ptr &in, Queue *queue ) :
    					Out( samplerate, channels, in, queue ),
    					samples_multiplier_( channels<<1 ), // Channels multiplied by 2 for the bitdepth
    					buffer_( new QIOConsumableBuffer( this ) )
    
    				{
    				}
    
    				/**
    				 * Delete this later
    				 */
    				virtual ~QTOut()
    				{
    					//if( audio_output_ )
    					//	audio_output_->deleteLater();
    					LOG_TRACE( logger() ) << __PRETTY_FUNCTION__;
    				}
    
    				/**
    				 * Create a shared pointer
    				 */
    				template<typename... Args>
    				static std::shared_ptr<QTOut> create( Args&&... args )
    				{
    					return( qthelper::make_shared<QTOut>( args... ) );
    				}
    
    				/**
    				 * Write PCM data to the QT object
    				 */
    				void writeSamples( const int16_t *pcm, size_t samples ) override
    				//void writePCM( const QByteArray &pcm )
    				{
    					// Make sure we don't write after we've been done
    					if( done_ )
    						throw std::runtime_error( "Audio out write after done." );
    
    					//LOG_TRACE( logger() ) << "Samples written " << samples;
    
    
    					// The size we are writin
    					const size_t bytes = samples*samples_multiplier_;
    
    					// Write the audio data
    					buffer_->write( (char*)pcm, bytes );
    					//const qint64 written = buffer_->write( (char*)pcm, bytes );
    
    					/*
    					if( written<0 )
    						LOG_TRACE( logger() ) << "Error: " << buffer_->errorString();
    					*/
    
    					// Do we have our out device?
    					//if( audio_state_==QAudio::StoppedState )
    					if( audio_state_!=QAudio::ActiveState )
    						audio_output_->start( buffer_ ); // Start us (again?)
    
    					//LOG_TRACE( logger() ) << "Bytes decoded: " << bytes << ", written: " << written << ", buffer size: " << buffer_->size();
    
    				}
    
    				/**
    				 * Implement cancel
    				 */
    				void cancel() override
    				{
    					
    					// Set the state
    					audio_state_ = QAudio::StoppedState;
    					// Stop the audio
    					audio_output_->stop();
    					// Quit now
    					reportDoneToQueue();
    				}
    
    				/**
    				 * Our initializer
    				 */
    				void init() override
    				{
    					// Do NOT double init
    					if( initted_ )
    						throw std::runtime_error( "Audio out already initialized." );
    
    					initted_ = true;
    
    					// init up
    					Out::init();
    
    					// Init our device
    					QAudioFormat format;
    
    					LOG_DEBUG( logger() ) << "Initializing " << samplerate_ << " Hz, " << channels_ << " Ch, 16-bit signed PCM";
    
    					format.setSampleRate( samplerate_ );
    					format.setChannelCount( channels_ );
    					format.setSampleSize( 16 ); // 16-bit audio, 2 bytes each
    					format.setCodec( "audio/pcm" );
    					format.setByteOrder( QAudioFormat::LittleEndian );
    					format.setSampleType( QAudioFormat::SignedInt );
    
    					// See if we support this
    					QAudioDeviceInfo info( QAudioDeviceInfo::defaultOutputDevice() );
    
    					if( !info.isFormatSupported( format ) )
    						throw std::runtime_error( "Audio format not supported by sound hardware: "+std::to_string( samplerate_ )+" Hz "+std::to_string( channels_ )+" Ch 16-bit Signed PCM" );
    
    					// Creat the audio object
    					audio_output_ = new QAudioOutput( format, this );
    
    					// Open our buffer
    					buffer_->open( QIODevice::ReadWrite );
    
    					// Grab the callback for state changes
    					connect( audio_output_, SIGNAL( stateChanged(QAudio::State) ), this, SLOT( handleStateChanged(QAudio::State) ) );
    
    					//LOG_TRACE( logger() ) << "Initted audio.";
    				}
    
    			private slots:
    
    				/**
    				 * State changed slotifier
    				 */
    				void handleStateChanged( QAudio::State state )
    				{
    					// Save the state
    					audio_state_ = state; // Using the new state
    
    					// Process the state
    					switch( state )
    					{
    						case QAudio::IdleState:
    						{
    							const qint64 bytes_available = buffer_->bytesAvailable();
    
    							LOG_DEBUG( queue().logger() ) << "Audio now in idle state. Bytes left=" << bytes_available;
    
    							const QAudio::Error error = checkAudioError();
    
    							// Are we done?
    							if( done_ and ( (error!=QAudio::NoError and error!=QAudio::UnderrunError) or bytes_available<=0 ) )
    							{
    								reportDoneToQueue();
    								return; // Do NOT process anymore because the class is GONE
    							}
    							else
    							{
    								LOG_DEBUG( queue().logger() ) << "Suspending...";
    							}
    
    							// Free the IO device. We'll get another one anyway.
    							//io_device_ = nullptr;
    
    							break;
    						}
    						case QAudio::ActiveState:
    							LOG_DEBUG( queue().logger() ) << "Audio now in active state.";
    							break;
    						case QAudio::SuspendedState:
    							LOG_DEBUG( queue().logger() ) << "Audio now in suspended state.";
    							checkAudioError();
    							break;
    						case QAudio::StoppedState:
    							LOG_DEBUG( queue().logger() ) << "Audio now in stopped state.";
    
    							//io_device_ = nullptr;
    							break;
    						default:
    							LOG_DEBUG( queue().logger() ) << "Unhandled audio state: " << state;
    						/*
    						case QAudio::InterruptedState:
    							LOG_TRACE( queue().logger() ) << "Audio now in interrupted state.";
    							break;
    						*/
    					}
    				}
    			public:
    				/**
    				 * Check for an audio error
    				 */
    				inline QAudio::Error checkAudioError()
    				{
    					QAudio::Error error = audio_output_->error();
    
    					if( error!=QAudio::NoError )
    						LOG_ERROR( queue().logger() ) << "Audio error:  " << audioError( error );
    
    					return( error );
    				}
    
    				/**
    				 * Get the audio error
    				 */
    				inline static const char *audioError( QAudio::Error error )
    				{
    					switch( error )
    					{
    						case QAudio::NoError:
    							return( "No errors have occurred." );
    						case QAudio::OpenError:
    							return( "An error occurred opening the audio device." );
    						case QAudio::IOError:
    							return( "An error occurred during read/write of audio device." );
    						case QAudio::UnderrunError:
    							return( "Audio data is not being fed to the audio device at a fast enough rate." );
    						case QAudio::FatalError:
    							return( "A non-recoverable error has occurred, the audio device is not usable at this time." );
    					}
    
    					return( "Unknown Error." );
    				}
    
    				/**
    				 * Override flush
    				 */
    				void flush() override
    				{
    					// Are we already done?
    					if( done_ )
    						throw std::runtime_error( "Audio out already flushed." );
    
    					LOG_TRACE( logger() ) << "Flushing";
    
    					// Set the done value
    					done_ = true;
    
    					// Are we already in an inactive state?
    					if( audio_state_!=QAudio::ActiveState )
    					{
    						const QAudio::Error error = audio_output_->error();
    						
    						if( error!=QAudio::NoError and error!=QAudio::UnderrunError and buffer_->bytesAvailable()>0 )
    						{
    							LOG_ERROR( queue().logger() ) << "Still more left. Waiting until audio empties buffer.";
    
    							/*
    							audio_output_->stop();
    							audio_output_->start( buffer_ ); // Start us (again?)
    							*/
    						}
    						else
    							reportDoneToQueue();
    					}
    				}
    			private:
    				QAudioOutput *audio_output_=nullptr; //!< Our QT audio output object
    
    				int samples_multiplier_; //!< How much to multiply the sample count by
    
    				bool initted_ = false; //!< Don't double init
    				bool done_ = false; //!< Whether or not we've been flushed
    				QAudio::State audio_state_ = QAudio::StoppedState; //!< our audio state
    				//QIODevice *io_device_=nullptr; //!< What we'll be writing to
    				QIOConsumableBuffer *buffer_;
    		};
    	}
    }
    
    #endif
    

  • Lifetime Qt Champion

    Hi,

    What version of Qt are you using ?
    On what version of macOS ?



  • Oh wow. Can't believe I didn't post that. Sorry.

    Qt 5.11.1 via homebrew on MacOS 10.13.6



  • @Michael-A.-Leonetti said in QAudioOutput becomes idle and stays idle when buffer underrun on MacOS:

      			if( audio_state_!=QAudio::ActiveState )
      				audio_output_->start( buffer_ ); // Start us (again?)
    

    The problem in the code was here. Start being called on the same buffer twice caused some sort of a lock up. Changing the code to make sure start only get called ONCE fixed the issue.

    So this issue is solved. Hopefully this saves somebody the days that it took me to figure out this simple problem.


  • Lifetime Qt Champion

    Glad you found out and thanks for sharing !

    Happy coding :)


Log in to reply