Trouble playing sound directly out of raw data vectors

[mrjj]

Hi
Im wondering if you have something that runs out of scope?

does it crash here

...
audio->start(input);
} <---

[beginner123]

I tried to run my program many times. Sometimes the program played the sound before it crashed, but many other times, it simply crashed without playing a sound. Also, when the sound was played, it sounded a bit weird and distorted.

I have to be honest that I don't totally understand all the codes that I put together here. I put together this program by borrowing snippets from other places that I searched at Qt Forum and elsewhere online.

One thing that puzzles me a lot is that the data type of my QVector is qreal, whereas in the sampleSize is set as 16 bits in the QAudioFormat object. This does not make sense to me. So, I tried to change both of them to be the same type (e.g., quint16 and 16 bits, or qreal and 64 bits ), but none of them worked.

Also, I want to say that, when the times that the sound got played out, it sounded a bit weird. Maybe the data were not converted correctly (from QVector to QByterArray) or the data did not agree with the audio format specified in this program, or something else.

I am getting very confused, and would really need someone's guidance to help me out.

Thank you so much for your time and willingness to help me out!!!

[beginner123]

@mrjj

This is a good comment. Thank you!

I don't think I have anything outside of the scope of this on_pushButton_clicked() function. I double checked. It is a simple empty Qt Widgets Application, with one pushbutton only.

To follow your comment, I temporally took out all the codes within this on_pushButton_clicked() function. This program ran with no crashes (and no sound output as well, as they got temporarily removed).

[mrjj]

Ok
if its not a out of scope problem, i think you are on right track that maybe
the actual data format and what you tell in QAudioFormat is not matching.
That could also crash.

I would insert some
qDebug() << "samples:" << n;

and have a look at the QAudioFormat settings with F1.

Also it does seems odd that
audioFormat.setSampleType(QAudioFormat::SignedInt);
but it seems to be qreal you have.
so i tried
qDebug() << audio->error() << "using " << deviceInfo.deviceName();
it seems it dont like my default output device for some reason.

I think you have to read the docs and debug it on your pc to make it play that sinus wave.
It seems close but some small error.

First i tried your code directly and it also crashed for me.
then i tried
audioFormat.setSampleType(QAudioFormat::Float);

and it stopped crashing but said
QAudioOutput: open error

[beginner123]

@mrjj

Thank you so much for taking time to read my codes, test them, and give me feedbacks! Greatly appreciated!!!

After many many trials and errors (and searching all resources that I could possibly find), I finally figured out a way to make this program to work - without crashing.

I ended up using a loop to go over each data point, and manually converted each data point from qreal (in QVector) to qint16 (in QByteArray). In our QAudioFormat, I set setSampleSize to 8 and SampleType to SignedInt.

Although this time the revised code worked, I did not like that fact that I needed to loop through each data point. It did not seem efficient. Is there a way to improve it?

Also, although the sound did sound better than before (when the program crashed easily), the pure tone still sounded a little distorted. I am not sure if I am too picky on sound quality. But is there is a way to further improve it?

Any comments, suggestions, or hints will be greatly appreciated!!!

For your review, I have pasted my revised code here.

void MainWindow::on_pushButton_clicked()
{

   // initialize parameters
   #define MAX_VAL_16BIT (2 ^ (16 - 1)) - 10   // Because our signal goes above and below zeros, the entire range of (2 ^ 16) needs to be divided by 2 ==> (2 ^ 16) / 2  ==> 2 ^ (16-1). Also, we substract 10 to avoid peak-clipping.
   qreal sampleRate = 10000;   // sample rate
   qreal duration = 1.000;     // duration in seconds
   qreal frequency = 1000;     // frequency
   const quint32 n = static_cast<quint32>(duration * sampleRate);   // number of data points

   // --- generate a QVector<qreal> that contains a sine wave signal ---
   QVector<qreal> sineWave(n);
   for (int i = 0; i < sineWave.size(); i++)
       sineWave[i] = (qreal)qSin(2.0 * M_PI * frequency * i / sampleRate);

   // --- transfer QVector data to QByteArray
   QByteArray* byteBuffer = new QByteArray();
   byteBuffer->resize(n);

   for (int i = 0; i < sineWave.size(); i++)
   {
       qreal y = sineWave[i];   // transfer data to y
       y *= MAX_VAL_16BIT;     // scale up y from +-1 to its full range
       (*byteBuffer)[i] = (qint16)y;   // use qint16 (instead of quint16), because our waveform goes above and below zeros.
   }

   // create and setup a QAudioFormat object
   QAudioFormat audioFormat;
   audioFormat.setSampleRate(static_cast<int>(sampleRate));
   audioFormat.setChannelCount(1);
   audioFormat.setSampleSize(8);   // set the sample size in bits. Because we are feeding in QByteArray data (which is one byte per data sample), we will have 8 bits for the sample size here (i.e., 1 byte = 8 bits).
   audioFormat.setCodec("audio/pcm");
   audioFormat.setByteOrder(QAudioFormat::LittleEndian);
   audioFormat.setSampleType(QAudioFormat::SignedInt);   // use SignedInt, because our waveform goes above and below zero.

   // create a QAudioDeviceInfo object, to make sure that our audioFormat is supported by the device
   QAudioDeviceInfo deviceInfo(QAudioDeviceInfo::defaultOutputDevice());
   if(!deviceInfo.isFormatSupported(audioFormat))
   {
       qWarning() << "Raw audio format not supported by backend, cannot play audio.";
       return;
   }

   // Make a QBuffer with our QByteArray
   QBuffer* input = new QBuffer(byteBuffer);
   input->open(QIODevice::ReadOnly);   // set the QIODevice to read-only

   // Create an audio output with our QAudioFormat
   QAudioOutput* audio = new QAudioOutput(audioFormat, this);

   // start the audio (i.e., play sound from the QAudioOutput object that we just created)
   audio->start(input);

}

[mrjj]

Hi
Good work.
And so good with the comments. +100 for that.

It also now works here and i got a nice tone. :)
It sounds pretty clear in my headphones but
im no true audiophile so very hard to say.

You could optimize a little and remove
QVector<qreal> sineWave(n); complete and do all in one loop directly to
byteBuffer saving one loop.

Also, since you new both QBuffer
and audio each time button is pressed,
it will leak memory so
You can use QAudioOutput::stateChanged signal to know when finished
so you can clean up.

    // initialize parameters
#define MAX_VAL_16BIT (2 ^ (16 - 1)) - 10   // Because our signal goes above and below zeros, the entire range of (2 ^ 16) needs to be divided by 2 ==> (2 ^ 16) / 2  ==> 2 ^ (16-1). Also, we substract 10 to avoid peak-clipping.
  qreal sampleRate = 10000;   // sample rate
  qreal duration = 1.000;     // duration in seconds
  qreal frequency = 1000;     // frequency
  const quint32 n = static_cast<quint32>(duration * sampleRate);   // number of data points

  // --- QByteArray  buffer (changed, move moved over)
  QByteArray* byteBuffer = new QByteArray();
  byteBuffer->resize(n);

  // --- generate a buffer that contains a sine wave signal --- (changed)
  for (int i = 0; i < n; i++) {
    qreal sinVal =  (qreal)qSin(2.0 * M_PI * frequency * i / sampleRate) ;
    sinVal *= MAX_VAL_16BIT;     // scale up y from +-1 to its full range
    (*byteBuffer)[i] = (qint16)sinVal;    // use qint16 (instead of quint16), because our waveform goes above and below zeros.
  }

  // create and setup a QAudioFormat object
  QAudioFormat audioFormat;
  audioFormat.setSampleRate(static_cast<int>(sampleRate));
  audioFormat.setChannelCount(1);
  audioFormat.setSampleSize(8);   // set the sample size in bits. Because we are feeding in QByteArray data (which is one byte per data sample), we will have 8 bits for the sample size here (i.e., 1 byte = 8 bits).
  audioFormat.setCodec("audio/pcm");
  audioFormat.setByteOrder(QAudioFormat::LittleEndian);
  audioFormat.setSampleType(QAudioFormat::SignedInt);   // use SignedInt, because our waveform goes above and below zero.

  // create a QAudioDeviceInfo object, to make sure that our audioFormat is supported by the device
  QAudioDeviceInfo deviceInfo(QAudioDeviceInfo::defaultOutputDevice());
  if(!deviceInfo.isFormatSupported(audioFormat)) {
    qWarning() << "Raw audio format not supported by backend, cannot play audio.";
    return;
  }

  // Make a QBuffer with our QByteArray
  QBuffer* input = new QBuffer(byteBuffer);
  input->open(QIODevice::ReadOnly);   // set the QIODevice to read-only

  // Create an audio output with our QAudioFormat
  QAudioOutput* audio = new QAudioOutput(audioFormat, this);

// connect up signal stateChanged to a lambda to get feedback (changed)
  connect(audio, &QAudioOutput::stateChanged, [audio, input](QAudio::State newState) {
    if (newState == QAudio::IdleState ) { // Finished playing (no more data)
      qDebug() << "finish:";
      delete audio;
      delete input;
    }
    // could / should also handle more states. like error
  } );

  // start the audio (i.e., play sound from the QAudioOutput object that we just created)
  audio->start(input);

I think
QByteArray* byteBuffer = new QByteArray();
leaks.
docs says
"QBuffer doesn't take ownership of the QByteArray." for the setBuffer function
so i assume giving it in construtor is the same.

[beginner123]

Thank you SO MUCH for testing my code, and helping me! Greatly appreciated, particularly on a Christmas Eve!!!

One thing that I cannot understand is this line of code here.

{
    ...
    (*byteBuffer)[i] = (qint16)sinVal;   <--- 
}

Although this program works now and does not crash, this line of code confuses me a lot. It seems that I cast a number to qint16, and then put it to a QByteBuffer, which is supposed to take in one byte at a time.

Does this mean that I am using only the first byte of this number?

Shall I break down this qint16 into two bytes, and store them separately in *(byteBuffer) and *(byteBuffer + 1), or something like that?

I think this may have something to do with the slight distortion in sound quality, but I am not 100% sure.

Thanks a lot for your time, and willingness to help me out!!!

[mrjj]

Hi
Good catch.
I dumped the values
Debug() << (int) byteBuffer[i] << "->" << (qint16)sinVal << " = " << sinVal;
0 -> 0 = -1.1817e-12
1 -> 1 = 1.76336
2 -> 2 = 2.85317
2 -> 2 = 2.85317
1 -> 1 = 1.76336
0 -> 0 = 2.91934e-12

So its clear we cap the value.

I tried with
(*byteBuffer)[i] = qRound(sinVal);

but i cant hear the difference.

[beginner123]

@mrjj

Thanks a lot for testing this out for me! Greatly appreciated!!!

However, I still cannot understand the logic here. We have a sine wave that has n data samples, and each sample is converted to qint16 (i.e., 2 bytes). At the same time, we also created a QByteArray and resized it to n only. I suppose that this QByteArray can only accommodate n bytes. Isn't that a mismatch?

Also, this brings in an issue and a technique that I really want to learn in Qt. I tried to see the value that a pointer is pointing to (in Debug Mode). However, many times, what I got was "<no such value>".

When I used "qDebug() << *byteBuffer", the value (where the pointer pointed to) was displayed properly in the Application Output pane. My inability to see the value (where a pointer is pointing to) has limited my ability to see what is actually going on in the code.

If someone can teach me how to see the value (that a pointer is pointing to) in the Debug Mode, it will be VERY helpful. So, I can see and evaluate the situation as the program progresses.

Here is a screen shot that shows my trouble. For clarity, I circled the "<no such value>" in red. I also framed the QByteArray values with a blue pen.

With an intent to see those values, I right-clicked on byteBuffer variable in Debug Mode ---> clicked "Change Value Display Format" --> From there, I tried several different types of display formats (e.g., raw data, compact float, hexadecimal integer, etc.). However, none of them showed me the data that made sense to me.

At the moment when this screen shot was taken, the program was in its Debug Mode, and the iteration index i was 4.

I guess I am still very confused. For example, when i was 2, the sinVal is 0.927051 (as you can see from the Application Output pan), which got converted to qint16 (i.e., 2 bytes). We then send this data sample into a QByteArray. The question now (for me) is whether both bytes were saved in QByteArray, or just one byte was saved. So, it would safe if I can see those values through our QByteArray pointer directly in the Debug Mode.

If anyone can teach me how to see those values (in the Debug Mode), that will VERY VERY helpful. So that I can become more independent developing a program on Qt.

Thanks a lot again for your time and kindness to help me out!!!

[beginner123]

I really want to learn how to play sound directly out of a raw data vector, because sound is what I do for my profession most of the time.

Due to my limited knowledge and skills to show the value that a pointer is pointing to, I added some snippet in this program to see the two bytes of a qint16.

Here is the snippet that I have inserted inside the loop where the sine wave was created and assigned to a QByteArray.

// --- transfer QVector data to QByteBuffer
QByteArray *byteBuffer = new QByteArray();  // create a new instance of QByteArray class (in the heap, dynamically arranged in memory), and set its pointer to byteBuffer
byteBuffer->resize(sizeof(qint16) * n);  // resize byteBuffer to the total number of bytes that will be needed to accommodate all the n data samples that are of type qint16

for (quint32 i = 0; i < n; i++)
    {
        qreal sinVal = (qreal)qSin(2.0 * M_PI * frequency * i / sampleRate);  // create sine wave data samples, one at a time
        sinVal *= MAX_VAL_16BIT;    // scale up sinVal from +-1 to its full range

        // break down one qint16 into two bytes
        qint16 sample = (qint16)sinVal;  // save one data sample in a local variable, so I can break it down into two bytes
        char *ptr = (char*)(&sample);  // assign a char* pointer to the address of this data sample (that gets casted into char* type as well).
        char byte00 = *ptr;         // first byte
        char byte01 = *(ptr + 1);   // second byte

        // put byte data into QByteArray, one byte at a time
        (*byteBuffer)[2 * i] = byte00;   // put first byte into QByteArray
        (*byteBuffer)[2 * i + 1] = byte01;   // put second byte into QByteArray
}

In order to accommodate a total of 2*n bytes, I resized our QByteArray accordingly.

By inserting this snippet and set a break point inside the loop, I was able to see the two bytes of a qint16. I found that, in my previous examples, only the first byte was saved in QByteArray. The second byte was discarded. So, this time, I added both bytes into the QByteArray buffer.

Through the setup of breaking down a qint16 into two separate bytes, I was able to verify that both bytes were saved in QByteArray properly - for each data sample.

So, I went ahead and ran this program.

However, the sound came out twice as long as sit should be, and its frequency was about half than it should be. I guess it was because the QByteArray size got doubled.

However, if I change the SampleSize in QAudioFormat from 8 to 16 bits, The program ran through, but no sound was heard at all.

I will keep trying, but if someone can guide me through this, I would greatly appreciate it.

[beginner123]

I got it. I got it. It WORKED!!!!

I am SO HAPPY!!!

Here is the revised program.

void MainWindow::on_pushButton_clicked()
{

    // initialize parameters
    qreal sampleRate = 40000;   // sample rate
    qreal duration = 1.000;     // duration in seconds
    qreal frequency = 1000;     // frequency
    const quint32 n = static_cast<quint32>(duration * sampleRate);   // number of data samples

    // --- transfer QVector data to QByteBuffer
    QByteArray *byteBuffer = new QByteArray();  // create a new instance of QByteArray class (in the heap, dynamically arranged in memory), and set its pointer to byteBuffer
    byteBuffer->resize(sizeof(float) * n);  // resize byteBuffer to the total number of bytes that will be needed to accommodate all the n data samples that are of type float

    for (quint32 i = 0; i < n; i++)
    {
        qreal sinVal = (qreal)qSin(2.0 * M_PI * frequency * i / sampleRate);  // create sine wave data samples, one at a time

        // break down one float into four bytes
        float sample = (float)sinVal;  // save one data sample in a local variable, so I can break it down into four bytes
        char *ptr = (char*)(&sample);  // assign a char* pointer to the address of this data sample
        char byte00 = *ptr;         // 1st byte
        char byte01 = *(ptr + 1);   // 2nd byte
        char byte02 = *(ptr + 2);   // 3rd byte
        char byte03 = *(ptr + 3);   // 4th byte

        // put byte data into QByteArray, one byte at a time
        (*byteBuffer)[4 * i] = byte00;       // put 1st byte into QByteArray
        (*byteBuffer)[4 * i + 1] = byte01;   // put 2nd byte into QByteArray
        (*byteBuffer)[4 * i + 2] = byte02;   // put 3rd byte into QByteArray
        (*byteBuffer)[4 * i + 3] = byte03;   // put 4th byte into QByteArray
    }

    // create and setup a QAudioFormat object
    QAudioFormat audioFormat;
    audioFormat.setSampleRate(static_cast<int>(sampleRate));
    audioFormat.setChannelCount(1);
    audioFormat.setSampleSize(32);   // set the sample size in bits. We set it to 32 bis, because we set SampleType to float (one float has 4 bytes ==> 32 bits)
    audioFormat.setCodec("audio/pcm");
    audioFormat.setByteOrder(QAudioFormat::LittleEndian);
    audioFormat.setSampleType(QAudioFormat::Float);   // use Float, to have a better resolution than SignedInt or UnSignedInt

    // create a QAudioDeviceInfo object, to make sure that our audioFormat is supported by the device
    QAudioDeviceInfo deviceInfo(QAudioDeviceInfo::defaultOutputDevice());
    if(!deviceInfo.isFormatSupported(audioFormat))
    {
        qWarning() << "Raw audio format not supported by backend, cannot play audio.";
        return;
    }

    // Make a QBuffer with our QByteArray
    QBuffer* input = new QBuffer(byteBuffer);
    input->open(QIODevice::ReadOnly);   // set the QIODevice to read-only

    // Create an audio output with our QAudioFormat
    QAudioOutput* audio = new QAudioOutput(audioFormat, this);

    // connect up signal stateChanged to a lambda to get feedback
    connect(audio, &QAudioOutput::stateChanged, [audio, input](QAudio::State newState)
    {
        if (newState == QAudio::IdleState)   // finished playing (i.e., no more data)
        {
            qDebug() << "finished playing sound";
            delete audio;
            delete input;
            //delete byteBuffer;  // I tried to delete byteBuffer pointer (because it may leak memories), but got compiler error. I need to figure this out later.
        }
        // should also handle more states, e.g., errors. I need to figure out on how to do this later.
    });

    // start the audio (i.e., play sound from the QAudioOutput object that we just created)
    audio->start(input);

}

Major changes that I have made are listed here.

• I used Float --> break it down into 4 bytes --> assign all four bytes into QByteArray (one byte at a time, in Little-Endian byte order)

• I set SampleSize to 32 bits

• I set SampleType to Float

And, TADA! I worked like a charm. The sound quality was very good. It is like a sound played out of a professional software.

It seems to me now that the SampleType of SignedInt (or UnSignedInt) only takes in one byte (and the second byte is discarded - at least I don't know how to get them to work yet), whereas Float takes in four bytes.

This is probably the reason that I cannot make qint16 to work, because the SampleSize was meant to be either 8 bits (for SignedInt or UnSignedInt that has only one byte) or 32 bits (for Float).

I guess as I have now made Float to work, I don't really need to worry about qint16. Float has a much better resolution than SignedInt. In my previous trials, the sound quality was not as good. I think it was due to the fact that we only used the resolution of one byte, not to mention that the only one byte we used contained only integer numbers, instead of floating numbers with decimal precisions.

To prevent people like me making the same mistakes later, I would suggest to add some explanation in the Qt HELP documentation for QAudioFormat::SetSampleSize(). As it is for now, it reads "Sets the sample size to the sampleSize specified, in bits.
This is typically 8 or 16, but some systems may support higher sample sizes."

This was misleading, as 32 bits should be used for Float. So, if people could add "use 32 bits for Float" or something like that, it would be great.

Again, I am VERY HAPPY that it WORKS.

Thank you SO MUCH!!! I don't think I can solve this problem without your help and inspiration!!!

I wish you all a Merry Christmas and a HAPPY NEW YEAR!!!

[mrjj]

@beginner123
Super!
Good work. Thank you for reporting back with new code
Merry Christmas and a happy new Year!