Incomplete data in QDataStream when reading from QTcpSocket

[RBLL]

So I have a strange issue that when I read data (QDataStream) on a QTcpSocket: some of the data seems to be missing. The bytesAvailable() function will return the proper amount of bytes to be read, but QDataStream doesn't seem to hold all the bytes.

First of all, this is how the data looks:

bufferX always contains 768 floats and bufferY always contains 5376 floats. Therefore, I would expect the total data to be sent to be (excluding block size) : int + 768 floats + 5376 floats = 4 + 3072 + 21504 = 24580 bytes.

Now, here is the sender code:

void ClientSocket::serverTaskResult()
{
    QByteArray block;
    QDataStream out(&block, QIODevice::WriteOnly);
    out.setVersion(QDataStream::Qt_DefaultCompiledVersion);

    out << quint16(0);

    out << mServerTask->getNPoints();

    for (size_t i = 0; i < BUFFERX_SIZE; i++)
        out << mServerTask->getBufferX(i);

    for (size_t i = 0; i < BUFFERY_SIZE; i++)
        out << mServerTask->getBufferY(i);

    out.device()->seek(0);
    out << quint16(block.size() - sizeof(quint16));

    write(block);

    out << quint16(0xFFFF);
}

And here is the receiver code:

void TestClient::recoverResult()
{
    QDataStream in(&mTcpSocket);
    in.setVersion(QDataStream::Qt_DefaultCompiledVersion);

    float wBufferX[BUFFERX_SIZE];
    float wBufferY[BUFFERY_SIZE];
    int wNPoints;

    forever{
        if (mNextBlockSize == 0) 
        {
            qint64 nBytesAvailable = mTcpSocket.bytesAvailable();
            if (nBytesAvailable < sizeof(quint16))
                break;
            in >> mNextBlockSize;
        }

        if (mNextBlockSize == 0xFFFF) 
        {
            closeConnection();
            break;
        }

        if (mTcpSocket.bytesAvailable() < mNextBlockSize)
            break;

        for (size_t i = 0; i < BUFFERX_SIZE; i++)
            in >> wBufferX[i];

        for (size_t i = 0; i < BUFFERY_SIZE; i++)
            in >> wBufferY[i];

        in >> wNPoints;

        mNextBlockSize = 0;
    }
}

Now, the first odd thing I'm noticing is that nBytesAvailable always has a value of 49158, which is about double of what I'm expecting. How is it that I'm receiving twice as many bytes as expected?

Secondly, since I have all these bytes available, I would expect the QDataStream to be able to properly fill in the buffers. However, after anywhere between 315 and 350 floats, the QDataStream seems to contain unavailable data. That is, wBufferX will have defined (and correct) values in its first 315-350 indexes and unknown values afterwards. I don't understand how that is since bytesAvailable() clearly indicates that almost 50 000 bytes are on the socket. What am I missing?

Your help is greatly appreciated! Thanks!

[Christian Ehrlicher]

wNPoints is written before the values but read after them - this does not look correct.

[RBLL]

That is a small typo I made when copying the code. I initially read if after the 2 buffers but when I did my small data illustration I put in front so I changed it in the code of the server but forgot to update the client. In the end, it shouldn't change anything related to the QDataStream not having all available bytes that are on the socket.

[Christian Ehrlicher]

@RBLL said in Incomplete data in QDataStream when reading from QTcpSocket:

out << quint16(0xFFFF);

This also does nothing.

I would also check the number of bytes written on the sender side - are you sure it's not called twice?

And since it's not the real code - did you maybe forgot something else?

[RBLL]

Ya that is again a mistake on my part. The line is supposed to go before the
write(block). I actually found the problem, but I'm still clueless as to what happens:

In my server, I get these bufferX and bufferY from another object like so:

for (size_t i = 0; i < BUFFERX_SIZE; i++)
{
	out << mServerTask->getBufferX(i);
	wBufferX[i] = mServerTask->getBufferX(i);
}

Now, if I execute this, I have the same behavior as I did previously: after index 315-350, values are unknown. However, executing this instead (note the commented line):

for (size_t i = 0; i < BUFFERX_SIZE; i++)
{
	//out << mServerTask->getBufferX(i);
	wBufferX[i] = mServerTask->getBufferX(i);
}

Will result in wBufferX containing all the data. It seems the QDataStream is interfering / corrupting the data?

[Christian Ehrlicher]

@RBLL said in Incomplete data in QDataStream when reading from QTcpSocket:

getBufferX

Is this function really const and does not modify the internal state of mServerTask ?
Check it by simply calling mServerTask->getBufferX(i); twice in your loop without doing something with the return value for the second call or look into the implementation.

[SGaist]

Hi,

Did you consider using read transactions ?

[Christian Ehrlicher]

@SGaist said in Incomplete data in QDataStream when reading from QTcpSocket:

Did you consider using read transactions ?

But then you have to make sure that your QDataStream stays alive all the time since this will not work with sequential devices otherwise:

void foo::onDataAvailable()
{
  QDataStream ds(m_socket);
  ds.startTransaction()
  if (readFromDataStream(ds))
    ds.commitTransaction();
  else
    ds.rollbackTransaction();
}

I had this error in my codebase once and it took some time until I figured out what went wrong.

[RBLL]

I just validated that calling mServerTask->getBufferX(i) twice does not modify the data. The function is a single line that returns a float. I'm not familiar with read transactions but I'll look into those to try and work around my issue. I'm still puzzled as to what happens with the QDataStream that makes my data "corrupted"...

[Christian Ehrlicher]

@RBLL said in Incomplete data in QDataStream when reading from QTcpSocket:

out << mServerTask->getBufferX(i);

So when getBuffer() is const this line can not do anything harmful to wBufferX

[RBLL]

The function was not declared const, but even when making it const, I have the same behavior. In any case, that is the get function : it does not modify the data:

const float ServerTask::getBufferX(size_t index)
{
	return mBufferX[index];
}

Now, for debugging purpose, I added an extra loop in front to see if I would have different results like so:

for (size_t i = 0; i < BUFFERX_SIZE; i++)
{
	wBufferX2[i] = ServerTask->getBufferX(i);
}

for (size_t i = 0; i < BUFFERX_SIZE; i++)
{
	out << ServerTask->getBufferX(i);
	wBufferX[i] = ServerTask->getBufferX(i);
}

Now, this is the what I have inside wBufferX2:

And this is what I have inside wBufferX:

[Christian Ehrlicher]

@RBLL said in Incomplete data in QDataStream when reading from QTcpSocket:

wBufferX

Is this memory big enough? Are you on Linux? I would take a look with valgrind since this really looks like a memory allocation problem

[RBLL]

No, this is on Windows. But both buffers are declared the same like so:

float wBufferX[BUFFERX_SIZE];
float wBufferX2[BUFFERX_SIZE];

It would be surprising if it had anything to do with memory allocation but at this point I don't see many other options...

[hskoglund]

Hi, it could be a transmission timing problem, about 350 floats is pretty close to MTU size (the normal Ethernet packet size). So it could be that while the first packet is xmitted, out<<... wreaks havoc with the rest of the data waiting to be xmitted (just guessing :-)

[RBLL]

When debugging I had a breakpoint before the write command. Nothing had been sent to the client yet when those 2 buffers had different values.

[hskoglund]

Maybe that QByteArray is the culprit, you could try rewrite into more vanilla standard:

...
{
    QByteArray block;
    QBuffer buffer(&block);
    buffer.open(QIODevice::WriteOnly);
    QDataStream out(&buffer);
...

at least you would expose more stuff to the debugger :-)