QImage streaming over TCP

robcont_

This is my first post on this forum so first of all: "Hi everyone and thanks to help grow this awesome community".
I'm working for an app for remote assistance. Once sampled the video stream and got the single frame as QImage, I have to send it to a client via TCP. Unfortunatelly, something is going wrong and I cannot get over it.
Here's my code.

/*
 * SERVER WRITE
 */
void VCameraServer::sendTcpData()
{
    if (mTcpClient->state() == QTcpSocket::ConnectedState) {

        // initialize stream
        QByteArray block; 
        QDataStream out(&block, QIODevice::WriteOnly);
        out.setVersion(QDataStream::Qt_5_11);

        // initialize data
        QImage image(filePath);
        qint64 size = image.sizeInBytes();
 
        // serialize
        out << size << filePath << image;

        qDebug() << "Sent:" << size << "bytes; img name:" << filePath << "image format:" << image.format();

        // send over TCP
        mTcpClient->write(block);
        mTcpClient->waitForBytesWritten(-1);
    }
}

/*
 * CLIENT READ
 */
void VCameraClient::newTcpDataRead()
{   
    // initialize stream
    QDataStream in(mTcpSocket);
    in.setVersion(QDataStream::Qt_5_11);
    
    in.startTransaction();

    // initialize data
    qint64  size;
    QString name;
    QImage  image;

    // read from stream
    in >> size >> name >> image;

    if ( !in.commitTransaction() ) return;

    qDebug() << "Recv:" << size << "bytes; img name:" << name << "; image format:" << image.format();   

    // data is complete, do something with it
    if (image.isNull) qDebug() << "Recv  image is null!";
    else update();
}

This is the log.

/*
 * SERVER 
 */
Sent: 1228800 bytes; img name: "/home/apps/Gallery/Screenshots/myImageTest" ; image format: 4

/*
 * CLIENT 
 */
QIODevice::ungetChar  (QTcpSocket): Called while transaction is in progress
QIODevice::ungetChar  (QTcpSocket): Called while transaction is in progress
QIODevice::ungetChar  (QTcpSocket): Called while transaction is in progress
QIODevice::ungetChar  (QTcpSocket): Called while transaction is in progress
QIODevice::ungetChar  (QTcpSocket): Called while transaction is in progress
QIODevice::ungetChar  (QTcpSocket): Called while transaction is in progress
QIODevice::ungetChar  (QTcpSocket): Called while transaction is in progress
QIODevice::ungetChar  (QTcpSocket): Called while transaction is in progress
QIODevice::ungetChar  (QTcpSocket): Called while transaction is in progress
QIODevice::ungetChar  (QTcpSocket): Called while transaction is in progress
QIODevice::ungetChar  (QTcpSocket): Called while transaction is in progress
QIODevice::ungetChar  (QTcpSocket): Called while transaction is in progress
QIODevice::ungetChar  (QTcpSocket): Called while transaction is in progress
QIODevice::ungetChar  (QTcpSocket): Called while transaction is in progress
QIODevice::ungetChar  (QTcpSocket): Called while transaction is in progress
QIODevice::ungetChar  (QTcpSocket): Called while transaction is in progress
QIODevice::ungetChar  (QTcpSocket): Called while transaction is in progress
QIODevice::ungetChar  (QTcpSocket): Called while transaction is in progress
Recv: 1228800 bytes; img name: "/home/apps/Gallery/Screenshots/myImageTest" ; image format: 0
Recv  image is null!

What am I doing wrong?

raven-worx

@robcont_
i am just guessing here, but it's very likely that the datastream (client) reads from the socket while it hasn't received all the data yet

you could try to read every data (once you have made sure that you have received all of it) from the socket to a QByteArray. Then use Qbuffer as an input QIODevice to the data stream and let it read from it.

robcont_

@raven-worx
Tnx for your reply, I'll try to implement your solution. Could you also share any sample please?

Anyway, I'm not understanding why my code does not work. From Fortune Client Example, docs says:

Now, TCP is based on sending a stream of data, so we cannot expect to get the entire fortune in one go. Especially on a slow network, the data can be received in several small fragments. QTcpSocket buffers up all incoming data and emits readyRead() for every new block that arrives, and it is our job to ensure that we have received all the data we need before we start parsing.
For this purpose we use a QDataStream read transaction. It keeps reading stream data into an internal buffer and rolls it back in case of an incomplete read. We start by calling startTransaction() which also resets the stream status to indicate that new data was received on the socket. We proceed by using QDataStream's streaming operator to read the fortune from the socket into a QString. Once read, we complete the transaction by calling QDataStream::commitTransaction(). If we did not receive a full packet, this function restores the stream data to the initial position, after which we can wait for a new readyRead() signal.

In my case the QImage is presumably sent over the network in different chunks. Why aren't they correctly rebuilt with the transaction mechanism?

J.Hilk

@robcont_

on the client side,
what does mTcpSocket->bytesAvailable(); return? I doubt it's not the expected ~1228800 bytes

If newTcpDataRead is called as soon as the socket emits the signal readyRead, than most likely all your data is not yet arrived and you have to buffer what allready arrived, like @raven-worx said.

robcont_

@J.Hilk

what does mTcpSocket->bytesAvailable(); return? I doubt it's not the expected ~1228800 bytes

Of course not. mTcpSocket->bytesAvailable() returns the size of every chunk received. Moreover, their sum is bigger than 1228800 bytes, due to the additional data used by QDataStream to encapsulate the stream (if I'm not mistaken).

If newTcpDataRead is called as soon as the socket emits the signal readyRead, than most likely all your data is not yet arrived and you have to buffer what allready arrived, like @raven-worx said.

Yes, I connected the newTcpDataRead() slot with the readyRead() signal of QTcpSocket. When I should call it to ensure that the transaction mechanism works properly?

raven-worx

@robcont_ said in QImage streaming over TCP:

When I should call it to ensure that the transaction mechanism works properly?

You can check if the received data ends with a "terminating data sequence".
When the client receives this data sequence, it removes it from the received data and continues processing.
A terminating data sequence could be anything you like (but it should be unique enough so that it isn't be contained in the data transferred)

Alternatively you could send the final data size in the very first 2-4 bytes for example. So the client knows how much data is left / when all the data was received.
I am not talking about the image byte size here, but rather the byte size of the data the QDataStream produced in the end.

robcont_

@raven-worx
Following your suggestion, now I can send/receive a QImage over TCP.
Here's my code.

/*
 * SERVER WRITE
 */
void VCameraServer::sendTcpData(/*const QImage &frame*/)
{
    if (mTcpClient->state() == QTcpSocket::ConnectedState) {

        // initialize stream
        QByteArray block;
        QDataStream out(&block, QIODevice::WriteOnly);
        out.setVersion(QDataStream::Qt_5_11);

        // initialize data
        QImage image(filePath);

        // serialize
        out << qint64(0) << image;
        out.device()->seek(0);
        out << (qint64)(block.size() - sizeof(qint64));  // the size of the block to be sent

        // send over TCP
        qint64 written = mTcpClient->write(block);
        mTcpClient->waitForBytesWritten(-1);
        qDebug() << "written" << written << "of" << block.size();
    }
}

/*
 * CLIENT READ
 */
void VCameraClient::newTcpDataRead()
{       
    QDataStream in(mTcpSocket);
    in.setVersion(QDataStream::Qt_5_11);

    // initialize data
    QImage image;
    static qint64 imageSize = 0;

    if ( 0 == imageSize ) {
        if ( mTcpSocket->bytesAvailable() < (int)sizeof(qint64) ) return;
        in >> imageSize;
        qDebug() << imageSize;
    }

    if ( mTcpSocket->bytesAvailable() < imageSize ) return;
    in >> image;

    // data is complete, do something with it
    imageSize = 0;
    if (image.isNull()) qDebug() << "QImage is null!";
    else {
        mTcpFrame = image;
        update();
    }
}

Now one more question: is this code correct? Is there a way to speed up sending/receiving data?
As I said on the first post, I need these frames to rebuild a video-streaming. Tnx in advance.

raven-worx

@robcont_ said in QImage streaming over TCP:

As I said on the first post, I need these frames to rebuild a video-streaming. Tnx in advance.

the QImage serialized over QDataStream is done by converting the image to a PNG data.
Of course this approach isn't very well suited for streaming "video".

It depends what your goals are . Is it ok to receive a "stuttering" video (due to your approach by transferring each frame)?
It may be ok for a small resolution though.

Normally classic video streams are optimized for their purpose of course.
Like proper compression, drops of frames which are too far in the past to display, etc.

But also this involves more research on this topic (video streaming).

sharath

@raven-worx I have also same problem. i'm ok with small resolution. can you please provide some sample code?