Slow client receiver with MBs of data sent

[enne9]

Hello,
I'm trying to send 20MB of data in json format from the server to the client. With a smaller amount of
data everything was fine, but now the readyRead signal seems to be called many times and the transfer is slow. I'm using QSslSocket.

Server code:

QDataStream socketStream(m_serverSocket);
socketStream.setVersion(QDataStream::Qt_5_7);
socketStream << byteArray;

Client code:

void Client::onReadyRead()
{
    QByteArray jsonData;
    QDataStream socketStream(m_clientSocket);
    socketStream.setVersion(QDataStream::Qt_5_7);
    while (true) {
        socketStream.startTransaction();
        socketStream >> jsonData;
        if (socketStream.commitTransaction()) {
            QJsonParseError parseError;
            const QJsonDocument jsonDoc = QJsonDocument::fromJson(jsonData, &parseError);
            if (parseError.error == QJsonParseError::NoError) {
                if (jsonDoc.isObject())
                    jsonReceived(jsonDoc.object());
            } else {
                byteArrayReceived(jsonData);
            }
        } else {
            break;
        }
    }
}

I try to search in the forum but wasn't able to find a solution, can anyone please help? Thanks.

[CKurdu]

You can use bytesAvailable method and read the data only when available bytes big then check the data if it is a valid json document. I think using a "while loop" is the wrong thing on readyread signal.

[SGaist]

Hi,

Take a look at the QDataStream documentation for an example of use of transactions.

[enne9]

@SGaist I understand now, with transactions I'm performing multiple incomplete reads and this is slowing down the whole process. Can you suggest me a clean and fast way to perform the read to reduce the delay to the minimum?

[enne9]

@CKurdu Do you mean checking if (m_clientSocket->bytesAvailable() < threshold) to reduce the incomplete reads? I'm struggling to understand where to put it... Do you have any suggestions?

[enne9]

@enne9 The problem is that I don't know the total amount of data that I'm gonna receive.

[mrjj]

@enne9
Hi
Do you control the server also?
If yes, I would consider adding an end of data marker to look for as trying to parse the
the incoming json might be a bit heavy on 20 MB files.

[CKurdu]

@enne9

Hi enne9
If I were you, first I prepare my data stream on the server as below.

QDataStream socketStream(m_serverSocket);
SerializeSize size;
quint64 json_size = size(json_content);
const qint64 end_of_stream = 4444; // what you prefer
socketStream.setVersion(QDataStream::Qt_5_7);
socketStream <<json_size<< json_content<<end_of_stream;

SerializeSize class is for calculating the size of the json_content.

class SerializeSize
{
  QBuffer m_data;
  QDataStream m_stream;
public:
  SerializeSize();
  template <typename T>
  quint64 operator ()(const T & t) {
    m_data.seek(0);
    m_stream << t;
    return m_data.pos();
  }
};
SerializeSize::SerializeSize()
{
    m_stream.setDevice(&m_data);
    m_data.open(QIODevice::WriteOnly);
};

On your client app, firstly read the json_content size and then start waiting to finish to came enough data. After that start deserialize content and parse json_content.

void Client::onReadyRead()
{
   
    if(m_exptected_json_size == 0)
    {
        if(!extract_content_size())
        {
            return; //wait to finish
        }
    }
   m_received_data.append(tcpSocket->readAll());
    if(m_exptected_json_size > 0 && m_received_data.size()>m_exptected_json_size )
    {
        if(parseJson())
        {
            qDebug()<<"Show Content" ;
            m_exptected_json_size = 0;
     
        }else{
            qDebug()<<"Continue Reading";
        }
    }
}

bool Client::extract_content_size()
{
    quint64 asize = tcpSocket->bytesAvailable();
    if(asize > treshold)
    {
       m_received_data.append(tcpSocket->readAll());
       QDataStream in;
       QBuffer in_buffer;
       in_buffer.setBuffer(&m_received_data);
       in_buffer.open(QIODevice::ReadOnly);
       in.setDevice(&in_buffer);
       in.setVersion(QDataStream::Qt_5_10);
       quint64 size = 0;
       in>>size;
       if(size>0)
       {
           m_exptected_json_size = size;
           in_buffer.close();
           return true;
       }
       in_buffer.close();
       return false;

    }
    return false;
}

bool Client::parseJson()
{
    int eof;
    QByteArray json_data;
    QDataStream in;
    m_buffer.setBuffer(&m_received_data);
    if(!m_buffer.open(QIODevice::ReadOnly))
        return false;
    in.setDevice(&m_buffer);
    in.setVersion(QDataStream::Qt_5_10);
    in.startTransaction();
    quint64 json_size;
    in>>json_size>>json_data>>eof;
    if(!in.commitTransaction())
    {
        m_buffer.close();
        return false;
    }
    m_buffer.close();
    if(eof != end_of_file)
    {
        return false;
    }
    QJsonParseError parseError;
    const QJsonDocument jsonDoc = QJsonDocument::fromJson(json_data, &parseError);
            if (parseError.error == QJsonParseError::NoError) {
                if (jsonDoc.isObject())
                    m_received_data.clear();
                    return true;
            } else {
                return false;
            }
    return true;
}

I tried the code with big size json data. It is working.
Source codes
Server App
Client App

[enne9]

@CKurdu Thank you very much, I adapted your code to my application and now it's really fast!

[CKurdu]

@enne9
You are welcome.