Important: Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

Slow client receiver with MBs of data sent



  • Hello,
    I'm trying to send 20MB of data in json format from the server to the client. With a smaller amount of
    data everything was fine, but now the readyRead signal seems to be called many times and the transfer is slow. I'm using QSslSocket.

    Server code:

    QDataStream socketStream(m_serverSocket);
    socketStream.setVersion(QDataStream::Qt_5_7);
    socketStream << byteArray;
    

    Client code:

    void Client::onReadyRead()
    {
        QByteArray jsonData;
        QDataStream socketStream(m_clientSocket);
        socketStream.setVersion(QDataStream::Qt_5_7);
        while (true) {
            socketStream.startTransaction();
            socketStream >> jsonData;
            if (socketStream.commitTransaction()) {
                QJsonParseError parseError;
                const QJsonDocument jsonDoc = QJsonDocument::fromJson(jsonData, &parseError);
                if (parseError.error == QJsonParseError::NoError) {
                    if (jsonDoc.isObject())
                        jsonReceived(jsonDoc.object());
                } else {
                    byteArrayReceived(jsonData);
                }
            } else {
                break;
            }
        }
    }
    

    I try to search in the forum but wasn't able to find a solution, can anyone please help? Thanks.



  • You can use bytesAvailable method and read the data only when available bytes big then check the data if it is a valid json document. I think using a "while loop" is the wrong thing on readyread signal.


  • Lifetime Qt Champion

    Hi,

    Take a look at the QDataStream documentation for an example of use of transactions.



  • @SGaist I understand now, with transactions I'm performing multiple incomplete reads and this is slowing down the whole process. Can you suggest me a clean and fast way to perform the read to reduce the delay to the minimum?



  • @CKurdu Do you mean checking if (m_clientSocket->bytesAvailable() < threshold) to reduce the incomplete reads? I'm struggling to understand where to put it... Do you have any suggestions?



  • @enne9 The problem is that I don't know the total amount of data that I'm gonna receive.


  • Lifetime Qt Champion

    @enne9
    Hi
    Do you control the server also?
    If yes, I would consider adding an end of data marker to look for as trying to parse the
    the incoming json might be a bit heavy on 20 MB files.



  • @enne9

    Hi enne9
    If I were you, first I prepare my data stream on the server as below.

    QDataStream socketStream(m_serverSocket);
    SerializeSize size;
    quint64 json_size = size(json_content);
    const qint64 end_of_stream = 4444; // what you prefer
    socketStream.setVersion(QDataStream::Qt_5_7);
    socketStream <<json_size<< json_content<<end_of_stream;
    

    SerializeSize class is for calculating the size of the json_content.

    class SerializeSize
    {
      QBuffer m_data;
      QDataStream m_stream;
    public:
      SerializeSize();
      template <typename T>
      quint64 operator ()(const T & t) {
        m_data.seek(0);
        m_stream << t;
        return m_data.pos();
      }
    };
    SerializeSize::SerializeSize()
    {
        m_stream.setDevice(&m_data);
        m_data.open(QIODevice::WriteOnly);
    };
    
    

    On your client app, firstly read the json_content size and then start waiting to finish to came enough data. After that start deserialize content and parse json_content.

    void Client::onReadyRead()
    {
       
        if(m_exptected_json_size == 0)
        {
            if(!extract_content_size())
            {
                return; //wait to finish
            }
        }
       m_received_data.append(tcpSocket->readAll());
        if(m_exptected_json_size > 0 && m_received_data.size()>m_exptected_json_size )
        {
            if(parseJson())
            {
                qDebug()<<"Show Content" ;
                m_exptected_json_size = 0;
         
            }else{
                qDebug()<<"Continue Reading";
            }
        }
    }
    
    bool Client::extract_content_size()
    {
        quint64 asize = tcpSocket->bytesAvailable();
        if(asize > treshold)
        {
           m_received_data.append(tcpSocket->readAll());
           QDataStream in;
           QBuffer in_buffer;
           in_buffer.setBuffer(&m_received_data);
           in_buffer.open(QIODevice::ReadOnly);
           in.setDevice(&in_buffer);
           in.setVersion(QDataStream::Qt_5_10);
           quint64 size = 0;
           in>>size;
           if(size>0)
           {
               m_exptected_json_size = size;
               in_buffer.close();
               return true;
           }
           in_buffer.close();
           return false;
    
        }
        return false;
    }
    
    bool Client::parseJson()
    {
        int eof;
        QByteArray json_data;
        QDataStream in;
        m_buffer.setBuffer(&m_received_data);
        if(!m_buffer.open(QIODevice::ReadOnly))
            return false;
        in.setDevice(&m_buffer);
        in.setVersion(QDataStream::Qt_5_10);
        in.startTransaction();
        quint64 json_size;
        in>>json_size>>json_data>>eof;
        if(!in.commitTransaction())
        {
            m_buffer.close();
            return false;
        }
        m_buffer.close();
        if(eof != end_of_file)
        {
            return false;
        }
        QJsonParseError parseError;
        const QJsonDocument jsonDoc = QJsonDocument::fromJson(json_data, &parseError);
                if (parseError.error == QJsonParseError::NoError) {
                    if (jsonDoc.isObject())
                        m_received_data.clear();
                        return true;
                } else {
                    return false;
                }
        return true;
    }
    
    

    I tried the code with big size json data. It is working.
    Source codes
    Server App
    Client App



  • @CKurdu Thank you very much, I adapted your code to my application and now it's really fast!



  • @enne9
    You are welcome.


Log in to reply