[SOLVED]QTcpSocket problem reading large data.



  • Hi, I'm having strange difficulties to read large data.
    Strangeness is that, when I send small data, it reads successfully, but when I send like 44kb of data, server crashes under release mode, but in debug mode it shows that I have to read 0 bytes data and when following to code, nothing was read, but fun part is that, it somehow already read data and showed in result.. I don't really understand why!

    I'm using custom protocol, sending length of data (qint64) followed by the data.

    Here's the code what I'm using right now.

    [code]#define chunk_size 1024
    void Thread::process()
    {
    qDebug() << "Started receiving data.";

    QByteArray buffer;
    char temp[chunk_size];
    qint64 bytesReceived, datalen;
    
    if((quint64)socket->bytesAvailable() < sizeof(quint64))
        return;
    
    socket->read((char *)&datalen, sizeof(qint64)); // when sending small amount of data, datalen is OK, but when I tried ~44kb of data, it is always 0. why?
    
    while(datalen)
    {
        memset(temp, 0, sizeof(temp));
    
        if(datalen >= chunk_size)
        {
            bytesReceived = socket->read(temp, chunk_size);
        }
        else if(datalen < chunk_size)
        {
            bytesReceived = socket->read(temp, datalen);
        }
    
        if(bytesReceived != -1)
        {
            buffer += temp;
            datalen -= bytesReceived;
        }
        else
        {
            qDebug() << "Error receiving data.";
            buffer.clear();
            return;
        }
    }
    
    if(socket->bytesAvailable())
        socket->readAll(); // ignore other packets.
    
    qDebug() << "Data:" << buffer;
    

    }[/code]

    Server is written in Qt and client in C++. Client side doesn't matter, because I'm using only send(sockfd, buffer, 44 * 1024, 0);
    Also, let me know if I'm doing something wrong.
    Regards.

    EDIT:
    Following error is shown: double free or corruption (!prev): 0x00007f49f40823c0 ***



  • Hi,

    are you sure you really send all your buffer?
    Check the amount of bytes really sent with send() reading its return value.

    The send() can send less bytes you request



  • Yes, all data is sent successfully.
    EDIT: am I doing it wrong?
    [code]unsigned long long size = 44 * 1024;
    char *buffer = (char *)malloc(size + 8);
    if(buffer)
    {
    memmove(buffer, (void *)&size, sizeof(unsigned long long));
    memmove(buffer + 8, 44kb_data, size);
    send(sockfd, buffer, size + 8, 0);
    }[/code]

    EDIT: This problem is related to this topic -> http://qt-project.org/forums/viewthread/25609 but whenever I'm doing it on QByteArray, it shows that result is always empty.



  • You MUST check what send() returns.
    It can return less bytes you provided, in this case you must sent the remaining data.

    For example:

    @
    size_t dataToSend = size + 8;
    size_t sent = 0;
    char *pBuffer = buffer;
    while (dataToSend > 0) {
    sent = send (sockfd, pBuffer, dataToSend);
    if (sent > 0) {
    pBuffer += sent;
    dataToSend -= sent;
    }
    else {
    // ERROR
    }
    }
    @



  • Sorry for long reply, I was traveling a bit..
    About the problem, seems I can't debug the code, because as far as I guess, readyRead called many times (because TCP is streaming protocol). do I am missing something? :-(

    [code]//Thread.h
    #ifndef THREAD_H
    #define THREAD_H

    #include <QObject>
    #include "tcpsocket.h"

    class Thread : public QObject
    {
    Q_OBJECT
    public:
    explicit Thread(TcpSocket *client, QObject *parent = 0);

    signals:
    void finished();

    public slots:
    void process();

    private:
    TcpSocket *socket; //inherits QTcpSocket.
    qint64 blockSize;
    QByteArray buffer;

    };

    #endif // THREAD_H[/code]

    [code]//Thread.cpp
    #include "thread.h"

    #include <QDebug>

    #define chunk_size 1024

    Thread::Thread(TcpSocket *client, QObject *parent) :
    QObject(parent), socket(client)
    {
    blockSize = 0;
    }

    void Thread::process() {
    qDebug() << "readyRead()";

    char temp[chunk_size];
    qint64 bytesReceived;
    
    if(socket->bytesAvailable() < (qint64)sizeof(qint64))
        return;
    
    if(blockSize == 0) {
        socket->read((char *)&blockSize, sizeof(qint64));
    }
    
    while(blockSize) {
        memset(temp, 0, sizeof(temp));
        bytesReceived = socket->read(temp, chunk_size);
        if(bytesReceived > 0) {
            buffer += temp;
            blockSize -= bytesReceived;
        } else {
            blockSize = 0;
            qDebug() << "Error receiving data.";
        }
    }
    
    qDebug() << "Data:" << buffer;
    buffer.clear();
    emit finished();
    

    }[/code]

    Sender part is exactly like that you posted above.

    When sending large data, it stops on readyRead and after step over, it still stays on readyRead twice or more and then just jumps to return and data is received successfully (can't see the receiving part while debugging), but on release mode, it just don't receive all data.. WHY? can someone do some tests please? :-(



  • Bump... Someone PLEASE explain me why this happens, I'm going to delete and forget about Qt soon!! Trying to understand this more than 2 weeks but I just can't find any documentation or answers about this, someone HELP PLEASE!



  • bq. Server is written in Qt and client in C++. Client side doesn’t matter, because I’m using only send(sockfd, buffer, 44 * 1024, 0);

    So, the server is C++ and the client is C++. Qt is library not a language. I will assume the first 8 bytes in buffer are a 64-bit int in the correct byte order.

    The general problem happens because you are subverting the normal flow of events in Qt.

    The readyRead() signal will be emitted when there is some data, note that's not all data, to be read from the socket. If that is less than the 8 byte size field then you return, fine.

    If you have more than 8 bytes you read the 8 byte size leaving the remainder. You then enter a loop trying to read chunks of 1024 bytes until the blocksize is exhausted. The thing is, blocksize will not be exhausted unless the entire packet was received and available when readyRead() was emitted. In general, blocks written in one send() will not be received in one packet because the network will fragment the data. For small payloads you may get away with it but you cannot rely on it.

    In a single threaded application your loop would never allow the Qt event loop to resume, and therefore no further data will be received. Your code implies you have threading involved, possibly in an attempt to receive data after you enter this loop.
    You do not require any form of threads to implement a network interfaces in Qt. Using them only adds potential places this can be coming unstuck.

    Here is a complete example that will receive single blocks in the form you describe. (No error handling etc.)
    @
    #include <QtCore>
    #include <QTcpServer>
    #include <QTcpSocket>

    class Receiver: public QObject
    {
    Q_OBJECT
    public:
    Receiver(QTcpSocket *socket, QObject *p = 0):
    QObject(p),
    sock(socket),
    bytesExpected(0)
    {
    connect(socket, SIGNAL(readyRead()), SLOT(readyRead()));
    }

    public slots:

    void readyRead()
    {
        if (bytesExpected == 0 && sock->bytesAvailable() >= sizeof(bytesExpected)) {
            sock->read((char *)&bytesExpected, sizeof(bytesExpected));
            qDebug() << "Expecting:" << &bytesExpected;
        }
    
        if (bytesExpected > 0 && sock->bytesAvailable() > 0) {
            QByteArray chunk = sock->read(qMin(bytesExpected, sock->bytesAvailable()));
            buffer += chunk;
            bytesExpected -= chunk.size();
            qDebug() << "Received chunk of:" << chunk.size();
    
            if (bytesExpected == 0) {
                qDebug() << "Received block of size:" << buffer.size();
                qDebug() << "Bytes left in socket:" << sock->bytesAvailable();
                sock->deleteLater();
                deleteLater();
            }
        }
    }
    

    private:
    QTcpSocket *sock;
    qint64 bytesExpected;
    QByteArray buffer;
    };

    class Server: public QObject
    {
    Q_OBJECT
    public:
    Server(QObject *p = 0): QObject(p)
    {
    server = new QTcpServer(this);
    connect(server, SIGNAL(newConnection()), SLOT(handleConnection()));
    server->listen(QHostAddress::Any, 12345);
    }
    public slots:
    void handleConnection()
    {
    while(server->hasPendingConnections())
    (void) new Receiver(server->nextPendingConnection(), this);
    }
    private:
    QTcpServer *server;
    };

    int main(int argc, char **argv)
    {
    QCoreApplication app (argc, argv);

    Server s;
    
    return app.exec();
    

    }
    #include "main.moc"
    @



  • Big thanks to you.. I was getting crazy day by day finding what caused the problem.. I finally got it that it was problem with threading, because after re-creating thread, blockSize would be always 0 (which constructor initializes).
    Many many many many thanks to you!


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.