QTcpSocket package loss [SOLVED]

Seraph

Hi there!

It's me again today... I'm using the QTcpSocket/QTcpServer to send some messages/bytes.

Here's my code:

Client side:
@void TcpClient::sendBytes(const QByteArray &b)
{
QByteArray block;
QDataStream out(&block, QIODevice::WriteOnly);
out.setVersion(QDataStream::Qt_4_8);

    out << (T_MSG_LENGTH)0;  //reserve 2 bytes for message length in bytes
    out << b;  //stream the bytes
    out.device()->seek(0); //go back to the beginning of the stream to set the message length
    out << (T_MSG_LENGTH)(block.size() - sizeof(T_MSG_LENGTH)); //set the message length

    _socket->write(block); //_socket is of type QTcpSocket*
 
}@

Server side:
@//! this method will be called, if the socket receives any bytes from any sender
void TcpServer::readBytes()
{
static T_MSG_LENGTH msgLength = 0; //a buffer for the message length

    QTcpSocket *senderSocket;

    //get the sending socket
    if(!(senderSocket = qobject_cast<QTcpSocket *> ( sender() ) ))
        return;

    QByteArray receivedBytes; //a buffer for the received bytes
    QDataStream in(senderSocket);
    in.setVersion(QDataStream::Qt_4_8);

    if(msgLength == 0)
    {
        //enough bytes received to read the message length?
        if(senderSocket->bytesAvailable() < (int)sizeof(T_MSG_LENGTH))
            return; //if not, return and wait for the next package

        in >> msgLength; //else read the message length
    }

    if(senderSocket->bytesAvailable() < msgLength) //all bytes received, yet?
        return; //if not, return and wait for the next package

    //in: now contains the full message
    in >> receivedBytes;

    //reset the message length to receive new messages/bytes
    msgLength = 0;

    //now commit the complete data for processing
    recievedData(receivedBytes);
}

//------

void TcpServer::recievedData(QByteArray &b)
{
    static T_UINT_64 count = 0;

    TRACE((++count)<<". Recv["<<b.size()<<"]: "<<QString(b).toStdString());
}@

T_UINT64 is a long
T_MSG_LENGTH is just a define to have better maintainance (its a short here)

This code works fine when i send packages in a low frequence.
But when I try to do smth dumm like this:

On client side:
@for(unsigned int i = 1; i <= 1000; i++)
client->sendBytes("HELLO WORLD");@

I loose like 99 percent of all messages:

Output from server (4 out of 1000 messages):
@1. Recv[13]: HELLO WORLD
2. Recv[13]: HELLO WORLD
3. Recv[13]: HELLO WORLD
4. Recv[13]: HELLO WORLD
@

Is there a way to get notified by a signal, that the server sent an ACK?
Or a function like:

@if(client->isReadyForSending() /or anything/)
client->sendBytes("HELLO WORLD");@

I already tried to use _socket->waitForBytesWritten() after _socket->write(b)
That increased the received messages from 4 to 27, but far away from 1000!

Thanks for ur patients!

Jeroentjehome

Hi there,
There is one big flaw in your loss of messages. It can not be the TCP connection itself, so the possibility is that the sending socket doesn't accept that much bytes to write.
Maybe check the waitForBytesWritten function or get the sending of a packet when the bytes have been send to the TCPip layer.
You could check the write functions return properties. It is lower then the bytes you want, or if -1 it went wrong!!
Greetz

Seraph

OK guys. Here is the solution for my problem (got it randomly :P )

Well, it is weird but it works without any loss!!!

I changed the send method from this:

@for(unsigned int i = 1; i <= 1000; i++)
client->sendBytes("HELLO WORLD");@

To this:

@ QByteArray b;
QDataStream out(&b, QIODevice::WriteOnly);

for(unsigned int i = 1; i <= 1000; i++)
{
out<<"HELLO WORLD";
cout<<"SEND[ "<<b.size()<<" ]: "<<b.data()<<endl; //I also tried QString(b).toStdString()
c->sendBytes(b);
}
@

Now it works fine! Testet it with 1*10^6 Packages!!! No loss!

The only problem is now, that b.data() is somehow empty. b.size() is but 13bytes????

The output lines of the client are now:

SEND[ 13 ]: //b IS EMPTY!!!
SEND[ 13 ]:
SEND[ 13 ]:
...
...

Do i have to read a QByteArray with a DataStream again? But what is the QByteArray::data() method for, then?

Seraph

Now I'm totally confused!!!

When I do it like this:

@ QByteArray b;
QDataStream out(&b, QIODevice::WriteOnly);

for(unsigned int i = 1; i <= 1000; i++)
{
out<<"HELLO WORLD";
cout<<"SEND[ "<<b.size()<<" ]: "<<b.data()<<endl; //I also tried QString(b).toStdString()
c->sendBytes(b);
}@

All messages will be received from the server, BUT the size of the byte array is increasing by the size of the message. Which means the output is:

SEND[ 13 ]: //b IS EMPTY!!!
SEND[ 26 ]:
SEND[ 39 ]:
SEND[ 52 ]:
…
…

I tried to solve this problem by resetting the byte array and it's stream, by doing smth like that:

@ QByteArray b;
QDataStream out(&b, QIODevice::WriteOnly);

for(unsigned int i = 1; i <= 1000; i++)
{
out<<"HELLO WORLD";
cout<<"SEND[ "<<b.size()<<" ]: "<<b.data()<<endl; //I also tried QString(b).toStdString()
c->sendBytes(b);

 //here the interesting code
 out->device->seek(0); //I also tried reset() and b.clear() in any combination!

}@

Now the byte array size is not increasing anymore. It remains constant at 13. So far so good, but now I have the message loss problem again! I tried the negative and byte size comparison with _socket->write() it allways returns the size of the bytes to be send! GRRRRRRR What is this? Do I get smth wrong here?

Seraph

Am I really the only one who has that problem? Maybe I don't understand the exact functionality of QDataStream and QByteArray!

Here again: Long story short:

I send 1000 QByteArrays via QTcpSocket::write() in a loop (see recent posts).
But only a view ByteArrays will be received!

If I do not reset the QDataStream or clear the QByteArray in the loop, the receiver receives all 1000 ByteArrays!!! The thing is, that the ByteArray which will be received is allways increasing it's size by the message length (that one is clear, because i do not reset the stream!)

What am I doing/getting wrong?

Thanks for every idea

koahnig

Did you have a look to the fortune server "client":http://qt-project.org/doc/qt-4.8/network-fortuneclient.html examples?
That pair of example code helps you to understand the different issues involved. Also increasing the number of messages sent will help you to understand what might go wrong.

[quote author="Jeroentje@home" date="1351257898"]Hi there,
There is one big flaw in your loss of messages. It can not be the TCP connection itself, so the possibility is that the sending socket doesn't accept that much bytes to write.
Maybe check the waitForBytesWritten function or get the sending of a packet when the bytes have been send to the TCPip layer.
You could check the write functions return properties. It is lower then the bytes you want, or if -1 it went wrong!!
Greetz[/quote]

This provides you already with the details of where to look. It must be in your code. Either in the sending or the receiving part. For others it will be very hard to point out the deficiency in your code.

Seraph

My Server/Client classes are exact copies of the fortune example!

Im meanwhile using waitForBytesWritten() and I am checking the write() functions return value!
Everything seams to be allright! The funny thing is, that only when I reset the QDataStream and the ByteArray (which is necessary to load new data inside to send it) the messages get lost!
If I do not reset the stream/device the receiver receives all 1000 Messages!!! But the size of the received data is increasing constant. So what I get on the receiver side, is of course bullsh1t !
But I have to reset the ByteArray and the DataStream to fill it with new data! That's the problem!

Could somebody please test this with the fortune example? Like just sending 1000 HELLO WORLD messages in a for or while loop and receive them !?
For me it is not working!

koahnig

[quote author="Seraph" date="1351332586"]My Server/Client classes are exact copies of the fortune example!

Im meanwhile using waitForBytesWritten() and I am checking the write() functions return value!
Everything seams to be allright! The funny thing is, that only when I reset the QDataStream and the ByteArray (which is necessary to load new data inside to send it) the messages get lost!
If I do not reset the stream/device the receiver receives all 1000 Messages!!! But the size of the received data is increasing constant. So what I get on the receiver side, is of course bullsh1t !
But I have to reset the ByteArray and the DataStream to fill it with new data! That's the problem!

Could somebody please test this with the fortune example? Like just sending 1000 HELLO WORLD messages in a for or while loop and receive them !?
For me it is not working![/quote]

I looks like you are confusing yourself.

This is a copy of code from above. Only the out statement of "HELLO WORLD" has been moved.
@ QByteArray b;
QDataStream out(&b, QIODevice::WriteOnly);

out<<"HELLO WORLD";

for(unsigned int i = 1; i <= 1000; i++)
{
cout<<"SEND[ "<<b.size()<<" ]: "<<b.data()<<endl; //I also tried QString(b).toStdString()
c->sendBytes(b);
}@
This will certainly prevent from accumulating text in your buffer. Another options would be:
@
for(unsigned int i = 1; i <= 1000; i++)
{
QByteArray b;
QDataStream out(&b, QIODevice::WriteOnly);

   out<<"HELLO WORLD";
   cout<<"SEND[ "<<b.size()<<" ]: "<<b.data()<<endl; //I also tried QString(b).toStdString()
   c->sendBytes(b);

}@

Seraph

No I already tried this...
Maybe somebody should test it with the simple fortune Server example himself.
Then we know if it is a bug and we report it!

I tried anything...

I also made a debug output in the receiving instance right after the readyRead() signal is emitted.
The receiver just don't get more messages! So it has to be the sender. I'm just wondering why the messages are sent if i do not reset the stream.

BTW: In the first version i just did it like this:

_socket->write("HELLO WORLD"); //same result!!!

koahnig

Either way you have to recognize that you basically send a very long string containing a 1000 times the information.
"HELLO WORLDHELLO WORLDHELLO WORLDHELLO WORLD...".
On the receiving side you have to understand that you are not receiving a 1000 messages as send them out. Depending on the speed of the connection and the speed of the computer you may receive anything from a single byte each time up to the whole information at once.

You have to decide what you like to do with the information. Probably the best is to buffer it. Each time you receive a signal you can read a part of the information. You add it to the buffer.
Certainly you should not reset the buffer in between. Because then you will delete information received.

Seraph

Yes, but how may i buffer smth which is simply not there?
And why is it there, when i do not reset the senders byte array after sending!? That is the point!
Did u test it yourself?

Seraph

BTW As u can see, i am buffering all incomeing bytes. I also wait until the message length is reached.

koahnig

[quote author="Seraph" date="1351358440"]Yes, but how may i buffer smth which is simply not there?
And why is it there, when i do not reset the senders byte array after sending!? That is the point!
Did u test it yourself?[/quote]
Yes, I have tried to modify the example code.
Yes, you are right. Packages are apparently lost. I assume that the changes within the last releases due to QNetwork stuff are the reason. I could not find the reason right away.

I have started using QTcpSocket much earlier than version 4.7 came up. It is not QTcpSocket's fault so. I would recommend using QTcpSocket and QTcpServer directly. The use of the QTcpSocket read and write on character basis is circumventing the QDataStream stuff. That is the way I am using it and it works.

Seraph

OK! Thanks!!

Hmm... Would it be to much? I mean... you don't have to but it would be nice:

To send me an working example code which does not lose any packages!?

Thanks a lot

koahnig

The easiest is to get the fortune server example extended to do what you like to see, since I have no example code readily available.

I have started another trial. I have used very old example code for the fortune examples. It is from version 4.3.1 downloaded "from here":ftp://ftp.qt.nokia.com/qt/source/qt-all-opensource-src-4.3.1.zip .

The same string is sent just 1000 times in here.
@

void Server::sendFortune()
{
QByteArray block;
QDataStream out(&block, QIODevice::WriteOnly);
out.setVersion(QDataStream::Qt_4_0);
out << (quint16)0;
out << fortunes.at(qrand() % fortunes.size());
out.device()->seek(0);
out << (quint16)(block.size() - sizeof(quint16));

QTcpSocket *clientConnection = tcpServer->nextPendingConnection();
connect(clientConnection, SIGNAL(disconnected()),
        clientConnection, SLOT(deleteLater()));

for ( int i = 0; i < 1000; ++i ) 
    clientConnection->write(block);

clientConnection->disconnectFromHost();

}
@

[edit: post shortened, koahnig]

koahnig

The client has some slight quick and dirty changes as well:
@

#include <QtGui>
#include <QtNetwork>

#include "client.h"

#include <fstream>

std::ofstream of ("test.txt");

Client::Client(QWidget *parent)
: QDialog(parent)
{
...
@

koahnig

@
void Client::readFortune()
{
QDataStream in(tcpSocket);
in.setVersion(QDataStream::Qt_4_0);

while ( tcpSocket->bytesAvailable() )
{
    if (blockSize == 0) {
        if (tcpSocket->bytesAvailable() < (int)sizeof(quint16))
            return;

        in >> blockSize;
    }

    if (tcpSocket->bytesAvailable() < blockSize)
        return;

    QString nextFortune;
    in >> nextFortune;
    blockSize = 0;

    of << nextFortune.toStdString() << std::endl;
}

//if (nextFortune == currentFortune) {
//    QTimer::singleShot(0, this, SLOT(requestNewFortune()));
//    return;
//}

//currentFortune = nextFortune;
//statusLabel->setText(currentFortune);
//getFortuneButton->setEnabled(true);

}
@

See readFortune and also the global fstream at the beginning. This version simply writes the received stuff in a file. I have tried to keep the changes as few as possible.

[edit, koahnig]

I did not expect that the source files are as long. Here are the changed "example files as zip on dropbox.":http://db.tt/ohgWvly0

Seraph

And with this example you received all 1000 messages?

OK.. Thanks i will read it through and give it a try.

koahnig

The example code contains also the information received. It has 1000 lines.
The fortune example is excellent. However, it could be the closing of the connection throwing you of the track. This is just guessing. My server apps are simply keeping the socket and therefore, they are keeping the connection open. In that respect the fortune server is different. It is a simply question and answer application.
However, check the example code out.

Seraph

OOOOOOKKKK! I finally got it!

So for everybody who is confused of the results of fortune client example for QT 4.7:

In the new fortune client example they dropped the while loop, which is looping until no bytes are available! This one is working, when you send the messages with delay (Like in an intervall of 100ms) or like in the example only one message.

The problem is fixed by adding this while loop again (like in koahnig's example):
@
void TcpServer::readBytes()
{
while ( _socket->bytesAvailable() )
{
if (blockSize == 0) {
if (tcpSocket->bytesAvailable() < (qint64)sizeof(quint16))
return;

        in >> blockSize;
    }

    if (tcpSocket->bytesAvailable() < blockSize)
        return;

    QString nextFortune;
    in >> nextFortune;
    blockSize = 0;

    of << nextFortune.toStdString() << std::endl;

}
}@

Thanks a lot to koahnig for figuring that out!!!

Bye and thanks again