UDP - Splitting datagrams



  • Hi there,

    I have again a problem with sockets and too fast write speed!

    I wrote a fragmenter for UDP datagrams which is able to split a QByteArray into several fragments of previous defined size.
    Each fragment has a leading header consisting of DATAGRAM-ID, FRAGMENT-ID, NUM-OF-FRAGMENTS and a CHECKSUM. Once the receiving instance receives all fragments of a specific host address and datagram id, it will merge all the fragments to one datagram. I tested the fragment system without sending anything via network and it worked fine.
    But if i send it via network (even via LOOPBACK), it results in a very high package loss!!!
    The fragmenter sends the 512 byte fragments within a while-loop which is too fast for the socket i guess!?
    I added QThread::usleep(1000) (sleep 1 ms) before the send call and it worked fine! This is not what i want, because it is too slow! 20MB of data would take about 47 seconds to be transmitted! I also tried QApplication::processEvents() after the send call with no success.

    Is there a recommended way to send fragmented data very fast via UDP?

    Thanks in advance.



  • Have you tried waiting for the BytesWritten signal, or using waitForBytersWritten (and then check that the full number of bytes is sent)?

    UDP is really more for streaming data (like for voice or video) where its not absolutely essential for every block to arrive.

    I think maybe you may want to use TCP sockets which provide a more guaranteed delivery (and in the correct order).

    Maybe you can post your code snippet? :)



  • I need UDP because i need multi cast! I have to stream to many instances simultaniously.

    But i think i solved the problem (I hope it is not a random success! ;) )
    Here's the thing:

    I wrote a method QByteArray getNextFragment(const QByteArray& b) which correctly retrieves the next fragment out of b, adds some header information and returns it. So my code was like this:

    @QByteArray fragment;
    while((fragment = getNextFragment(bytesToSend)).size() > 0)
    {
    _socket->writeDatagram(fragment.data(), fragment.size(), _hostAddress, _port);
    _socket->waitForBytesWritten();
    }@
    The problem with this code is, that the QByteArray fragment has not really (only sometimes) been overwritten with the return value of getNextFragment(). So i changed the method getNextFragment() to bool getNextFragment(const QByteArray& b, QByteArray* fragmentBuffer). This method now first clears the fragmentBuffer and resizes it to the actual fragment size. At last it swaps the retrieved fragment bytes into the fragmentBuffer.
    So i can use it like this:
    @QByteArray fragment;
    while(getNextFragment(bytesToSend, &fragment))
    {
    _socket->writeDatagram(fragment.data(), fragment.size(), _hostAddress, _port);
    _socket->waitForBytesWritten();
    }@
    Without taking a closer look at this, i can say at least all fragments arrive at the clients!!! I will now do further testing to see if it is reliable! All checksums are matching until now!



  • Hmm... I still loose around 0.01% of all Packages! Even via loop back!!!
    My fragments are 512Bytes each.

    I guess it depends on how many times the receiving instance is slower then the sending instance. If my processPendingDatagrams() method on my receiving instance has to do more work (like unpack the byte array, read the header, put it to the fragment list) while the sending instance is just sending it in a fast loop, the receiving instance looses lots of packages!
    e.g.: I just added a small console output for each received fragment and i lost 75% of all packages!
    I also tried to change the connection type of the connection between the readyRead() signal of my socket and the processPendingDatagrams() slot, with no success.

    I thought that the QUdpSocket has a kind of datagram queue which should not loose any incomeing data on loop back!?



  • bq. I thought that the QUdpSocket has a kind of datagram queue which should not loose any incomeing data on loop back!?

    Yes you are right it does, certainly if you are just a frame or two slower you should not have any problems.

    Maybe lets take a look at the receive side If you are happy all blocks are sent ok.

    I am again not at my PC with any of my code (so no examples), but from memory when you receive readReady() you go ahead and read from the QUdpSocket, there my be a case when you are too slow and two arrive before you have read it so you need somthing like:

    @
    whlie (mySocket.datagramPending())
    {
    mySocket.readDatagram(...);
    }
    @

    However, I seem to recall something not quite right about this setup (maybe it was just me) but I seem to have found it unreliable. I went with:

    @
    whlie (mySocket.bytesAvailable ())
    {
    ba = mySocket.read(512);
    }
    @

    Maybe you can paste your receive side code. Also you can number your frame and confirm that you both send and receive the data and check which side is losing the data (also use wireshark to verify the middle :o



  • Hi code_folder,

    OK, yes i did it like your first meta-code:

    @void UdpRecvSocket::processPendingDatagramms()
    {
    QByteArray data;
    QDataStream in(&data, QIODevice::ReadOnly);

        qint64 datagramSize = -1;
        QHostAddress* addressBuffer = NULL;
        while(_socket->hasPendingDatagrams())
        {
            in.device()->seek(0);
    
            datagramSize = _socket->pendingDatagramSize();
            data.resize(datagramSize);
            _socket->readDatagram(data.data(), datagramSize, addressBuffer);
    
            //do smth with the data
            processDatagram(data);
        }
    }
    

    @
    The processPendingDatagramms() method is connected to the readyRead() signal of my underlying _socket.

    And the second approach works differently you think?



  • Well, it should not behave differently.... but .... :o

    I notice that you have a lot of code there to do the read. you should be able to just do _udpSocket.read

    I think one of the reasons that I used the "bytesReceived" and "read" is that they are more intuitive and QByteArray friendly : )

    I would replace your code with:
    @
    void UdpRecvSocket::processPendingDatagramms()
    {

    whlie (mySocket.bytesAvailable ())
    {
        processDatagram(mySocket.read(512));
    }
    

    }
    @

    Unless I missed something, that is what you are wanting to achieve?, but just keeping it simple (not sure if you have a subtle problem with the resizing and seeking or something....). It would def be easier to debug it as well as you don't have to worry about all that resizing, seeking, pointers etc...

    Its just my thoughts : )



  • The thing is that i need the address of the sender. Maybe there is a way to get it with your approach, too. But that is not the problem.
    The main problem is, that the receiving instance misses lots of packages if it is a bit slower as the sending instance. That's why i cannot really debug it. Once i stop the receiving instance at a breakpoint, the whole rest gets lost. That's why I'm a bit confused. What is then the datagram queue for? Or is this queue also running in the gui thread? Which would be bad, because it would not be able to receive new datagrams until i proceeded the last one! There must be a more reliable way to do this!

    It would be cool if someone could test the same thing. Just make a sending and receiving QUdpSocket instance and send 50000*512Bytes-Datagrams in a loop.
    Smth like that:
    @for(int i = 0; i < 50000; i++)
    socket->writeDatagram(b.data(), 512, QHostAddress("127.0.0.1"), 1564);@

    I would be interested in your results!



  • If you pause your program (for debug) I think you will no longer receive anything. They way to test your buffer is to intentionally miss a few reads.

    You could send, say 10 datagrams and on the receive side set a timer and do nothing for, say 2 seconds, and then see what's in your receive buffer... it should contain 10 x 512 bytes.

    In the mean time I will try your set-up when I get a chance, which I should do tomorrow :)



  • That would be great, thanks!

    At the moment i am testing it by just collecting the fragments (by pusihng them on a map (datagramID to fragments). I wrote a function which simply prints out all received fragments sorted by datagramID and sender address. And then i can see that i lost many fragments. As i said, how many fragments i loose, depends on how much slower the receiving instance is compared to the sending instance. I tested it with a QThread::usleep(1000) before each sendDatagram call and i received 100% of all fragments, so my system itself should work.
    I am curious what you get.



  • Hey Dude,

    Sorry, I did not get a chance to look at this for you :( - I thought I had spare time over the weekend, but it turns out that I did not.

    I just thought, maybe you can send me your project/projects as a zip and I'll run it up here and/or make tweaks to it. It would probably be easier to see what's going on as well.

    I can post you my email to send the zip file (just clear out any output files so it is nice and small) :)


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.