Reading UDP socket without using an auxiliar buffer



  • Hi everyone!

    The question I'm about to make is quite basic, but I havent been able to figure it out. I have an application that recieves a massive ammount of information through an UdpSocket (1280 bytes at a time), and needs to save them to disk. Is crucial that I dont lose any datagram, so the program must be as fast as it can get.
    In order to save the information, im using a separate thread with the lowest priority, but im still sometimes losing information, so I need to make my program faster.

    My approach to saving information on the RAM is this one (the one suggested by the documentation)

    @
    void Server::readPendingDatagrams()
    {
    while (udpSocket->hasPendingDatagrams())
    {
    QByteArray aux;
    aux.resize(udpSocket->1280);
    udpSocket->readDatagram(aux.data(), aux.size());
    GlobalArray.append(aux);
    }
    }

    @

    Where GlobalArray is a Global QByteArray that stores the information, and once it reaches 10MB I save it to disk using a separate thread.
    The thing here is that once I recieve a datagram, im saving it to an auxiliar buffer, and then appending this buffer to my main buffer. This is a waste of time, I was wondering if theres a way to directly read and append it to my global array. Something like this (that of course, doesnt work)

    @
    void Server::readPendingDatagrams()
    {
    while (udpSocket->hasPendingDatagrams())
    {
    GlobarArray.resize(GlobalArray.size()+1280);
    udpSocket->readDatagram(GlobalArray.data(), 1280);
    }
    }
    @

    the .data() method from QByteArray doesnt point to the end of the data, so I guess thats why this approach isnt working. So my question would be, is there a way to append information that comes through an UdpSocket to an already existing QByteArray without using an auxiliar buffer?
    And, as im already annoying people with questions, I will add another one :D
    Is there a way to increase the priority of the udpSocket, and decrease the priority of the rest of the program? I dont care if the visual part doesnt work well, the only thing that I care for is not losing information.



  • There are some design flaws here, but answering your question first, basically, use something like (although it most likely won't solve your problem or even introduce crashes):
    @ void Server::readPendingDatagrams()
    {
    while (udpSocket->hasPendingDatagrams())
    {
    GlobarArray.resize(GlobalArray.size()+1280);
    udpSocket->readDatagram(GlobalArray.data() + GlobalArray.size(), 1280);
    }
    }
    @

    But basically, it's the design that seems flawed. Usually one would handle such stuff with queues - just make a datagram queue, to which you push received datagrams and from which you pop in the disk-writing thread.

    If you have to use QByteArray as a buffer, do consider using a double buffer - one is used exclusively by the UDP receiver, the second being written to disk or unused, then once the one used by the receiver fills swap them.

    I'm not low-level enought to be sure, but doesn't your system buffer the socket up to some point? And if you overflow, you overflow at some point anyways, worst case running out of memory.

    Also, please be aware that unless you use QByteArray::reserve(int) appending is a potentially time consuming operation - if there is not enough space at the end of the array the program has to allocate a new, bigger array somewhere else and move the whole, say, 8 megs to the new one.



  • "But basically, it’s the design that seems flawed. Usually one would handle such stuff with queues – just make a datagram queue, to which you push received datagrams and from which you pop in the disk-writing thread"

    Im not quite sure, im doing it like this to keep it as simple as it can get, a queue sounds reasonable, but would require more processing. If i cant get this program to work at the speed that i require, I will probably have to start over from a different approach.

    "If you have to use QByteArray as a buffer, do consider using a double buffer "

    Actually, im already doing this, only that better. Im using 5 buffers, I didnt put this in the post because i thought it wasnt relevant, but my GlobalArray, is actually GlobalArray[ 5], and I keep rotating them all the time. As i dont know how much time it takes to save to disk on a low priority thread, I decided to compensate by simply placing more buffers

    "I’m not low-level enought to be sure, but doesn’t your system buffer the socket up to some point? And if you overflow, you overflow at some point anyways, worst case running out of memory."

    I here think same as you, but I do know that im losing data, dont know why. To make sure I have 2 computers connected with a cable (no swithc, nothing in the middle) and with one computer I can count exactly how much data I have sent, and right now every 400MB that i send, im losing like 8KB (i know this isnt much, but I need to improve it)

    I didnt know that the .append method was time consuming, now that I read it, it sounds reasonable. Gonna read the reserve method.

    Thanks for your input!



  • Ive tested the .reserve method, and im not sure if its working.
    If I do:

    @
    QByteArray aux;
    aux.reserve(50000000);
    aux.append("This is a test");
    @

    In this case aux.size() returns 15, (the size of the string I placed), but if I open the windows task manager, and check the program memory usage, its definitly not 50MB, so is that space actually reserved? How can it be reserved if the program is not using that amount of RAM?


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.