How to Store UDP packet for some second in RAM



  • Hi
    I have a problem that confused me. If any body can help me , I will be glad :).
    I have 1G Ethernet card (one port) that connected to server on the other side. Server send continuously data to my PC with UDP connection type, I wrote Client-base app with Qudpsocket and read incoming data and Store in QByteArray().(Qt 64 bit GCC )

    My problem is :
    I want to store for 15 Second all packet in RAM and then store in HDD in *.txt file. (15 second * 800Mb=12Gb)
    When capacity of QByteArray increase this Error will be appear "What() : bad_alloc()".
    How can i solve this problem? How can i manage huge data and store big data to RAM when capacity pass the variable size?
    please say the true and best structure.


  • Moderators

    Store the data in several QByteArrays instead of a single one :-) Simple and should work fine (and cross platform).

    Or store it in a file in RAM (ram file system). Some info here and here.



  • 1- is it possible to create dynamic QByteArray object in loop ?like below ,Because I want to increase time of storing in the next step.

    //your code here
    for (int i = 50; i >=0;i--){
            QByteArray *datagram'i' = new QByteArray();
    }
    

    2- in this solution , you say to store all received datagram in temp file in RAM?



  • @Alexanov 1Gbit ethernet isn't 1GBytes/second, it's 1GBits/second which is 1/8 of 1GBytes/second.

    It's of course possible to create QByteArrays in a loop with new, however you have to store the pointers somewhere. Or maybe you can use QByteArrayList.


  • Qt Champions 2016

    @Alexanov said in How to Store UDP packet for some second in RAM:

    I want to store for 15 Second all packet in RAM and then store in HDD in *.txt file. (15 second * 800MB =12GB)

    Do you have that amount of memory (in addition to what the system requires), as your error seems to indicate you don't?



  • You can use an easier solution to allocate QByteArray

    std::vector<QByteArray> byte_arrays(50);
    //access it like this
    qDebug()<<byte_arrays[0]; //0 mean you access first QByteArray in the vector
    //no need to free the QByteArray because std::vector do it for you, who say resource management is hard in c++?
    

    In most of the cases, following statement is correct
    allocate resource on stack is better than smart pointer, allocate on smart pointer is better than raw pointer



  • @tham said in How to Store UDP packet for some second in RAM:

    allocate resource on stack is better than smart pointer, allocate on smart pointer is better than raw pointer

    Yes, although with Qt we have to remember that smart pointers are objects which manage lifetime of other objects, and in Qt the QObject based class system does it with parents. On the other hand some important classes (like QByteArray) are not QObject based. In Qt programming you can do

    new MyQObjectBasedClass(pointerToParent);
    

    and the new pointer isn't get lost. But this is getting offtopic...


  • Lifetime Qt Champion

    Hi,

    Why not just send them by batch rather than trying to read them all there ?

    [Moved in from duplicate thread ~kshegunov]



  • @kshegunov said in How to Store UDP packet for some second in RAM:

    addition

    Yes , I have 128G RAM and 1TB HDD. I did not have any problem with my Hardware.
    Although I saw the task manager ,when my program allocate 1G Memory , OS terminate that process and send signal to Qt and say "bad alloc()".

    My Qustion is: what do you do? if you want to accumulate UDP received packet for a few second and then store in HDD???



  • @Eeli-K said in How to Store UDP packet for some second in RAM:

    create

    yes I knew that . and in my question wrote wrong, thanks.
    is it possible to show me how we can store pointer for some QByteArrays that were created in loop ?



  • @SGaist said in How to Store UDP packet for some second in RAM:

    Why not just send them by batch rather than trying to read them all the

    Hi
    Can you explain more ? I did not understand very well !! :(


  • Lifetime Qt Champion

    Rather than store that much data in RAM and then dump everything in your HDD, make the chunk smaller and write more regularly.



  • Okay ,I did it and worked fine but , this method is stopgap solution,
    because at second Step i want to replace 1G Ethernet card to 10G. When I replaced cards , "file.write" line (in my code) worked very slowly than packets were coming !(I lost some packets when my program was writing in file , the speed of write is more slower than speed of incoming data)!!!!!
    do you have any solution for this ?
    Thanks :)


  • Lifetime Qt Champion

    Only one: adapt your hardware.

    You can't expect to dump GB worth of data each second on a single spinning disks hard drive (even solid state drive). Use something like a RAID to withstand the throughput.



  • @SGaist
    for this reason i want to fill all received packets in RAM and after a few second
    store in my HDD.(is it possible?)


  • Moderators

    @Alexanov said in How to Store UDP packet for some second in RAM:

    the speed of write is more slower than speed of incoming data

    In addition to what SGaist said, also consider moving various tasks into separate threads. So that your network thread will continue reading data into buffer while your IO thread writes to HDD.

    If, however, you constantly (per second) get more data than your hard disks can write... there is no software solution for this (unless you can go clever with the data: analyze it and extract only important parts, compress it etc.).



  • @tham
    Thank you so much, your solution is very helpful for me ;)
    Finally, I used Qvector to create dynamic array of QByteArrays type.


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.