How to Store UDP packet for some second in RAM

Alexanov

Hi
I have a problem that confused me. If any body can help me , I will be glad :).
I have 1G Ethernet card (one port) that connected to server on the other side. Server send continuously data to my PC with UDP connection type, I wrote Client-base app with Qudpsocket and read incoming data and Store in QByteArray().(Qt 64 bit GCC )

My problem is :
I want to store for 15 Second all packet in RAM and then store in HDD in *.txt file. (15 second * 800Mb=12Gb)
When capacity of QByteArray increase this Error will be appear "What() : bad_alloc()".
How can i solve this problem? How can i manage huge data and store big data to RAM when capacity pass the variable size?
please say the true and best structure.

sierdzio

Store the data in several QByteArrays instead of a single one :-) Simple and should work fine (and cross platform).

Or store it in a file in RAM (ram file system). Some info here and here.

Alexanov

1- is it possible to create dynamic QByteArray object in loop ?like below ,Because I want to increase time of storing in the next step.

//your code here
for (int i = 50; i >=0;i--){
        QByteArray *datagram'i' = new QByteArray();
}

2- in this solution , you say to store all received datagram in temp file in RAM?

Eeli K

@Alexanov 1Gbit ethernet isn't 1GBytes/second, it's 1GBits/second which is 1/8 of 1GBytes/second.

It's of course possible to create QByteArrays in a loop with new, however you have to store the pointers somewhere. Or maybe you can use QByteArrayList.

kshegunov

@Alexanov said in How to Store UDP packet for some second in RAM:

I want to store for 15 Second all packet in RAM and then store in HDD in *.txt file. (15 second * 800MB =12GB)

Do you have that amount of memory (in addition to what the system requires), as your error seems to indicate you don't?

tham

You can use an easier solution to allocate QByteArray

std::vector<QByteArray> byte_arrays(50);
//access it like this
qDebug()<<byte_arrays[0]; //0 mean you access first QByteArray in the vector
//no need to free the QByteArray because std::vector do it for you, who say resource management is hard in c++?

In most of the cases, following statement is correct
allocate resource on stack is better than smart pointer, allocate on smart pointer is better than raw pointer

Eeli K

@tham said in How to Store UDP packet for some second in RAM:

allocate resource on stack is better than smart pointer, allocate on smart pointer is better than raw pointer

Yes, although with Qt we have to remember that smart pointers are objects which manage lifetime of other objects, and in Qt the QObject based class system does it with parents. On the other hand some important classes (like QByteArray) are not QObject based. In Qt programming you can do

new MyQObjectBasedClass(pointerToParent);

and the new pointer isn't get lost. But this is getting offtopic...

SGaist

Hi,

Why not just send them by batch rather than trying to read them all there ?

[Moved in from duplicate thread ~kshegunov]

Alexanov

@kshegunov said in How to Store UDP packet for some second in RAM:

addition

Yes , I have 128G RAM and 1TB HDD. I did not have any problem with my Hardware.
Although I saw the task manager ,when my program allocate 1G Memory , OS terminate that process and send signal to Qt and say "bad alloc()".

My Qustion is: what do you do? if you want to accumulate UDP received packet for a few second and then store in HDD???

Alexanov

@Eeli-K said in How to Store UDP packet for some second in RAM:

create

yes I knew that . and in my question wrote wrong, thanks.
is it possible to show me how we can store pointer for some QByteArrays that were created in loop ?

Alexanov

@SGaist said in How to Store UDP packet for some second in RAM:

Why not just send them by batch rather than trying to read them all the

Hi
Can you explain more ? I did not understand very well !! :(

SGaist

Rather than store that much data in RAM and then dump everything in your HDD, make the chunk smaller and write more regularly.

Alexanov

Okay ,I did it and worked fine but , this method is stopgap solution,
because at second Step i want to replace 1G Ethernet card to 10G. When I replaced cards , "file.write" line (in my code) worked very slowly than packets were coming !(I lost some packets when my program was writing in file , the speed of write is more slower than speed of incoming data)!!!!!
do you have any solution for this ?
Thanks :)

SGaist

Only one: adapt your hardware.

You can't expect to dump GB worth of data each second on a single spinning disks hard drive (even solid state drive). Use something like a RAID to withstand the throughput.

Alexanov

@SGaist
for this reason i want to fill all received packets in RAM and after a few second
store in my HDD.(is it possible?)

sierdzio

@Alexanov said in How to Store UDP packet for some second in RAM:

the speed of write is more slower than speed of incoming data

In addition to what SGaist said, also consider moving various tasks into separate threads. So that your network thread will continue reading data into buffer while your IO thread writes to HDD.

If, however, you constantly (per second) get more data than your hard disks can write... there is no software solution for this (unless you can go clever with the data: analyze it and extract only important parts, compress it etc.).

Alexanov

@tham
Thank you so much, your solution is very helpful for me ;)
Finally, I used Qvector to create dynamic array of QByteArrays type.