[Solved] Allocate and read from memory and write to a File

[pushpa416]

Hi There,

I got the data stored in memory in a 2D format which is 8000rows and 70000Columns, both rows and columns are variable numbers, the maximum rows i can go up to 8192 and maximum columns can go upto 200000columns.

now I have to read and store the data in a txt file for 8000rows and 70000Columns.

By doing like following I could be able to read (8000rows and 33000Columns) or (5000rows and 63000columns) without any Problem.

@
unsigned short buf;
int fd;
buf = (unsigned short )malloc(60010241024);

fd = FOS_Open(0);
FOS_Read_Data((uint32 *)buf, 0, 0, sel_last_column, sel_last_row, 2); // read function provided
FOS_Close(fd);

uint32 i,j;

QFile res("cotdr.txt"); // writing the resultant array of values to a txt file
if (res.open(QIODevice::WriteOnly | QIODevice::Truncate)) {
QTextStream result(&res);
for(i = 0; i < (lel_last_column)*(sel_last_row);i+=sel_last_column)
{
for (j=sel_first_column; j<(sel_last_column); j++)
{
result << buf[i+j] << QChar(',');
}
result << "\n";
}
free(buf);
@

everything is on Linux based system and i am doing all this under push_button_clicked.
if at all I increase the malloc size or number of rows or number of columns, Gui Crashes immeadiately with the above code.
Could someone please suggest me how i could overcome this, example is much appreciated.

Thanks

[edit: added missing coding tags @ SGaist]

[leonardo.ramosantos]

Hi there,

Crash can happen because malloc gets a wide free memory space. Thus, malloc will look for an area of memory that fits the size you want. However, you may not have all the space you want, then your system goes down.
What you can do is test the buf variable to make sure if malloc worked. Try this:

@ buf = (unsigned short ) malloc(60010241024);
if (buf) {
/ here you put your code */
} else {
qDebug() << "malloc didn't work!!!";
}@

If Buf is not valid, in fact, malloc failed to allocate memory.

I hope I have helped you.

[SGaist]

Hi,

Are you also aware that you are trying to allocate 600MB at a time ? Is your application 32bit ? Do you have other such allocation in it ?

Also, since you need to re-parse the data anyway, why not read it one row at a time ?

[pushpa416]

Hi both,

when i print the buf message it doesn't show the fail message still failed(Gui Exits). it is a 32 bit system. i don't have any other allocations of 600MB.

SGaist, does reading one row at time means (used while loop instead of for loop still fails), sorry I am bit new to the programming(believe that was correct).

How Else i can read the data without allocating 600MB? if you don't mind could you please give me an example?

Thanks,

[SGaist]

I don't know how FOS_Read_Data works but yes the idea is to call it repeatedly to get the data you want line by line.

Also, your current allocation is a bit strange

@
unsigned short buf;
buf = (unsigned short )malloc(60010241024); // You don't take into account the size of short

fd = FOS_Open(0);
FOS_Read_Data((uint32 *)buf, 0, 0, sel_last_column, sel_last_row, 2); // You are telling that the buffer is uint32 however from your allocation it should be unsigned short which size is not be the same as uint32
@

[pushpa416]

Hi SGaist,

Thank you for your help, in the FOS_Read_Data structure the buf was defined as default unsigned32, so i have to use as it is. and the values stored in the memory was 16bit value so i was used unsigned short to read. but anyway i could be able to solve it finally.

Thank you