Qt arrays with more than [500000] indexes
-
Hello,
right now im creating a software fro my university and we want to analise two .csv files which have both about 300.000 lines! So i wanna put each of them into a QStringList array like this:
QStringList arr1[300000], arr2[300000];
And exactly there is where my problem starts. I tested it and the prolem is, when I creat arraays which indexes are summed up over ~500.000 then the pogramm will not start! Following program works:
QStringList arr1[250000], arr2[250000];
Thats actually my main problem, can please somebody help me?
Thaks a lot!
-
Google is very helpful:
https://www.geeksforgeeks.org/stack-vs-heap-memory-allocation/ -
First determine size of QString() when empty. Then multiply this by 500000. Determine if this is bigger than your stack size (most likely). You are most likely experiencing a stack overflow. You can only allocate so much space on the stack before you run out of room. If you want big sets of data you want to allocate on the heap.
-
Hi.
have you considered using QTextStream? CSV files are text, unless for some reason you need to keep those files in memory there is no need to load them all at once. -
First determine size of QString() when empty. Then multiply this by 500000. Determine if this is bigger than your stack size (most likely). You are most likely experiencing a stack overflow. You can only allocate so much space on the stack before you run out of room. If you want big sets of data you want to allocate on the heap.
-
Hi.
have you considered using QTextStream? CSV files are text, unless for some reason you need to keep those files in memory there is no need to load them all at once. -
@artwaw
Thanks for ur answer but for my project it is important to safe the files, thatswhy i prefere Stringlist, but actually im more interested to solve the problem i told because i dont understand why i dont can create an aray of e.g. 1.000.000...@nifroidl But you can read/write transparently using streams. They just take care of I/O operations for you, once properly associated with the files, and it is hard to run out of memory too. Of course, using them is a bit more complicated than simple string list. Perhaps if you could share the algorithm/workflow we could be more helpful?
-
Google is very helpful:
https://www.geeksforgeeks.org/stack-vs-heap-memory-allocation/ -
Google is very helpful:
https://www.geeksforgeeks.org/stack-vs-heap-memory-allocation/ -
Hi,
@nifroidl said in Qt arrays with more than [500000] indexes:
QStringList arr1[250000], arr2[250000];
Do you realize that you are declaring two big arrays of QStringList ?
Using two QVector of QStringList if you want to store there the content of your files will get you further.
-
Hello,
right now im creating a software fro my university and we want to analise two .csv files which have both about 300.000 lines! So i wanna put each of them into a QStringList array like this:
QStringList arr1[300000], arr2[300000];
And exactly there is where my problem starts. I tested it and the prolem is, when I creat arraays which indexes are summed up over ~500.000 then the pogramm will not start! Following program works:
QStringList arr1[250000], arr2[250000];
Thats actually my main problem, can please somebody help me?
Thaks a lot!
@nifroidl Hi i have a code, where i scanning all my system check files and sum files(lines) and paste all files to file.
You big files it database. Big file it type you data.Big file have format text(1 format for all lines) You ptogram it methods for find and more in you data.
Check themes iterator, maps,Qfile, threads,ios(tream). my scan check all my files where 1-2-3 millions lines for maximum 1 minute, and you have more methods + boost library for more methods you filesystems and check files.
you index its 1 format for every line. this speed
But if you create *arr 1gb and you put to memory this array it cache file it not way for you and not speed method
