How can we handle large files in qt?

[aurora]

Is there any good way for handling text files of hundred of bytes?
If there any such please tell me...

I my GUI program, i need to read very large files (around 200MB..!!!) and need to perform search operation on that...
I am reading line by line and displaying them in QTableWidget but as the file size increases, program crashes or hangs....what should i do, in this case?
How can i load such big files in QTableWidget?
How can i perform searching efficiently on them?
Does Qt provides any good mechanism for these kind of problems?

[qxoz]

I've never tried to work with such large files. But maybe it's worth a try with ModelView architecture. QTableView and QStandardItemModel(or some other models or write your own model).

[lgeyer]

If you really have to handle large amounts of data and you want your application to be conservative memory wise you should probably implement your own QAIM which supports streaming and caching.

[andre]

Memory-mapping the file would be a start, I guess. In no case, try to load it all in one go. Also note that depending on the encoding of the original file, your memory consumption can be quite a bit higher than just the size of the original file. Still, 200MB should be doable. It is not that big.

[Guest]

Hi aurora,
You can use qsql for this issue and it is more efficient to manipulate data like retrieving, searching. refer this link i think it will help for u http://developer.qt.nokia.com/doc/qt-4.8/qsqlquery.html

[aurora]

Thank u all for your valuable suggestion, i'll go through all the alternatives that u specified and come back to if there is any more problem

[andre]

[quote author="cdeepak" date="1329191204"]Hi aurora,
You can use qsql for this issue and it is more efficient to manipulate data like retrieving, searching. refer this link i think it will help for u http://developer.qt.nokia.com/doc/qt-4.8/qsqlquery.html[/quote]

That would really depend on the kind of data you are manipulating, and the input format you have been given to work with. If your data is suitable for this: by all means, use it, but not all data is. Putting a huge text file in a database for instance doesn't really work.

[Guest]

yes, Of course Andre is right. if your data is suitable u can use this.

[deisik]

[quote author="aurora" date="1329131080"]I am reading line by line and displaying them in QTableWidget but as the file size increases, program crashes or hangs….what should i do, in this case? How can i load such big files in QTableWidget?[/quote]

No way. Possibly you could try QTableView but your mileage may vary

[quote author="aurora" date="1329131080"]How can i perform searching efficiently on them?[/quote]

You should build something like a binary search tree on the data

[quote author="aurora" date="1329131080"]Does Qt provides any good mechanism for these kind of problems? [/quote]

The cruel answer is no, qt won't help you in any of these problems. You should rely only on your sheer C/C++ skills in such questions if you want to create something workable. In short, you will have to write your own file engine or use an existing one to work directly off disk (relational DBMS won't help you either). And probably you will have to exclude qt from everything that deals with large datasets. For sure somebody will get up here and point a finger at me but before blowing fire read "this":http://bugreports.qt-project.org/browse/QTBUG-18490