How can I read large files to QByteArray?



  • QT 5.8.0

    I read file from disk:

    QFile CurrentFile(FullPath);
    if(!CurrentFile.open(QIODevice::ReadOnly)) return;
    QByteArray DataFile = CurrentFile.readAll();
    int S = DataFile.size();
    

    But the size() return type int and I get 0, if the file size is greater than 2 147 483 647 bit (~270 MB). How can i read large files uses QByteArray?


  • Moderators

    @AlekseyB You're wrong: as explained in the documentation QFile::size() returns an quin64, not int. And size is in byte not bits.
    See http://doc.qt.io/qt-5/qfile.html#size. Use quing64 instead of int for S.



  • @jsulm I use the size() for variable DataFile, type which is QByteArray.


  • Moderators

    @AlekseyB 2 147 483 647 bytes are around 2Gbyte, not 270MB. How big is the files you're trying to load? Keep in mind that you are loading huge amount of data into RAM.



  • @jsulm I made a mistake with my calculations maximum size data. I read the file, which size is 3.1 GB



  • Why do you need to load so much data in RAM?



  • @VRonin I encrypt files.

    As I understand QbyteArray is not designed for the size as I needed. So, I decided to create needed class. Thank you for all!


  • Moderators

    @AlekseyB In such situations you should use memory mapped files. This way you avoid to loading everything at once into RAM but still can access the whole file content. See http://doc.qt.io/qt-5/qfiledevice.html#map



  • If you just need to encrypt the file you can use a crypto library like Crypto++ (Boost License so fits 99.9999999% of the projects). That library explicitly include an example of encrypting and decrypting an arbitrarily big file. No need to load the file in memory



  • @AlekseyB If you decide to create your own class instead of QByteArray, that is fine.
    Still loading anything over 2 Gig into your ram is a bad idea, even 2 is stretching it.
    What happens when your user has less ram available than the file size that has to be worked on?
    Why don't you read it one sector at a time, do your encrypting -> write that sector -> move on to the next one.


Log in to reply