Fastest way to read part of 300 Gigabyte binary file

JonB

@jsulm
Your answer is in principle correct. However, should we warn the OP that I'm thinking this will only "work" if the return result from the QFile::map() he calls (given his offsets) is suitably aligned at a 32-bit boundary for qint32 * to address without segmenting?? I don't see the Qt docs mentioning whether this is the case for the normally-uchar * return result?

jsulm

@JonB Could be, I'm not sure

J.Hilk

@JonB
well if you take a look at the loop so far:

for(qint64 i = 0; i < N; i++){
            FFID[i] = memory[i*Nb];
            FFID[i+1] = memory[i*Nb+1];
            FFID[i+2] = memory[i*Nb+2];
            FFID[i+3] = memory[i*Nb+3];
        }

no checks inside the loop nor before, so it's going to hard crash any way, when the file is not int32_t aligned.

JonB

@J-Hilk
Umm, no, I don't see that. His current uchar *memory means it's only picking up bytes from there. And he made his FFID be QVector<uchar>. So he is copying one byte at a time (which is what I think he wants to get rid of), and current code won't have odd-boundary-memory-alignment issue. But new code with qint32* for uchar* could have problem....

If his offset is always like the example 7996 so it's divisible by 4 always then I would guess the return result from QFile::map() will not show any problem. This is an issue which does not arise when reading numbers from file, only from mapping, so just to be aware.

J.Hilk

@JonB
really? And what guarantees, that memory[i*Nb+3]; will be part of the valid memory ?

I assume this, is, what the OP wants to do

QVector<uchar> FFID(N*4); -> QVector<qint32> FFID(N);
uchar *memory -> qint32 *memory

and 
for(qint64 i = 0; i < N; i++){
            FFID[i] = memory[i*Nb];
        }

Please_Help_me_D

@jsulm thank you, that works!
@JonB @J-Hilk I think I see what you are discussing and I keep that in mind.
If I map the part of a file that is is not equal to N*4 (like in the code below) my program doesn't output any error or command line. Compiler says that it was succesfully built and application output throws that it started and one second later it is terminated.

#include <QCoreApplication>
#include <QFile>
#include <QVector>
//#include <QIODevice>
#include <armadillo>
using namespace arma;

int main()
{
    char segyFile[]{"C:/Users/tasik/Documents/Qt_Projects/raw_le.sgy"};
    QFile file(segyFile);
    if (!file.open(QIODevice::ReadOnly)) {
         //handle error
    }
    //qint32 *memory = new qint32;
    //(uchar*)&memory;
    uchar* memory = file.map(3608, file.size()-3607); // Here the mappable part file.size()-3607 has some remainder of the division by 4 
    (qint32*) memory;
    if (memory) {
        std::cout << "started..." << std::endl;
        wall_clock timer;
        qint64 fSize = file.size();
        qint64 N = 44861;
        qint64 Nb = 661*4;
        QVector<qint32> FFID(N);
        (uchar *)&FFID;
        timer.tic();
        for(qint64 i = 0; i < N; i++){
            FFID[i] = memory[i*Nb];
            /*FFID[i+1] = memory[i*Nb+1];
            FFID[i+2] = memory[i*Nb+2];
            FFID[i+3] = memory[i*Nb+3];*/
            std::cout << FFID[i] << std::endl;
        }
        double n0 = timer.toc();
        std::cout << n0 << std::endl;
        std::cout << "finished!" << std::endl;
    }
}

JonB

@Please_Help_me_D said in Fastest way to read part of 300 Gigabyte binary file:

and application output throws that it started and one second later it is terminated.

Yes, that was my point. You won't get a compilation error. You would get a run-time "crash" on something like line FFID[i] = memory[i*Nb];. Under Linux you'd get a core dump (if enabled), under Windoze I don't know but would have thought it would bring up a message box of some kind.

However, I haven't got time, I don't think the code you've written reflects this. For a start statements (qint32*) memory; and (uchar *)&FFID; are No-Ops (turn compiler warnings level up, you might get a warning of "no effect" for these lines, you should always develop with highest warning level you can). You haven't changed over the memory to qint32*, what you seem to think is how to do casts is wrong. This is C/C++ stuff. You'll want something more like

~~qint32* memory = static_cast<qint32*>(file.map(3608, file.size()-3607));~~

qint32* memory = reinterpret_cast<qint32*>(file.map(3608, file.size()-3607));

but I haven't got time to sort you out. And if you do that you need to understand how to then index it, it won't be the same offsets as you used when it was uchar*. Don't try to change to qint32* for your accesses if you don't know what you're doing cast-wise in C/C++! :)

Please_Help_me_D

@JonB said in Fastest way to read part of 300 Gigabyte binary file:

qint32* memory = static_cast<qint32*>(file.map(3608, file.size()-3607));

thank you but this sends me an error:

main.cpp:17:22: error: static_cast from 'uchar *' (aka 'unsigned char *') to 'qint32 *' (aka 'int *') is not allowed

J.Hilk

@Please_Help_me_D
@JonB meant to write reinterpret_cast not static_cast there are few uses for reinterpret_cast but this is one :)

Please_Help_me_D

@J-Hilk ok, now it works :)

Please_Help_me_D

@SGaist said in Fastest way to read part of 300 Gigabyte binary file:

Did you consider mapping only the parts that are pertinent to what you want to read ?

I don't know how to do that but I saw something like this in BOOST C++ documentation . Here is writen:
What is a memory mapped file?
File mapping is the association of a file's contents with a portion of the address space of a process. The system creates a file mapping to associate the file and the address space of the process. A mapped region is the portion of address space that the process uses to access the file's contents. A single file mapping can have several mapped regions, so that the user can associate parts of the file with the address space of the process without mapping the entire file in the address space, since the file can be bigger than the whole address space of the process (a 9GB DVD image file in a usual 32 bit systems). Processes read from and write to the file using pointers, just like with dynamic memory.

Maybe if I could map only regions of my file that I need to read then it would speed up my application? Does Qt provide something like that?

SGaist

Well... As already said, the map function takes an offset in your file and a size so you can map several regions of it with that. It's nowhere written that you have to passe an offset of zero and the full file size.

Please_Help_me_D

@SGaist I understand that I have offset and size parameters and actually I use them as a single valued numbers. If I want to map several regions of a file then I should use multiple offsets and multiple size but the example below doesn't work:

    qint64 offset[] = {100, 200, 300};
    qint64 size[] = {4, 4, 4};
    qint32* memory = reinterpret_cast<qint32*>(file.map(offset, size));

The error I get is:
main.cpp:19:57: error: cannot initialize a parameter of type 'qint64' (aka 'long long') with an lvalue of type 'qint64 [3]'
qfiledevice.h:127:23: note: passing argument to parameter 'offset' here

SGaist

You can't just replace an input type by an array of the same type. That's not how it's working. And in any case, the returned value of map is the address you'll have to pass to the unmap function.

You won't avoid using a form of loop or another.

JonB

@Please_Help_me_D
As @SGaist has said. You will need to make multiple calls to QFileDevice::map(), one for each of the distinct regions you want mapped.

Please_Help_me_D

@SGaist when I heard the word "loop" then I finnaly got the idea :)
Here is my code:

#include <QCoreApplication>
#include <QFile>
#include <QVector>
//#include <QIODevice>
#include <armadillo>
using namespace arma;

int main()
{
    char segyFile[]{"D:/STACK1_PRESTM.sgy"};
    QFile file(segyFile);
    qint64 fSize = file.size();
    qint64 N = 1734480;
    qint64 Nb = 2059*4;
    if (!file.open(QIODevice::ReadOnly)) {
         //handle error
    }
    //qint32* memory = reinterpret_cast<qint32*>(file.map(3608, file.size()-3608));
    qint32* memory = new qint32;
    QVector<qint32> FFID(N);
    std::cout << "started..." << std::endl;
    wall_clock timer;
    timer.tic();
    for (int i = 0; i < N; i++){
        memory = reinterpret_cast<qint32*>(file.map(3600+i*Nb, 1));
        FFID[i] = *memory;
        //std::cout << *memory << std::endl;
    }
    double n0 = timer.toc();
    std::cout << n0 << std::endl;
    std::cout << "finished!" << std::endl;
}

Is it possible to create to store in memory all the values that I need? Now I only have a pointer to the single value in var memory. Then I could avoid to use assigning values to FFID.
The timing result is almost the same:
SSD internal

QFile::map 97 Seconds (previously it was 86)

HDD internal

QFile::map 223 Seconds (previously it was 216)

To check the reliability of the results I also made the experiments with whole file mapping as I did before and the timings is the same. So there is no big difference whether to map the whole file or many regions of it

SGaist

Is it possible to create to store in memory all the values that I need? Now I only have a pointer to the single value in var memory. Then I could avoid to use assigning values to FFID.

I am not sure I understand that question. memory is a pointer to the start of the region you mapped. In any case, you are still not un-mapping anything in your code which is a bad idea.

In order to be able to answer your question, please explain what your are you going to do with the values you want to retrieve from that file.

Please_Help_me_D

@SGaist I forgot to unmap...
Here to get an array (or vector) of values in FFID I use:

    for (int i = 0; i < N; i++){
        memory = reinterpret_cast<qint32*>(file.map(3600+i*Nb, 1));
        FFID[i] = *memory;
        //std::cout << *memory << std::endl;
    }

I do that because memory is a pointer to a single value. If I could write something like:

    for (int i = 0; i < N; i++){
        memory[i] = reinterpret_cast<qint32*>(file.map(3600+i*Nb, 1));
    }

then I could avoid using FFID.
It doesn't make much sense in this case but i'm just interested and maybe this information would be useful in future in other situations.

jsulm

@Please_Help_me_D said in Fastest way to read part of 300 Gigabyte binary file:

memory is a pointer to a single value

No. It is a pointer to a chunk of memory (bytes), you can interpret that memory as you like. You can use memory as array (as in C/C++ an array is simply a pointer to first element):

FFID[i] = memory[i];

So, there is really no need to map inside the loop.

JonB

@jsulm
I'm afraid this is not what he means/how he is using memory. There are quite distinct, separate, non-contiguous areas of his memory-mapped file he wishes to access. He wishes to call QFile::map() many times, each one mapping a separate area of memory. He will need to retain those mapped addresses so that he can later unmap() them.

He should change to an array/list of memoryMappings. I'm not a C++ expert, but his code should be more like:

QVector<qint32*> memoryMappings(N);
for (int i = 0; i < N; i++){
    memoryMappings[i] = reinterpret_cast<qint32*>(file.map(3600+i*Nb, 1));
    FFID[i] = *memoryMappings[i]
}