General question about arrays and I/o

mrjj

Didn't understand that..
I meant that using whatever HDF5 uses to allow reading those large files might just work out of the box and then maybe no need for your own memmap file or similar. Was just saying you need something extra to drive such large files and it seems HDF5 does give that via its chunked file design.

(loops)
Ahh, That way. Well, there is a thing with loops in c++/Qt.
If you fill very large array in main thread, it will lag your program's interface.
But besides that, loops are fast in c++. (generally speaking)

(Qfile map)
QFile map function should also work in window as far as i know.
Windows does support it natively and i think Qfile map uses that.

The https://github.com/ess-dmsc/h5cpp
provides a c++ wrapper for a c library.
This is often done to allow for object orientated programming with the
c library and maybe hide details behind more easy to use classes than raw C code.
you dont need to use the wrapper if you feel good with c code.
However,

I dont have experience with HDF5 format but looking over the docs, it really seems the way to go as it should provide you with anything you need to make a c++ program that can consume and produce such giga files.

and the c api dont really look that bad
https://support.hdfgroup.org/ftp/HDF5/examples/examples-by-api/hdf5-examples/1_10/C/H5D/h5ex_d_chunk.c

Please_Help_me_D

@mrjj Thank you for information
I am working on I/O with information you provided :)

Please_Help_me_D

@mrjj one more question. I get an error: array subscript is not an integer:

int data[5] = {1, 2, 3, 4, 5};
int ind[3] = {0, 1, 2};
int data2 = data[ind]; // here is that error . **ind** is highlighted by red

I declare ind as integer but still can't get access to those elements of an array...

And here is similar problem expression is not determined by a constant. Failure caused by reading a variable beyond its lifetime:

    std::string str_file = "C:\\Users\\Tasik\\Documents\\Qt_prj\\proba.bin";
    int n = str_file.length();
    char char_file[n]; // here is that error. It appears only when I launch the application

SGaist

Hi,

ind is not an integer, it's an array of 3 integers.

Depending on your compiler you will have to allocate your char_file array on the heap using new and then delete when done with that array.

Please_Help_me_D

@SGaist Hello
Thank you for answer
So is there a way to extract few elements from an array at the same time without loop?

Depending on your compiler you will have to allocate your char_file array on the heap using new and then delete when done with that array.

My compiler is MSVC 2017. Could you write an example of this?

jsulm

@Please_Help_me_D said in General question about arrays and I/o:

So is there a way to extract few elements from an array at the same time without loop?

Not with plain C arrays.
But you can do this with QVector: https://doc.qt.io/qt-5/qvector.html#mid
There is something you can do without copying anything: an array is just a pointer to first element, so:

int data[5] = {1, 2, 3, 4, 5};
int *data2 = &data[2]; // data2 is now [3, 4, 5].

Do you really need to copy to data2? You can simply have a variable "length" containing the length of the sub-array in data.

"My compiler is MSVC 2017. Could you write an example of this?":

char *char_file = new char[n]; // Allocate on the heap
...
delete[] char_file; // Delete when not needed anymore

Please_Help_me_D

@jsulm thank you for the answer
The problem is that usually I have I know indexes are maybe like:

int data[4] = {1, 2, 3, 4, 5};
int ind[3] = {4, 2, 3};

and then I need to get access to those elements like:

int data2[3] = data[ind];

Now I read about QVector, I hope it is able to do that.

You know both examples that you wrote seems to me don't work properly.

    int data[5] = {1, 2, 3, 4, 5};
    int *data2 = &data[2]; // data2 is now [3].

And:

    int n = 5;
    char char_file = new char[n]; // error: cannot initialize a variable of type 'char' with an rvalue of type 'char *'
    delete[] char_file; // Delete when not needed anymore

Where can I read about '*' and '&' signs when using in such ways? What it gives?

jsulm

@Please_Help_me_D said in General question about arrays and I/o:

seems to me don't work properly

In what way? &data[2] points to 3 in data, so data2[0] == 3, data2[1] == 4 and data2[2] == 5

Please read about pointers in C/C++:

// It must be *char_file not just char_file
char *char_file = new char[n];

I edited my previous post as I forgot *

Please_Help_me_D

@jsulm

In what way? &data[2] points to 3 in data, so data2[0] == 3, data2[1] == 4 and data2[2] == 5

I attach the picture below. data2 is now is equal to 3 and that is it. Is it correct?
After I added * pointer the program works but seems to me that the length of char_file doesn't depend on n. If n=4 then length of char_file=32, n=5 then char_file=32. Is it ok?

jsulm

@Please_Help_me_D said in General question about arrays and I/o:

data2 is now is equal to 3 and that is it. Is it correct?

Yes it is, you can treat a pointer as an array (actually in C/C++ an array is simply a pointer to first element of the array). So, data2[0] == 3, data[1] == 4...
Just do

qDebug() << data2[1];

and see.

Regarding second question: this is debugger view. Your array is for sure 4 char in size. To verify do

char_file[4] = 1;

your app should crash.

Please_Help_me_D

@jsulm Yes that works:

    int data[5] = {1, 2, 3, 4, 5};
    int *data2 = &data[2]; // data2 is now [3, 4, 5].
    qDebug() << data2[2];

But what's the magic behind that?:) I debug I can see that data2 has only a single number.

But this doesn't crash and I can see some output in terminal (it is not 100 but some letters or signs as I think it is char) even if I lauch the program not in debug mode:

    int n = 2;
    char *char_file = new char[n]; // error: cannot initialize a variable of type 'char' with an rvalue of type 'char *'
    delete[] char_file; // Delete when not needed anymore

    char_file[5] = 100;

    std::cout << char_file[5];

Are there in Qt the possibility to use command line when the program stopped in debug mode? For example if it's stopped and I want to do something in real time (while the program is topped)? Like in Matlab command line in debug mode

JonB

@Please_Help_me_D said in General question about arrays and I/o:

Are there in Qt the possibility to use command line when the program stopped in debug mode? For example if it's stopped and I want to do something in real time (while the program is topped)? Like in Matlab command line in debug mode

No, this is a C++ compiled program (nothing to do with Qt), not Matlab/an interpreted language! You can print out values, and even at a pinch poke a value into a variable, but you can't start "telling" the debugger/program to go perform actions :)

jsulm

@Please_Help_me_D said in General question about arrays and I/o:

But what's the magic behind that?:) I debug I can see that data2 has only a single number.

Pointer magic :-) data2 is defined as pointer to int, that's why debugger only shows one value. But you as developer know that it's actually pointing to an array of int. Writing data2[1] is same as *(data2 + 1).
*(data2 + 1) means: give me the value in memory at the position (data2 + 1) is pointing to.
Keep in mind that in this case (data2 + 1) increments the pointer by 4 as sizeof(int) == 4.

This should actually crash as you're accessing memory which was already freed:

delete[] char_file; // Delete when not needed anymore
char_file[5] = 100;
std::cout << char_file[5];

Please_Help_me_D

@JonB thank you! Now I know that:)
@jsulm ok that is interesting. I need to read about pointers
I read about QVector and I can't solve the situation when I need to extract {4, 2, 3} elements from data (without loop) so that:
data2[0] = data[4] = 5,
data2[1] = data[2] = 3,
data2[2] = data[3] = 4 ,
where data = {1, 2, 3, 4, 5};
Seems to me that QVector::mid(int pos, int length = ...) can't do that. But of coarse:

    QVector<int> data = {1, 2, 3, 4, 5}; 
    QVector<int> data2 = data.mid(1,2); // data2 = {2, 3}

this is also good to know for me.
I'm trying to avoid loops here because it is like the main standart operation for me and I feel there should be a way to do that.

JonB

@Please_Help_me_D
I will say one thing about "loops": although you may try to avoid, it is likely that even if there is a library call that will do a loop. (Not always true, some adjacent elements may be implemented by a memmove or similar.) Even your Matlab or whatever may present an an operation as "non-loop" as far as you are concerned, but under the hood that is what it will have to do. C++ is "lower-level" and more "literal" about what is going on/has to go on than a higher level language like Matlab may suggest to you. Not saying you shouldn't ask, or try to avoid, but be aware it may be inevitable.

In your example, btw, technically there is no loop. It's just what it looks like: 3 separate statements (which cannot be optimized for non-adjacent data). And someone here will tell you these will take about 3 nanoseconds to perform.

Please_Help_me_D

@JonB Yes I understand that many function that don't use explicitly loops they use them under the hood. But I believe that in computer science there are some tricks that I don't know and that provide high perfomance of operations and specialists who write Qt functions (or Matlab functions) they implement those tricks to achieve good optimization of code.
I'm just starting learning Qt I think I just need experience in C++ :)

jsulm

@Please_Help_me_D said in General question about arrays and I/o:

I need to extract {4, 2, 3}

You will need to do it by yourself. There is nothing for that neither in Qt nor in C++ stdlib. Even if there would be it would do nothing else as iterating over {4, 2, 3} in a loop. So, I doubt you can optimise here much.