How to pass a 2D array without memory issue ?

TremblayGIS

Hello,
i am working on image analysis.
I need to transfert the pixels values into an array for specific grid analysis.

So i have a function in an external class called gdalfile like this :

GDAL FILE**

float **gdalfile::GetFileAsAnArray()
 {

.....
     float *pafScanline;
.....
 // nbPixel = nb column * nb line
//pafScanline  is a buffer used by the gdal librairie to get the image value
     pafScanline = (float *) CPLMalloc(sizeof(float)*nbPixel);

// GDAL function which put the pixel value in pafScanline 
 ....
//--> SO FAR NO PROBLEM


// Creat the array
 int count(0);
 int nColonnes=nXSize;
 int  nLignes=nYSize;

//dynamic 2D array
 float **t;
  t = new float* [ nLignes ];
  for (int i=0; i < nLignes; i++)
    t[i] = new float[ nColonnes ];

 // fill the array with pafScanline value
  for (int i=0; i < nLignes; i++)
    for (int j=0; j < nColonnes; j++)
    {
    t[i][j] = pafScanline[count];
    count++;
    }

  return t ;

Here is the main.cpp
MAIN**

//constructors 
........

// Get the array values

    float** p= gdalTest.GetFileAsAnArray();

//Some analysis
     for (int i=0; i < gdalTest.getYLine(); i++)
       for (int j=0; j < gdalTest.getXColonne(); j++)
       {

       p[i][j]= .......
......

       }

After around 100000 loop, i got this :
ERROR 2: CPLMalloc(): Out of memory allocating 65536 bytes.

I guess this comes from the way i pass mya array.

Could you help me ?

Regards

jsulm

@TremblayGIS What loop do you mean? Do you have a loop in main where you constantly allocating memory? You can just free the memory before allocating new:

for(...) {
float** p= gdalTest.GetFileAsAnArray();

//Some analysis
for (int i=0; i < gdalTest.getYLine(); i++)
for (int j=0; j < gdalTest.getXColonne(); j++)
{

   p[i][j]= .......
}
delete[] p;
}
......

   }

jsulm

@TremblayGIS If this array always has same dimensions, then no need to allocate all the time: just allocate once and then fill it with data.

VRonin

p[i][j]= ....... the problem is somewhere in the ....... part. Looks like you are having a memory leak that is eating away all your available RAM.

on the other hand, this is 2017 and it's C++, why are you using such a convoluted code?

#include <boost/multi_array.hpp>
#include <memory>
#include <cassert> //optional, for assert()
boost::multi_array<float, 2> gdalfile::GetFileAsAnArray(){
const int nColonnes=nXSize;
const int  nLignes=nYSize;
std::unique_ptr<float[]> pafScanline(new float[nbPixel]);
// GDAL function which put the pixel value in pafScanline (use pafScanline.get() for the raw pointer)
assert(nLignes*nColonnes<=nbPixel);
boost::multi_array<float, 2> t(boost::extents[nLignes][nColonnes]);
int count=0;
for (int i=0; i < nLignes; i++)
    for (int j=0; j < nColonnes; j++)
        t[i][j] = pafScanline[count++];
}
return t;
}

or if you don't want to use boost:

#include <vector>
#include <memory>
#include <cassert> //optional, for assert()
std::vector< std::vector<float> > gdalfile::GetFileAsAnArray(){
const int nColonnes=nXSize;
const int  nLignes=nYSize;
std::unique_ptr<float[]> pafScanline(new float[nbPixel]);
// GDAL function which put the pixel value in pafScanline (use pafScanline.get() for the raw pointer)
assert(nLignes*nColonnes<=nbPixel);
std::vector<std::vector<float> > t(nLignes, std::vector<float>(nColonnes));
int count=0;
for (int i=0; i < nLignes; i++)
    for (int j=0; j < nColonnes; j++)
        t[i][j] = pafScanline[count++];
}
return t;
}

@jsulm said in How to pass a 2D array without memory issue ?:

delete[] p;

Small note, malloc needs free, not delete

TremblayGIS

Thank you so much for your help
i'm gone a try that !

regards

TremblayGIS

Hello, in fact i tried tis

cpp :

...

// 2D : column & lines dimensions
     const int nColonnes;
     const int  nLignes;

 //array. nbPixel = nLignes*nColonnes
    float pafScanline[nbPixel];

    // fill pafScanline with trivial value
      for(int i(0); i<nbPixel; ++i)
      {
          pafScanline[i] = i*3 ;
      }

//fill  vector with pafScanline  
 std::vector<std::vector<float> > t(nLignes, std::vector<float>(nColonnes));
 int count=0;
 for (int i=0; i < nLignes; i++)
     for (int j=0; j < nColonnes; j++)
        {
         t[i][j] = pafScanline[count++];
        }

return t

main :

int compteur(0);
std::vector< std::vector<float> > p;
p= gdalTest.GetFileAsAnArray();

 for (int i=0; i < gdalTest.getYLine(); i++)
   for (int j=0; j < gdalTest.getXColonne(); j++)
   {
   p[i][j]= p[i][j]+1500;
   compteur++;
       qDebug()<< p[i][j];
   }

but after a while my program crashed (compteur = about 100 000 ) i got a crashed. i gess it's due to a memory issue.

so i tried this to optimize the memory by passing the reference not the values :

CPP :

std::vector<std::vector<float> >* gdalfile::GetFileAsAnArray()
 {
// 2D : column & lines dimensions
     const int nColonnes;
     const int  nLignes;

 //array. nbPixel = nLignes*nColonnes
    float pafScanline[nbPixel];
  
    // fill pafScanline with trivial value
      for(int i(0); i<nbPixel; ++i)
      {
          pafScanline[i] = i*3 ;
      }

std::vector<std::vector<float> >* t(nLignes, std::vector<float>(nColonnes)); 

 int count=0;
//loop to fill the vector
 for (int i=0; i < nLignes; i++)
 {
       t[i]=new std::vector<float>(nColonnes);
     for (int j=0; j < nColonnes; j++)
        {
         t[i][j] = pafScanline[count++];
        }
}

  return t ;

I GOT THIS ERROR :
// D:\Documents\codeblocks\testTableau\gdalfile.cpp|43|error: expression list treated as compound expression in initializer [-fpermissive]|
//D:\Documents\codeblocks\testTableau\gdalfile.cpp|43|error: cannot convert 'std::vector<float>' to 'std::vector<std::vector<float> >*' in initialization|

would you have some ideas about what is wrong. I just want to pass the reference of the vector in order to avoid to copy it.

Regards

VRonin

this std::vector<std::vector<float> >* t(nLignes, std::vector<float>(nColonnes)); should become auto* t=new std::vector<std::vector<float> >(nLignes, std::vector<float>(nColonnes)); and this t[i]=new std::vector<float>(nColonnes); should be t[i]=std::vector<float>(nColonnes); but this will not improve anything in your program. std::vector is move constructable and move assignable so it doesn't make a copy when you call p= gdalTest.GetFileAsAnArray();

your problem is somewhere else as the piece of code that crashes does not allocate any memory. Since you removed all my assertions from your code I'll have to ask you to check what nbPixel , nLignes and nColonnes contain

TremblayGIS

Ok thank you for your prompt help
i'm gone a try that
regards

VRonin

@TremblayGIS said in How to pass a 2D array without memory issue ?:

i'm gone a try that

Try what? my point was that you code does not need to be changed but that the error was coming from somewhere else, like nbPixel ,nLignes or nColonnes