QVector::realloc() throwing SIGSEGV when very large Surface3D is rendered

Tom asso

Does anyone know of a QVector::realloc() bug that occurs when realloc'ing very large memory blocks?

My app invokes methods in the Qt DataVisualization library to build and render a QSurface3DSeries object (i.e. x,y,z data), and those Qt apps invoke QVector methods. My app reads a topography dataset from a file, builds the QSurface3DSeries and passes it to a QML Surface3D object for display. In many cases my app renders the dataset correctly, but for very large datasets (e.g. 15962 rows x 14815 cols), QVector<QVector3D>::realloc() throws SIGSEGV in the QSGRenderThread (shown by gdb backtrace), only after main() calls QGuiApplication::exec(). Yet the app doesn’t throw this error for smaller datasets, e.g. 1450 rows x 1113 cols works ok).

I have a class TopographicSeries that is a subclass of Surface3DSeries, and here is its function that reads the dataset - latitude, longitude, and elevation - from a “grid” object and loads it into a QSurfaceDataArray:

void TopographicSeries::setTopography(void *gmtApi, GMT_GRID *grid)
{
  // Reset min/max latitude, longitude, height
  resetDataLimits();
 
 
  // Latitudes
  double *latit = grid->y;
 
  // Longitudes
  double *longit = grid->x;
 
  // This holds the data - uses QVector?
  QSurfaceDataArray *dataArray = new QSurfaceDataArray();
 
  // Reserve space for rows (latitudes) - uses QVector?
  dataArray->reserve(nRows);
 
  for (int row = 0; row < nRows; row++) {
    // Allocate a row - uses QVector?
    QSurfaceDataRow *newRow = new QSurfaceDataRow(nColumns); 
    for (int col = 0; col < nColumns; col++) {
 
      int index = GMT_Get_Index(gmtApi, grid->header, row, col);
      float dataValue = (float )grid->data[index];
      
      (*newRow)[col].setPosition(QVector3D(longit[col], dataValue, latit[row]));
 
      // Check longitude range
      m_minLongit = std::min<double>(m_minLongit, longit[col]);
      m_maxLongit = std::max<double>(m_maxLongit, longit[col]);
 
      // Check data value range
      m_minHeight = std::min<double>(m_minHeight, dataValue);
      m_maxHeight = std::max<double>(m_maxHeight, dataValue);
    }
    *dataArray << newRow;
 
    // Check latitude range
    m_minLatit = std::min<double>(m_minLatit, latit[row]);
    m_maxLatit = std::max<double>(m_maxLatit, latit[row]);
  }
 
  // Set Surface3DSeries data
  dataProxy()->resetArray(dataArray);
}

The above function is called by this code, invoked by main(), which loads the dataset and passes the TopographicSeries object to a Surface3D QML object:

// Load topographic dataset
m_topographicSeries->setTopography(); 
 
// Pass TopographicSeries to QML Surface3D object
m_surface3D->addSeries(m_topographicSeries);

main() then invokes QGuiApplication::exec(), and SIGSEGV is thrown after that; here is the ‘gdb’ backtrace, though the Qt code doesn’t include debugging symbols:

The Qt libraries don’t include debug info, but here’s the ‘gdb’ backtrace:

Thread 8 "QSGRenderThread" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff7ac27700 (LWP 19271)]
0x00007ffff6df70cf in QVector<QVector3D>::realloc(int, QFlags<QArrayData::AllocationOption>) ()
   from /home/oreilly/Qt5.14.2-ok/5.14.2/gcc_64/lib/libQt5DataVisualization.so.5
(gdb) list
1	<built-in>: No such file or directory.
(gdb) bt
#0  0x00007ffff6df70cf in QVector<QVector3D>::realloc(int, QFlags<QArrayData::AllocationOption>) ()
    at /home/oreilly/Qt5.14.2-ok/5.14.2/gcc_64/lib/libQt5DataVisualization.so.5
#1  0x00007ffff6e02fb5 in QVector<QVector3D>::resize(int) ()
    at /home/oreilly/Qt5.14.2-ok/5.14.2/gcc_64/lib/libQt5DataVisualization.so.5
#2  0x00007ffff6e0242d in QtDataVisualization::SurfaceObject::setUpSmoothData(QList<QVector<QtDataVisualization::QSurfaceDataItem>*> const&, QRect const&, bool, bool, bool) ()
    at /home/oreilly/Qt5.14.2-ok/5.14.2/gcc_64/lib/libQt5DataVisualization.so.5
#3  0x00007ffff6db47d5 in QtDataVisualization::Surface3DRenderer::updateObjects(QtDataVisualization::SurfaceSeriesRenderCache*, bool) ()
    at /home/oreilly/Qt5.14.2-ok/5.14.2/gcc_64/lib/libQt5DataVisualization.so.5
#4  0x00007ffff6dbf84e in QtDataVisualization::Surface3DRenderer::updateData() () at /home/oreilly/Qt5.14.2-ok/5.14.2/gcc_64/lib/libQt5DataVisualization.so.5
#5  0x00007ffff6dc5865 in QtDataVisualization::Abstract3DController::synchDataToRenderer() ()
    at /home/oreilly/Qt5.14.2-ok/5.14.2/gcc_64/lib/libQt5DataVisualization.so.5
#6  0x00007ffff6dc23e5 in QtDataVisualization::Surface3DController::synchDataToRenderer() ()
    at /home/oreilly/Qt5.14.2-ok/5.14.2/gcc_64/lib/libQt5DataVisualization.so.5
#7  0x00007fffc3bad39f in QtDataVisualization::AbstractDeclarative::synchDataToRenderer() ()
    at /home/oreilly/Qt5.14.2-ok/5.14.2/gcc_64/qml/QtDataVisualization/libdatavisualizationqml2.so
#8  0x00007ffff58d45f0 in void doActivate<false>(QObject*, int, void**) ()
    at /home/oreilly/Qt5.14.2-ok/5.14.2/gcc_64/lib/libQt5Core.so.5
#9  0x00007ffff7306278 in QQuickWindowPrivate::syncSceneGraph() ()
    at /home/oreilly/Qt5.14.2-ok/5.14.2/gcc_64/lib/libQt5Quick.so.5
#10 0x00007ffff72afd08 in QSGRenderThread::sync(bool, bool) ()
    at /home/oreilly/Qt5.14.2-ok/5.14.2/gcc_64/lib/libQt5Quick.so.5
#11 0x00007ffff72b1df7 in QSGRenderThread::syncAndRender(QImage*) ()
    at /home/oreilly/Qt5.14.2-ok/5.14.2/gcc_64/lib/libQt5Quick.so.5
#12 0x00007ffff72b56fb in QSGRenderThread::run() ()
    at /home/oreilly/Qt5.14.2-ok/5.14.2/gcc_64/lib/libQt5Quick.so.5
#13 0x00007ffff56bb375 in QThreadPrivate::start(void*) ()
    at /home/oreilly/Qt5.14.2-ok/5.14.2/gcc_64/lib/libQt5Core.so.5
---Type <return> to continue, or q <return> to quit---
#14 0x00007ffff46e46db in start_thread (arg=0x7fff7ac27700)
    at pthread_create.c:463
#15 0x00007ffff4d3f88f in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)

Has anyone seen a similar problem when allocating very large datasets with QVector?
Thanks!

jsulm

@Tom-asso So, you're allocating 236477030 * sizeof(QVector3D) bytes. Since QVector needs contiguous chunk of memory it can simply mean that there is no contiguous memory chunk of that size available anymore. You should check memory consumption of your app. Is your app 32bit or 64bit? Why do you allocate such a huge dataset?

Tom asso

@jsulm - My app is 64-bit. The app displays datasets (aka grid files) of seafloor bathymetry as aQtDataVisualization Surface3D object, and some of those bathymetry files are very large. How to check how much contiguous memory is available at run time?
Thanks!

JonB

@Tom-asso said in QVector::realloc() throwing SIGSEGV when very large Surface3D is rendered:

How to check how much contiguous memory is available at run time?

You can't.

Tom asso

Too bad QVector doesn't fail more gracefully. The SIGSEGV happens in the render thread after I've called methods in the main thread to reserve memory for the data. If I at least could predict in the main thread that QVector::realloc() would subsequently fail in the render thread I could do something about it - e.g. decimate the data for zoomed out view or something like that. Does anyone have a suggested strategy?
Thanks!

Tom asso

@jsulm - as a test, in the main thread I estimate bytes needed as nRows * nColumns * sizeof(QVector3D), then check if malloc(bytesNeeded) succeeds. I find that malloc() does succeed for that large dataset of 15962 rows x 14815 cols - 2837724360 bytes. I tried malloc() of five times that amount, and it still succeeds.
If I can predict that QVector will fail in the render thread, then I could take other measures...

Tom asso

Ah. QVector documentation says:

The current version of QVector is limited to just under 2 GB (2^31 bytes) in size.

My big dataset that triggers the SIGSEGV is 15962 rows x 14815 cols, and assuming that each of those points is represented as QVector3D (12 bytes) that means the dataset requires 15962 x 14815 x 12 = 2.8 GB - so exceeds the maximum QVector size.

If my app continues to use the DataVisualization AP I think I am stuck with this QVector limitation. :-(

SGaist

Hi,

You might want to check Qt 6 as there has been work done there on QVector.

fcarney

Can you subdivide the data into "chunks" and use multiple Surface3D objects?