An error occurred: bad allocation

SPlatten

I am sending a large file via TCP, the file has been stored in a database in 256MB chunks. I have a query that takes the records one at a time and sends the chunk via TCP. The problem is after two chunks are sent I get an error message displayed in the console:

An error occurred: bad allocation

Here is the code:

for( long lngBlockNo=0; lngBlockNo<(long)ulngTotalBlocks; lngBlockNo++ )
        {
            QSqlQuery query;
            query.prepare("SELECT"
                          " `binChunk`"
                          " FROM"
                          " `rdf`"
                          " WHERE"
                          " `biDataset`=?"
                          " AND"
                          " `intBlockNo`=?");
            query.addBindValue(lngDatasetID);
            query.addBindValue(lngBlockNo);
            if ( Trainer::queryDB(query) != true )
            {
                break;
            }
            while(query.next())
            {
                QSqlRecord record(query.record());
                int intTotal(record.count());
                if ( intTotal > 0 )
                {
                    QSqlField field(record.field(0));
                    QVariant varValue(field.value());
                    if ( varValue.isValid() == true )
                    {
                        QByteArray baBLOB(varValue.toByteArray());
    //Sum up size
                        ulngCalcSize += baBLOB.size();
qdbg() << "baBLOB.length: " << baBLOB.length() << ", ulngCalcSize: " << ulngCalcSize;
    //Send binary data
                        pTrainee->sendBinary(baBLOB);
                    }
                }
            }
        }

Any suggestions on how I can avoid this error?

jsulm

@SPlatten said in An error occurred: bad allocation:

pTrainee

I don't know what this is, but I guess you get out of memory because data is sent slower over network compared to you passing the data. You should call sendBinary() when the previous chunk was completely sent.

SPlatten

@jsulm , thank you, I'm not sure what you mean, baBLOB contains 268435456 bytes 256MB.

pTrainee is a pointer to an instance of a class that presents a client, here is the sendBinary function:

void TraineeMonitor::sendBinary(const QByteArray& crbaData) {
    if ( mpTCPsocket != nullptr ) {
       mpTCPsocket->write(crbaData); 
   }
}

JonB

@SPlatten
Two small comments while I look at your code, both of which are probably not important. I know you are trying squeeze efficiency out of your code. Nothing to do with your problem, but:

Any reason you make separate SQL calls for each blob in a loop? Seems to me same could be achieved via SELECT ... WHERE biDataset=? ORDER BY intBlockNo.
You choose to use QSqlRecord, QSqlField, QVariant on these large returned blobs. Do these level of calls impose any extra overhead (e.g. extra copying of 256MB) compared to the lowest level calls available?

JonB

@SPlatten said in An error occurred: bad allocation:

mpTCPsocket->write(crbaData);

So at a guess wait till QAbstractSocket::waitForBytesWritten() or QIODevice::bytesWritten() (see note especially if Windows) there to make sure the write is not complaining too much has not been written yet when next chunk is sent? You should debug to see when An error occurred: bad allocation arrives.

SPlatten

@JonB , thank you, will give it a go.

jsulm

@SPlatten said in An error occurred: bad allocation:

I'm not sure what you mean

What I mean is that you are writing data to the socket faster than it can send it...

SPlatten

@jsulm , ok, thank you, I've added a call to waitForBytesWritten to the sendBinary function.

JonB

@SPlatten said in An error occurred: bad allocation:

I've added a call to waitForBytesWritten to the sendBinary function.

Are you on Linux or Windows?

SPlatten

@JonB This project is on Windows 10.

JonB

@SPlatten So I wrote:

or QIODevice::bytesWritten() (see note especially if Windows)

There is a reason I wrote that for you, I thought you would look at it before implementing. Did you read QAbstractSocket::waitForBytesWritten

Note: This function may fail randomly on Windows. Consider using the event loop and the bytesWritten() signal if your software will run on Windows.

? Up to you, just be aware.

SPlatten

@JonB , thank you, will implement bytesWritten too.

JonB

@SPlatten
:) Of course be aware you cannot have both in your running code, they will get in each other's way, one or the other!

SPlatten

@JonB , looking at the documentation:
https://doc.qt.io/qt-5/qiodevice.html#write-2

write returns the number of bytes written, how can this work when you call a function like waitForBytesWritten after calling write?

JonB

@SPlatten

For buffered devices, this function [waitForBytesWritten()] waits until a payload of buffered written data has been written to the device

I think there is some distinction between write() accepting the bytes initially into a buffer versus those buffered bytes being sent to the device. Easy enough to use the signal to check, or look at whatever the source code does for waitForBytesWritten().

In any case: if you quickly try waitForBytesWritten() (and it works under Windows, I don't know, else use the signal) and your error message goes away you know where you are. If it does not, you do not.