An error occurred: bad allocation
-
I am sending a large file via TCP, the file has been stored in a database in 256MB chunks. I have a query that takes the records one at a time and sends the chunk via TCP. The problem is after two chunks are sent I get an error message displayed in the console:
An error occurred: bad allocationHere is the code:
for( long lngBlockNo=0; lngBlockNo<(long)ulngTotalBlocks; lngBlockNo++ ) { QSqlQuery query; query.prepare("SELECT" " `binChunk`" " FROM" " `rdf`" " WHERE" " `biDataset`=?" " AND" " `intBlockNo`=?"); query.addBindValue(lngDatasetID); query.addBindValue(lngBlockNo); if ( Trainer::queryDB(query) != true ) { break; } while(query.next()) { QSqlRecord record(query.record()); int intTotal(record.count()); if ( intTotal > 0 ) { QSqlField field(record.field(0)); QVariant varValue(field.value()); if ( varValue.isValid() == true ) { QByteArray baBLOB(varValue.toByteArray()); //Sum up size ulngCalcSize += baBLOB.size(); qdbg() << "baBLOB.length: " << baBLOB.length() << ", ulngCalcSize: " << ulngCalcSize; //Send binary data pTrainee->sendBinary(baBLOB); } } } }Any suggestions on how I can avoid this error?
@SPlatten said in An error occurred: bad allocation:
pTrainee
I don't know what this is, but I guess you get out of memory because data is sent slower over network compared to you passing the data. You should call sendBinary() when the previous chunk was completely sent.
-
@SPlatten said in An error occurred: bad allocation:
pTrainee
I don't know what this is, but I guess you get out of memory because data is sent slower over network compared to you passing the data. You should call sendBinary() when the previous chunk was completely sent.
@jsulm , thank you, I'm not sure what you mean, baBLOB contains 268435456 bytes 256MB.
pTrainee is a pointer to an instance of a class that presents a client, here is the sendBinary function:
void TraineeMonitor::sendBinary(const QByteArray& crbaData) { if ( mpTCPsocket != nullptr ) { mpTCPsocket->write(crbaData); } }Kind Regards,
Sy -
I am sending a large file via TCP, the file has been stored in a database in 256MB chunks. I have a query that takes the records one at a time and sends the chunk via TCP. The problem is after two chunks are sent I get an error message displayed in the console:
An error occurred: bad allocationHere is the code:
for( long lngBlockNo=0; lngBlockNo<(long)ulngTotalBlocks; lngBlockNo++ ) { QSqlQuery query; query.prepare("SELECT" " `binChunk`" " FROM" " `rdf`" " WHERE" " `biDataset`=?" " AND" " `intBlockNo`=?"); query.addBindValue(lngDatasetID); query.addBindValue(lngBlockNo); if ( Trainer::queryDB(query) != true ) { break; } while(query.next()) { QSqlRecord record(query.record()); int intTotal(record.count()); if ( intTotal > 0 ) { QSqlField field(record.field(0)); QVariant varValue(field.value()); if ( varValue.isValid() == true ) { QByteArray baBLOB(varValue.toByteArray()); //Sum up size ulngCalcSize += baBLOB.size(); qdbg() << "baBLOB.length: " << baBLOB.length() << ", ulngCalcSize: " << ulngCalcSize; //Send binary data pTrainee->sendBinary(baBLOB); } } } }Any suggestions on how I can avoid this error?
@SPlatten
Two small comments while I look at your code, both of which are probably not important. I know you are trying squeeze efficiency out of your code. Nothing to do with your problem, but:-
Any reason you make separate SQL calls for each blob in a loop? Seems to me same could be achieved via
SELECT ... WHERE biDataset=? ORDER BY intBlockNo. -
You choose to use
QSqlRecord,QSqlField,QVarianton these large returned blobs. Do these level of calls impose any extra overhead (e.g. extra copying of 256MB) compared to the lowest level calls available?
-
-
@jsulm , thank you, I'm not sure what you mean, baBLOB contains 268435456 bytes 256MB.
pTrainee is a pointer to an instance of a class that presents a client, here is the sendBinary function:
void TraineeMonitor::sendBinary(const QByteArray& crbaData) { if ( mpTCPsocket != nullptr ) { mpTCPsocket->write(crbaData); } }@SPlatten said in An error occurred: bad allocation:
mpTCPsocket->write(crbaData);So at a guess wait till
QAbstractSocket::waitForBytesWritten()orQIODevice::bytesWritten()(see note especially if Windows) there to make sure the write is not complaining too much has not been written yet when next chunk is sent? You should debug to see whenAn error occurred: bad allocationarrives. -
@SPlatten said in An error occurred: bad allocation:
mpTCPsocket->write(crbaData);So at a guess wait till
QAbstractSocket::waitForBytesWritten()orQIODevice::bytesWritten()(see note especially if Windows) there to make sure the write is not complaining too much has not been written yet when next chunk is sent? You should debug to see whenAn error occurred: bad allocationarrives. -
@jsulm , thank you, I'm not sure what you mean, baBLOB contains 268435456 bytes 256MB.
pTrainee is a pointer to an instance of a class that presents a client, here is the sendBinary function:
void TraineeMonitor::sendBinary(const QByteArray& crbaData) { if ( mpTCPsocket != nullptr ) { mpTCPsocket->write(crbaData); } }@SPlatten said in An error occurred: bad allocation:
I'm not sure what you mean
What I mean is that you are writing data to the socket faster than it can send it...
-
@SPlatten said in An error occurred: bad allocation:
I'm not sure what you mean
What I mean is that you are writing data to the socket faster than it can send it...
-
@jsulm , ok, thank you, I've added a call to waitForBytesWritten to the sendBinary function.
-
@SPlatten said in An error occurred: bad allocation:
I've added a call to waitForBytesWritten to the sendBinary function.
Are you on Linux or Windows?
-
@SPlatten So I wrote:
or
QIODevice::bytesWritten()(see note especially if Windows)There is a reason I wrote that for you, I thought you would look at it before implementing. Did you read QAbstractSocket::waitForBytesWritten
Note: This function may fail randomly on Windows. Consider using the event loop and the
bytesWritten()signal if your software will run on Windows.? Up to you, just be aware.
-
@SPlatten So I wrote:
or
QIODevice::bytesWritten()(see note especially if Windows)There is a reason I wrote that for you, I thought you would look at it before implementing. Did you read QAbstractSocket::waitForBytesWritten
Note: This function may fail randomly on Windows. Consider using the event loop and the
bytesWritten()signal if your software will run on Windows.? Up to you, just be aware.
-
@SPlatten
:) Of course be aware you cannot have both in your running code, they will get in each other's way, one or the other! -
@JonB , looking at the documentation:
https://doc.qt.io/qt-5/qiodevice.html#write-2write returns the number of bytes written, how can this work when you call a function like waitForBytesWritten after calling write?
For buffered devices, this function [
waitForBytesWritten()] waits until a payload of buffered written data has been written to the deviceI think there is some distinction between
write()accepting the bytes initially into a buffer versus those buffered bytes being sent to the device. Easy enough to use the signal to check, or look at whatever the source code does forwaitForBytesWritten().In any case: if you quickly try
waitForBytesWritten()(and it works under Windows, I don't know, else use the signal) and your error message goes away you know where you are. If it does not, you do not.
