Database query, front end update, suggestions?
-
I have a lot of data stored in a database (MariaDB). I query the data by emitting a signal queryBlock:
for( long lngBlockNo=0; lngBlockNo<lngTotalBlocks; lngBlockNo++ ) { if ( blnOnline() != true ) { //Trainee has gone offline, stop transfer break; } //Push each query to a stack then pop each query from the stack QThread::msleep(TraineeMonitor::mscuintDelayBetweenQueries); emit queryBlock(mlngDatasetID, lngBlockNo); }queryBlock is connected to the slot onQueryBlock:
void TraineeMonitor::onQueryBlock(long lngDatasetID, quint32 uint32BlockNo) { QSqlQuery query; query.prepare("SELECT" " `binChunk`" " FROM" " `rdf`" " WHERE" " `biDataset`=?" " AND" " `intBlockNo`=?"); query.addBindValue(lngDatasetID); query.addBindValue(uint32BlockNo); if ( Trainer::queryDB(query) != true ) { return; } //Whilst there is a block to send and the Trainee is online if ( query.next() ) { QByteArray baData(query.value(0).toByteArray()); if ( baData.isEmpty() == true ) { return; } //Send binary data sendBinary(uint32BlockNo, baData); } }Here is the sendBinary function:
void TraineeMonitor::sendBinary(quint32 uint32BlockNo, const QByteArray& crbaData) { if ( crbaData.size() == 0 ) { //Nothing to send! return; } if ( mpTCPsocket == nullptr ) { //Create TCP socket for sending binary data to trainee mpTCPsocket = new QTcpSocket(this); QObject::connect(mpTCPsocket, &QTcpSocket::connected, [this]() { qdbg() << QString("%1 TCP(%2) connected") .arg(QTime::currentTime().toString(Qt::ISODateWithMs)) .arg(mstrHostname); } ); QObject::connect(mpTCPsocket, &QTcpSocket::disconnected, [this]() { if ( mpTCPsocket == nullptr ) { return; } qdbg() << QString("%1 TCP(%2) disconnected") .arg(QTime::currentTime().toString(Qt::ISODateWithMs)) .arg(mstrHostname); mpTCPsocket->close(); mpTCPsocket->deleteLater(); mpTCPsocket = nullptr; } ); } if ( mpTCPsocket->isOpen() != true ) { mpTCPsocket->connectToHost(mstrHostname, SckServer::uint16PortTCP()); } if ( !(mpTCPsocket != nullptr && mpTCPsocket->isOpen() == true) ) { return; } qint64 int64BytesWritten(mpTCPsocket->write(crbaData)); if ( int64BytesWritten > 0 && int64BytesWritten == crbaData.length() ) { mpTCPsocket->waitForBytesWritten(SckServer::mscuint16DataWriteTO); emit binarySent(strHostName(), uint32BlockNo, int64BytesWritten); } }binarySent is a signal connected to onUpdate:
void SendRDF::onUpdate(const QString& crstrTraineeMAC, quint32 uint32BlockNo, quint64 uint64BytesWritten) { quint8 uint8TraineeCount((quint8)mhmpTrainees.size()); tBlocksHash::iterator itTrainee(mhmpTrainees.find(crstrTraineeMAC)); if ( itTrainee == mhmpTrainees.end() ) { //This trainee isn't listed yet, add now! addTrainee(crstrTraineeMAC); } if ( mblnTIP != true || (uint8TraineeCount > 0 && muint8TraineeCount != uint8TraineeCount) ) { emit showDialog(); return; } //Update block count itTrainee.value() = uint32BlockNo; #if defined(TRAINER) //Is block number to trainee so it can update the receiver //dialog QJsonObject objBlockInfo(SckServer::prepareMsg( SckServer::mscszMTdsBlockInfo)); QJsonValue valBlockNo(QString::number(uint32BlockNo)); objBlockInfo.insert(SckServer::mscszBlockNo, valBlockNo); SckServer::queueUDP(objBlockInfo); #endif #if !defined(TRAINER) int intTimeout; if ( uint32BlockNo == 0 ) { intTimeout = SendRDF::mscintRxInitialTimeout; } else { intTimeout = SendRDF::mscintRxInitialTimeout; } //Kick Rx Timeout timer to prevent it from timing out mtmrRxTO.start(intTimeout); #endif QProgressBar* ppgbStatus(qobject_cast<QProgressBar*> (pFindTraineeControl(SendRDF::mscszTraineeProgBarPrefix, crstrTraineeMAC))); if ( ppgbStatus != nullptr ) { const int cintMaximum(ppgbStatus->maximum()); if ( cintMaximum != muint64Filesize ) { //This must be the first time in, set the initial state of the //progress bar ppgbStatus->setMaximum(muint64Filesize); } ppgbStatus->setValue(uint64BytesWritten); //Is transfer complete? if ( uint64BytesWritten >= muint64Filesize ) { #if !defined(TRAINER) //Change text on button to inform user of status mpui->pbtnAbort->setText(SendRDF::scstrTransferComplete()); //Notify application that transfer has completed DatasetTransfer* pTransfer(DatasetTransfer::pCurrentTransfer()); if ( pTransfer != nullptr ) { emit pTransfer->transferComplete(mstrFullPath); } #else //Add entry to audit log QString strAudit(QString("%1 %2 %3") .arg(mpui->plblFilename->text()) .arg(tr("transfer complete, time taken")) .arg(mpui->plblElapsed->text())); QSqlQuery queryAudit; queryAudit.prepare("CALL addAuditEntry(?);"); queryAudit.addBindValue(strAudit); Trainer::queryDB(queryAudit); emit transferComplete(); #endif removeAfterDelay(); } } mpui->plblFilename->setText(mstrFilename); //Update trainee specific data QLabel* plblBlock(qobject_cast<QLabel*> (pFindTraineeControl(SendRDF::mscszTraineeBlockLabelPrefix, itTrainee.key()))); if ( plblBlock != nullptr ) { plblBlock->setText(QString::number(itTrainee.value())); } QLabel* plblName(qobject_cast<QLabel*> (pFindTraineeControl(SendRDF::mscszTraineeNameLabelPrefix, itTrainee.key()))); if ( plblName != nullptr ) { plblName->setText(itTrainee.key()); } }The problem is that the dialog that displays the progress appears, empty, completely blank and does not start to update until the queries have completed and the binary sent. What can I do to improve this and make the update more fluid?
@SPlatten said in Database query, front end update, suggestions?:
The problem is that the dialog that displays the progress appears, empty, completely blank and does not start to update until the queries
Of course! You are blocking the event loop with your loop and QThread::msleep! Why are you doing this? You should know that this is something you should NEVER do in the main thread!
Simply use a QTimer to fire the signal... -
@SPlatten said in Database query, front end update, suggestions?:
The problem is that the dialog that displays the progress appears, empty, completely blank and does not start to update until the queries
Of course! You are blocking the event loop with your loop and QThread::msleep! Why are you doing this? You should know that this is something you should NEVER do in the main thread!
Simply use a QTimer to fire the signal... -
@SPlatten said in Database query, front end update, suggestions?:
The problem is that the dialog that displays the progress appears, empty, completely blank and does not start to update until the queries
Of course! You are blocking the event loop with your loop and QThread::msleep! Why are you doing this? You should know that this is something you should NEVER do in the main thread!
Simply use a QTimer to fire the signal... -
@jsulm , I've removed the QThread::msleep, the dialog still doesn't update until all the calls to sendBinary have completed.
@SPlatten As I already said: you also need to remove the loop! Remove it and use a QTimer instead...
-
@SPlatten As I already said: you also need to remove the loop! Remove it and use a QTimer instead...
-
@jsulm , since the OS is windows, using a timer is going to slow things right down, one of the requirements is for a fast transfer, is there a better / alternative to using a timer that would achieve the same?
@SPlatten said in Database query, front end update, suggestions?:
using a timer is going to slow things right down
Why? The slow part is the query, not timer. And with QThread::msleep() you basically did exactly same (but also blocking the event loop) - why do you think timer will be slot?
If you want to do it better, then wait for currently running query before sending next one. -
@jsulm , since the OS is windows, using a timer is going to slow things right down, one of the requirements is for a fast transfer, is there a better / alternative to using a timer that would achieve the same?
@SPlatten said in Database query, front end update, suggestions?:
since the OS is windows, using a timer is going to slow things right down, one of the requirements is for a fast transfer, is there a better / alternative to using a timer that would achieve the same?
I fail to see the connection to the OS and the timer usage ?
You originally, before you removed it, send your thread to sleep, that would be the same time as a timer would use!?
Be aware of the Qt Code of Conduct, when posting : https://forum.qt.io/topic/113070/qt-code-of-conduct
Q: What's that?
A: It's blue light.
Q: What does it do?
A: It turns blue. -
@SPlatten said in Database query, front end update, suggestions?:
since the OS is windows, using a timer is going to slow things right down, one of the requirements is for a fast transfer, is there a better / alternative to using a timer that would achieve the same?
I fail to see the connection to the OS and the timer usage ?
You originally, before you removed it, send your thread to sleep, that would be the same time as a timer would use!?
@J-Hilk, Unless it's improved with later versions timing has never been one of Windows strong points. Timer accuracy use to be around 50ms, despite being able to set the timers to 1ms, if you hook up a scope you would see its no-where close, other operating systems are capable of not only ms but microsecond timer accuracy.
-
Ok, if I do see this correctly, the slow part is the TCP creation connection and sending.
Why don't you make that socket a class member and keep it alive and connected until everything is done.
Than you can also, easily, listen to the bytesWritten Signal, and use that to proceed in your "loop" instead of timers or infinite loops
Be aware of the Qt Code of Conduct, when posting : https://forum.qt.io/topic/113070/qt-code-of-conduct
Q: What's that?
A: It's blue light.
Q: What does it do?
A: It turns blue. -
Ok, if I do see this correctly, the slow part is the TCP creation connection and sending.
Why don't you make that socket a class member and keep it alive and connected until everything is done.
Than you can also, easily, listen to the bytesWritten Signal, and use that to proceed in your "loop" instead of timers or infinite loops
@J-Hilk , thats exactly what it does already, I don't reconnect for every packet. Its rather complex in that there is a mixture of UDP and TCP, the UDP data is broadcast by the application to multiple clients notifying them that a new data set is available. Each client can issue a request for the dataset to be sent, then this is sent individually to each client using TCP.
Kind Regards,
Sy -
@J-Hilk , thats exactly what it does already, I don't reconnect for every packet. Its rather complex in that there is a mixture of UDP and TCP, the UDP data is broadcast by the application to multiple clients notifying them that a new data set is available. Each client can issue a request for the dataset to be sent, then this is sent individually to each client using TCP.
@SPlatten so everyone on
lngTotalBlocksis a different receiver and requires different TCP connection ? thats unfortunate.Unless it's improved with later versions timing has never been one of Windows strong points. Timer accuracy use to be around 50ms, despite being able to set the timers to 1ms, if you hook up a scope you would see its no-where close, other operating systems are capable of not only ms but microsecond timer accuracy
are you talking about the actual data packet transmitted via your tcp connection? that you observe via the scope?
That of course is rather os depending.
But the accuracy of QTimer and QThread:sleep should be the same as both require the OS to pass the execution back to your program
Be aware of the Qt Code of Conduct, when posting : https://forum.qt.io/topic/113070/qt-code-of-conduct
Q: What's that?
A: It's blue light.
Q: What does it do?
A: It turns blue. -
@SPlatten so everyone on
lngTotalBlocksis a different receiver and requires different TCP connection ? thats unfortunate.Unless it's improved with later versions timing has never been one of Windows strong points. Timer accuracy use to be around 50ms, despite being able to set the timers to 1ms, if you hook up a scope you would see its no-where close, other operating systems are capable of not only ms but microsecond timer accuracy
are you talking about the actual data packet transmitted via your tcp connection? that you observe via the scope?
That of course is rather os depending.
But the accuracy of QTimer and QThread:sleep should be the same as both require the OS to pass the execution back to your program
-
Ok, that was probably my bad, due to formatting issues, I missed the nullptr check before the call to new QTcpSocket.
Makes more sense that way :D and would indicate that everything goes to one receiver.
the problem you now have, my guess at least, is your call to this blocking function
mpTCPsocket->waitForBytesWritten(SckServer::mscuint16DataWriteTO);use the bytesWritten Signal instead.
after that, use your onUpdate function, that is called once all bytes are written, signalled by the bytesWritten signal, to proceed, instead of thefor( long lngBlockNo=0; lngBlockNo<lngTotalBlocks; lngBlockNo++ )loop
Be aware of the Qt Code of Conduct, when posting : https://forum.qt.io/topic/113070/qt-code-of-conduct
Q: What's that?
A: It's blue light.
Q: What does it do?
A: It turns blue. -
Ok, that was probably my bad, due to formatting issues, I missed the nullptr check before the call to new QTcpSocket.
Makes more sense that way :D and would indicate that everything goes to one receiver.
the problem you now have, my guess at least, is your call to this blocking function
mpTCPsocket->waitForBytesWritten(SckServer::mscuint16DataWriteTO);use the bytesWritten Signal instead.
after that, use your onUpdate function, that is called once all bytes are written, signalled by the bytesWritten signal, to proceed, instead of thefor( long lngBlockNo=0; lngBlockNo<lngTotalBlocks; lngBlockNo++ )loop
