Database query, front end update, suggestions?

[SPlatten]

I have a lot of data stored in a database (MariaDB). I query the data by emitting a signal queryBlock:

for( long lngBlockNo=0; lngBlockNo<lngTotalBlocks; lngBlockNo++ )
    {
        if ( blnOnline() != true )
        {
    //Trainee has gone offline, stop transfer
            break;
        }
    //Push each query to a stack then pop each query from the stack
        QThread::msleep(TraineeMonitor::mscuintDelayBetweenQueries);
        emit queryBlock(mlngDatasetID, lngBlockNo);
    }

queryBlock is connected to the slot onQueryBlock:

void TraineeMonitor::onQueryBlock(long lngDatasetID, quint32 uint32BlockNo)
{
    QSqlQuery query;
    query.prepare("SELECT"
                  " `binChunk`"
                  " FROM"
                  " `rdf`"
                  " WHERE"
                  " `biDataset`=?"
                  " AND"
                  " `intBlockNo`=?");
    query.addBindValue(lngDatasetID);
    query.addBindValue(uint32BlockNo);
    if ( Trainer::queryDB(query) != true )
    {
        return;
    }
//Whilst there is a block to send and the Trainee is online
    if ( query.next() )
    {
        QByteArray baData(query.value(0).toByteArray());
        if ( baData.isEmpty() == true )
        {
            return;
        }
//Send binary data
        sendBinary(uint32BlockNo, baData);
    }
}

Here is the sendBinary function:

void TraineeMonitor::sendBinary(quint32 uint32BlockNo,
                                const QByteArray& crbaData)
{
    if ( crbaData.size() == 0 )
    {
    //Nothing to send!
        return;
    }
    if ( mpTCPsocket == nullptr )
    {
//Create TCP socket for sending binary data to trainee
        mpTCPsocket = new QTcpSocket(this);
        QObject::connect(mpTCPsocket, &QTcpSocket::connected,
            [this]()
            {
                qdbg() << QString("%1 TCP(%2) connected")
            .arg(QTime::currentTime().toString(Qt::ISODateWithMs))
            .arg(mstrHostname);
            }
        );
        QObject::connect(mpTCPsocket, &QTcpSocket::disconnected,
            [this]()
            {
                if ( mpTCPsocket == nullptr )
                {
                    return;
                }
                qdbg() << QString("%1 TCP(%2) disconnected")
            .arg(QTime::currentTime().toString(Qt::ISODateWithMs))
            .arg(mstrHostname);
                mpTCPsocket->close();
                mpTCPsocket->deleteLater();
                mpTCPsocket = nullptr;
            }
        );
    }
    if ( mpTCPsocket->isOpen() != true )
    {
        mpTCPsocket->connectToHost(mstrHostname, SckServer::uint16PortTCP());
    }
    if ( !(mpTCPsocket != nullptr && mpTCPsocket->isOpen() == true) )
    {
        return;
    }
    qint64 int64BytesWritten(mpTCPsocket->write(crbaData));
    if ( int64BytesWritten > 0 && int64BytesWritten == crbaData.length() )
    {
        mpTCPsocket->waitForBytesWritten(SckServer::mscuint16DataWriteTO);
        emit binarySent(strHostName(), uint32BlockNo, int64BytesWritten);
    }
}

binarySent is a signal connected to onUpdate:

void SendRDF::onUpdate(const QString& crstrTraineeMAC,
                       quint32 uint32BlockNo, quint64 uint64BytesWritten)
{
    quint8 uint8TraineeCount((quint8)mhmpTrainees.size());
    tBlocksHash::iterator itTrainee(mhmpTrainees.find(crstrTraineeMAC));
    if ( itTrainee == mhmpTrainees.end() ) {
    //This trainee isn't listed yet, add now!
        addTrainee(crstrTraineeMAC);
    }
    if ( mblnTIP != true || (uint8TraineeCount > 0 && muint8TraineeCount != uint8TraineeCount) )
    {
        emit showDialog();
        return;
    }
    //Update block count
    itTrainee.value() = uint32BlockNo;
#if defined(TRAINER)
    //Is block number to trainee so it can update the receiver
    //dialog
    QJsonObject objBlockInfo(SckServer::prepareMsg(
                                 SckServer::mscszMTdsBlockInfo));
    QJsonValue valBlockNo(QString::number(uint32BlockNo));
    objBlockInfo.insert(SckServer::mscszBlockNo, valBlockNo);
    SckServer::queueUDP(objBlockInfo);
#endif
#if !defined(TRAINER)
    int intTimeout;
    if ( uint32BlockNo == 0 )
    {
        intTimeout = SendRDF::mscintRxInitialTimeout;
    }
    else
    {
        intTimeout = SendRDF::mscintRxInitialTimeout;
    }
    //Kick Rx Timeout timer to prevent it from timing out
    mtmrRxTO.start(intTimeout);
#endif
    QProgressBar* ppgbStatus(qobject_cast<QProgressBar*>
            (pFindTraineeControl(SendRDF::mscszTraineeProgBarPrefix,
                                 crstrTraineeMAC)));
    if ( ppgbStatus != nullptr )
    {
        const int cintMaximum(ppgbStatus->maximum());
        if ( cintMaximum != muint64Filesize )
        {
    //This must be the first time in, set the initial state of the
    //progress bar
            ppgbStatus->setMaximum(muint64Filesize);
        }
        ppgbStatus->setValue(uint64BytesWritten);
    //Is transfer complete?
        if ( uint64BytesWritten >= muint64Filesize )
        {
#if !defined(TRAINER)
    //Change text on button to inform user of status
            mpui->pbtnAbort->setText(SendRDF::scstrTransferComplete());
    //Notify application that transfer has completed
            DatasetTransfer* pTransfer(DatasetTransfer::pCurrentTransfer());
            if ( pTransfer != nullptr ) {
                emit pTransfer->transferComplete(mstrFullPath);
            }
#else
    //Add entry to audit log
            QString strAudit(QString("%1 %2 %3")
                                .arg(mpui->plblFilename->text())
                                .arg(tr("transfer complete, time taken"))
                                .arg(mpui->plblElapsed->text()));
            QSqlQuery queryAudit;
            queryAudit.prepare("CALL addAuditEntry(?);");
            queryAudit.addBindValue(strAudit);
            Trainer::queryDB(queryAudit);
            emit transferComplete();
#endif
            removeAfterDelay();
        }
    }
    mpui->plblFilename->setText(mstrFilename);
    //Update trainee specific data
    QLabel* plblBlock(qobject_cast<QLabel*>
            (pFindTraineeControl(SendRDF::mscszTraineeBlockLabelPrefix,
                                 itTrainee.key())));
    if ( plblBlock != nullptr )
    {
        plblBlock->setText(QString::number(itTrainee.value()));
    }
    QLabel* plblName(qobject_cast<QLabel*>
            (pFindTraineeControl(SendRDF::mscszTraineeNameLabelPrefix,
                                 itTrainee.key())));
    if ( plblName != nullptr )
    {
        plblName->setText(itTrainee.key());
    }
}

The problem is that the dialog that displays the progress appears, empty, completely blank and does not start to update until the queries have completed and the binary sent. What can I do to improve this and make the update more fluid?

[jsulm]

@SPlatten said in Database query, front end update, suggestions?:

The problem is that the dialog that displays the progress appears, empty, completely blank and does not start to update until the queries

Of course! You are blocking the event loop with your loop and QThread::msleep! Why are you doing this? You should know that this is something you should NEVER do in the main thread!
Simply use a QTimer to fire the signal...

[SPlatten]

@jsulm, not sure why I added that, will remove it and try.

[SPlatten]

@jsulm , I've removed the QThread::msleep, the dialog still doesn't update until all the calls to sendBinary have completed.

[jsulm]

@SPlatten As I already said: you also need to remove the loop! Remove it and use a QTimer instead...

[SPlatten]

@jsulm , since the OS is windows, using a timer is going to slow things right down, one of the requirements is for a fast transfer, is there a better / alternative to using a timer that would achieve the same?

[jsulm]

@SPlatten said in Database query, front end update, suggestions?:

using a timer is going to slow things right down

Why? The slow part is the query, not timer. And with QThread::msleep() you basically did exactly same (but also blocking the event loop) - why do you think timer will be slot?
If you want to do it better, then wait for currently running query before sending next one.

[J.Hilk]

@SPlatten said in Database query, front end update, suggestions?:

since the OS is windows, using a timer is going to slow things right down, one of the requirements is for a fast transfer, is there a better / alternative to using a timer that would achieve the same?

I fail to see the connection to the OS and the timer usage ?

You originally, before you removed it, send your thread to sleep, that would be the same time as a timer would use!?

[SPlatten]

@J-Hilk, Unless it's improved with later versions timing has never been one of Windows strong points. Timer accuracy use to be around 50ms, despite being able to set the timers to 1ms, if you hook up a scope you would see its no-where close, other operating systems are capable of not only ms but microsecond timer accuracy.

[J.Hilk]

Ok, if I do see this correctly, the slow part is the TCP creation connection and sending.

Why don't you make that socket a class member and keep it alive and connected until everything is done.

Than you can also, easily, listen to the bytesWritten Signal, and use that to proceed in your "loop" instead of timers or infinite loops

[SPlatten]

@J-Hilk , thats exactly what it does already, I don't reconnect for every packet. Its rather complex in that there is a mixture of UDP and TCP, the UDP data is broadcast by the application to multiple clients notifying them that a new data set is available. Each client can issue a request for the dataset to be sent, then this is sent individually to each client using TCP.

[J.Hilk]

@SPlatten so everyone on lngTotalBlocks is a different receiver and requires different TCP connection ? thats unfortunate.

Unless it's improved with later versions timing has never been one of Windows strong points. Timer accuracy use to be around 50ms, despite being able to set the timers to 1ms, if you hook up a scope you would see its no-where close, other operating systems are capable of not only ms but microsecond timer accuracy

are you talking about the actual data packet transmitted via your tcp connection? that you observe via the scope?

That of course is rather os depending.

But the accuracy of QTimer and QThread:sleep should be the same as both require the OS to pass the execution back to your program

[SPlatten]

@J-Hilk , why do you say a different receiver what makes you say that?

[J.Hilk]

@SPlatten

Ok, that was probably my bad, due to formatting issues, I missed the nullptr check before the call to new QTcpSocket.

Makes more sense that way :D and would indicate that everything goes to one receiver.

the problem you now have, my guess at least, is your call to this blocking function
mpTCPsocket->waitForBytesWritten(SckServer::mscuint16DataWriteTO);

use the bytesWritten Signal instead.
after that, use your onUpdate function, that is called once all bytes are written, signalled by the bytesWritten signal, to proceed, instead of the for( long lngBlockNo=0; lngBlockNo<lngTotalBlocks; lngBlockNo++ ) loop

[SPlatten]

@J-Hilk , thank you, I will look into it.