copy entire SQLite DB trought soket

[JonB]

@gfxx
I mentioned about for server because you can look at examples Fortune Server Example versus Threaded Fortune Server Example.

Your question is way too complicated/vague for people to answer specifically. But if @SGaist & @Christian-Ehrlicher are saying they do not think you need threading on the server side that is good for me. One thing you should not do is use threads and change to using blocking calls in them, that is just a waste, when Qt's signal/slots are non-blocking and do not need separate threads.

[gfxx]

@JonB said in copy entire SQLite DB trought soket:

Your question is way too complicated/vague for people to answer specifically. But if @SGaist & @Christian-Ehrlicher are saying they do not think you need threading on the server side that is good for me. One thing you should not do is use threads and change to using blocking calls in them, that is just a waste, when Qt's signal/slots are non-blocking and do not need separate threads.

thanks ..... understand the point .... for certain I have a one more doubt: can connect multiple client on one QTcpserver & socket at same time? or need to start more than one server (1 server to one client? or can be more client to one server + socket?) .... these because I read a lot of post these week and some they confused me a bit.....

[Christian Ehrlicher]

@gfxx said in copy entire SQLite DB trought soket:

can connect multiple client on one QTcpserver & socket at same time?

Yes, that's the task for a server.
QTcpServer::newConnection()

[SGaist]

Out of curiosity, why do you want to copy database files from multiple clients to a server ?

It sounds like you are trying reinvent networked database system like PostgreSQL.

[gfxx]

@SGaist said in copy entire SQLite DB trought soket:

Out of curiosity, why do you want to copy database files from multiple clients to a server ?
It sounds like you are trying reinvent networked database system like PostgreSQL.

Because on machinery there are installed SQLite db ..... and it is not networked database .... actually I evaluate if is more convenient change totally database system on machinery or try to copy database on machinery (as try in these moment) ..... plus server is not everytime power ON so indipendent database on single machine prevent loss of data and can work in disconnected mode without issue.

[JonB]

@gfxx said in copy entire SQLite DB trought soket:

and every 50-60 min need to copy for every client a little DB from client TO server ... so sever can have an identical copy of single DB every hour

Assuming users are connected to their SQLite database files and you copy these files while they may be updated or in partial state I'm not sure you will get "reliable" backups?

[SGaist]

Then you should rather implement a proper backup strategy. Having a bunch of machines with each their own SQLite file that you copy at some random time is not going to provide the resiliency you seem to be after.

You do not need to have one central server for all your machines to connect to if it's not possible but you really need to study the architecture of your setup.

[gfxx]

@JonB @SGaist ..... because the real control is on client .... the simply base ideas was client that decide when send copy of DB file .... example when no one read or write on client sqlite DB. These is quite easy .... because soft plc is connected on qt app that runs on realtime system. Plc write on DB, so plc can decide when send DB copy without c++ strategy for copy ... plc decide if qt app can write or read on DB .... and can block writing and readig action ..... more safe more easy and vice versa ;) .....

[gfxx]

Any how now my Qtcpserver work on mainwindos .... now I have some issue on managin send and receive messages (a sort off proto-protocol for exchange data) ..... based on other post I subclass QTcpSocket for have trace of every client that have connection... is the right strategy or better is to use "nextpendingconnection" macro only? what you can suggest based on your knowelge?

[SGaist]

It's not a macro it's a function. Anyway, if you have a protocol that requires multiple communication between your client and server you will have to manage a list of connections.

You should also properly model the communication and maybe use a state machine to handle the various steps required from start to finish of your database copy.

[gfxx]

@SGaist said in copy entire SQLite DB trought soket:

It's not a macro it's a function. Anyway, if you have a protocol that requires multiple communication between your client and server you will have to manage a list of connections.

Ok noted ... thanks

You should also properly model the communication and maybe use a state machine to handle the various steps required from start to finish of your database copy.

Not really understand .... never use state machines but know it .... but why need to use state machine? Why need various step? Explain me better please .... in my ideas, one time client choose that is ok to copy DBfile, send a messages to server for advice about data transfer .... server send "ok go" messages and client send.

So QByteArray sender = file.readAll() client side ..... and server side ...

    while (client->bytesAvailable()){
        QByteArray buffer;
        buffer.resize( client->bytesAvailable() );
        client->read( buffer.data(), buffer.size() );
        if(buffer.size() < 150){
            qDebug() << "Server is receiving ...."<<buffer;
        }
        else{
            QFile filew("/home/mypc/.DBmem/DB_mine.sqlite3");
            if(!(filew.open(QIODevice::Append)))
            {
                qDebug("....file not open....");
            }
            else{
                filew.write(buffer);
                //filew.
                filew.close();
                qDebug("... write the file ....");
            }

        }
    }

perhaps .... now work ok and file is received in right dimension (667,7Kb like client side) ..... but sqlite browser send these messages: FILE COULD NOT BE OPEN: FILE IS NOT A DATABASE ..... but not know why .... strange because is exactly same dimension than client side ....

client side file is write on socket in these way:

QFile file("/home/mypc/.DBmem/DB_mine.sqlite3");
if (!file.open(QFile::ReadOnly)){
        qDebug() << "File not open .....";
    }
 else
}
    QByteArray mydata = file.readAll();
    QByteArray block;
    QDataStream stOut(&block, QIODevice::WriteOnly);
    stOut.setVersion(QDataStream::Qt_5_15);
    stOut << (quint32)0;
    stOut << mydata;
    stOut.device()->seek(0);
    stOut << quint32((block.size() -sizeof(quint32)));;
    tcpSocket->write(block);
}
file.close();

file client side is 667,7kb ... and server side the same ... but server side could not open ..... I'm in error about file reading client side? ... need to use QtextStream?

Appreciate any help.

regards
bkt

[SGaist]

You are not handling the block size you are sending server side.

As for the state machine, it allows you to automate the protocole handling. It's not mandatory though.

That said, you don't seem to do any error handling.

[gfxx]

@SGaist said in copy entire SQLite DB trought soket:

ou are not handling the block size you are sending server side.

Ok thanks ... you means in client code? or on server code?... because client send and server receive .....and my english is not good enought :))

As for the state machine, it allows you to automate the protocole handling. It's not mandatory though.

oh server side ... like plc do client side .... but can start a soft plc server side too for that ... i Think is the same result .... you means that right?

That said, you don't seem to do any error handling.

You are in right ... because it is only self learning code .... but think is better add that because maybe i loose something .....

[SGaist]

Server side. But as I already suggested: use a transaction to transfert your data.

[gfxx]

@SGaist you means somethings like the pieces of code?

void Reader::slotOnReadyRead()
{
    mStream.startTransaction();

    quint64 availableBytesForReading = mStream.device()->bytesAvailable();
    QByteArray binaryDataBlock;
    char *tmp = new char[availableBytesForReading];
    mStream.readRawData(tmp, availableBytesForReading);
    binaryDataBlock.append(tmp, availableBytesForReading);
    delete[] tmp;
    tmp = nullptr;

    if (mStream.commitTransaction())
    {
        emit signalNewData(binaryDataBlock);
    }
}

not my code only an example and not try it.

[SGaist]

The fortune client example implements that in a simpler manner.

[gfxx]

@SGaist ...... seems not so symple ....

//####################  MAINWINDOWS SERVER START ########################################################

    server = new QTcpServer(this);
    client = new QTcpSocket();

    if (!server->listen(QHostAddress("192.111.1.111"), 50000)) {
        QString ipi = "Server error";
        ui->textEdit_Server->setText(ipi);
    }
    else{
        QString  ipAddress = QHostAddress("192.111.1.111").toString();
        QString ipa = "Il server start on IP: " + ipAddress + "  port:  " + QString::number(server->serverPort());
        ui->textEdit_Server->setText(ipa);
        connect(server, &QTcpServer::newConnection, this, [=](){client = server->nextPendingConnection(); new_con();});
        connect(ui->pbtn_STOP, SIGNAL(released()), this, SLOT(shotValue()));
    }


//####################  MAINWINDOWS SERVER START END SECTION SERVER MANAGEMENT #########################


void MainWindow::new_con()
{

    connect(client, &QTcpSocket::readyRead, this, &MainWindow::readSocket);
    connect(client, &QTcpSocket::disconnected,this, &MainWindow::discardSocket);
    connect(client, &QTcpSocket::errorOccurred, this, &MainWindow::sokError);


    if(shotVal != initTX1){
        QByteArray block;
        QDataStream out(&block, QIODevice::WriteOnly);
        out.setDevice(client);
        out.setVersion(QDataStream::Qt_5_15);

        out << shotVal;

        client->write(block);
        client->waitForDisconnected();
        client->disconnectFromHost();
        client->close();

        initTX1 = shotVal;
        qDebug() << "... send these string ...." << initTX1;
    }
    else{
        qDebug() << "... no news to send  ....";
    }
}

void MainWindow::readSocket()
{

        QByteArray buffer;
        qint64 fileSize = client->bytesAvailable();
        QDataStream stream(client);
        stream.setVersion(QDataStream::Qt_5_15);
        if(fileSize < 150){
            qDebug() << "Server is receiving ...." << buffer;
        }
        else{
            QFile filew("/home/mypc/.DBc/testFile.sqlite3");
            if(!filew.open(QIODevice::WriteOnly))
            {
                qDebug() << ".... file not exist ....";
            }
            else{
                stream.startTransaction();
                if(!stream.commitTransaction()){
                    qDebug() << ".... file can not be tranfered or corruption ....";
                }

                stream >> buffer;
                filew.write(buffer);

                qDebug() << "... file is written ...." << buffer;

                filew.close();
            }

        }

}

void MainWindow::discardSocket()
{
    client->deleteLater();
}



void MainWindow::sokError(QAbstractSocket::SocketError sErr){
    qDebug() << ".... errore socket ...." << sErr;
}

so now receive error .... can see that : server try to receive more than one time the file send by client ..... these is my received error from qtcreator reading my debug point ....

.... errore socket .... QAbstractSocket::RemoteHostClosedError
... send these string .... "££stop££"
... no news to send  ....
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
...file is written .... ""
.... errore socket .... QAbstractSocket::RemoteHostClosedError

so buffer seems empty now ..... plus remote host (my client) seems close at start of transaction and at commit too ... try to read more on datastream and socket .... but appreciate that someone can show me my MACRo error ... actually not see it ...

regards

[gfxx]

@SGaist plus you have suggested to me sometings like these (state machine):

#include "testclient.h"
#include <QState>
#include <QStateMachine>
#include <QTcpSocket>
#include <QThread>      // Sleep
#include <QTimer>

//-----------------------------------------------------------------------------
// PUBLIC METHODS
//-----------------------------------------------------------------------------
TestClient::TestClient(QObject *parent) : QObject(parent)
{
    m_socket = new QTcpSocket(this);
    m_timer = new QTimer(this);
    m_timer->setInterval(100);
    connect(m_timer, &QTimer::timeout, this, &TestClient::tryConnect);
    connect(m_socket, &QAbstractSocket::connected,this, &TestClient::onSocketConnected);
    connect(m_socket, &QAbstractSocket::disconnected,this, &TestClient::onSocketDisconnected);
    connect(m_socket, &QIODevice::bytesWritten,this, &TestClient::onSocketBytesWritten);
    connect(m_socket, QOverload<QAbstractSocket::SocketError>::of(&QAbstractSocket::error), this, &TestClient::onSocketError);
}
void TestClient::start()
{
    createClientFsm();
    m_clientFsm->start();
}
//-----------------------------------------------------------------------------
// TCP CONNECTION MANAGEMENT SLOTS
//-----------------------------------------------------------------------------
void TestClient::onSocketConnected()
{
    m_timer->stop();
    qDebug() << "connected...";
    emit fsmEvtConnected();
}
void TestClient::onSocketDisconnected()
{
    qDebug() << "disconnected...";
    emit fsmEvtError();
}
void TestClient::onSocketBytesWritten(qint64 bytes)
{
    qDebug() << bytes << " bytes written...";
}
void TestClient::onSocketError(QAbstractSocket::SocketError err)
{
    qDebug() << "socket error " << err;
}
//-----------------------------------------------------------------------------
// FSM MANAGEMENT
//-----------------------------------------------------------------------------
void TestClient::createClientFsm()
{
    m_clientFsm = new QStateMachine(this);
    // Create states
    QState* sConnect = new QState();
    QState* sTransmit = new QState();
    // Add transitions between states
    sConnect->addTransition(this, SIGNAL(fsmEvtConnected()), sTransmit);
    sTransmit->addTransition(this, SIGNAL(fsmEvtError()), sConnect);
    // Add entry actions to states
    connect(sConnect, &QAbstractState::entered, this, &TestClient::onfsmConnectEntered);
    connect(sTransmit, &QAbstractState::entered, this, &TestClient::onfsmTransmitEntered);
    // Create state machine
    m_clientFsm->addState(sConnect);
    m_clientFsm->addState(sTransmit);
    m_clientFsm->setInitialState(sConnect);
}
void TestClient::tryConnect(){
    m_socket->connectToHost("localhost", 11000);
}
void TestClient::onfsmConnectEntered()
{
    m_timer->start();
}
void TestClient::onfsmTransmitEntered()
{
    qDebug() << "sending data...";
    m_socket->write("TEST MESSAGE");
}

(for my memory only add .h too ... code is from stackoverflow eyllanesc user as sample for other user)

#ifndef TESTCLIENT_H
#define TESTCLIENT_H

#include <QObject>

class QTcpSocket;
class QStateMachine;
class QTimer;

#include <QAbstractSocket>

class TestClient : public QObject
{
    Q_OBJECT
public:
    explicit TestClient(QObject *parent = nullptr);
public slots:
    void start();
signals:
    // FSM events
    void fsmEvtConnected();
    void fsmEvtError();
private slots:
    void onSocketConnected();                       // Notify connection to TCP server
    void onSocketDisconnected();                    // Notify disconnection from TCP server
    void onSocketBytesWritten(qint64 bytes);        // Notify number of bytes written to TCP server
    void onSocketError(QAbstractSocket::SocketError err);
    // FSM state enter/exit actions
    void onfsmConnectEntered();
    void onfsmTransmitEntered();
private:
    // Member variables
    QTcpSocket*         m_socket;       // TCP socket used for communications to server
    QStateMachine*      m_clientFsm;      // FSM defining general client behaviour
    QTimer*             m_timer;
private:
    void createClientFsm();             // Create client FSM
    void tryConnect();
};

#endif // TESTCLIENT_H

[SGaist]

Before adding any thread and state machine, understand that with regards to the fortune client and server examples, the roles are reversed here.

In the example, the client receives the fortune from the server. In your case, the server will receive the data from the client. It's in that direction that you have to implement your logic.

[Kent-Dorfman]

scp, ftp, cfdp, rsync...the list goes on.