copy entire SQLite DB trought soket

gfxx

I try to copy fileDB.sqlite3 trought QTcpSocket ..... would make a copy from client to server .... so client syde:

void MainWindow::writeFileToServer()
{

    QFile file("/home/mypc/.DBmem/DB_mine.sqlite3");
    if (!file.open(QFile::ReadOnly)){
        qDebug() << "File not open .....";
    }
    QByteArray mydata = file.readAll();
    QDataStream stOut(&mydata, QIODevice::WriteOnly);
    stOut.setVersion(QDataStream::Qt_5_15);
    stOut.device()->seek(0);
    stOut << (quint64) mydata.size();
    stOut << mydata;
    if(tcpSocket->waitForConnected()){
        tcpSocket->write(mydata);
        qDebug() << "...try to send entyre DB ....";
    }
}

and server syde i write a test code inside a QThread ...

void MyThread::run()
{
    QTcpSocket tcpSocket;

    if (!tcpSocket.setSocketDescriptor(socketDescriptor)) {
        emit error(tcpSocket.error());
        return;
    }
 
    if(tcpSocket.waitForReadyRead(5000)){
        if(tcpSocket.bytesAvailable() < 150){
            arrData = tcpSocket.readAll();
            qDebug() << "...receive less than 150 byte ....";
            readClient();
        }
        else if(tcpSocket.bytesAvailable() > 160){
            blockw.clear(); 
            qDebug() << "...we receive more than 160 byte ....";
            QFile filew("/home/serverPc/.DBcopy/DB_beckup.sqlite3");
            if(!(filew.open(QIODevice::Append)))
            {
                qDebug("....file not open ....");
            }
            else{
                blockw = tcpSocket.readAll();
                filew.write(blockw);
                filew.close();
                qDebug("...seems we write a file ....");
            }

        }

    }

    QByteArray block;   /* these is only for write little string as "custom protocol" like ....*/
    QDataStream out(&block, QIODevice::ReadWrite);
    out.setVersion(QDataStream::Qt_5_15);
    out << text;
    tcpSocket.write(block);
    tcpSocket.disconnectFromHost();
    tcpSocket.waitForDisconnected();
    arrData.clear();


}

It works .... but copy of database could not open on SQLite Browser 3.31.1 .... it return messages like: not possible open file it si not sqlite file .... or similar ...by the way it seems an empty database like 7,7Kb instead the original one of 670Kb .... so obviuslly somethings works in wrong way .... someone can suggest my error?

thanks in advances
bkt

SGaist

Hi,

One thing that looks off is how you manage the receiving end.

You should rather use the asynchronous nature of Qt rather than use the blocking API.

As for the transfert itself, you should use QDataStream's transaction system.

Christian Ehrlicher

And don't use a thread on the receiver side - it's not needed. Why do all people think threads are needed for such kind of things?

gfxx

@Christian-Ehrlicher because from 4.0 time all peaple suggest to use thread instead for gui app? not all people can have time to upgrade knowelge about .... so some people can use old system to do the thing .... with erroneus culture ... im sorry for that ....

I think a better help to improve people culture is explain why is not needed thread ... when note that thise is. I think you can understand what I means .... real thanks

bkt

Christian Ehrlicher

@gfxx said in copy entire SQLite DB trought soket:

why is not needed thread

The same reason which is stated in every topic here when someone without basic knowledge about threads is using threads in Qt with e.g. QProcess, Q(Tcp/Ud/whatever)Socket, Qfoo- Qt system is asynchronous signals and slots is the way to go to not mess around with threading issues for no reasons. There is no QThread usage in QProcess/QTcpSocket/... documentation so why use threads? Because it sounds fancy I would guess...

gfxx

@Christian-Ehrlicher said in copy entire SQLite DB trought soket:

The same reason which is stated in every topic here when someone without basic knowledge about threads is using threads in Qt with e.g. QProcess, Q(Tcp/Ud/whatever)Socket, Qfoo- Qt system is asynchronous signals and slots is the way to go to not mess around with threading issues for no reasons. There is no QThread usage in QProcess/QTcpSocket/... documentation so why use threads? Because it sounds fancy I would guess...

Not for sure ..... normally I use qthread for OpenCv calculation only or for modbus/canbus operation, so in my non-comproved ideas when use

waitForReadyRead(5000)

I immagine is better use qthread instead mainwindows Gui .... but I think you learn me that these waiting is not blocking on mainwindows GUI so I can perform it on mainwindows .... Is right these?

thanks for these advice.
bkt

JonB

@gfxx
The whole point is not to use the waitFor...() methods, use only the signals & slots for sockets. Then your UI does not block and freeze, and you do not need a separate thread, At least on the client side.

gfxx

@JonB thanks .... but on CLIENT side there are not Qthread .... I use signal&slot only on main tharead aka mainwindows ..... on SERVER side (other GUI app with mainwindows too) I use Qthread because in my original ideas we connect more than one client, and every 50-60 min need to copy for every client a little DB from client TO server ... so sever can have an identical copy of single DB every hour .... I means 10 client max .... server app must manage the data trought QChart .... and data need to be processed .... similar data for 10 client print on same QChartView .... i have many QChartView .... so think is better to have local data instead remote query ....
So if in your experience and knowelge, is preferred to use SIGNAL&SLOT at SERVER side mainwindows GUI .... I work on it for sure .... because less code for make same things .... less is better .... appreciate you opinion on these sketch of project .... : is better manage signal & slot from mainwindows in these situation too? No freeze gui. Perfect I love it. But is possible? Sorry but never work with soket so really not have experience on these .... and I'm intrested on your opinion. If possible i change immediately strategy and abandone Qthread.

regards
bkt

JonB

@gfxx
I mentioned about for server because you can look at examples Fortune Server Example versus Threaded Fortune Server Example.

Your question is way too complicated/vague for people to answer specifically. But if @SGaist & @Christian-Ehrlicher are saying they do not think you need threading on the server side that is good for me. One thing you should not do is use threads and change to using blocking calls in them, that is just a waste, when Qt's signal/slots are non-blocking and do not need separate threads.

gfxx

@JonB said in copy entire SQLite DB trought soket:

Your question is way too complicated/vague for people to answer specifically. But if @SGaist & @Christian-Ehrlicher are saying they do not think you need threading on the server side that is good for me. One thing you should not do is use threads and change to using blocking calls in them, that is just a waste, when Qt's signal/slots are non-blocking and do not need separate threads.

thanks ..... understand the point .... for certain I have a one more doubt: can connect multiple client on one QTcpserver & socket at same time? or need to start more than one server (1 server to one client? or can be more client to one server + socket?) .... these because I read a lot of post these week and some they confused me a bit.....

Christian Ehrlicher

@gfxx said in copy entire SQLite DB trought soket:

can connect multiple client on one QTcpserver & socket at same time?

Yes, that's the task for a server.
QTcpServer::newConnection()

SGaist

Out of curiosity, why do you want to copy database files from multiple clients to a server ?

It sounds like you are trying reinvent networked database system like PostgreSQL.

gfxx

@SGaist said in copy entire SQLite DB trought soket:

Out of curiosity, why do you want to copy database files from multiple clients to a server ?
It sounds like you are trying reinvent networked database system like PostgreSQL.

Because on machinery there are installed SQLite db ..... and it is not networked database .... actually I evaluate if is more convenient change totally database system on machinery or try to copy database on machinery (as try in these moment) ..... plus server is not everytime power ON so indipendent database on single machine prevent loss of data and can work in disconnected mode without issue.

JonB

@gfxx said in copy entire SQLite DB trought soket:

and every 50-60 min need to copy for every client a little DB from client TO server ... so sever can have an identical copy of single DB every hour

Assuming users are connected to their SQLite database files and you copy these files while they may be updated or in partial state I'm not sure you will get "reliable" backups?

SGaist

Then you should rather implement a proper backup strategy. Having a bunch of machines with each their own SQLite file that you copy at some random time is not going to provide the resiliency you seem to be after.

You do not need to have one central server for all your machines to connect to if it's not possible but you really need to study the architecture of your setup.

gfxx

@JonB @SGaist ..... because the real control is on client .... the simply base ideas was client that decide when send copy of DB file .... example when no one read or write on client sqlite DB. These is quite easy .... because soft plc is connected on qt app that runs on realtime system. Plc write on DB, so plc can decide when send DB copy without c++ strategy for copy ... plc decide if qt app can write or read on DB .... and can block writing and readig action ..... more safe more easy and vice versa ;) .....

gfxx

Any how now my Qtcpserver work on mainwindos .... now I have some issue on managin send and receive messages (a sort off proto-protocol for exchange data) ..... based on other post I subclass QTcpSocket for have trace of every client that have connection... is the right strategy or better is to use "nextpendingconnection" macro only? what you can suggest based on your knowelge?

SGaist

It's not a macro it's a function. Anyway, if you have a protocol that requires multiple communication between your client and server you will have to manage a list of connections.

You should also properly model the communication and maybe use a state machine to handle the various steps required from start to finish of your database copy.

gfxx

@SGaist said in copy entire SQLite DB trought soket:

It's not a macro it's a function. Anyway, if you have a protocol that requires multiple communication between your client and server you will have to manage a list of connections.

Ok noted ... thanks

You should also properly model the communication and maybe use a state machine to handle the various steps required from start to finish of your database copy.

Not really understand .... never use state machines but know it .... but why need to use state machine? Why need various step? Explain me better please .... in my ideas, one time client choose that is ok to copy DBfile, send a messages to server for advice about data transfer .... server send "ok go" messages and client send.

So QByteArray sender = file.readAll() client side ..... and server side ...

    while (client->bytesAvailable()){
        QByteArray buffer;
        buffer.resize( client->bytesAvailable() );
        client->read( buffer.data(), buffer.size() );
        if(buffer.size() < 150){
            qDebug() << "Server is receiving ...."<<buffer;
        }
        else{
            QFile filew("/home/mypc/.DBmem/DB_mine.sqlite3");
            if(!(filew.open(QIODevice::Append)))
            {
                qDebug("....file not open....");
            }
            else{
                filew.write(buffer);
                //filew.
                filew.close();
                qDebug("... write the file ....");
            }

        }
    }

perhaps .... now work ok and file is received in right dimension (667,7Kb like client side) ..... but sqlite browser send these messages: FILE COULD NOT BE OPEN: FILE IS NOT A DATABASE ..... but not know why .... strange because is exactly same dimension than client side ....

client side file is write on socket in these way:

QFile file("/home/mypc/.DBmem/DB_mine.sqlite3");
if (!file.open(QFile::ReadOnly)){
        qDebug() << "File not open .....";
    }
 else
}
    QByteArray mydata = file.readAll();
    QByteArray block;
    QDataStream stOut(&block, QIODevice::WriteOnly);
    stOut.setVersion(QDataStream::Qt_5_15);
    stOut << (quint32)0;
    stOut << mydata;
    stOut.device()->seek(0);
    stOut << quint32((block.size() -sizeof(quint32)));;
    tcpSocket->write(block);
}
file.close();

file client side is 667,7kb ... and server side the same ... but server side could not open ..... I'm in error about file reading client side? ... need to use QtextStream?

Appreciate any help.

regards
bkt

SGaist

You are not handling the block size you are sending server side.

As for the state machine, it allows you to automate the protocole handling. It's not mandatory though.

That said, you don't seem to do any error handling.