send sqlite database file trought tcp

gfxx · 25 Oct 2022, 06:06

I know is difficult but need to copy entire sqlite db trought tcp .... not understand reading online post the best approch ... for sure I understand that using BEGIN IMMEDIATE can pause db connection so can freely copy db on Qfile and send it ..... problem seems packet .... so there are a way to not corrupt sqlite file during tranfer via tcp?

regards

jsulm · G gfxx 25 Oct 2022, 05:56

@gfxx said in send sqlite database file trought tcp:

so there are a way to not corrupt sqlite file during tranfer via tcp?

Close all db connections, read the db file and send its content over TCP/IP connection.
"problem seems packet" - don't know what this means.

gfxx · wrote on 23 Dec 2022, 03:14

In the past Seems my problem was solved .... but now I reuse the same call in other app an the result of qtcp transfer file become the same junk file trasmission ... for better understand it start in these manner:

"\x00\x00\x00\x00\x00\x14\xA0\x04\x00\x14\xA0\x00SQLite format 3\x00\x10\x00\x01\x01\x00@  \x00\x00G[\x00\x00\x01J\x00\x00\x00\x05\x00\x00\x01""2\x00\x00\x02\xD1\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00G[\x00.G\xA8\x05\x00\x00\x00\x01\x0F\xFA\x00\x00\x00\x00""c\x0F\xFA\x0F\xEE\x0F\xEE\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x

and that sequence of \x00 continue but everytime there are some redable string that contain database column description or table description .... plus database file in origin was 1.3Mb but at end of transmission obtain only 50kb (probabily last packet).

In the past I use this code for send:

    Database.removeDatabase("/home/de/.DB/mydb.sqlite3");

    if(tcpSocket->state() == QAbstractSocket::ConnectedState){
        QFile file("/home/de/.DB/mydb.sqlite3");
        if (!file.open(QIODevice::ReadOnly)){
            qDebug() << "file not open .....";
        }
        else{
                if(tcpSocket->state() == QAbstractSocket::ConnectedState){
                QByteArray block;
                while(1){
                    block.clear();
                    block = file.read(32768*8);
                    if(block.size() == 0) break;
                    tcpSocket->write(block);
                    tcpSocket->waitForBytesWritten();
                    block.clear();
                }
                 file.close();
            }

code for receive:

qint64 fileSize = client->bytesAvailable();
else if (fileSize > 200 && !fileBR)
        {
            QFile filew("/home/gr/.DB/receivedb.sqlite3");
            filew.resize(0);
            if(!filew.open(QIODevice::Append))
            {
                qDebug() << "....file not open  ....";
            }
            else{
                QByteArray read;
                //qDebug() << "Read    : " << read;
                while(client->bytesAvailable()){
                    read = client->read(client->bytesAvailable());
                    filew.write(read);
                    qDebug() << "Read this data   : " << read;
                }
             filew.close();

but these time I obtain these junk file instead the original copy ..... i try with buffer with same result .... problem seems to become from sender ....

Christian Ehrlicher · 23 Dec 2022, 18:57

Your tcp socket handling is a mess - esp. on the receiver side since client->bytesAvailable() can be 0 even tough not all bytes were received. Use proper signal/slots, open the file and hold it as member and send the file size in the first 8 bytes so you know when you received all data.

gfxx · 24 Dec 2022, 06:47

@Christian-Ehrlicher said in send sqlite database file trought tcp:

Your tcp socket handling is a mess - esp. on the receiver side since client->bytesAvailable() can be 0 even tough not all bytes were received. Use proper signal/slots, open the file and hold it as member and send the file size in the first 8 bytes so you know when you received all data.

ok .... so means if LAN was really good can work, but if "normal" impossible to obtain good result ... right? ....

any how really thans ...

I'm teaching myself, so that any suggestion is valuable .... and I'm not touchy .... a question why byteavaible can be "0" (as it seems to be in fact) ... ie what mechanism produces this? .... you can also send me links where it is explained .... in my head the sockets work with datagrams of predefined size and with a minimum of control .... but I understand that it's just my fantasy .. ..

Christian Ehrlicher · wrote on 23 Dec 2022, 19:30

A tcp stream is a ... stream.
So you have to define your own protocol to send data over your stream and be able to decrypt it on the receiver side. Or you use e.g. QDataStream which implements a protocol you can use. There are a lot of questions and answers about this in the forum -> forum search

ChrisW67 · 24 Dec 2022, 07:48

@gfxx said in send sqlite database file trought tcp:

a question why byteavaible can be "0" (as it seems to be in fact) ... ie what mechanism produces this? ....

Sender sends 20000 bytes. This will go across the network in a series of packets of up to a network path specific maximum size. Let's say 1460 bytes, a typical maximum segment size in TCP.
Receiver receives some of these bytes.
The bytesAvailable() call returns 1460 and you read them before looping back.
If you are lucky then more of the 20000 bytes has arrived in the short time it took to read and save the first chunk, and you can read, save, and loop again.
If you are not lucky then no more of the 20000 bytes has arrived and your bytesAvailable() call returns zero.

gfxx · wrote on 24 Dec 2022, 07:48

@ChrisW67 said in send sqlite database file trought tcp:

Sender sends 20000 bytes. This will go across the network in a series of packets of up to a network path specific maximum size. Let's say 1460 bytes, a typical maximum segment size in TCP.
Receiver receives some of these bytes.
The bytesAvailable() call returns 1460 and you read them before looping back.
If you are lucky then more of the 20000 bytes has arrived in the short time it took to read and save the first chunk, and you can read, save, and loop again.
If you are not lucky then no more of the 20000 bytes has arrived and your bytesAvailable() call returns zero.

real thanks ... good explain.

gfxx · wrote on 26 Dec 2022, 17:31

@Christian-Ehrlicher

completely missing ....

	if (client->bytesAvailable() <=0)    
		return;

think too tired......