soket datastream empty file problem

gfxx

I try to send from client a file to serverr using tcp and socket + datastream .... in my case client send string OR file and server must store it on Qstring OR on existing QFile .... depends on fileSize .... my readyread code server side ......

void MainWindow::readSocket()
{

        QByteArray buffer;
        qint64 fileSize = client->bytesAvailable();
        sqlFile.setDevice(client);
        sqlFile.setVersion(QDataStream::Qt_5_15);
        sqlFile.startTransaction();

        if(fileSize < 40 && fileSize > 0){
            if(!sqlFile.commitTransaction()){
                qDebug() << "....String is not tranfered ....";
            }

            sqlFile >> buffer;
            client->waitForBytesWritten(-1);
            qDebug() << " ... Server is receiving ...." << buffer;
            client->flush();
        }
        else{
            QFile filew("/home/mypc/.DB/DBmine.txt");
            //if(!(filew.open(QIODevice::Append)))
            if(!filew.open(QIODevice::WriteOnly))
            {
                qDebug() << ".... file not exist  ....";
            }
            else{

                if(!sqlFile.commitTransaction()){
                    qDebug() << ".... file is not possible to tranfer  ....";
                }

                sqlFile >> buffer;
                filew.write(buffer);
                client->waitForBytesWritten(-1);
                qDebug() << "... received file ...." << buffer;
                filew.close();
                client->flush();
            }
        }
}

These work perfect if send from client a little string like " bla bla bla" ..... if I filtered (fileSize < 40 && fileSize > 20) instead (fileSize < 40 && fileSize > 0) .... "bla bla bla" stream is written on file in correct manner on QFile .... but if my file is more than a string app return an empty ...... think because wrong management of qintsize on server .... but maybe some other stupid error .... and because I'm ignorant on these theme not see it and solution .... someone can help my on these? I'm working on some hour 2 days without understand the problem ...

best regards

Kent-Dorfman

you are working under the false assumption that readyread() gets called once for the whole session. As a session is a stream of "chunks" readyread() may be called many times for a connected session.

gfxx

@Kent-Dorfman said in soket datastream empty file problem:

As a session is a stream of "chunks" readyread() may be called many times for a connected session.

Real Thanks .... ! :)) .... :)) ..... :)) ..... so it seems that I have lost many pieces of the matter ...... :)))

So need to put my code in some sort of loop until fileSize become = 0 ???

JonB

@gfxx
Essentially yes, though really without any kind of "loop". Receive some data, increment the count received, that's it. When it reaches the announced file size you are done.

If you want to deal with possible error conditions you could think about what happens if you were to receive too much data, or if you have not received all the data you should have but a certain amount of time has passed without receiving any more.

gfxx

@JonB said in soket datastream empty file problem:

Essentially yes, though really without any kind of "loop". Receive some data, increment the count received, that's it. When it reaches the announced file size you are done.
If you want to deal with possible error conditions you could think about what happens if you were to receive too much data, or if you have not received all the data you should have but a certain amount of time has passed without receiving any more.

so need to send for first quint64 filesize, than filedata from client to server .... server for first receive qint64 filesize than packet of filedata .... everytime receive a packet need to decrement filesize quint64 or everitime need to send filesize too? for better explain me .... what of these two option?

[is not code .... is flow]

1- (client send) qint_filesize + filedata (all QfileData on stream or bytearray in one shot only)
1- (server receive) qint_filesize + filedata_packet
for(qint_filesize > -1) /* or ((qint_filesize - qint_actual_packet_size) > -1) */ {
qint_filesize = qint_filesize - 1;
read filedata_packet => filedata_packet => append to QFile}

2- (client send) qint_filesize + filedata_packet ..... (packet of QfileData on stream or bytearray in more than one time)
2- (server receive) qint_filesize + filedata_packet
for(qint_filesize > -1)/* or ((qint_filesize - qint_actual_packet_size) > -1) */ {
qint_filesize = qint_filesize - 1;
read filedata_packet => filedata_packet => append to QFile}

Than is :

a) for(qint_filesize  > -1) => qint_filesize = qint_filesize - 1;
or 
b) for((qint_filesize - qint_actual_packet_size) > -1)

regards and thanks

JonB

@gfxx
You seem to be overcomplicating/overthinking. You do not need to divide file transfer into chucks with sizes and counts, unless you want to. All you need to do, while you already had, is a filesize sent at the start and then you know when you have received all data when you have received that many bytes.

gfxx

@JonB thanks ... finally think Understand .... so without datastream .... first step ... after can try with datastream way (but is the right way ??)

Server DB receive:

void MainWindow::readSocket()
{

        QByteArray buffer;
        qint64 fileSize = client->bytesAvailable();


        if(fileSize < 40 && fileSize > 0 && !sokFirstRead){

            while(client->waitForReadyRead())
            {
                buffer.clear();
                buffer = client->read(client->bytesAvailable());

            }
        if(fileSize < 40 && fileSize > 0 && !sokFirstRead){
            qDebug() << " ... Server is receiving ...." << buffer << "...and size..." << buffer.size();
            client->flush();
            sokFirstRead = true;
        }
        else{
            QFile filew("/home/mypc/.DBc/DB_test.sqlite3");
            filew.resize(0);  /* because everytime revrite completely sqlite db server side ... than connect these re-new db to server db*/
            if(!filew.open(QIODevice::Append))
            {
                qDebug() << ".... file could not open  ....";
            }
            else{
                while(client->waitForReadyRead())
                {
                    buffer.clear();
                    buffer = client->read(client->bytesAvailable());
                    qDebug() << "Read    : " << buffer.size();
                    qDebug() << "Written : " << filew.write(buffer);
                }
                qDebug() << "... write the file content...." << buffer << "....size of file ...." <<  fileSize;
                client->flush();
            }
            filew.close();
            client->flush();
        }
}

CLIENT side when send client db:

void MainWindow::writeFileToServer()
{
    requestNewFortune();
    QFile file("/home/mypc_client/.DBclient/DB_client_test.sqlite3");
    if (!file.open(QIODevice::ReadOnly)){
        qDebug() << " ... not possible to open file .....";
    }
    else{
        if(tcpSocket->state() == QAbstractSocket::ConnectedState){
/* these code not work correct an is not call because everytime need to wait new connection ... so never connectedstate = true ... I can cancel it */
            //QByteArray byte = file.readAll();
            QByteArray block;
            QDataStream Sout(&block, QIODevice::WriteOnly);
            Sout.setVersion(QDataStream::Qt_5_15);
            //Sout.setDevice(tcpSocket);
            Sout << (quint64)0;
            Sout << file.readAll();
            Sout.device()->seek(0);
            Sout << quint64((block.size() - sizeof(quint64)));
            tcpSocket->write(block);
            tcpSocket->waitForBytesWritten();
            //tcpSocket->disconnectFromHost();

             qDebug() << "....connect ... write the file ??.....";
             file.close();
        }
        else { /* these work perfect*/
            tcpSocket->waitForConnected();
            QByteArray block1;

            while(1){
                block1.clear();
                block1 = file.read(32768*8); /* not find notice what is this ... for sure file size 16bit  like ...  but why 32768 "multiply" 8  ?? ....  really not know these */
                if(block1.size() == 0) break;
                tcpSocket->write(block1);
                tcpSocket->waitForBytesWritten();
                block1.clear();
            }

             qDebug() << "....wait forconnect ... send the file??.....";
            file.close();
        }
    }
    tcpSocket->disconnectFromHost();
}

There are some to wrong in these? because it work .... but think can make it better .... think to receive more or less 12 db from 12 different client every 5min ... more or less .... every client db can be max 20Mb. ...

JonB

@gfxx
You should not be writing (the receiving side) anything like this. No wait...() calls, no loops. You should just attach your slot to readyRead() and that should do whatever with whatever bytes are available each time it is called. Outline (this does not mean copyable code):

// member variables
int fileLen, bytesReceived.

bytesReceived = 0;
fileLen = // get the sender's fileLen (adjust `onReadyRead()` for this or whatever necessary)
connect(socket, readyRead, this, onReadyRead);

onReadyRead()
{
    byte buffer[];
    int received = socket.readAll(buffer);
    // do whatever with bytes, e.g. append them to a file if that's what you're doing
    bytesReceived += received;
    if (bytesReceived == fileLen)
        emit finishedReceive();  // do whatever when full file received
}

The important thing is we don't have a loop sitting getting all the bytes. We just have a slot which is called each time some new bytes arrive, and it does whatever with them, counting how many received till it reaches the originally-sent file length to expect.

gfxx

@JonB so these code (server side that receive the file) ...

                while(client->waitForReadyRead())
                {
                    buffer.clear();
                    buffer = client->read(client->bytesAvailable());
                    qDebug() << "Read    : " << buffer.size();
                    qDebug() << "Written : " << filew.write(buffer);
                }

Is not happy ..... client side (send db file) is ok for you or can do it better?

real thanks for helps.
bkt

jsulm

@gfxx said in soket datastream empty file problem:

while(client->waitForReadyRead())

Please don't do this!
@JonB already told you that you should connect a slot to readyRead() signal and read from client in that slot (without any loops and waitFor... calls!). Each time that slot is called you read what is available and add it to your buffer until your buffer size is >= fileSize.

JonB

@gfxx said in soket datastream empty file problem:

while(client->waitForReadyRead())

As @jsulm says. I wrote earlier:

No wait...() calls, no loops.

And yet you have both of these and ask if it's correct!

Please stop pasting code and just think about what we are trying to do here. Take a deep breath, stop typing for a bit and take the time to understand! Unlike what you might write in a non-event-driven program we do not want to write some loop which sits and gets all the bytes being transferred. Instead, we just allow the bytes to arrive, and as they do we count off how many till we have reached the number expected. Then we know we have finished, and can proceed with whatever we want to do at that point. No looping, no waiting, just react when you are told some new data has arrived. That is event driven programming, and the signals-slots paradigm in Qt.

gfxx

@JonB said in soket datastream empty file problem:

As @jsulm says. I wrote earlier:

No wait...() calls, no loops.

Sorry I ask you only because previously talk about these only receiver side .... so I ask for sender too .... I understood wery well that loops is the hell on receiver side .... Not sure is the hell too on sender side .... In my image (but not real understand if right or not) qt socket have your proper buffer .... so when send can load all data in that in one time ... after receiver side can read that data when chunk arrived .... Now understand the right way to do is for first send file info (filesize) than data ... so can compare filesize with total amount of byte received .... when filesize = total amount .... all is received. Ok thanks .... but is the same on sender side or can send all file in one time? Is right to think qsoket have a propetary buffer? and witch size is? Or simply buffer not exist and need to think about sender send and receiver receive only chunk data? honestly these is not understand.

Anyway real thanks ..... I'm really out of by these things, and I learn it now.

gfxx

@JonB an other question ..... but during receiving period (using only signal/slot) can send some data to same client from server? O need to make a statemachine for stop sending during receiving?

regards
bkt

JonB

@gfxx
Good, you getting there :)

Sender is simpler. You should be able to write and forget, no need for chunking or your own buffer. Qt and/or OS should take care of whatever "buffering" may be needed.

If and only if you are sending a huge file --- GBs --- it might be an issue. I believe Qt/OS must copy your written bytes somewhere since it (QIODevice::write()) does not say your written buffer has to stay in scope till bytes written. In that case I can only guess that it would have to allocate GBs of memory, which would not be good; or maybe the write() blocks till enough data has been sent in this case. Then I might break the write-sends into smaller chunks to alleviate this issue. *However, do not do this now, get what you have working with no buffering/chucks, regard it as an advanced issue if you test to encounter it.

JonB

@gfxx said in soket datastream empty file problem:

but during receiving period (using only signal/slot) can send some data to same client from server?

In principle that is fine. Sockets are bi-directional: in middle of sending file bytes from client to server if you want to send something in other direction from server to client (e.g. perhaps a status of "so many bytes received so far) you can.

However in this case it won't/may not be any use if the client has set off a single write() of whole file. Depends how the write() works, whether it returns immediately in client but bytes are actually then being written in background. In this case you might want to "chunk" the writes after all, e.g. next chunk on sender getting void QIODevice::bytesWritten(qint64 bytes) signal.