Unsolved Not able to transfer large files on QTcpSocket
-
Hi everybody, I'm trying to code a server-client application that automatically transfers files from a client to a server but I'm not able to send files largen than 8-10 Mb to the server.
On Windows it seems to work well, in fact I'm able to transfer files with size of more or less 5 GB, but on unix/macOS I cannot: the readFile function triggers only few times and doesn't receive the whole transmitted bytes.This is the code:
Client-side:void client::sendFile(std::string path_to_watch) { file = new QFile(QString::fromStdString(path_to_watch)); if (!file->open(QIODevice::ReadWrite)) { qDebug()<<"Couldn't open the file"; return; } qint64 file_size = file->size(); QFileInfo fileInfo(file->fileName()); QJsonObject file_created{ {"action", 1}, {"file_name", fileInfo.fileName()}, {"file_size", file_size} }; t->sendJson(file_created, address, 50505); QByteArray block; connect(t->getTcpSocket(), SIGNAL(disconnected()), t->getTcpSocket(), SLOT(deleteLater())); qint64 sentData = 0; qint64 leftData = file->size()-sentData; while(leftData>0){ if(leftData > 100000000){ block = file->read(100000000); }else{ block = file->read(leftData); } qint64 written = t->getTcpSocket()->write(block); sentData+=written; leftData = file->size()-sentData; t->getTcpSocket()->waitForBytesWritten(); block.remove(0, block.size()); } file->close(); return; }
Server-side:
void server::receive(){ in.startTransaction(); QString nextFortune; in >> nextFortune; QJsonDocument jsonResponse = QJsonDocument::fromJson(nextFortune.toLatin1()); QJsonArray jsonArray = jsonResponse.array(); if(!jsonArray.isEmpty()) { QJsonObject jsonObject = jsonArray.first().toObject(); if (!in.commitTransaction()) return; int c = jsonObject.value("action").toInt(); QJsonArray jsarray; switch (c) { ... case 1: { std::cout<<"entro in switch 1..."<<std::endl; file_name = jsonObject.value("file_name").toString(); file_size = jsonObject.value("file_size").toVariant().toLongLong(); disconnect(clientConnection, &QIODevice::readyRead, 0, 0); connect(clientConnection, &QIODevice::readyRead, this, &server::readFile); } break; ... } } } void server::readFile(){ if(firstTry){ QByteArray b; blocks.append(b); total=0; firstTry = false; } if(blocks.last().size() > 1400000000){ QByteArray b1; blocks.append(b1); } if(total < file_size){ qint64 available = clientConnection->bytesAvailable(); total += available; std::cout<<"Available: "<<available<<std::endl; std::cout<<"Total: "<<total << std::endl; blocks.last().append(clientConnection->read(available)); if(total<file_size){ return; }else{ firstTry=true; dataReceived(); } } }
blocks is a QVector of QByteArray but it's temporary, I'll change it as soon as I'm able to transmit the files correctly
Any idea?
Thank you in advance -
@dual
Not sure, because you have a heck of a lot of code and no comments/debug information.Your code looks incredibly complex. It should be miles shorter. I wouldn't expect to see any
waitFor...()
s,bytesAvailable()
,firstTry
s, or half the other stuff. I am not sure about your transaction either. There is lots of code missing. I presume you are compiling 64-bit, and all your relevant variables used here are correctly 64-bit (e.g. not plainint
).disconnect(clientConnection, &QIODevice::readyRead, 0, 0); connect(clientConnection, &QIODevice::readyRead, this, &server::readFile);
I don't like the look of the above. You are switching slots while a transfer is underway. The client sends a stream of data, you cannot assume you receive them in "boundaries" which have any match to what calls you used to send them. I am concerned that by the time you start getting
readFile()
called you may have already received some of the bytes. (Maybe your transaction prevents that, I don't know.)If it were me, I would start again and make the code much simpler...! :)
-
@JonB
Thanks for your answer.
My code became complex because I tried to make it work on Windows machines at least.
But let me explain some choices I made:disconnect(clientConnection, &QIODevice::readyRead, 0, 0); connect(clientConnection, &QIODevice::readyRead, this, &server::readFile);
I used these two functions because I need to differentiate the receiving of a Json (which provides the different commands -like login, sign up, sign out, infos about the file size, etc-) from the receiving of the file itself.
qint64 available = clientConnection->bytesAvailable();
This function was necessary because readAll()/read() will stop working when receiving a large sized file. I think here there's room for improvement, in fact I accept any suggestion in order to manage this issue.
if(firstTry){ ... }
I'm using the boolean firstTry in order to initialize the data struct I will use for receiving the file, in fact the readFile will be called many times and I need to initialize it only once
-
Hi,
From the looks of it you have a protocol to follow. You should implement that logic properly rather than relying on some hazardous state.
It looks like you should have something along the line of:- send json data
- get ack from server
- send file
- get ack from server
Rince and repeat.
You might want to take a look at Amazon's S3 chunked API. It has a pretty good design and implements a pretty nice way to upload heavy files.