Solved QTcpServer + QTcpSocket with multi-threading: can never read the whole incoming data
-
Hi @Violet-Giraffe,
no, you don't need big TCP buffers, both ends negotiate the buffer size and send data in smaller but more chunks if needed.
You should use Wireshark to monitor the network traffic and get an understanding what's going on.
I don't immediately see something wrong in your code, but with threaded servers things mostly get complicated. I'd suggest to try a simple non-threaded server first.
Regards
-
@Violet-Giraffe
why don't you overlay your own protocol on top?Byte[0] chunkIndex; of Byte[1] totalAmount of Chunks; Byte[2] & Byte[3] bytes to follow; ......
that way you could send your 100kBytes in nice small 10k byte packages.
-
@J.Hilk
I guess I'll have to, but I expected the sender to implement that behavior already (only without explicit headers). I may need to use Wireshark after all, to double-check whether the data is actually being sent. Thanks for the suggestions. -
@Violet-Giraffe said in QTcpServer + QTcpSocket: can never read more than 14600 bytes:
14 600 bytes. Where is this magic value coming from?
I recall this figure is a fundamental TCP max physical packet size, at the lowest level. Nothing to do with Qt, or sockets.
-
I fired up Wireshark, and there's something interesting going on. I'm no expert on networking and TCP in particular, but I do think this could be an issue in my TCP server thought, or perhaps even in the Qt TCP layer. Here's what it looks like, you can see that the data was transmitted:
Here's the capture file if someone can be persuaded to look at it: https://mega.nz/#!luoRlRjY!pQG_nr1nIxMkwSx1D4WN9qJjB2e5tkIxgclfhAftRG4
Transmitter was 192.168.1.2, receiver (the Qt TCP server) is 192.168.1.85.On the receiver (Qt TCP socket) end, 1460 bytes was obtained, and then naught more. And here's the specific code I used:
void IndexerServer::incomingConnection(qintptr handle) { if (!handle) return; _threadPool.enqueue([this, handle](){ qInfo() << "New incoming connection."; QTcpSocket socket; socket.setSocketDescriptor(handle); socket.waitForReadyRead(); QDataStream inStream(&socket); const int dataSize = 100000; QByteArray data; for (qint64 bytesAvailable = socket.bytesAvailable(); data.size() < dataSize; bytesAvailable = socket.bytesAvailable()) { if (bytesAvailable > 0) { data.append(socket.readAll()); qInfo() << bytesAvailable; } std::this_thread::sleep_for(std::chrono::milliseconds(333)); } qInfo() << data.size(); qInfo() << QString::fromUtf8(data.data(), 100); socket.close(); }); }
So what could be wrong, and how to get the data that seems to have been transmitted?
-
you don't have a
waitForReadyRead
in your loop? That could prohibit the TCP buffer handling in the background ... I'm not sure.if you wanne have a look at some simple example transferring megabytes:
http://doc.qt.io/qt-5/qtnetwork-loopback-example.html
Regards
-
@aha_1980 said in QTcpServer + QTcpSocket: can never read the whole incoming data (Updated with new findings on 23.09):
you don't have a
waitForReadyRead
in your loop? That could prohibit the TCP buffer handling in the background ... I'm not sure.Hmm, you might be right! Simply adding
socket.waitForReadyRead();
inside the loop solves the problem. I wish the docs would mention that, I wasted so much time debugging this issue (and looking for the source in the wrong places, e. g. the sender code). The docs only said this method may fail randomly on Windows, so I didn't see any point using it. Meh.Thank you!
-
@Violet-Giraffe said in QTcpServer + QTcpSocket: can never read the whole incoming data (Updated with new findings on 23.09):
I wish the docs would mention that
The docs clearly say you should not use blocking calls but signals and slot to get data from a QTcpSocket (or any other Qt class which retrieves data from sockets)
-
@Christian-Ehrlicher said in QTcpServer + QTcpSocket: can never read the whole incoming data (Updated with new findings on 23.09):
The docs clearly say you should not use blocking calls but signals and slot to get data from a QTcpSocket (or any other Qt class which retrieves data from sockets)
"Clearly"? I disagree. Where do the docs say that any kind of blocking use case is prohibited and using signals is mandatory?
-
Hi @Violet-Giraffe,
I'm glad you solved your issue.
Qt is fully event based, so it needs the event loop for correct operation. This might not always and not everywhere be stated, but its just a fundamental fact.
There is even a new blog series, where already part one has this topic: https://www.cleanqt.io/blog/crash-course-in-qt-for-c%2B%2B-developers,-part-1
In case you use threads, the
waitForXxx()
functions often do the background processing to keep the event loop alive. (However, it is also possible to use signals&slots in threads ... so many possibilities :)) -
@aha_1980 said in QTcpServer + QTcpSocket: can never read the whole incoming data (Updated with new findings on 23.09):
Qt is fully event based, so it needs the event loop for correct operation. This might not always and not everywhere be stated, but its just a fundamental fact.
I understand that perfectly, and there is an event loop. I have a
QApplication::exec()
in main.cpp, and I'm not blocking the main thread that does the processing, I'm only blocking my own worker thread which I expect to only "look" at the state of the program, not affect it so drastically. I don't understand why I can't block my thread and what it is thatwaitForReadyRead()
does. Or, more precisely, if that's howQTcpSocket
is designed, and ifwaitForReadyRead()
is the special method to call in this use case for the socket to update/process, I don't understand why this issue is not documented directly. It would only be logical, sinceQTcpServer
does inherently support multithreading and there are multithreading examples for it all over the web (including the official examples). But never mind, I'm glad to put this behind me and move on to implementing the interesting things. -
It is not documented directly because it's an essential part of the Qt design. To handle events Qt needs an eventloop (per thread) - how should a socket event should be retrieved by Qt otherwise when there is no eventloop to retrieve it from the os?
http://doc.qt.io/qt-5/threads-qobject.html#per-thread-event-loop