QTcpServer + QTcpSocket with multi-threading: can never read the whole incoming data

[Violet Giraffe]

@J.Hilk
I guess I'll have to, but I expected the sender to implement that behavior already (only without explicit headers). I may need to use Wireshark after all, to double-check whether the data is actually being sent. Thanks for the suggestions.

[JonB]

@Violet-Giraffe said in QTcpServer + QTcpSocket: can never read more than 14600 bytes:

14 600 bytes. Where is this magic value coming from?

I recall this figure is a fundamental TCP max physical packet size, at the lowest level. Nothing to do with Qt, or sockets.

[Violet Giraffe]

I fired up Wireshark, and there's something interesting going on. I'm no expert on networking and TCP in particular, but I do think this could be an issue in my TCP server thought, or perhaps even in the Qt TCP layer. Here's what it looks like, you can see that the data was transmitted:

Here's the capture file if someone can be persuaded to look at it: https://mega.nz/#!luoRlRjY!pQG_nr1nIxMkwSx1D4WN9qJjB2e5tkIxgclfhAftRG4
Transmitter was 192.168.1.2, receiver (the Qt TCP server) is 192.168.1.85.

On the receiver (Qt TCP socket) end, 1460 bytes was obtained, and then naught more. And here's the specific code I used:

void IndexerServer::incomingConnection(qintptr handle)
{
	if (!handle)
		return;

	_threadPool.enqueue([this, handle](){
		qInfo() << "New incoming connection.";

		QTcpSocket socket;
		socket.setSocketDescriptor(handle);
		socket.waitForReadyRead();

		QDataStream inStream(&socket);
		const int dataSize = 100000;
		QByteArray data;
		for (qint64 bytesAvailable = socket.bytesAvailable(); data.size() < dataSize; bytesAvailable = socket.bytesAvailable())
		{
			if (bytesAvailable > 0)
			{
				data.append(socket.readAll());
				qInfo() << bytesAvailable;
			}

			std::this_thread::sleep_for(std::chrono::milliseconds(333));
		}

		qInfo() << data.size();
		qInfo() << QString::fromUtf8(data.data(), 100);
		socket.close();
	});
}

So what could be wrong, and how to get the data that seems to have been transmitted?

[aha_1980]

@Violet-Giraffe

you don't have a waitForReadyRead in your loop? That could prohibit the TCP buffer handling in the background ... I'm not sure.

if you wanne have a look at some simple example transferring megabytes:

http://doc.qt.io/qt-5/qtnetwork-loopback-example.html

Regards

[Violet Giraffe]

@aha_1980 said in QTcpServer + QTcpSocket: can never read the whole incoming data (Updated with new findings on 23.09):

@Violet-Giraffe

you don't have a waitForReadyRead in your loop? That could prohibit the TCP buffer handling in the background ... I'm not sure.

Hmm, you might be right! Simply adding socket.waitForReadyRead(); inside the loop solves the problem. I wish the docs would mention that, I wasted so much time debugging this issue (and looking for the source in the wrong places, e. g. the sender code). The docs only said this method may fail randomly on Windows, so I didn't see any point using it. Meh.

Thank you!

[Christian Ehrlicher]

@Violet-Giraffe said in QTcpServer + QTcpSocket: can never read the whole incoming data (Updated with new findings on 23.09):

I wish the docs would mention that

The docs clearly say you should not use blocking calls but signals and slot to get data from a QTcpSocket (or any other Qt class which retrieves data from sockets)

[Violet Giraffe]

@Christian-Ehrlicher said in QTcpServer + QTcpSocket: can never read the whole incoming data (Updated with new findings on 23.09):

The docs clearly say you should not use blocking calls but signals and slot to get data from a QTcpSocket (or any other Qt class which retrieves data from sockets)

"Clearly"? I disagree. Where do the docs say that any kind of blocking use case is prohibited and using signals is mandatory?

[aha_1980]

Hi @Violet-Giraffe,

I'm glad you solved your issue.

Qt is fully event based, so it needs the event loop for correct operation. This might not always and not everywhere be stated, but its just a fundamental fact.

There is even a new blog series, where already part one has this topic: https://www.cleanqt.io/blog/crash-course-in-qt-for-c%2B%2B-developers,-part-1

In case you use threads, the waitForXxx() functions often do the background processing to keep the event loop alive. (However, it is also possible to use signals&slots in threads ... so many possibilities :))

[Violet Giraffe]

@aha_1980 said in QTcpServer + QTcpSocket: can never read the whole incoming data (Updated with new findings on 23.09):

Qt is fully event based, so it needs the event loop for correct operation. This might not always and not everywhere be stated, but its just a fundamental fact.

I understand that perfectly, and there is an event loop. I have a QApplication::exec() in main.cpp, and I'm not blocking the main thread that does the processing, I'm only blocking my own worker thread which I expect to only "look" at the state of the program, not affect it so drastically. I don't understand why I can't block my thread and what it is that waitForReadyRead() does. Or, more precisely, if that's how QTcpSocket is designed, and if waitForReadyRead() is the special method to call in this use case for the socket to update/process, I don't understand why this issue is not documented directly. It would only be logical, since QTcpServer does inherently support multithreading and there are multithreading examples for it all over the web (including the official examples). But never mind, I'm glad to put this behind me and move on to implementing the interesting things.

[Christian Ehrlicher]

It is not documented directly because it's an essential part of the Qt design. To handle events Qt needs an eventloop (per thread) - how should a socket event should be retrieved by Qt otherwise when there is no eventloop to retrieve it from the os?
http://doc.qt.io/qt-5/threads-qobject.html#per-thread-event-loop