QTcpServer + QTcpSocket with multi-threading: can never read the whole incoming data

Violet Giraffe

SOLUTION: I was missing socket.waitForReadyRead() call in my threaded connection handler, which is required, apparently. Thanks to @aha_1980 for the solution.

I'm experimenting with QTcpServer and QTcpSocket. Eventually, I'll need to receive large chunks of data (and send small chunks in response). I have a Node.js application that sends an ASCII string 100 000 characters long, and QTcpSocket has to receive it. I've tried various approaches, and all of them only read 14 600 bytes. Where is this magic value coming from?

Here's my server, it's multithreaded by means of my own thread pool class:

class IndexerServer : public QTcpServer {
public:
	IndexerServer();
	void start(quint16 port);
protected:
	void incomingConnection(qintptr handle) override;
private:
	CWorkerThreadPool _threadPool;
};

IndexerServer::IndexerServer() : _threadPool(16, "Indexer server thread pool")
{
}

void IndexerServer::start(quint16 port)
{
	if (isListening())
		return;

	if (!listen(QHostAddress::Any, port))
	{
		qInfo() << "Starting server on port" << port << "failed:" << errorString();
		return;
	}

	qInfo() << "Listening on port" << port;
}

void IndexerServer::incomingConnection(qintptr handle)
{
	if (!handle)
		return;

	_threadPool.enqueue([this, handle](){
		qInfo() << "New incoming connection.";

		QTcpSocket socket;
		socket.setSocketDescriptor(handle);
		socket.waitForReadyRead();

		QDataStream inStream(&socket);
		QByteArray data;
		for (qint64 read = 0; read < 50000;)
		{
                        const  QByteArray newData = socket.readAll();
			qInfo() << read;
			std::this_thread::sleep_for(std::chrono::milliseconds(333));
		}

// Nope, doesn't work either

//              QByteArray data(50000, Qt::Uninitialized);
//		do {
//			inStream.startTransaction();
//			inStream.readRawData(data.data(), 50000);
//		} while (!inStream.commitTransaction());

		qInfo() << data.size();
		qInfo() << QString::fromUtf8(data.data(), 100);

		socket.close();
	});
}

I'm only trying to read 50 000 characters which is half of what's being sent, and I always get 14 600 with either method that I try.
Tried setting the socket read buffer size to 1M, still no luck. What's wrong with this code and how can I read my 100k bytes?

There is a slight possibility that the Node.js application is not sending the 100k bytes as requested, but there are no errors or exceptions, and I have checked that the Buffer object I'm writing to the socket indeed contains 100k bytes.

J.Hilk

@Violet-Giraffe
the max size of tcp transfer is negotiated between sender and reciver during the connect handshake. To garantie a certain amount of buffer size, you need to set the send and recive buffer to that size before the connect attempt.

Violet Giraffe

@J.Hilk
I understand that data is transmitted in chunks and I don't expect all the data to be sent in a single chunk, that's why I rely on knowing the data size beforehand and assembling the data from multiple chunks received over time.

Does this transfer size limit the size of a single chunk, or that of the whole "session" / connection / whatever it's called? What's the right way to send large amounts of data? I'm not looking to have a 1G RAM buffer in order to receive 1 gig of data.

aha_1980

Hi @Violet-Giraffe,

no, you don't need big TCP buffers, both ends negotiate the buffer size and send data in smaller but more chunks if needed.

You should use Wireshark to monitor the network traffic and get an understanding what's going on.

I don't immediately see something wrong in your code, but with threaded servers things mostly get complicated. I'd suggest to try a simple non-threaded server first.

Regards

J.Hilk

@Violet-Giraffe
why don't you overlay your own protocol on top?

Byte[0] chunkIndex; of Byte[1] totalAmount of Chunks; Byte[2] & Byte[3] bytes to follow; ......

that way you could send your 100kBytes in nice small 10k byte packages.

Violet Giraffe

@J.Hilk
I guess I'll have to, but I expected the sender to implement that behavior already (only without explicit headers). I may need to use Wireshark after all, to double-check whether the data is actually being sent. Thanks for the suggestions.

JonB

@Violet-Giraffe said in QTcpServer + QTcpSocket: can never read more than 14600 bytes:

14 600 bytes. Where is this magic value coming from?

I recall this figure is a fundamental TCP max physical packet size, at the lowest level. Nothing to do with Qt, or sockets.

Violet Giraffe

I fired up Wireshark, and there's something interesting going on. I'm no expert on networking and TCP in particular, but I do think this could be an issue in my TCP server thought, or perhaps even in the Qt TCP layer. Here's what it looks like, you can see that the data was transmitted:

0_1537701194027_f3e3ef7f-48da-4667-8b43-dda780ff47ef-image.png

Here's the capture file if someone can be persuaded to look at it: https://mega.nz/#!luoRlRjY!pQG_nr1nIxMkwSx1D4WN9qJjB2e5tkIxgclfhAftRG4
Transmitter was 192.168.1.2, receiver (the Qt TCP server) is 192.168.1.85.

On the receiver (Qt TCP socket) end, 1460 bytes was obtained, and then naught more. And here's the specific code I used:

void IndexerServer::incomingConnection(qintptr handle)
{
	if (!handle)
		return;

	_threadPool.enqueue([this, handle](){
		qInfo() << "New incoming connection.";

		QTcpSocket socket;
		socket.setSocketDescriptor(handle);
		socket.waitForReadyRead();

		QDataStream inStream(&socket);
		const int dataSize = 100000;
		QByteArray data;
		for (qint64 bytesAvailable = socket.bytesAvailable(); data.size() < dataSize; bytesAvailable = socket.bytesAvailable())
		{
			if (bytesAvailable > 0)
			{
				data.append(socket.readAll());
				qInfo() << bytesAvailable;
			}

			std::this_thread::sleep_for(std::chrono::milliseconds(333));
		}

		qInfo() << data.size();
		qInfo() << QString::fromUtf8(data.data(), 100);
		socket.close();
	});
}

So what could be wrong, and how to get the data that seems to have been transmitted?

aha_1980

@Violet-Giraffe

you don't have a waitForReadyRead in your loop? That could prohibit the TCP buffer handling in the background ... I'm not sure.

if you wanne have a look at some simple example transferring megabytes:

http://doc.qt.io/qt-5/qtnetwork-loopback-example.html

Regards

Violet Giraffe

@aha_1980 said in QTcpServer + QTcpSocket: can never read the whole incoming data (Updated with new findings on 23.09):

@Violet-Giraffe

you don't have a waitForReadyRead in your loop? That could prohibit the TCP buffer handling in the background ... I'm not sure.

Hmm, you might be right! Simply adding socket.waitForReadyRead(); inside the loop solves the problem. I wish the docs would mention that, I wasted so much time debugging this issue (and looking for the source in the wrong places, e. g. the sender code). The docs only said this method may fail randomly on Windows, so I didn't see any point using it. Meh.

Thank you!

Christian Ehrlicher

@Violet-Giraffe said in QTcpServer + QTcpSocket: can never read the whole incoming data (Updated with new findings on 23.09):

I wish the docs would mention that

The docs clearly say you should not use blocking calls but signals and slot to get data from a QTcpSocket (or any other Qt class which retrieves data from sockets)

Violet Giraffe

@Christian-Ehrlicher said in QTcpServer + QTcpSocket: can never read the whole incoming data (Updated with new findings on 23.09):

The docs clearly say you should not use blocking calls but signals and slot to get data from a QTcpSocket (or any other Qt class which retrieves data from sockets)

"Clearly"? I disagree. Where do the docs say that any kind of blocking use case is prohibited and using signals is mandatory?

aha_1980

Hi @Violet-Giraffe,

I'm glad you solved your issue.

Qt is fully event based, so it needs the event loop for correct operation. This might not always and not everywhere be stated, but its just a fundamental fact.

There is even a new blog series, where already part one has this topic: https://www.cleanqt.io/blog/crash-course-in-qt-for-c%2B%2B-developers,-part-1

In case you use threads, the waitForXxx() functions often do the background processing to keep the event loop alive. (However, it is also possible to use signals&slots in threads ... so many possibilities :))

Violet Giraffe

@aha_1980 said in QTcpServer + QTcpSocket: can never read the whole incoming data (Updated with new findings on 23.09):

Qt is fully event based, so it needs the event loop for correct operation. This might not always and not everywhere be stated, but its just a fundamental fact.

I understand that perfectly, and there is an event loop. I have a QApplication::exec() in main.cpp, and I'm not blocking the main thread that does the processing, I'm only blocking my own worker thread which I expect to only "look" at the state of the program, not affect it so drastically. I don't understand why I can't block my thread and what it is that waitForReadyRead() does. Or, more precisely, if that's how QTcpSocket is designed, and if waitForReadyRead() is the special method to call in this use case for the socket to update/process, I don't understand why this issue is not documented directly. It would only be logical, since QTcpServer does inherently support multithreading and there are multithreading examples for it all over the web (including the official examples). But never mind, I'm glad to put this behind me and move on to implementing the interesting things.

Christian Ehrlicher

It is not documented directly because it's an essential part of the Qt design. To handle events Qt needs an eventloop (per thread) - how should a socket event should be retrieved by Qt otherwise when there is no eventloop to retrieve it from the os?
http://doc.qt.io/qt-5/threads-qobject.html#per-thread-event-loop