QTcpServer with multiple persistent connections
-
@JonB I think what he wanted to say is that your protocol shouldn't allow sending huge packets of data in one big piece at all, and that – if you want to handle such amounts – you will have to implement it in the proper way so that everything is fine. Because if you send 1 GB to a socket using the default behavior (which will probably result in the reveiver's socket buffer filling up to 1 GB until you can read it out), you will have to read it into memory before. And you shouldn't have done that in the first place.
@l3u_
I have written TCP client/server in previous products to allow transfer of files. They seem to have worked fine for years. Neither the client nor the server read the entire file into memory to send or receive. That's all I meant.Because if you send 1 GB to a socket using the default behavior (which will probably result in the reveiver's socket buffer filling up to 1 GB until you can read it out),
Umm, will it?? I don't think so... Pipes wouldn't do well...
-
@l3u_
I have written TCP client/server in previous products to allow transfer of files. They seem to have worked fine for years. Neither the client nor the server read the entire file into memory to send or receive. That's all I meant.Because if you send 1 GB to a socket using the default behavior (which will probably result in the reveiver's socket buffer filling up to 1 GB until you can read it out),
Umm, will it?? I don't think so... Pipes wouldn't do well...
@JonB I'm totally fine with everything and everybody here – I'm the one that surely has the least experience with all this here ;-) And I won't be transferring stuff that is bigger than a few hundred characters of JSON data, so the problem is surely interesting, but I don't think I have to worry about it in my case.
-
@l3u_
I have written TCP client/server in previous products to allow transfer of files. They seem to have worked fine for years. Neither the client nor the server read the entire file into memory to send or receive. That's all I meant.Because if you send 1 GB to a socket using the default behavior (which will probably result in the reveiver's socket buffer filling up to 1 GB until you can read it out),
Umm, will it?? I don't think so... Pipes wouldn't do well...
@JonB said in QTcpServer with multiple persistent connections:
Umm, will it?? I don't think so... Pipes wouldn't do well...
Probably, yes. Qt has its own buffering on top of the socket layer, so unless you change that, by default the socket's buffer will accommodate it provided you have enough memory.
@l3u_ said in QTcpServer with multiple persistent connections:
I'm the one that surely has the least experience with all this here
Don't worry, give it a decade or two and you'd be the one giving the advice. ;P
Read and abide by the Qt Code of Conduct
-
@JonB said in QTcpServer with multiple persistent connections:
Umm, will it?? I don't think so... Pipes wouldn't do well...
Probably, yes. Qt has its own buffering on top of the socket layer, so unless you change that, by default the socket's buffer will accommodate it provided you have enough memory.
@l3u_ said in QTcpServer with multiple persistent connections:
I'm the one that surely has the least experience with all this here
Don't worry, give it a decade or two and you'd be the one giving the advice. ;P
@kshegunov said in QTcpServer with multiple persistent connections:
Qt has its own buffering on top of the socket layer
So if you then use socket "transactions" on top of the socket, I read that as doing the buffering in memory. Do you mean the transactions have to add another layer of buffering, or do they use whatever the
QTcpSocketis already using? -
@kshegunov said in QTcpServer with multiple persistent connections:
Qt has its own buffering on top of the socket layer
So if you then use socket "transactions" on top of the socket, I read that as doing the buffering in memory. Do you mean the transactions have to add another layer of buffering, or do they use whatever the
QTcpSocketis already using?@JonB said in QTcpServer with multiple persistent connections:
So if you then use socket "transactions" on top of the socket, I read that as doing the buffering in memory. Do you mean the transactions have to add another layer of buffering, or do they use whatever the QTcpSocket is already using?
They use the QAbstractSocket's buffer directly. Think of the transaction as only adding atomicity to the reading from the Qt's buffer. They do stuff similar to what @VRonin wrote about splitting your big data chunk: try to read it in the known format, if that fails, rewind the buffer's pointer. If it succeeds cut the internal buffer with what you read.
-
Sorry to jump in late here, but I've been grooving on WebSockets lately
(a stupid name for a good protocol). Very easy to setup and use in Qt 5. Connect a couple of signals and slots and you can reliably send and receive text and/or binary data. You can ignore the grind of TCP/IP and just mostly deal with your text. -
Sorry to jump in late here, but I've been grooving on WebSockets lately
(a stupid name for a good protocol). Very easy to setup and use in Qt 5. Connect a couple of signals and slots and you can reliably send and receive text and/or binary data. You can ignore the grind of TCP/IP and just mostly deal with your text.That is true. There are 2 conditions to do so, however:
- You want to commit to the websockets protocol, which isn't always the case.
- You are willing to pull a module outside of qtbase, which is fine most of the time.
To expand on 1:
In the project I'm currently working on I have to support multiple protocols, so websockets does me no good. You may be surprised, but that case is more common than not. -
Sorry to jump in late here, but I've been grooving on WebSockets lately
(a stupid name for a good protocol). Very easy to setup and use in Qt 5. Connect a couple of signals and slots and you can reliably send and receive text and/or binary data. You can ignore the grind of TCP/IP and just mostly deal with your text.@Dick-Balaska Hey, thanks for the hint with WebSockets! I never heard about this until now. At first sight, it looks promising, because it already has the methods I need for sending and receiving text (I use JSON documents for my communication).
But would have using it an actual benefit for my use-case, after already having it got to work? What I did until now is (and it was surprisingly easy to implement it, as soon as someone points you towards the right way):
- Writing an abstract class that cares about the data transmission and receiving, using
QByteArrayandQDataStream, similar to the chat example approach . It provides asendMessage(const QString &message)function, and reads incoming messages, calling a virtual handler - Create a client derived from it that can connect to a server
- Create a server that manages an object derived from it for each connection
So … am I right that the only thing I could have cheaped out using WebSockets is that abstract class, because the WebSocket classes already have functions for sending and receiving text?
- Writing an abstract class that cares about the data transmission and receiving, using
-
@Dick-Balaska Hey, thanks for the hint with WebSockets! I never heard about this until now. At first sight, it looks promising, because it already has the methods I need for sending and receiving text (I use JSON documents for my communication).
But would have using it an actual benefit for my use-case, after already having it got to work? What I did until now is (and it was surprisingly easy to implement it, as soon as someone points you towards the right way):
- Writing an abstract class that cares about the data transmission and receiving, using
QByteArrayandQDataStream, similar to the chat example approach . It provides asendMessage(const QString &message)function, and reads incoming messages, calling a virtual handler - Create a client derived from it that can connect to a server
- Create a server that manages an object derived from it for each connection
So … am I right that the only thing I could have cheaped out using WebSockets is that abstract class, because the WebSocket classes already have functions for sending and receiving text?
@l3u_ If you have already hardened [1] your QTcpSocket implementation, then there's no reason to switch, except for maybe ease of SSL support.
For new work, these days I try to use WebSockets. I, too, use JSON. Each asynchronous message is just a command token followed by a JSON string.[1] You handle message fragmentation and reconnects, right?
- Writing an abstract class that cares about the data transmission and receiving, using
-
@l3u_ If you have already hardened [1] your QTcpSocket implementation, then there's no reason to switch, except for maybe ease of SSL support.
For new work, these days I try to use WebSockets. I, too, use JSON. Each asynchronous message is just a command token followed by a JSON string.[1] You handle message fragmentation and reconnects, right?
@Dick-Balaska said in QTcpServer with multiple persistent connections:
[1] You handle message fragmentation and reconnects, right?
Of course I don't – or at least, I don't know – or I'm not sure ;-)
Would you please explain what you mean? If a "reconnect" is loss of a client connection followed by a new connect, I handle it (the client is removed from the server's client list as soon as it disconnects and is simply added like a new one when it reconnects).
If "message fragmentation" is that "you can't expect all data to arrive in one piece", I think this is already handled by the chat example (cf.
ChatClient::onReadyRead()).I don't need encryption and such, the protocol will only be used in a closed LAN consisting of two or three computers, the data itself is not confidential, and I also don't need to protect against manipulation etc. (somebody trying this would shoot himself in the foot as the program probably wouldn't work as expected in this case).
-
@Dick-Balaska said in QTcpServer with multiple persistent connections:
[1] You handle message fragmentation and reconnects, right?
Of course I don't – or at least, I don't know – or I'm not sure ;-)
Would you please explain what you mean? If a "reconnect" is loss of a client connection followed by a new connect, I handle it (the client is removed from the server's client list as soon as it disconnects and is simply added like a new one when it reconnects).
If "message fragmentation" is that "you can't expect all data to arrive in one piece", I think this is already handled by the chat example (cf.
ChatClient::onReadyRead()).I don't need encryption and such, the protocol will only be used in a closed LAN consisting of two or three computers, the data itself is not confidential, and I also don't need to protect against manipulation etc. (somebody trying this would shoot himself in the foot as the program probably wouldn't work as expected in this case).
@l3u_ I see class
ChatClientreferenced in two examples (linux Qt 5.11.2), bluetooth, which I don't think you're doing, and WebChannel, which actually sits on top of WebSockets. So, I'm not sure what your base is.WebSockets guarantees you get a whole message at a time. A regular socket (QTcpSocket) you just know you have x bytes available and it's up to you to frame them in a message. (Like http knows that \n\n is the end of the request headers).
Clearly you're well beyond that.
If you're unsure about message fragmentation, make a class with a >2k byte object and see if you get it in one message. (hardwired lan TCP is 1536 bytes per packet. Wifi is about 400 bytes per packet.) -
@l3u_ I see class
ChatClientreferenced in two examples (linux Qt 5.11.2), bluetooth, which I don't think you're doing, and WebChannel, which actually sits on top of WebSockets. So, I'm not sure what your base is.WebSockets guarantees you get a whole message at a time. A regular socket (QTcpSocket) you just know you have x bytes available and it's up to you to frame them in a message. (Like http knows that \n\n is the end of the request headers).
Clearly you're well beyond that.
If you're unsure about message fragmentation, make a class with a >2k byte object and see if you get it in one message. (hardwired lan TCP is 1536 bytes per packet. Wifi is about 400 bytes per packet.)@Dick-Balaska said in QTcpServer with multiple persistent connections:
I see class ChatClient referenced in two examples (linux Qt 5.11.2), bluetooth, which I don't think you're doing, and WebChannel, which actually sits on top of WebSockets. So, I'm not sure what your base is.
Scroll up. The OP is using as a base the WIP example from the wiki I sourced in my first post.
@l3u_ said in QTcpServer with multiple persistent connections:
If "message fragmentation" is that "you can't expect all data to arrive in one piece", I think this is already handled by the chat example (cf. ChatClient::onReadyRead()).
Yes, this is correct.
-
@Dick-Balaska said in QTcpServer with multiple persistent connections:
I see class ChatClient referenced in two examples (linux Qt 5.11.2), bluetooth, which I don't think you're doing, and WebChannel, which actually sits on top of WebSockets. So, I'm not sure what your base is.
Scroll up. The OP is using as a base the WIP example from the wiki I sourced in my first post.
@l3u_ said in QTcpServer with multiple persistent connections:
If "message fragmentation" is that "you can't expect all data to arrive in one piece", I think this is already handled by the chat example (cf. ChatClient::onReadyRead()).
Yes, this is correct.
@kshegunov said in QTcpServer with multiple persistent connections:
Scroll up. The OP is using as a base the WIP example from the wiki I sourced in my first post.
Ok. Wrapping "read the socket" in a transaction is a neat trick. But, hmm, it determines end-of-message-received by whether it can parse valid JSON out of what it has. It seems to me that if you send a malformed message, the socket becomes dead. There is no way to clear the existing buffer. It will continue growing the buffer ad infinitum. I'd rather see it buffer until it receives a marker like \0\0, with a limit of 64KB on a message. Then if the JSON doesn't parse you can note the error and move on. Unless there is some application level feedback (ACK) involved, so the client can kill and restart the socket, neither side will know this socket is on its way to tipping over the app.
I know, you shouldn't be sending malformed JSON, but doing so shouldn't be fatal.
-
@kshegunov said in QTcpServer with multiple persistent connections:
Scroll up. The OP is using as a base the WIP example from the wiki I sourced in my first post.
Ok. Wrapping "read the socket" in a transaction is a neat trick. But, hmm, it determines end-of-message-received by whether it can parse valid JSON out of what it has. It seems to me that if you send a malformed message, the socket becomes dead. There is no way to clear the existing buffer. It will continue growing the buffer ad infinitum. I'd rather see it buffer until it receives a marker like \0\0, with a limit of 64KB on a message. Then if the JSON doesn't parse you can note the error and move on. Unless there is some application level feedback (ACK) involved, so the client can kill and restart the socket, neither side will know this socket is on its way to tipping over the app.
I know, you shouldn't be sending malformed JSON, but doing so shouldn't be fatal.
@Dick-Balaska said in QTcpServer with multiple persistent connections:
But, hmm, it determines end-of-message-received by whether it can parse valid JSON out of what it has.
This is incorrect. It determines end-of-message based on the
QByteArrayit receives. This isn't strictly a text protocol, as the buffer's size is known beforehand (you haveQDataStreamwithQByteArrayon both sides). If the JSON parsing (after the message is read) fails, then the server just continues its merry way, it doesn't even report the error, which it probably should ... -
@kshegunov said in QTcpServer with multiple persistent connections:
Scroll up. The OP is using as a base the WIP example from the wiki I sourced in my first post.
Ok. Wrapping "read the socket" in a transaction is a neat trick. But, hmm, it determines end-of-message-received by whether it can parse valid JSON out of what it has. It seems to me that if you send a malformed message, the socket becomes dead. There is no way to clear the existing buffer. It will continue growing the buffer ad infinitum. I'd rather see it buffer until it receives a marker like \0\0, with a limit of 64KB on a message. Then if the JSON doesn't parse you can note the error and move on. Unless there is some application level feedback (ACK) involved, so the client can kill and restart the socket, neither side will know this socket is on its way to tipping over the app.
I know, you shouldn't be sending malformed JSON, but doing so shouldn't be fatal.
PS.
You raise a good point for the general case, however - how to validate messages that come through a binary stream. But I fear this is a topic that goes beyond the purpose of this rather simple example.Read and abide by the Qt Code of Conduct
-
PS.
You raise a good point for the general case, however - how to validate messages that come through a binary stream. But I fear this is a topic that goes beyond the purpose of this rather simple example.@kshegunov said in QTcpServer with multiple persistent connections:
But I fear this is a topic that goes beyond the purpose of this rather simple example.
Keep on with your conversation if you don't mind – I (and everybody else reading this) is definitely learning all the stuff I wished to find somewhere before starting my project!
-
@kshegunov said in QTcpServer with multiple persistent connections:
But I fear this is a topic that goes beyond the purpose of this rather simple example.
Keep on with your conversation if you don't mind – I (and everybody else reading this) is definitely learning all the stuff I wished to find somewhere before starting my project!
Well the conventional wisdom when working with a developed binary protocol (which the aforementioned example does not pretend to posses), is to add a message header (usually a fixed size one) and a checksum for it. So you know when reading the header and calculating the checksum if your header's valid. Thereafter you can read the payload somewhat safely, for which (if necessary) you can also add a hash to be sure data's valid. After that it can be parsed, if that's required.
Read and abide by the Qt Code of Conduct
-
Well the conventional wisdom when working with a developed binary protocol (which the aforementioned example does not pretend to posses), is to add a message header (usually a fixed size one) and a checksum for it. So you know when reading the header and calculating the checksum if your header's valid. Thereafter you can read the payload somewhat safely, for which (if necessary) you can also add a hash to be sure data's valid. After that it can be parsed, if that's required.
@kshegunov But after all, we can still be sure that a message is processed completely by using QDataStream transactions, can't we? Connecting my server using telnet from a console, the JSON greeting is prepended by an additional character, so I think the datastream also adds some header and knows if it's complete?
If I read the code correctly, the chat example makes sure that a whole message is received (and the socket buffer is emptied after reading it out) and tries to parse the JSON data inside it. If it's actually JSON, and it's valid, some action is triggered, if not, nothing is done – but the buffer is still emptied. Is this correct?
-
@kshegunov But after all, we can still be sure that a message is processed completely by using QDataStream transactions, can't we? Connecting my server using telnet from a console, the JSON greeting is prepended by an additional character, so I think the datastream also adds some header and knows if it's complete?
If I read the code correctly, the chat example makes sure that a whole message is received (and the socket buffer is emptied after reading it out) and tries to parse the JSON data inside it. If it's actually JSON, and it's valid, some action is triggered, if not, nothing is done – but the buffer is still emptied. Is this correct?
@l3u_ said in QTcpServer with multiple persistent connections:
@kshegunov But after all, we can still be sure that a message is processed completely by using QDataStream transactions, can't we?
Well, yes. But you're not protected from a malformed message (i.e. some client you didn't write sending you a ridiculous message).
Connecting my server using telnet from a console, the JSON greeting is prepended by an additional character, so I think the datastream also adds some header and knows if it's complete?
Yes, it adds the
QByteArray's size before sending the actual contents of the byte array. However you have no way of knowing if this size (or the data) isn't corrupted somehow. That's really rare, but it's a possibility.For example we don't check the byte array's size before trying to read the data. If someone sends you an integer indicating a message of 10GB, the code will just eat it. It doesn't check if that's too big.
If I read the code correctly, the chat example makes sure that a whole message is received (and the socket buffer is emptied after reading it out) and tries to parse the JSON data inside it. If it's actually JSON, and it's valid, some action is triggered, if not, nothing is done – but the buffer is still emptied. Is this correct?
That is correct.
Read and abide by the Qt Code of Conduct
-
@l3u_ said in QTcpServer with multiple persistent connections:
@kshegunov But after all, we can still be sure that a message is processed completely by using QDataStream transactions, can't we?
Well, yes. But you're not protected from a malformed message (i.e. some client you didn't write sending you a ridiculous message).
Connecting my server using telnet from a console, the JSON greeting is prepended by an additional character, so I think the datastream also adds some header and knows if it's complete?
Yes, it adds the
QByteArray's size before sending the actual contents of the byte array. However you have no way of knowing if this size (or the data) isn't corrupted somehow. That's really rare, but it's a possibility.For example we don't check the byte array's size before trying to read the data. If someone sends you an integer indicating a message of 10GB, the code will just eat it. It doesn't check if that's too big.
If I read the code correctly, the chat example makes sure that a whole message is received (and the socket buffer is emptied after reading it out) and tries to parse the JSON data inside it. If it's actually JSON, and it's valid, some action is triggered, if not, nothing is done – but the buffer is still emptied. Is this correct?
That is correct.
@kshegunov Okay, then I'm safe. It's a protocol to be used by two or three computers in a LAN, and everybody using it wants it to function – so nobody will try to fake a message or trick it with wrong headers.
I protect the integrity against whatever may go wrong with a checksum: Each change request contains a checksum the whole data has to have afterwards. If the checksum is correct after a client applied the change, everything is fine. If not, the connection is closed and the client is told to re-connect, as a syncing with the server is done when a new client connects. Ans this one also contains the checksum verification.
At least, I hope that I'm safe this way ;-)
