Transferring data...
-
@artwaw , thank you, I am using UDP as the protocol because of the larger packets it is capable of sending, however there is a two way transaction for every packet. In the system I've developed, the server sends a message to the clients, notifying them that a transfer is ready, this consists of a JSON packet:
{"DSID" : 1, /*Data Set ID */ "RDF" : "/folder/name.rdf", /*Path and name of file to transfer*/ "Filesize". : 1234, /*File size in bytes*/ "Totalblocks" : 1234} /*Number of blocks, where a block consists of N bytes */Each client will then start to issue requests for each block, where a block will contain N bytes of the file as stored in a database. Client request:
{"DSID" : 1, /*Data Set ID */ "BlockNo" : 0} /*Block number to request 0 to Totalblocks-1*/The server will respond to each request with:
{"DSID" : 1, /*Data Set ID */ "BlockNo" : 0} /*Block number to request 0 to Totalblocks-1*/ "Checksum" : 0x1234, /*Checksum of hex bytes for validation*/ "Chunk". : "hex bytes"}. /*String containing hex nibbles*/The client requests each block until all blocks have been received. The client will also verify that the received data is correct by recalculating the checksum and comparing with the received checksum.
This process isn't quick and takes typical around 20 minutes to transmit a GB file.
@SPlatten said in Transferring data...:
This process isn't quick and takes typical around 20 minutes to transmit a GB file.
And how long does it take if you use TCP instead of "simulating" TCP with your own implementation?
-
@SPlatten said in Transferring data...:
This process isn't quick and takes typical around 20 minutes to transmit a GB file.
And how long does it take if you use TCP instead of "simulating" TCP with your own implementation?
-
@jsulm , since TCP packets are limited to 1.5K I can only assume it will take significantly longer.
@SPlatten said in Transferring data...:
I can only assume it will take significantly longer
That's my point - you are assuming, you actually do not know.
-
@SPlatten said in Transferring data...:
I can only assume it will take significantly longer
That's my point - you are assuming, you actually do not know.
-
@jsulm , all I can go on is the data thats in front of me. TCP packets can send 1.5K, UDP packets can send 64K, what isn't clear?
@SPlatten
What is not clear is:-
How do you know sending 64K packets actually transfers at ~40 x the speed of 1.5K packets?
-
How do you know given your own protocol which "simulates" TCP packet transfer reliability what overhead that places on your assumed timings?
We have said before it is not clear that UDP will be superior to TCP for reliable transfer. Maybe it is, maybe it isn't. All people are saying is why not take the time to find out for yourself how it compares instead of stating it will be more efficient? It's up to you.
-
-
@SPlatten
What is not clear is:-
How do you know sending 64K packets actually transfers at ~40 x the speed of 1.5K packets?
-
How do you know given your own protocol which "simulates" TCP packet transfer reliability what overhead that places on your assumed timings?
We have said before it is not clear that UDP will be superior to TCP for reliable transfer. Maybe it is, maybe it isn't. All people are saying is why not take the time to find out for yourself how it compares instead of stating it will be more efficient? It's up to you.
@JonB , thank you, I will try. I'm using Sockets, is there a built in tried and tested method of transferring files over TCP that I can try?
Because without this I would implement the same protocol I've adopted for UDP but using TCP sockets.
Kind Regards,
Sy -
-
@JonB , thank you, I will try. I'm using Sockets, is there a built in tried and tested method of transferring files over TCP that I can try?
Because without this I would implement the same protocol I've adopted for UDP but using TCP sockets.
@SPlatten said in Transferring data...:
I would implement the same protocol I've adopted for UDP but using TCP sockets
Well I don't think so because you spend time explaining you do chunks, checksums, retries and re-ordering. You wouldn't do any of that with TCP, would you? So presumably your code will be simpler.
I think when I have done this before I just did protocol of: client requests file, sender sends back leading long of total file size, then client reads till that many bytes received. I'm not sure whether an alternative could be to send a "0-packet-size" message as terminator. There will be stuff out there for this, even particularly for Qt.
I'm not as expert over UDP as others. What about: put the simplest TCP alternative in place. Time it (a few times) for one client to one server. Then time it for two clients, and half a dozen clients. Then see where you are so far?
As I said before: maybe you are right about efficiency, especially for multiple clients, I don't know. I just have a nagging feeling that with your retries etc. you are duplicating what TCP would do for you, and if this was so much better why haven't I heard about this UDP approach as a "thing" for multiple transfer?
You may be right in one situation. Not my area (I am strictly legal!), but how do these "torrent" streamer things work? I believe they download & upload files to/from your client, like in "batches" over time. Maybe that does simulate "efficiently distributing big files to/from many clients", does it use UDP, I don't know. Then again, I think they certainly are not about speed, more maybe about reducing total network traffic? UPDATE @jsulm has looked it up, and they use TCP, just multiple connections (between client and server for one file) and small packets. That might give you some food for thought... :)
-
@JonB , thank you, I will try. I'm using Sockets, is there a built in tried and tested method of transferring files over TCP that I can try?
Because without this I would implement the same protocol I've adopted for UDP but using TCP sockets.
@SPlatten Do not use UDP for transferring the data that can't be lost! @JonB is very right about you simulating TCP/IP control flow, don't do that. You will not get better results than using plain TCP/IP. While frame size of TCP/IP is smaller, it might actually be faster. The test I mentioned above, I'd see it using QTcpSocket with minimum control flow connected to signals of like connected(), stateChanged() and errorOccured(), plus monitoring (maybe) bytesToWrite() on the sending part and bytesAvailable() on the receiving end.
With the socket and QDataStream you can actually use
<<operators - and that's where the size of the data chunk I mentioned above comes in, you can actually output timings from QElapsedTimer on both ends to find the bottlenecks. Don't go into to deep level of implementation since that's where usually the unnecessary complications occur. Please try to use the highest implementation level possible and allow Qt to do its parts.Socket documentation (I know that you've read it already but just allow me to hang in here my reference point for clarity of the post): https://doc.qt.io/qt-5/qabstractsocket.html
Word on recalculating the CRC - I would not calculate those for the parts, since TCP/IP makes sure that for any given continuous transfer CRC of the parts/frames is sound and that the parts are ordered correctly. Calculate it once at the very end, once you've got the whole file.
-
@SPlatten
Putting aside the TCP / UDP discussion for a minute, the simplicity-driven speed of UDP is basically limited by your physical environment. The copy_calc speed is based on the assumption that the given bandwidth is fully available between your traffic endpoints. USB traffic (w/o USB HUBs!) is the only use case where this assumption always holds true. You are sending UDP broad- or multicasts across your network. The first bottle neck is your server's ethernet adapter. Your max outbound speed is the server's adapter speed / number of clients. Next bottle neck is your router/hub/switch, which comes with a number of different potential factors which will slow down your traffic:(1) Physical limitations
Depending how new and fast your router/hub/switch is(2) Configuration
Protocols, ports and endpoints may or may not be prioritized.
Every package must be inspected to determine its priority.
So even if your UDP traffic enjoys top priority, the inspection itself slows it down.(3) Logical factors
How connections are rendered between two endpoints largely depends on internal algorithms in your router/hub/switch hardware. Some switches have UDP optimizers which accept basically a package and send it to all multi-/broadcast addresses at the same time. Others buffer and serialize.
The speed of optimized UDP fully unfolds only if it is truly unidirectional. Since your environment works with bi-directional features (basically a limited TCP re-implementation), this may slow down your hardware performance additionally. Since UDP optimizers take the risk of package loss for the sake of speed, your hardware configuration may be a trigger for your bi-directional features so you end up in a vicious circle: UDP-optimization => package loss => bi-directional features => more package loss => more bi-directional features etc.Having said that: While your case is certainly interesting, I am not surprised that you don't get the speed you want. Debugging your bi-directional features (e.g. requests to resend missing packages) will give you an idea where the troublemaker is to be located. Your hardware is most likely punishing you for using TCP like features on UDP. Moreover, you loose TCP based optimization features which your hardware may even provide.
This is probably not the solution you expected, however, I hope it sheds some light on the matter.
C++ and Python walk into a bar. C++ reuses the first glass.
-
@SPlatten
Putting aside the TCP / UDP discussion for a minute, the simplicity-driven speed of UDP is basically limited by your physical environment. The copy_calc speed is based on the assumption that the given bandwidth is fully available between your traffic endpoints. USB traffic (w/o USB HUBs!) is the only use case where this assumption always holds true. You are sending UDP broad- or multicasts across your network. The first bottle neck is your server's ethernet adapter. Your max outbound speed is the server's adapter speed / number of clients. Next bottle neck is your router/hub/switch, which comes with a number of different potential factors which will slow down your traffic:(1) Physical limitations
Depending how new and fast your router/hub/switch is(2) Configuration
Protocols, ports and endpoints may or may not be prioritized.
Every package must be inspected to determine its priority.
So even if your UDP traffic enjoys top priority, the inspection itself slows it down.(3) Logical factors
How connections are rendered between two endpoints largely depends on internal algorithms in your router/hub/switch hardware. Some switches have UDP optimizers which accept basically a package and send it to all multi-/broadcast addresses at the same time. Others buffer and serialize.
The speed of optimized UDP fully unfolds only if it is truly unidirectional. Since your environment works with bi-directional features (basically a limited TCP re-implementation), this may slow down your hardware performance additionally. Since UDP optimizers take the risk of package loss for the sake of speed, your hardware configuration may be a trigger for your bi-directional features so you end up in a vicious circle: UDP-optimization => package loss => bi-directional features => more package loss => more bi-directional features etc.Having said that: While your case is certainly interesting, I am not surprised that you don't get the speed you want. Debugging your bi-directional features (e.g. requests to resend missing packages) will give you an idea where the troublemaker is to be located. Your hardware is most likely punishing you for using TCP like features on UDP. Moreover, you loose TCP based optimization features which your hardware may even provide.
This is probably not the solution you expected, however, I hope it sheds some light on the matter.
@AxelVienna , thank you, I'm developing on a supplied laptop where presently the client and server are on the same system, this isn't how it will be when rolled out.
Kind Regards,
Sy -
@AxelVienna , thank you, I'm developing on a supplied laptop where presently the client and server are on the same system, this isn't how it will be when rolled out.
@SPlatten said in Transferring data...:
this isn't how it will be when rolled out.
I was going to ask about this earlier. One trouble I foresee is how you will know how your approach fares in another environment, particularly about required retries. With TCP you may not know the speed but you do know it will be reliable. With your UDP I don't know how you can anticipate its performance in a distributed environment.
-
@SPlatten said in Transferring data...:
this isn't how it will be when rolled out.
I was going to ask about this earlier. One trouble I foresee is how you will know how your approach fares in another environment, particularly about required retries. With TCP you may not know the speed but you do know it will be reliable. With your UDP I don't know how you can anticipate its performance in a distributed environment.
-
@JonB , I'm going to look into using QTCP now, is there any good quick start example I can look at or is it simply write the data to the Socket ?
@SPlatten From the documentation I posted above:
https://doc.qt.io/qt-5/qtnetwork-fortuneclient-example.html
https://doc.qt.io/qt-5/qtnetwork-blockingfortuneclient-example.html
-
@JonB , I'm going to look into using QTCP now, is there any good quick start example I can look at or is it simply write the data to the Socket ?
-
@SPlatten From the documentation I posted above:
https://doc.qt.io/qt-5/qtnetwork-fortuneclient-example.html
https://doc.qt.io/qt-5/qtnetwork-blockingfortuneclient-example.html
-
-
@AxelVienna , thank you, I'm developing on a supplied laptop where presently the client and server are on the same system, this isn't how it will be when rolled out.
@SPlatten said in Transferring data...:
@AxelVienna , thank you, I'm developing on a supplied laptop where presently the client and server are on the same system, this isn't how it will be when rolled out.
With client and server hosted on the same machine, the vicious circle I described is still likely to happen: Blocking code or performance issues on the client side, will cause package loss.
-
I have been working on a C++ class to transfer a large amount of data to a clients, presently I'm working on the server and client being on the same system which is a 100MB/s network. The file is 1.17GB, what is the best way to transfer this data as presently it takes around 20 minutes to transfer.
I've seen various online calculators which give 1 minute 45 seconds as the transfer time.
https://techinternets.com/copy_calc
I can't see how this can be.
@SPlatten said in Transferring data...:
presently it takes around 20 minutes to transfer.
Your implementation must be rather dubious, I'd say. A 10/100 network (the typical cat5(e) UTP without much noise on the channel) will easily give you ~10 MB/s transfer speed (in reality, not theoretical) over plain TCP, which should sum up to just about under 2 minutes.
Read and abide by the Qt Code of Conduct
-
@SPlatten said in Transferring data...:
presently it takes around 20 minutes to transfer.
Your implementation must be rather dubious, I'd say. A 10/100 network (the typical cat5(e) UTP without much noise on the channel) will easily give you ~10 MB/s transfer speed (in reality, not theoretical) over plain TCP, which should sum up to just about under 2 minutes.
@kshegunov said in Transferring data...:
under 2 minutes
Yes, it looks the expected value... See this calculator for instance.
1.7 GB over 100 Mbps with 10% overhead -> 2m 33secInstead of developing your own file transfer protocol over UDP, have you consider TFTP for example?
It's quite used for initial remote file transfer/configuration of network devices (cable modems, IP phones, etc.)
I guess you can even have already implemented TFTP servers for free. -
@jsulm , all I can go on is the data thats in front of me. TCP packets can send 1.5K, UDP packets can send 64K, what isn't clear?
@SPlatten said in Transferring data...:
@jsulm , all I can go on is the data thats in front of me. TCP packets can send 1.5K, UDP packets can send 64K, what isn't clear?
What isn't clear is why you keep making this claim.
TCP uses a window for flow control rather than a packet size, because a TCP stream represents a sequence of bytes. An implementation may send that sequence via one or more IP packets. The window field in the TCP header is 16 bits, allowing the sender to advertise 64 kilobytes of available space. Window scaling effectively extends it to 32 bits.
Asking a question about code? http://eel.is/iso-c++/testcase/
