Networking c++ TCP BOOST stream data limit

Question

0.00/5 (No votes)

See more:

Hi,
I have a question about tcp data streams in c++. I understand that since a 32-bit integer is used to count the number of bytes sent in the stream there must be a limit of sending 2^32 bytes (4 GB) per stream, or am I wrong?

Is this a problem one must consider when using e.g. boost::asio or is the implementation managing this itself. I'm writing an application which will stream data of sizes possible lager than 4GB so I started to thinking if I should split the data into some several streams if it's size exceeds 4GB. Note, this is not for performance.

**edit**
The TCP protocol itself does contain an integer in order to count the number of bits sent, from what I understand that limits the underlying TCP stream itself from sending more data than 4GB. So I wonder if the implementation of ASIO has solved this issue behind the scenes or if you should keep track of it yourself.

The data isn't stored in memory for very long, so not being able to address it in memory isn't a problem. The data is also "generated" in small chunks so that's neither a problem for the sender.
********

Regards
WaZoX

Posted 7-Feb-13 6:19am

WaZoX

Updated 7-Feb-13 8:59am

v2

Add a Solution

2 solutions

Solution 1

On a 32 bit system, an int is 32 bits and any 'size' value will be determined by this as an upper limit (actually an unsigned int, not a signed int) but you will run into other limits far sooner than that. The 32 bit int (unsigned) also defines address space, and you don't have all of that available to you to store your data if the data needs to be in memory.

Since you are talking about streams, I assume that you do not want to put all of the data in memory at the same time.

The simplest way to count bytes in the stream is to use a 64 bit int. This should give you many years of countability (did I just invent a new word?).

Posted 7-Feb-13 7:01am

H.Brydon

Comments

WaZoX 7-Feb-13 14:57pm

I'm not sure you understood my question correctly? The TCP protocol itself does contain an integer in order to count the number of bits sent, from what I understand that limits the underlying TCP stream itself from sending more data than 4GB. So I wonder if the implementation of ASIO has solved this issue behind the scenes or if you should keep track of it yourself.

The data isn't stored in memory for very long, so not being able to address it in memory isn't a problem. The data is also "generated" in small chunks so that's neither a problem for the sender. I should have been more clear about that, thanks for pointing that out and for your answer.

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Matthew Faithfull · Accepted Answer · 2013-02-07T14:17:00

I wouldn't worry about TCP, that can take care of larger streams than 4GB. It does so every time I download a new Linux Distro these days, let alone when I'm playing with a few hundred hours of video footage.
Asio is another question but I seriously doubt if you'll break it with anything like that. The only way to be 100% sure though would be to try it because Asio is complex and there's more than one way to use it so even asking the Asio devs might not give you a precise answer.
To be honest I wouldn't worry about it at all unless you're already seeing a problem. The last time I went hunting for a bug in Asio it nearly melted my brain and the fault turned out to be in the embedded TCP implementation of the source the stream was coming from. I've used Asio indirectly to process ~1TB of TCP data in 12 streams that were continuous over several days while simultaneously wiresharking the whole mess and it stood up.