|
I dont think you've made it clear exactely what your issue is with large files
The general approach is to
1. read n buffers of size x from file
2. transmit the 1..n buffers over the wire
3. re-assemble them in the correct order
obviously a huge buffer size (x) isnt efficient - have you tried sending n smaller buffers with a sequence tag and then reassembling them ?
'g'
|
|
|
|
|
To be honest, I am not sure what the problem is.
I think that I am not sending all n buffers, as you suggested because my file when it is recieved on the client side is always smaller than the original, hence I must not be sending it correctly.
Not quite sure which one of the techniques I used was best, or even whether I implememnted them correctly.
I am pretty much open to all suggestions as I am quite new to c#.
Thanks for the message! Appreciated.
|
|
|
|
|
|
Hi
Thanks for the links...have gone through some one of them already from a previous Google search.
One of the other links contains good explanations, and more links, which I checked too.
Still finding it difficult to recreate the same file on the other end. 
Thanks
|
|
|
|
|
If you have a method for sending files that are small them it should work for larger files, it will just take longer to transmit the file. It would help if you specified the file sizes for what you are calling small and large files.
All of the methods for sending data via TCP load/convert the data to byte[] and send that so avoid converting the file to Base64 as this just increases the amount of information that has to be sent over the network, which then takes longer to transmit.
|
|
|
|
|
The small file was 11kb, and the larger file was 1.5MB (not that large).
I initially used the same method as sending text (via byte streaming) to send the files, but as the files I tested got larger, it wouldnt work any longer, which has lead me to testing different methods of sending files.
|
|
|
|
|
I've sent files up to 5MB using something similar to;
Byte[] data = System.Text.Encoding.ASCII.GetBytes(message);
TcpClient client = new TcpClient(server, port);
NetworkStream stream = client.GetStream();
stream.Write(data, 0, data.Length);
client.Close();
It did take a while to transfer though of the order of 3-5 minutes.
For receiving the file look at the TcpListener Class[^]
|
|
|
|
|
Surely that would then mean converting to base64 first?
Thought that this might not be ideal or practical as files become larger.
But I will give it a go...
Thanks
|
|
|
|
|
You don't need to convert to Base64 first, just load your file into a byte[] and send it, then when receiving don't do any byte to string conversions either, just save the received bytes to a byte[] that keeps growing or save the bytes to a file on disk.
|
|
|
|
|
Sorry for being such a noob here...this is the code I used:
System.IO.FileStream inFile = new System.IO.FileStream(theCompressedFile, System.IO.FileMode.Open, System.IO.FileAccess.Read);
byte[] binaryData = new Byte[inFile.Length];
long bytesRead = inFile.Read(binaryData, 0, (int)inFile.Length);
inFile.Close();
Byte[] data = System.Text.Encoding.ASCII.GetBytes(binaryData);
serverStream.Write(data, 0, data.Length); client.Close();
"System.Text.Encoding.ASCII.GetBytes" won't accept binaryData (byte[]) as an input.
ClientSide:
byte[] message = new byte[4096];
int bytesRead = 0;
bytesRead = clientStream.Read(message, 0, 4096);
file = encoder.GetString(message, 0, bytesRead);
FileStream fs = new FileStream(theFileName, FileMode.Create, FileAccess.ReadWrite);
BinaryWriter bw = new BinaryWriter(fs);
bw.Write(file);
bw.Close();
Thanks for the help...and sorry if this is annoying. 
|
|
|
|
|
No problem change the send to;
serverStream.Write(binaryData, 0, binaryData.Length);
drop the data byte[], then try;
FileStream fs = new FileStream(theFileName, FileMode.Create, FileAccess.ReadWrite);
BinaryWriter bw = new BinaryWriter(fs);
byte[] message = new byte[4096];
int totalBytesRead = 0;
int bytesRead = clientStream.Read(message, totalBytesRead, 4096);
while (bytesRead > 0)
{
bw.Write(message, totalBytesRead, bytesRead);
message = new byte[4096];
totalBytesRead += bytesRead;
bytesRead = clientStream.Read(message, totalBytesRead, 4096);
}
bw.Close();
on the receive.
modified on Tuesday, April 6, 2010 8:54 AM
|
|
|
|
|
Woah, cool...that works.
For some reason, the files I test are slightly larger than the original, and won't open on the received client side due to corruption.
On days like these, just want to drop the PC and never touch it again...but I can't, I can't get away from the use of a PC. 
Thanks Again for all your efforts, I really appreciate it!
|
|
|
|
|
There shouldn't be any difference in the file size.
Also on the receive side you could drop the BinaryWriter and just write the bytes received direct to the FileStream.
FileStream fs = new FileStream(theFileName, FileMode.Create, FileAccess.Write);
byte[] message = new byte[4096];
int totalBytesRead = 0;
int bytesRead = clientStream.Read(message, totalBytesRead, 4096);
while (bytesRead > 0)
{
fs.Write(message, totalBytesRead, bytesRead);
message = new byte[4096];
totalBytesRead += bytesRead;
bytesRead = clientStream.Read(message, totalBytesRead, 4096);
}
fs.Flush();
fs.Close();
|
|
|
|
|
With this new method, an exception is thrown when reading from the clientStream
i.e. once in the while loop, it cannot read from clientStream
Exception: "Specified argument was out of the range of valid values. Parameter name: size"
|
|
|
|
|
Sorry late night  ;
FileStream fs = new FileStream(theFileName, FileMode.Create, FileAccess.Write);
byte[] message = new byte[4096];
int bytesReceived = 0;
int bytesRead = clientStream.Read(message, 0, 4096);
while (bytesRead > 0)
{
fs.Write(message, 0, bytesRead);
bytesReceived += bytesRead;
message = new byte[4096];
bytesRead = clientStream.Read(message, 0, 4096);
}
fs.Close();
|
|
|
|
|
Hi There
Was a long day for me yesterday too...I got it sorted just before leaving work yesterday.
Thanks again for all the help, I really appreciate it.
Enjoy the rest of ur day! 
|
|
|
|
|
Sorry, I'm not going to read such amount of unformatted code. Please use PRE tags to preserve code formatting.
|
|
|
|
|
Sorry about that, I only figured out how to do that now.
I have edited the question.
Thanks
|
|
|
|
|
Hi,
I have looked into your different approaches, and here are some comments:
1.
you should avoid reading all file data into memory at once; that is quite acceptable for small sizes, it becomes a bottleneck on large files though. Streams are the fundamental solution to this, as they offer a way to process data while it comes along, without ever needing all of it at once.
So your attempts #1 and #2 are out.
2.
you should also avoid base64 encoding, unless your communication channel is only working properly with printable ASCII characters; example: some serial communications (I mean over RS232C) only work properly with simple text, arbitrary byte values might upset the serial driver and/or modems involved in the communication.
base64 is both calculation intensive and it expands the data, as it uses one character to encode 6 bits (which means 3 bytes become 4 characters), something you want to avoid especially when the amount of data is large.
3.
and finally you should avoid copying all the data as your transmitter #2 does. It is expensive and does not bring you anything.
4.
your receiver #1 does not fit your transmitter #1 as it relies on a receiving buffer of fixed size (4096) whereas the sender uses a single, possibly huge, buffer. Now base64 will append up to 2 fill characters while encoding, and the decoder will not understand the data as it is chopped into 4096-byte packets, lacking the padding characters that should go with that.
5.
your third approach looks most promising, however you should keep it simple. There is no need to calculate the number of packets in advance, it is just adding complexity (and working against the streaming idea, as you are relying on total size, which implies all the data has to be present somewhere).
Simply make a loop reading some data (from the file) and sending that data (to the network), until there is no more data, at which point you close the output stream. Should work like a charm.
Your receiver #3 looks fine.
|
|
|
|
|
Thank you so much for the insight, will be sure to use that.
I have been working on solution with the help of Rod Kemp.
Basically just streaming the file across. Not certain about the efficiency of the code, but at this point, just getting something to work spot on is my aim.
Server: Sends
System.IO.FileStream inFile = new System.IO.FileStream(theCompressedFile, System.IO.FileMode.Open, System.IO.FileAccess.Read);
byte[] binaryData = new Byte[inFile.Length];
long bytesRead = inFile.Read(binaryData, 0, (int)inFile.Length);
inFile.Close();
serverStream.Write(binaryData, 0, binaryData.Length);
client.Close();
Client: Recieves
FileStream fs = new FileStream(theFileName, FileMode.Create, FileAccess.ReadWrite);
BinaryWriter bw = new BinaryWriter(fs);
byte[] theFile = new byte[1024];
while (clientStream.Read(theFile, 0, 1024) > 0)
{
bw.Write(theFile);
theFile = new byte[1024];
}
fs.Close();
bw.Close();
Thanks again for the response!
|
|
|
|
|
You're getting close to the solution, you still should use a smaller buffer in the transmitter; just implement a loop (as you did in the receiver), and best make sure your transmit buffer is not larger than your receive buffer.
In the end receiver and transmitter will look very similar, which shouldn't come as a surprise: they both transfer data from one stream to another (file->network and network->file).
|
|
|
|
|
Is this better:
Server:
System.IO.FileStream inFile = new System.IO.FileStream(theCompressedFile, System.IO.FileMode.Open, System.IO.FileAccess.Read);
byte[] binaryData = new Byte[inFile.Length];
long bytesRead = inFile.Read(binaryData, 0, (int)inFile.Length);
inFile.Close();
int bytesSent = 0;
while (bytesSent < binaryData.Length)
{
serverStream.Write(binaryData, 0, 1024);
bytesSent += 1024;
}
serverStream.Flush();
client.Close();
Client:
FileStream fs = new FileStream(theFileName, FileMode.Create, FileAccess.ReadWrite);
BinaryWriter bw = new BinaryWriter(fs);
byte[] theFile = new byte[1024];
while (clientStream.Read(theFile, 0, 1024) > 0)
{
bw.Write(theFile);
theFile = new byte[1024];
}
fs.Flush();
fs.Close();
bw.Flush();
bw.Close();
** When I remove the line of code indicated, the file is sent as is and can be opened on client side. However, because I removed this, there is no end to the stream (I think) and the client keeps waiting.
But when this line is there, the file is not sent exactly as is for some reason (it is always slightly larger - sometimes 1kb) and the file is corrupted and won't open.
Can you please help?
|
|
|
|
|
You still have byte[] binaryData = new Byte[inFile.Length]; which isn't necessary at all; you are NOT interested in file size!
Try this:
System.IO.FileStream inFile = new System.IO.FileStream(theCompressedFile, System.IO.FileMode.Open, System.IO.FileAccess.Read);
byte[] buffer = new byte[4096];
for(;;) {
int byteCount=inFile.Read(buffer, 0, buffer.Length);
if (byteCount==0) break;
serverStream.Write(buffer, 0, byteCount);
}
serverStream.Close();
inFile.Close();
see, no file.Length.
also, no Flush() as Close() implies Flush().
FileStream fs = new FileStream(theFileName, FileMode.Create, FileAccess.ReadWrite);
BinaryWriter bw = new BinaryWriter(fs);
byte[] buffer= new byte[4096];
for(;;) {
int byteCount=clientStream.Read(buffer, 0, buffer.Length);
if (byteCount==0) break;
bw.Write(buffer, 0, byteCount);
}
bw.Close();
fs.Close();
Now see how similar they are!
FYI: it still needs some Dispose() calls, and then some of it can be made more elegant by using a using construct.
|
|
|
|
|
You have saved my day!
Thanks so much.
Every time I think that I have learnt sufficiently, there are things that catch me out (especially having to learn a new language).
Thanks for the lesson in coding, appreciate it.
Google can't replace good experience of a human! 
|
|
|
|
|
show the good answer in this page
[enter link description here][1]
[1]: http:
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
