|
I am creating an application that copies files from one location to another sometimes locally and sometimes over a network.
I am using buffered streams to accomplish desired functionality and it is working as expected but I am unsure on what size my buffer should be. I have experimented with different sizes and I have had varied results.
Recap:
What is an optimal size for the buffer when copying files?
Note:
File size varies from 1 byte to 1.5Gb never more.
When the game is over the Pawn and the King go into the same box.
|
|
|
|
|
MicealG wrote: What is an optimal size for the buffer when copying files?
Depends. Optimal size for reading/writing to/from Disk is based on the OS and hardware I/O specifications. Optimal size for transporting network packets is based on network protocols and hardware however the network stack you are using, like TCP/IP does that buffering for you so you don't deal with that issue.
led mike
|
|
|
|
|
Thanks for the reply.
Is there any way I could work out an optimal buffer size to use depending on the file size?
When the game is over the Pawn and the King go into the same box.
|
|
|
|
|
MicealG wrote: Is there any way I could work out an optimal buffer size to use depending on the file size?
No. It has nothing to do with the file size. The networking software/hardware determines the optimal buffer size. For Disk I/O it is dependent on the Hardware / OS File System combination. Always use an exponent of 2 since the systems do. When testing Disks I have found that 2k or 4k will prove out as the optimal size. I have never looked at network transfers but if you are doing Disk I/O it seems that should be your determining factor.
led mike
|
|
|
|
|
The streams are buffered by the file system and the network layer, so a buffer smaller than one cluster or smaller than one network packet, doesn't have any buffering effect at all.
Other than that, the buffer size shouldn't matter that much.
Have you tried using two buffers and asynchronous reading and writing, so that you read to one buffer while writing from the other?
Despite everything, the person most likely to be fooling you next is yourself.
|
|
|
|
|
Guffa wrote: Other than that, the buffer size shouldn't matter that much.
Not true for file I/O. Play around with it and see. 
led mike
|
|
|
|
|
What the others have said is true - it depends on many things.
However - even though an optimal buffer size can only be determined by implementing a complex (and thus error-prone - just look at Vista) self-tuning algorithm of some kind, I guess you're just asking for an arbitrary but "reasonable" value. Correct?
I've used 64K (65536) bytes for the last five years or so. Equals the maximum stripe size of popular RAID controllers, the maximum block size of popular tape devices and the default maximum window size of the Windows TCP stack (pre-RSS).
The world is changing, though, so I'm considering reconsidering that. Whenever I have a budget allowing finer points, I make buffer sizes configurable (still with a 64K default, though).
On occasion I've experimented with 256K or 512K and it seems to work OK as well. Being old enough to remember computers with less RAM than that, it always gives me an eerie feeling - but given the RAM in computers of today, 512KB is really not a lot of memory to set aside for a buffer.
In any case: Like someone said, powers of two still feel like the right thing to use. Actually moot when it comes to network traffic (TCP couldn't care less), but disks still stick to that for block sizes, allocation units and stripe sizes.
Later,
--
Peter
|
|
|
|
|
PeterTheSwede wrote: On occasion I've experimented with 256K or 512K and it seems to work OK as well.
On consideration here is that an object larger than 85 kB will be allocated on the large objects heap instead of the regular heap. The large objects heap is not compacted in the way that the regular heap is, so the application might not be able to return that memory to the system when you have released the buffer.
Not a very serious concern as it's not very much memory, but it might be good to know.
PeterTheSwede wrote: Being old enough to remember computers with less RAM than that
Those were the days... My first programming was done on a Sinclair ZX81 with 4 kB of memory. My first own computer was an Atari 600XL with 16 kb of memory. After that I got an Atari 130XE with massive 128 kB of memory. It had a special chip for swapping memory banks, as the CPU wasn't able to address more than 64 KB of memory... 
Despite everything, the person most likely to be fooling you next is yourself.
|
|
|
|
|
Hi,
On the "large objects heap": I have managed to miss that somehow. Thanks a lot for the enlightenment! Will keep it in mind for the future.
On RAM sizes: My first programming was done on a Norsk Data ND-100 minicomputer with 512KB RAM (IIRC) and an 80MB disk running the SINTRAN-III OS. It had an ND-50 number-cruncher for a side-car (same one that did the F-16 simulations a few years earlier). We were approximately 200 student groups writing, compiling and testing BASIC and Pascal programs on it (50-60 groups at a time, on DEC VT-100 and ND terminals using current loop cables from the terminal room to the computer room). I ended up being an assistant teacher there for a few years... we're talking 1981 or so.
My first experience with a PC:
- Hmmm... wonder if it's like Unix: C:\>ls
- No, let's try SINTRAN III syntax: C:\>LIST-FILES
- Nope, not that. Let's do VMS then: C:\>DIR
- WOW, it worked. Let's run it in the background: C:\>BG DIR
- Nope. Maybe some Unix influence after all: C:\>DIR &
- Not that. Now let's see... where's the manual. WHAT?!?! No multi-tasking?!?!? Are you f*ing serious?!?!
And I had just promised writing actual software for the thing... I have never felt so lost in my life...
Later,
--
Peter
|
|
|
|
|
Thank you for the response, it is very helpful, many many thanks.
When the game is over the Pawn and the King go into the same box.
|
|
|
|
|
I have been looking at http://msdn.microsoft.com/en-us/library/aa288453(VS.71).aspx trying to figure this out but I just seem to get it.
I have a function that I would like it to return the information in an array or something like it, but I can't figure out how to add the data. The only way I can think to explain it is more like XML. One function will return different items from a XML feed, and the other will return one record from an XML feed with the information
Example for multiple records:
[Message]
[ID]44562[/ID]
[/Message]
[Message]
[ID]44562[/ID]
[/Message]
[Message]
[ID]44562[/ID]
[/Message]
Example for single records:
[Message]
[ID]44562[/ID]
[From]Scott[/From]
[To]Scott[/To]
[Subject]Test[/Subject]
[/Message]
|
|
|
|
|
Brad Wick wrote: I have a function that I would like it to return the information in an array or something like it, but I can't figure out how to add the data.
If you can't figure out how to add data to an array then I suggest you get a beginners book. Look for book reviews here on the site.
led mike
|
|
|
|
|
Adding items to an array I can do, but not a Multidimensional Array. Some people do not read books, but search the internet and learn from each time they tackle something. I have been playing around in C# for some time and can do pretty much anything that is needed, but trying to learn some other items. Why dont you just respond to every post "go buy a book" that is so much more helpful then actually helping someone out.
|
|
|
|
|
Brad Wick wrote: Some people do not read books, but search the internet and learn from each time they tackle something.
Ok, good luck.
led mike
|
|
|
|
|
Just look at your XML examples (hierarchy). It's just a name/value pair array.
|
|
|
|
|
So you think it would be better to store the information in XML to pass between different functions/objects?
|
|
|
|
|
No. I don't know your application so it's hard to say what's better. What I was getting at was that if you're confused about multi-dimensional arrays, just look at your XML to visualize what it would look like.
In 2.0 forward, you could use a generic list, but I think you said you're using 1.0+
[Edit] No, the 1.0 was from another question. Sorry.
|
|
|
|
|
The way I normally handle this is to create a class (or struct if more appropriate) for a single item, then create a class that is a generic list of single items.
e.g.
public class MyMessage
{
}
using System.Collections.Generic;
public class MyMessageCollection : List<MyMessage>
{
public MyMessage FindItemFromID(int id)
{
MyMessage found = this.Find(c => c.ID == id);
return found != null ? found : new MyMessage();
}
}
Dave
|
|
|
|
|
Note: This method is new in the .NET Framework version 2.0. and I am using .NET Framework version 2.0.50727.
http://msdn.microsoft.com/en-us/library/1ffy6686(VS.80).aspx
Array.Resize(ref m_yData, m_points);
I get this compile error message "'System.Array' does not contain a definition for 'Resize'".....What am I missing or doing wrong?
Thanks
|
|
|
|
|
C++NewBe wrote: What am I missing or doing wrong?
It's a Generic method so you need a type initializer. The example code in the documentation appears to be wrong also.
int[] ints = { 1, 2, 3 };
Array.Resize<int>(ref ints, 5);
led mike
|
|
|
|
|
You don't need to specify the generic type for a generic method call - you can do this, but in most cases the compiler is smart enough to figure it out. Explicitly specifying the generic type in a method call is only necessary in ambiguous cases where the compiler can't compute the type.
e.g.,
Array.Resize(ref ints, 5);
will work just fine.
David Anton
http://www.tangiblesoftwaresolutions.com
C++ to C# Converter
C++ to VB Converter
C++ to Java Converter
VB to Java Converter
Java to VB & C# Converter
Instant C#: VB to C# converter
Instant VB: C# to VB converter
Instant C++: convert VB, C#, or Java to C++/CLI
|
|
|
|
|
David Anton wrote: You don't need to specify the generic type for a generic method call - you can do this, but in most cases the compiler is smart enough to figure it out. Explicitly specifying the generic type in a method call is only necessary in ambiguous cases where the compiler can't compute the type.
Sweet. Missed that one and it makes a huge difference. ![Beer | [beer]](https://codeproject.global.ssl.fastly.net/script/Forums/Images/beer.gif)
led mike
|
|
|
|
|
Use a collection instead of an arraylist. Then you don't have to resize manually.
"Why don't you tie a kerosene-soaked rag around your ankles so the ants won't climb up and eat your candy ass..." - Dale Earnhardt, 1997
-----
"...the staggering layers of obscenity in your statement make it a work of art on so many levels." - Jason Jystad, 10/26/2001
|
|
|
|
|
I use sql 2005 along with DAAB using asp.net (c#.Net)
But when i attempt to run the app, i got this error " Instance Failure".
How do I resolve this and what could be responsible.
Thanks
|
|
|
|
|
Banjo Ayorinde wrote: How do I resolve this
I would start by debugging
led mike
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
