|
I am writing the contents of a huge DataTable (60000 rows, 15 columns) into a text file.
Method 1: I am appending the entire content of the DataTable to a StringBuilder and then writing it to a text file in one go.
RUM TIME: 25 mins
Method 2: I am writing one row at a time to the file. ( I am still using a StringBuilder, but I am clearing the String builder after every row)
RUN TIME: 30 secs
Why this huge difference?
I have a theory, but i wouldn't want to influence your thoughts.
|
|
|
|
|
rsunilbabu wrote: Why this huge difference?
I don't know, but if I wanted to know (note: "know" does not mean "guess"), I would use a profiler to discover why.
|
|
|
|
|
IO is always faster when writing big packages instead of many small ones. I assume you use a StreamWriter to write the file. Try increasing its buffer to one megabyte (default should be one kilobyte) and method 2 should become faster.
|
|
|
|
|
Method 2 is faster....the second method took "30 seconds" only....
|
|
|
|
|
 I should read more carefully. I read sec on both methods...
Theory A: You are using some kind of threading in method 1. As far as I remember StringBuilder is allergic against that.
Theory B: Physical memory gets low and thus it is swapped to hdd.
Theory C (my favourite): Every now and then the StringBuilder allocates more space. Everytime it does this it must copy the content it had before to the new memory space (like an Arraylist). This could be easily tested by giving the StringBuilder an appropriate capacity into the constructor.
Theory D: I don't have a f***ing clue 
Robert
|
|
|
|
|
My theory is close to your "Theory C".
The process was slowing down as I add more and more rows. May be larger swaps?
But hard to figure out a 24 minute difference
I am still wondering about Theory A. I am doing this in a timer_elapsed event in a windows service. That may be a thread in the background!?! It may be adding to the time as well.
How the hell I am going to explain to people that the program which is was running for 25 mins now only needs 25 secs. I don't have to......but its embarrassing.
|
|
|
|
|
rsunilbabu wrote: How the hell I am going to explain to people that the program which is was running for 25 mins now only needs 25 secs.
Just tell them you are a genius but didn't want to show off with this right from the beginning... 
Robert
|
|
|
|
|
Set the StringBuilders Capacity to the estimated filesize when u construct it. Or better, use the StringWriter.
|
|
|
|
|
The reason that the string builder gets slow when you put a lot of data in it, is that every time that it's capacity has to be increased, another buffer is allocated that is twice the size of the current buffer, and all the data is copied to the new buffer.
Method 3: Just write it all directly to a stream.
File output is buffered on at least two levels, so there is no reason for you to add another level of buffering. A FileStream uses a buffer so that it doesn't have to call the system for every byte written, and the system uses a buffer before writing anything to disk, as data can only be written in chunks of a cluster at a time (usually between one and eight kb). Then there is a write cache that works more or less as another buffer before the data is finally written to disk.
---
single minded; short sighted; long gone;
|
|
|
|
|
As leppie already indicated, StringBuilder.Capacity is the issue.
If you don't take care, a StringBuilder will allocate some buffer, and when it comes
to exceed its capacity, it will allocate a new buffer and copy everything.
IIRC it doubles the size every time.
The statement "use StringBuilder to speed up string operations" often does not mention
"and allocate ample capacity".
Writing smaller amounts to file is slower in theory, but using a stream should take care of
that based on its own buffering. (Warning: writing bytes one at a time would be much
slower because of the overhead involved in calling all the methods!).
So I would not doubt it a second, and write one row at a time. It simply makes no sense
to spend a lot of memory to collect things that do not need to be collected at all.
Luc Pattyn [Forum Guidelines] [My Articles]
this weeks tips:
- make Visual display line numbers: Tools/Options/TextEditor/...
- show exceptions with ToString() to see all information
- before you ask a question here, search CodeProject, then Google
|
|
|
|
|
Yes. But I don't see the point in using the StringBuilder at all, just write the darn data to the file.
|
|
|
|
|
I agree an SB does not bring anything here.
I tried to explain the difference between the two proposals though.
Luc Pattyn [Forum Guidelines] [My Articles]
this weeks tips:
- make Visual display line numbers: Tools/Options/TextEditor/...
- show exceptions with ToString() to see all information
- before you ask a question here, search CodeProject, then Google
|
|
|
|
|
Yes, and as long as he's clearing the StringBuilder (that's already been extended) rather than instantiating a new one each too.
|
|
|
|
|
Luc Pattyn wrote: The statement "use StringBuilder to speed up string operations" often does not mention
"and allocate ample capacity".
Probably because the StringBuilder performs pretty well even if you don't set the capacity.
25 minutes may seem a lot, but it's not that bad considering the amount of data. Based on the amount of data and the time it took for the StringBulder, some approximate calculations tells me that doing the same using string concatenation would need around 11 years to run.
---
single minded; short sighted; long gone;
|
|
|
|
|
Guffa wrote: 25 minutes may seem a lot
it is a lot: it is much more than one would expect, independent of the approach, given the
simple job of writing a table to a file.
Guffa wrote: string concatenation would need around 11 years to run
hope your PC and power company will run that long without failure...
Luc Pattyn [Forum Guidelines] [My Articles]
this weeks tips:
- make Visual display line numbers: Tools/Options/TextEditor/...
- show exceptions with ToString() to see all information
- before you ask a question here, search CodeProject, then Google
|
|
|
|
|
Luc Pattyn wrote: hope your PC and power company will run that long without failure...
Well, you could build some protection into the program. If it saves the string to a temporary file every now and then, it could continue from the last save point in case of a failure.
---
single minded; short sighted; long gone;
|
|
|
|
|
On the other hand, there may be better ways[^] to spend idle CPU cycles...
Luc Pattyn [Forum Guidelines] [My Articles]
this weeks tips:
- make Visual display line numbers: Tools/Options/TextEditor/...
- show exceptions with ToString() to see all information
- before you ask a question here, search CodeProject, then Google
|
|
|
|
|
private void checkBox1_CheckedChanged(object sender, EventArgs e)
{
TextBox tr2 = new TextBox();
tr2.Location = new Point(400, 163);
this.Controls.Add(tr2);
TextBox tr1 = new TextBox();
tr1.Location = new Point(495, 163);
this.Controls.Add(tr1);
textBox1.Text = Convert.ToInt32(tr2.Text) * Convert.ToInt32(tr1.Text) ;
}
where is the error that??
|
|
|
|
|
The error comes clearly from the last line. You try to convert the text of newly created TextBoxes to a number. As you haven't assigned anything to them they will contain an empty string. What value would you expect them to have?!?
|
|
|
|
|
Where do you set tr1.Text and tr2.Text? Based on the code you provided, they will both be an empty string.
|
|
|
|
|
Hello,
I'm trying to make a C# program that, in part, sends and receives emails. I'm using Gmail as the account, and can send emails from this program that would show as if I'd sent it from my Gmail account because it sends a command to Gmail containing my name and password, the email itself, who it goes to, etc.
That's fine, but the other half of the transfer is what's got me stumped--how to get the emails off that same Gmail account and show them on the screen.
Does anybody have any experience with this that can guide me in the right direction?
Thanks for your time,
Michael Fritzius
|
|
|
|
|
You need to use POP3 or IMAP protocols. Check out here[^] and here[^]. You will also need to enable POP3/IMAP access in your gmail account (through the gmail interface under settings).
Take care,
Tom
-----------------------------------------------
Check out my blog at http://tjoe.wordpress.com
|
|
|
|
|
Thanks! I'm glad to see that one of those links was visited already so I guess I was on the right track. Those links were a ton of help.
Thanks again!
Michael Fritzius
|
|
|
|
|
Hi mates...i try to find a way to creating 2 or more texbox with 1 button for example if the user click the button i wanna create 2 or more textbox help plssss.
|
|
|
|
|
Write the following code in the event handler for the button:
TextBox textBox1 = new TextBox();
textBox1.Location = new Point(10, 10);
TextBox textBox2 = new TextBox();
textBox1.Location = new Point(10, 20);
this.Controls.Add(textBox1);
this.Controls.Add(textBox2);
That's it. You can of course set the other properties for the textboxes as well.
Virtual1ty
"Any fool can learn from his own mistakes, but a wise man learns from mistakes of others"
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
