Click here to Skip to main content
15,867,995 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
I have 8GB of RAM on my machine, with a significant amount free.
Trying to read 300MB file into a System String gives me an Out of Memory error. Why? Is there a hotfix? I remember a similar problem with emailing attachments over a moderate size with .NET.

MSIL
System.IO.StreamReader myFile = new System.IO.StreamReader("C:\\Z1_struct.xml");
System.String File_As_String = myFile.ReadToEnd();
myFile.Close();
Posted

300 MB XML likely encoding UTF-8 -> 600 MB UTF-16 string

ReadToEnd uses a StringBuilder internally, it returns sb.ToString().
That alone results in at least 1200 MB being allocated at one point. There is also a bit of buffering and the memory used to convert from UTF-8 to UTF-16.

If you are doing this in ASP.Net the application pool has usually a default memory limit - usually resulting in the pool being recycled.

If you are running under a 32-bit os those 8 GB will not help at all, and even if you are running under a 64-bit os, you have to make sure that the process is executed as a 64-bit process. Running it under Visual Studio defaults to 32-bit execution ...

I would try another method of processing a 300 MB xml file.

A working SAX based approach would probably be better - I haven't tested this, but it is perhaps worth a try:
Towards a Declarative SAX Framework : Part 1 - A Simple SAX-to-C#-Mapping[^]

Best regards
Espen Harlinn
 
Share this answer
 
v2
Comments
Abhinav S 20-Jul-11 13:16pm    
Fair answer! My 5.
Espen Harlinn 20-Jul-11 13:20pm    
Thank you, Abhinav!
T2102 20-Jul-11 14:12pm    
Thanks for now, I am downloading data in a series of much smaller csv files. It will take me a few hours, but then I avoid the xml problem. I an in 64 bit OS with Pro version of Visual Studio 2010.
Sergey Alexandrovich Kryukov 21-Jul-11 11:11am    
Good answer, my 5.
--SA
Espen Harlinn 21-Jul-11 11:58am    
Thank you, Sergey!
This[^] could help you.
 
Share this answer
 
Comments
Espen Harlinn 20-Jul-11 13:01pm    
ReadToEnd() does something like this:
StringBuilder sb = new StringBuilder(charLen - charPos);
do {
sb.Append(charBuffer, charPos, charLen - charPos);
charPos = charLen;
ReadBuffer();
} while (charLen > 0);
return sb.ToString();

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900