.NET (Core and Framework)

23-Aug-05 14:37

Hello,

I have a PC with 2GB RAM, Windows XP and .NET 1.1 framework. My .NET application has huge memory requirements.

It crashes (ignoring my try/catch) because of an "unhandled out of memory exception", when it uses approx. 2GB RAM of pure RAM memory. (says the Windows Taks Manager, Performance, Physical Memory).

As I did set Windows XP (Control Panel - System - Advanced - Advanced - Virtual Memory - 2047 MB) to use 2extra GBRAM of virtual memory on my hard disk, it looks like .NET does not follow Windows, and refuses to use the extra virtual memory.

(1) Is there some kind of default (Windows XP or .NET) that I should know of, that will allow my application to use more memory than the physical 2GB RAM ?

(2) Is there any way to programmatically tell .NET that I will use virtual memory, instead of physical memory?

--Max

Steve Pullan23-Aug-05 14:54

Steve Pullan

23-Aug-05 14:54

Max,

There should be no practical difference between virtual and real memory usage. What you are probably seeing is that even though there is a 4GB programming address space in XP, only 2GB is ever available to any application - the other 2GB is for the o/s.

You can change this with a startup option (I think it is /3GB) which will increase the address space available to non-o/s programs to 3GB and reduce the o/s address space accordingly to 1GB. You'll need to look this up on Google/Microsoft etc for the exact syntax and further info.

Having said that, I would be attempting to use much less memory by storing data on disk in a database etc. Maybe your application must use memory, maybe not? It's worth considering anyway.

...Steve

Silberztein23-Aug-05 17:24

23-Aug-05 17:24

Steve,

I used the /3GB startup option to launch Windows XP, but the "Out of Memory" exception fires exactly at the same time, i.e. when the program uses 2GB RAM Physical Memory (as told by Windows Task Manager / Performance / Physical Memory).

Is there anything else I have to do to get more than 2GB RAM of data under .NET ?

Max

Steve Pullan23-Aug-05 18:27

Steve Pullan

23-Aug-05 18:27

Max,

I found a hit via Google on this problem...

Google is my friend

Basically you may be hitting a threshold limit for the ArrayList structure so even if you give it more memory there is no guarantee that it will use it all.

I strongly suggest that you look at virtualizing this data because even if you do somehow happen to get it running with 3GB memory, you may find that just one small change to the input will push the limits again.

Another approach may be to limit the size of each list entry by compressing each entry so that the ArrayListitself uses less memory.

...Steve

Silberztein24-Aug-05 7:51

24-Aug-05 7:51

Steve,

Do you think that even if I get a Windows 64 machine with 16GBRAM or more, I will have access only to 2GB RAM data for each .NET object(ArrayList, Hashtable, etc.) ?

What about pure arrays: are they limited too to the int32 limitation?

Will .NET 2.0 fix this ?

Now for my problem: actually I did not realize how expensive using ArrayList is...

Each of the 15 million+ State are very small; probably 90% have less than 2 outgoing transitions.

I replaced the embedded ArrayList {char,int,char,int...} with two synch pure arrays char[] and int[], and now the whole thing gets stored in approx. 600 MB, which is fine (for now).

Thanks
Max

Dave Kreskowiak23-Aug-05 14:58

23-Aug-05 14:58

There's nothing "tunable" or configurable as far as the .NET memory manager is concerned.

Also, running the system out of handles can also cause the same problem. If you'r not freeing an unmanaged resource or not Disposing of some types of objects properly, you'll run into the same problem.

What are you using that's taking up all this memory?

RageInTheMachine9532
"...a pungent, ghastly, stinky piece of cheese!" -- The Roaming Gnome

Silberztein23-Aug-05 15:19

23-Aug-05 15:19

I have a Hungarian dictionary (several dozen millions of word forms) that I need to store into a Search tree.

The search tree is basically an ArrayList of states; each state has the following fields:

public class DState
{
ArrayList CharsToDests;
int infonb;
int canonical;
(... and then the methods)
}

in which CharsToDests contains alternatively characters, and their corresponding outgoing state index, e.g.:

('a',23,'b',3456,'c',1200023,'d',57)

For some reason I don't understand, my ANTS profiler says each state consumes 20 bytes (I would have thought 4x3 = 12).

BTW, do you know how much memory such an ArrayList takes? would it be better to use two synchronized type-consistent ArrayList, e.g.:

ArrayList Characters;
ArrayList OutgoingStateNb;

or something else (I cannot use an array because these lists grow: I just need a list of chars and a list of int).

Anyways, after having stored 5 million words,
that gives me approx. 50 million states, with the connections in the Arraylist => I get approx. 2GBRAM and then the Out of Memory exception (that I cannot catch).

--Max

Steve Pullan23-Aug-05 15:31

Steve Pullan

23-Aug-05 15:31

Sounds like you need to use a database rather than in-storage arrays.

...Steve

Dave Kreskowiak24-Aug-05 6:10

24-Aug-05 6:10

Silberztein wrote:
('a',23,'b',3456,'c',1200023,'d',57)

For some reason I don't understand, my ANTS profiler says each state consumes 20 bytes (I would have thought 4x3 = 12).

Did you also account for the boundry padding between fields and array elements?

Also, an ArrayList, or any other array for that matter, can only be indexed by a 32 bit integer and, therefore, can only hold a maximum of Int32.MaxValue, which is about 2.147 billion, elements. Try and exceed this limit, and you'll get an OutOfMemory exception.

RageInTheMachine9532
"...a pungent, ghastly, stinky piece of cheese!" -- The Roaming Gnome

Silberztein24-Aug-05 8:00

24-Aug-05 8:00

> Also, an ArrayList, or any other array for that
> matter, can only be indexed by a 32 bit integer
> and, therefore, can only hold a maximum of
> Int32.MaxValue, which is about 2.147 billion,
> elements. Try and exceed this limit, and you'll
> get an OutOfMemory exception.

is this limit also true for Windows 64?
for .NET 2.0 ?

--Max

Dave Kreskowiak24-Aug-05 11:22

24-Aug-05 11:22

Don't know. I would suspect the indexer limit would be the same size as the processor data width.

RageInTheMachine9532
"...a pungent, ghastly, stinky piece of cheese!" -- The Roaming Gnome

Silberztein24-Aug-05 7:57

System.Diagonastics.Process.GetProcesses throws error if the user does not have admin rights.

24-Aug-05 7:57

> What are you using that's taking up all this memory?

I was using an array of 15 million states:

public class State
{
ArrayList transitions; // {char,state,char,state...}

int infonb;
int canonicalstate;
}

I did not realize how expensive the ArrayList is. Since most states have less than 2 outgoing transitions, I replaced the arraylist with two synch pure arrays:

public class State
{
char[] char;
int[] targetstate;
int infonb;
int canonicalstate;
}

and that came to 1/3 of the previous memory usage.

--Max

manivannan.p23-Aug-05 5:16

manivannan.p

23-Aug-05 5:16

Hi,

I get error
"Couldn't get process information from remote machine" when i try to run Process.GetProcesses().I understand that this is because i dont have administration rights.How can i solve this problem.Is there a way where in i can give some permissions to the user to execute that part of code alone.

Hope to get replies soon.
Thanks

Re: System.Diagonastics.Process.GetProcesses throws error if the user does not have admin rights.

Dave Kreskowiak24-Aug-05 6:11

Office Programming: Insert or Merge Password Protected Documents in a Word Add-in

24-Aug-05 6:11

The user account running the code must have Admin rights to the remote machine in order to get it's Process list. There is no way around this.

RageInTheMachine9532
"...a pungent, ghastly, stinky piece of cheese!" -- The Roaming Gnome

tartancli22-Aug-05 4:14

tartancli

22-Aug-05 4:14

When developing Word Add-in programs and I want to insert a file into my currently open file I normally do the following:

object nRef = Type.Missing;
doc.Bookmarks.Item(ref oEndOfDoc).Range.InsertFile("c:\\temp\\filetobeinserted.doc",ref nRef,ref nRef,ref nRef,ref nRef);

My question is how do I do this when I want to insert a password protected file into my open file? I know the password for the file so I can supply this programmatically (I'm not concerned with possible security issues associated with this approach).

Any help would be much appreciated.

About Properties

chandru_inbox22-Aug-05 4:03

chandru_inbox

22-Aug-05 4:03

Hai,

In my Application i need to give the chance to the change the Property to any controls (say textbox,datagrid etc.,) Is it Possible to show the Property Window in Runtime... Help Me.

Christian Graus22-Aug-05 11:01

Christian Graus

22-Aug-05 11:01

No, the property window is just enumerating the properties of the control. You'd have to write your own window to do the same thing in your code.

Christian Graus - Microsoft MVP - C++

[Marc]22-Aug-05 17:42

[Marc]

22-Aug-05 17:42

Right-Click the ToolBox --> Add/Remove Items --> Check PropertyGrid (namespace System.Windows.Forms) --> Press OK --> Drag it to your Form.
The PropertyGrid.SelectedObject property is the object who's properties will be shown. In designtime you can only select a Component, but it can be any object you wish if you specify it at runtime.

Hope this helps! Cool | :cool:

"..Commit yourself to quality from day one..it's better to do nothing at all than to do something badly.."
-- Mark McCormick

Christian Graus22-Aug-05 18:54

Christian Graus

22-Aug-05 18:54

Really ? Wow. I did not know that...

Christian Graus - Microsoft MVP - C++