|
|
I have a process running on a 64bits server OS.
It does use a lot of memory and I sometimes get outOfMemory exception....
However, at best, I'm using a few tens of Megabytes at once, say 60Mb. Or even 200 Mb (if memory representation of disk data takes more space)
How come I have out of memory?
I understand there could be memory fragmentation and such, but our server have 8GB, plus, with virtual memory, process should have something like 16GB of available memory, or so I believed.
That leaves plenty of room...
Can anyone explain to me what's probably going on or how much memory is available to .NET process?
A train station is where the train stops. A bus station is where the bus stops. On my desk, I have a work station....
_________________________________________________________
My programs never have bugs, they just develop random features.
|
|
|
|
|
It means that you're using too much memory in a process. What is the app doing ?
Christian Graus
Driven to the arms of OSX by Vista.
Read my blog to find out how I've worked around bugs in Microsoft tools and frameworks.
|
|
|
|
|
I'm between a rock and a hard place.
What I'm doing is:
Reading data in a DataTable
Writing the DataTable in a "MapXtreme Table".
Writing to MapXtreme is godamn slow and if I read line by line in the reader and directly write line by line in the MapXtreme table, I will certaimnly reduce memory consumption. But it might take up to half an hour to write all the data in a MapXtreme table, and I have SQL Timeout.
My SQL timeout is already 10 minute and I'm not eager to increase it more.
So I decided to read all the data in a DataTable first (quick), close the SQl connection and then write it to MapXtreme.
I have OutOfMemory of the DataTable.Load()...
Mmhh... I'm thinking I can load my data in chunk perhaps? tricky but feasible..
But then I also get out of memory in MapXtreme Table.Write (Row i.e. Feature)
I can't really do anything there... MapXtreme support is appealing so I won't get much help from their either 
But anyway, the MapXtreme files are, at most, about 30~40 Mb, so it stands to reason that the memory used can't be much more than that.
40Mb is not that much in a 64bit server with 8GB memory + virtual memory, why do I get that?
A train station is where the train stops. A bus station is where the bus stops. On my desk, I have a work station....
_________________________________________________________
My programs never have bugs, they just develop random features.
|
|
|
|
|
I once worked on a Win App using vb.net. Well i got that problem and when i get back in my app, i found out that there was a prt of my code that was in a infinite loop. Variable and objects were created and never used. I suggest you go back to your code and check if you are not misusing the memory from there. another thing after you use object just set them to null in the Finally block. if you don't have finally block you must consider using it.
Vuyiswa Maseko,
Few companies that installed computers to reduce the employment of clerks have realized their expectations.... They now need more and more expensive clerks even though they call them "Developers" or "Programmers."
C#/VB.NET/ASP.NET/SQL7/2000/2005/2008
http://www.vuyiswamaseko.somee.com
vuyiswa@its.co.za
http://www.itsabacus.co.za/itsabacus/
|
|
|
|
|
Hi,
This is a standard text of mine, as the question pops up regularly:
----------------------------------------------------
I know of three ways to get an OOM Exception:
1.
the most surprising one: some method throw OOM on bad inputs, most notoriously: Image.FromFile().
2.
you are really running out of memory; typically that means you are preventing objects from being collected. You can watch your app grow, roughly through Task Manager; more correctly by incorporating a display of Environment.WorkingSet in your app.
One way of keeping objects around is by storing references to lots of, or all, objects you create; maybe for debugging, for statistical information, as a cache, whatever. If so, consider the WeakReference class.
3.
you have objects larger than 80KB; those get allocated on the "large-object-heap", which never gets compacted, and hence could become fragmented, resulting in an OOM even when lots of free memory is still around (but each chunk is smaller than the size needed).
Remember: lots of collections (ArrayList, List< T>, etc) are stored as arrays, and such arrays get doubled in size and copied each time their capacity gets exceeded.
IMO you can't reliably design a long running .NET app if you need large objects, unless you:
- either make them all the same size;
- or perform the entire memory management for large objects yourself.
Both of which may be tedious. A third possibility is making your app restart periodically all by itself.
----------------------------------------------------
As you say your app is both small and slow, there must be something really wrong. Why don't you show all relevant code (in PRE tags!)? My best guess is you are building very long strings (or collections) and your app's memory is getting fragmented. That is possible, no matter what Windows version, Framework version and amount of memory/virtual memory you have.
Luc Pattyn [Forum Guidelines] [My Articles]
The quality and detail of your question reflects on the effectiveness of the help you are likely to get.
Show formatted code inside PRE tags, and give clear symptoms when describing a problem.
|
|
|
|
|
The code is hard to share if only because I extensively using a 3rd party GIS library called MapXtreme, which is damn slow to write its data files.
Sometimes I can get tens of thousands row in a select, and then I write a feature table with same size. In fact I'm generating (spatial) report in the background and I will get many such case.
Here is relevant method.
Which is invoked continuously in a never ending process until all trucks travel for the period (usually day) have been accounted for.
static void AddFeatures(ReportTask report, Guid taskId, out int nRow, out Table table)
{
table = null;
nRow = 0;
var miStyle = TabUtility.MakeMapInfoStyle(report.Style);
var dbtable = new DataTable();
try
{
using (var cmd = new SqlCommand())
using (var conn = Config.GetOpenedConnection())
{
cmd.Connection = conn;
cmd.CommandType = CommandType.StoredProcedure;
cmd.CommandTimeout = Config.SqlTimeout;
cmd.CommandText = report.StoredProc;
cmd.AddParameter("uidTaskID", SqlDbType.UniqueIdentifier, taskId);
using (var reader = cmd.ExecuteReader())
{
if (reader.HasRows)
dbtable.Load(reader);
}
}
}
catch (Exception ex)
{
ex.Data["ReportName"] = report.Name;
ex.Data["ReportSP"] = report.StoredProc;
throw;
}
try
{
long TotalWKBRead = 0;
foreach (DataRow row in dbtable.Rows)
{
try
{
var wkb = row[GeomColumnName] as byte[];
if (wkb == null || wkb.Length == 0)
continue;
var geom = WKB.ToFeatureGeometry(wkb);
if (geom == null)
continue;
TotalWKBRead = wkb.Length;
if (table == null)
{
table = CreateTable(report.BasePath, report.Name, miStyle, dbtable);
table.BeginAccess(TableAccessMode.Write);
}
var feature = new Feature(table.TableInfo.Columns);
feature.Geometry = geom;
feature.Style = miStyle;
TabUtility.SetFeatureValues(feature, row);
table.InsertFeature(feature);
nRow++;
}
catch (Exception ex)
{
var wkb = row[GeomColumnName] as byte[];
ex.Data["TaskID"] = taskId;
ex.Data["ReportName"] = report.Name;
ex.Data["table.Row.Count"] = nRow;
ex.Data["WKB.Length"] = wkb != null ? wkb.Length : 0;
ex.Data["TotalWKB.Length"] = TotalWKBRead;
throw;
}
}
}
finally
{
if (table != null)
table.EndAccess();
}
}
A train station is where the train stops. A bus station is where the bus stops. On my desk, I have a work station....
_________________________________________________________
My programs never have bugs, they just develop random features.
|
|
|
|
|
Hi,
I agree problem 3 is the most likely.
I happen to be researching fragmentation and OOM; it used to be very simple and predictable; I had a 20-line test app that demonstrates it in a matter of milliseconds, but something has changed recently, and now I get it sometimes, and somehow .NET works around it on some occasions. I'll have to spend more time before I can draw any conclusions.
From earlier fragmentation observations, I would avoid all unnecessary objects larger than 80KB, that includes huge strings (always bad!), and large collections (see http://www.perceler.com/articles1.php?art=capacity1[^]) hence also your DataTable. I'd rather use an enumeration that doesn't require all the rows to be present in memory at once
I'm not a DB specialist, however probably you should use an ExecuteScalar to get a count when relevant, and a DataReader might be better to get one row at a time.
Luc Pattyn [Forum Guidelines] [My Articles]
The quality and detail of your question reflects on the effectiveness of the help you are likely to get.
Show formatted code inside PRE tags, and give clear symptoms when describing a problem.
|
|
|
|
|
Thanks for your time
I found those 2 links:
Garbage Collection[^]
Investigating Memory Issues[^]
And this interesting property:
GCSettings.LatencyMode[^]
Which I have to investigate tomorrow.
And also I have to check that those MapXtreme table are really closed... (I do call the Close() method but apparently, according to the documentation, it is unreliable!)
A train station is where the train stops. A bus station is where the bus stops. On my desk, I have a work station....
_________________________________________________________
My programs never have bugs, they just develop random features.
|
|
|
|
|
I've gotten this problem and solved it by adding dispose () calls to free large blocks of memory.
The fact that .NET is garbage collected doesn't mean you can ignore memory issues. As someone else here mentioned, the large-object heap does NOT get compacted, so can be fragmented, reducing the memory available.
I make sure large memory blocks are disposed in C#, just like I made sure they were deleted in C++.
|
|
|
|
|
Is there a way to detect if a folder is open.
Regards
|
|
|
|
|
A folder does not change state because it is 'open' in another window. Therefore, the answer is no. Why do you want to ?
Christian Graus
Driven to the arms of OSX by Vista.
Read my blog to find out how I've worked around bugs in Microsoft tools and frameworks.
|
|
|
|
|
|
Thank You,That seems to be what I'm looking for.
Regards
Rick
|
|
|
|
|
This will only work for a top level window, and could give false hits ( if a lower down folder of the same name, is open, or an app has the same name as the folder ). It's as close as you can get, but it's very fragile.
I asked you this before - why do you need to do this ?
Christian Graus
Driven to the arms of OSX by Vista.
Read my blog to find out how I've worked around bugs in Microsoft tools and frameworks.
|
|
|
|
|
The code as written will return false positives. Multiple parent folders may have seperate child folders with the same name. Which one is really open??
The only way to detemine this would be to enumerate the Explorer windows, treating them like Internet Explorer windows and getting the URL they are showing.
|
|
|
|
|
Sorry I've completely forgotten about this , I was hurried to much for my lunch before the post.In order to avoid confusion I've just deleted my previous post.I agree that my fast method by using FindWindow wasn't sophisticated.
Here is my C++ function that solves the solution.It easy could be converted to .NET too.
<br />
BOOL IsOpenedNyWindowsExplorer(CString FolderName)<br />
{<br />
ATLENSURE(FolderName);<br />
<br />
BOOL RESULT=FALSE;<br />
IShellWindows *psw;<br />
HRESULT hr=0;<br />
<br />
hr = CoCreateInstance(CLSID_ShellWindows, NULL, CLSCTX_ALL,IID_IShellWindows,(void**)&psw);<br />
<br />
if (SUCCEEDED(hr))<br />
{<br />
long lCount=0;<br />
psw->get_Count(&lCount);<br />
for (int i = 0; i < (int)lCount; i++)<br />
{<br />
_variant_t vtIndex( (long)i);<br />
IDispatch* pDisp=NULL;<br />
hr=psw->Item(vtIndex,&pDisp);<br />
if(SUCCEEDED(hr))<br />
{<br />
IWebBrowser2* pBrowser=NULL;<br />
hr=pDisp->QueryInterface(IID_IWebBrowser2,(void**)&pBrowser);<br />
if(SUCCEEDED(hr))<br />
{<br />
BSTR Loc=SysAllocStringLen(NULL,270);<br />
hr=pBrowser->get_LocationURL(&Loc);<br />
if(SUCCEEDED(hr))<br />
{<br />
TCHAR* Dest=new TCHAR[270];<br />
DWORD Buff=0; <br />
CString str(Loc);<br />
if(AtlUnescapeUrl(str,Dest,&Buff,270))<br />
{<br />
str.ReleaseBuffer();<br />
str=Dest;<br />
<br />
str.TrimLeft(L"file:///").Replace('/','\\');<br />
if(!FolderName.CompareNoCase(str)) <br />
RESULT=TRUE;<br />
<br />
str.ReleaseBuffer();<br />
}<br />
str.ReleaseBuffer();<br />
delete[] Dest;<br />
pBrowser->Release();<br />
}<br />
SysFreeString(Loc);<br />
}<br />
pDisp->Release();<br />
}<br />
}<br />
<br />
psw->Release();<br />
}<br />
<br />
return RESULT;<br />
}<br />
|
|
|
|
|
Hi,
Am writing a function which should return a dataset to bind to a grid control. It is a linq query, Can I type cast var type variable to DataSet and return it.
Here is my funtion below.
public DataSet GetSites()
{
SiteTerDBDataContext ctx = new SiteTerDBDataContext();
ctx.Connection.ConnectionString = ConfigurationManager.ConnectionStrings["DB_ConnectionString"].ConnectionString;
var res = from sites in ctx.tblSites select sites;
return (DataSet)res;
}
Any early replies will be appreciated.
regards
Sajid
|
|
|
|
|
Sayed Sajid wrote: Any early replies will be appreciated.
You're in a hurry, but you didn't think to try this and see what happened, or check res in the debugger to see what type it is ?
I would be stunned if it came back as a dataset. I expect it would be a collection.
Christian Graus
Driven to the arms of OSX by Vista.
Read my blog to find out how I've worked around bugs in Microsoft tools and frameworks.
|
|
|
|
|
Mr Chris,
Thanks for your answer.
What would be the appropiate way of returning var type?
This is my intension to do actually.
public var GetSites()
{
SiteTerDBDataContext ctx = new SiteTerDBDataContext();
ctx.Connection.ConnectionString = ConfigurationManager.ConnectionStrings["DB_ConnectionString"].ConnectionString;
var res = from sites in ctx.tblSites select sites;
return res;
}
Many thanks.
Sajid
|
|
|
|
|
You cannot return var type since it's not a type like int or string.
It simply allows the compiler to infer the real type.
Regards
David R
---------------------------------------------------------------
"Every program eventually becomes rococo, and then rubble." - Alan Perlis
|
|
|
|
|
Sayed Sajid wrote: Can I type cast var type variable to DataSet and return it.
No.
var is just a syntactic sugar. In your case, res will be enumerable of ctx.tblSites's type. You can't cast it directly to a DataSet.
Create a DatTable, Iterate the contents of res in a loop and fill data into DataTable. Once data table is ready, add it to a DataSet and return.
|
|
|
|
|
Hey Everyone,
I'm looking for some ideas on how to find the indexes of nested parentheses () in a string.
Basically the user is going to type a complex mathematical formula and i would like to know where all the nested ( and )'s are.
I have considered RegEx, but i had some real trouble with the nested part and i dont want to remove that functionality)
Any Ideas?
If at first you don't succeed ... post it on The Code Project and Pray.
|
|
|
|
|
Have a look at Expresso[^] - it is a regex creation / testing tool. One of its pre-built expressions is:
\(
(?>
[^()]+
| \( (?<number>)
| \) (?<-number>)
)*
(?(number)(?!))
\)
Which is a parentheses matcher. May be a good start for you?
No trees were harmed in the sending of this message; however, a significant number of electrons were slightly inconvenienced.
This message is made of fully recyclable Zeros and Ones
|
|
|
|
|
Thanks for the reply, this a start for sure. I could use something like this then just recursively go through the matches finding all the nested pairs.
Thanks for the reply
If at first you don't succeed ... post it on The Code Project and Pray.
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
