|
Hi all.
in my app i have to load one or two million of strings from the database to an array.
after loading i close the database connection.
i run a FOR loop and load the array
but memory reaches 125 MB
how to make it low?
TheMrProgrammer
http://www.icbse.com/2009/funny-exam-answers-school-students
http://download.cnet.com/TheCalcMan/3000-2094_4-10958266.html
|
|
|
|
|
Don't load so much data into memory.
What are you trying to achieve that you need so much data loading into memory from a database?
Steve Jowett
-------------------------
Real programmers don't comment their code. If it was hard to write, it should be hard to read.
|
|
|
|
|
My main motive is to save the data in the database and then on loading the application ALL the data from the database is loaded into an array of string datatype.
well it is really essential to load all the data, so i'll have to load it.
what i am looking for is a method to lower the memory utilization.
i tried to minimize the window and then restore it. This solves the memory problem but i want to use it as the last available option.
can you(or anyone) tell me how to lower the memory(RAM) utilazation?
TheMrProgrammer
http://www.icbse.com/2009/funny-exam-answers-school-students
http://download.cnet.com/TheCalcMan/3000-2094_4-10958266.html
|
|
|
|
|
Have you tryed to trim strings before load them in the array?
|
|
|
|
|
if you mean using the Trim() function, there is no need because there are no
spaces(leading or trailing).
thanks anyway for the suggestion.
TheMrProgrammer
http://www.icbse.com/2009/funny-exam-answers-school-students
http://download.cnet.com/TheCalcMan/3000-2094_4-10958266.html
|
|
|
|
|
TheMrProgrammer wrote: well it is really essential to load all the data, so i'll have to load it.
I am not sure why you need to do this. Are you doing this to make further data fetching to be from memory and do you think it is going to be faster than querying database for required data? If you need frequent data retrieval, array won't be the right data structure as it takes linear time for search.
TheMrProgrammer wrote: what i am looking for is a method to lower the memory utilization.
Do you have 2 million unique strings? Strings are interned by the CLR and same strings can share the memory. Have you profiled your application? Use CLR profiler and find out which area is using more memory.
So hard to tell more without knowing details about your application.
|
|
|
|
|
Yes I need frequent data retrieval.
can you tell me waht will be the right data structure to do so.
Thanks
TheMrProgrammer
http://www.icbse.com/2009/funny-exam-answers-school-students
http://download.cnet.com/TheCalcMan/3000-2094_4-10958266.html
|
|
|
|
|
You don't need to worry about it. What you did by minimizing the app and restoring it tells me that you only looked in TaskManager for the memory statistic. Don't. What you're seeing the memory the .NET CLR has RESERVED for your application, not how much your app is actually using. Use Performance Monitor instead to see what your app is really using.
You do not need to worry about the extra memory being reserved. The .NET Memory Manager keeps extra memory in the pool to allocate new objects your app wants as fast as possible. WIthout this, the Memory Manager has to request more memory from Windows and add it to the Managed Pool, which takes time. So, yeah, you can reduce this extra pool, but you're only hurting your apps performance by doing so.
Now, if Windows needs that memory back, the Memory Manager will be more than happy to return any extra memory it can back to Windows, without you doing anything.
|
|
|
|
|
hey thanks fer tha info
TheMrProgrammer
http://www.icbse.com/2009/funny-exam-answers-school-students
http://download.cnet.com/TheCalcMan/3000-2094_4-10958266.html
|
|
|
|
|
TheMrProgrammer wrote: how to make it low?
Don't load 2 million strings!
Try to find a logical way to subdivide the dataset and process it a bit at a time.
Without knowing what it is that you need to do with the data it is difficult to give more detailed advice. 
Henry Minute
Do not read medical books! You could die of a misprint. - Mark Twain
Girl: (staring) "Why do you need an icy cucumber?"
“I want to report a fraud. The government is lying to us all.”
|
|
|
|
|
I don't see other ways than that already suggested you, but you can check if it is possible to elaborate strings directly in the database. I don't know if it's possible, and if yes, how to do that.
|
|
|
|
|
Hi,
Working on big data sets will probably end up being terribly slow. If you can stream the data, it might be orders of magnitude faster.
You haven't told anything useful so far, explain what kind of data it is, where it comes from, what you want to do with it, how long it is allowed to take, and exactly why you want to close the database.
Luc Pattyn
Have a look at my entry for the lean-and-mean competition; please provide comments, feedback, discussion, and don’t forget to vote for it! Thank you.
Local announcement (Antwerp region): Lange Wapper? Neen!
|
|
|
|
|
My app saves data (string) in the database which is of .mdb format.
At the startup all the data from the database is loaded into the memory
because the app needs to iterate through the data many times.
I think it will be faster to iterate through the string array rather than
iterating through the database, thats why i load it in a string array. am I correct in thinking so?
I use ADODB to connect to the database.
Is there a way to have the strings in the memory, without eating up the RAM?
how do i convert the data into stream?
well what is stream anyway?
TheMrProgrammer
http://www.icbse.com/2009/funny-exam-answers-school-students
http://download.cnet.com/TheCalcMan/3000-2094_4-10958266.html
|
|
|
|
|
TheMrProgrammer wrote: I think it will be faster to iterate through the string array rather than
iterating through the database, thats why i load it in a string array. am I correct in thinking so?
No. A simple DB query with required filter in where condition will be faster than iterating an array. As I said, in worst case, searches in arrays takes O(n) time. This becomes pretty slow when you have huge number of data. Databases are clever to handle searches and can perform well than a linear storage like array.
TheMrProgrammer wrote: because the app needs to iterate through the data many times.
If you still need data in memory, load it to a right data structure that supports faster searches. You can try a SortedList which will do searches in O(log n) time as it uses binary search internally. This can improve the search time drastically.
If you have 2 million unique strings, HashSet(T) is also a good choice. It uses hash tables internally and searches are done in constant (O(1)) time.
|
|
|
|
|
TheMrProgrammer wrote: I think it will be faster to iterate through the string array rather than
iterating through the database, thats why i load it in a string array. am I correct in thinking so?
If by iterating you mean you will need each row once, then loading them all at once is the worst thing you could do.
TheMrProgrammer wrote: Is there a way to have the strings in the memory, without eating up the RAM?
Magic powder perhaps.
Luc Pattyn
Have a look at my entry for the lean-and-mean competition; please provide comments, feedback, discussion, and don’t forget to vote for it! Thank you.
Local announcement (Antwerp region): Lange Wapper? Neen!
|
|
|
|
|
Luc Pattyn wrote:
Magic powder perhaps.
You are wrong! Magic powder sits out of managed heap and there is no way for it to reduce memory allocated for managed strings.
|
|
|
|
|
If I were you I would check into how to call garbage collection more often or call it explicitly. Also something you might find useful is the use of memory pressure.
Thanks,
Humble Programmer
|
|
|
|
|
Hello Everbody,
M getting a problem on textBox Keypress.When i pass 1st Character in textBox on keypress it takes textBox.text="" value.Can Anybody provide Solution to the problem please its menance bug.
Thanks in advance
Thanks Christian Graus
Problem [Solved]
modified on Saturday, September 12, 2009 3:45 AM
|
|
|
|
|
The keypress event allows you to block the text from going on to the textbox, Handle the keyup event instead, if you want the key's contents to already be reflected in the textboxes contents. It's not a bug, at all.
(edit) as I think about it, the textchanged event is the one you want, if you want to know when the text in the textbox changes.
Christian Graus
Driven to the arms of OSX by Vista.
Read my blog to find out how I've worked around bugs in Microsoft tools and frameworks.
|
|
|
|
|
use KeyUp event.
Keypreess fires just before the Text is changed. ![Rose | [Rose]](https://codeproject.global.ssl.fastly.net/script/Forums/Images/rose.gif)
|
|
|
|
|
After scouring the net for info on using VB.net to generate graphic output, specifically a series of rectangles at mathematically derived intervals (a lattice, in fact), I'm now wondering if it's the best way to go. I've found nothing that demonstrates what I'm trying to do. I'm just starting out on a long, long road which I've been meaning to start on for a few years. I'd like to be able to create utilities as needed some day but wanted to start with this project to get going. Is VB (and GDI+) a bad choice for this type of work? I've made spreadsheets to do the math of laying out lattices but seeing them is just as important. Steer me in the right direction if there's a better way to go. I chose VB based on my thinking it's a relatively easy way to quickly get into programming.
|
|
|
|
|
Member 4196521 wrote: Is VB (and GDI+) a bad choice for this type of work?
Not even remotely. It's the most trivial way I can see of doing it in a meaningful language. You just handle the paint event of your form, and you're passed a Graphics object. This object has methods for drawing shapes onto the form, using other objects like Brushes and Pens.
I think your best bet is to look at the MSDN for help on the Graphics object, I suspect that once you know that's where to look, you'll find that the documentation is going to lead you from there.
Christian Graus
Driven to the arms of OSX by Vista.
Read my blog to find out how I've worked around bugs in Microsoft tools and frameworks.
|
|
|
|
|
Much obliged, Christian. Fears assuaged and whole bottle of eye drops at the ready. Once more unto the breach..
|
|
|
|
|
By all means, ask more specific questions if you still get stuck, but I think that's the boost you needed to get you on your way.
One thing, if you create any brushes, etc, they need to be disposed of, you call Dispose() on them, after using them. Using a system brush like Brushes.Black, does not have this requirement. Nor should you Dispose of the graphics object passed to you on the event args, although if you ever create your own, you should dispose of them, too.
Christian Graus
Driven to the arms of OSX by Vista.
Read my blog to find out how I've worked around bugs in Microsoft tools and frameworks.
|
|
|
|
|
Uncanny timing, I had just read about 'dispose' on MSDN. Too soon to tell if I will be using a brush or coordinate points-to-lines to create filled rectangles : {, which may be far more work but seems to offer easier translation from the resultant calculations to be performed in determining spacing, typically to thousandths of inches to reduce accumulated error. The graphics don't require such precision, merely visual feedback, but no sense not using it if it's there I figure.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
