|
Hello
Your stucture is 7 bytes long. That means that 50 million records should take about 350Mb of memory. Still not effecient. Such amount shouldn't be placed in memory all at once. Maybe you got 512Mb or even 1Gb memory and 2-4Gb of virtual memory or so, but many users may not. Even if they have such memory they won't appreciate all this memory going to your program all of a sudden -specially if they know more about memory and what goes behind the scenes-. At least you could notify them that this process will consume large amounts of memory, and they should close all open programs. Otherwise use temorary files to store output as it's parsed.
Anyway, for your issue. As I said you should be using only 350Mb. The rest is wasted because you use ArrayList, and not arrays. ArrayList are complex reference types, and contain more than just the data you give them.
Two suggestions:
1- Make it unsafe code, and use pointers to make up a Linked list.
2- Use array, and dynamically allocate more as you need -not easy and a waste of processing time-. After allocation you'd have to copy the old array into the new one. This will take more time if your GrowthBy factor is small.
They bound to be more suggestions, but I'd go for the linked list.
Regards ![Rose | [Rose]](https://www.codeproject.com/script/Forums/Images/rose.gif)
|
|
|
|
|
Look at the StructLayoutAttribute, specifically the Pack parameter. Setting that to 1/2 may help. Also rearranging the struct can help with the alignment. I cant recall the exact details, but a look in the EMCA spec will tell you.
Another optimization is to specify an initial size for the ArrayList, given you can estimate the number of lines given the input size.
The ideal option would be to use an array of the struct, if you can be sure you will have enough space (and not waste to much of it).
Further more, you can likely truncate the ranges of the fields, eg surely rating cant have 256 different values. What about id? What are the limits of those numbers? It depends on the input. Finally you pack them into a 32-bit sized structure (aka int/BitVector32).
Please note all/most of these optimizations are to optimize memory usage, and not performance increase!
|
|
|
|
|
There is mainly two things that waste space in your construction:
1. The ArrayList will grow by a certain percent each time it's full, so it will allocate larger and larger arrays. After a while the arrays will be too large to fit in the regular heap, so they will be allocated as large objects, and the space allocated for large objects never shrink, even if the objects in it are garbage collected.
2. As you are storing structures in an array list, each structure will be boxed inside an object. The metadata that the objects use to keep track of your structure is larger than the structure itself, so you will end up storing more metadata than actual data.
To get around those two problems, you can store the structures in small typed arrays (arrays of allInfo), and store the arrays in an ArrayList. That would make the ArrayList significantly smaller, and the structures doesn't need to be boxed.
Make sure that the arrays are smaller than 85000 bytes, so that they don't end up as large objects. 1000 items is a good round figure which is large enough to keep the overhead low and still way below the large objects threshold.
---
b { font-weight: normal; }
|
|
|
|
|
I am also having trouble with assigning the array to the array list:
while (((read = s.ReadLine()) != null)&&(entryCounter <= 100480507))//100480507
{
//Console.WriteLine(entryCounter+"\n");
split = read.Split(splitter);
id = uint.Parse(split[0]);
movieId = ushort.Parse(split[1]);
rating = byte.Parse(split[2]);
values.id = id;
values.movieId =movieId;
values.rating =rating;
loc = entryCounter-1;
masterArray.Insert(entryCounter,values);
split=null;
entryCounter++;
}
after thats done all:
Console.WriteLine("ID:"+((allInfo[])masterArray[20])[0].id+"\n");
Console.WriteLine("ID:"+((allInfo[])masterArray[21])[0].id+"\n");
Console.WriteLine("ID:"+((allInfo[])masterArray[22])[0].id+"\n");
Console.WriteLine("ID:"+((allInfo[])masterArray[23])[0].id+"\n");
all produce the same id. I checked and they should be different, so do the arrays in the array list point to the array?
|
|
|
|
|
You haven't shown enough of your code for me to say anything about what might be wrong. Are you creating separate arrays at all?
Here is a short example of creating and accessing an ArrayList containing arrays of integers:
ArrayList list = new ArrayList();
int[] array = new int[2];
list.Add(array);
array[0] = 4;
array[1] = 7;
array = new int[2];
list.Add(array);
array[0] = 42;
array[1] = -1;
int v0 = ((int[])list[0])[0];
int v1 = ((int[])list[0])[1];
int v2 = ((int[])list[1])[0];
int v3 = ((int[])list[1])[1];
---
b { font-weight: normal; }
|
|
|
|
|
kshet26 wrote: Thanks for your help. You have cleared a bunch of things up for me. I have one more question. If I initialize the ArrayList (ie ArrayList list = new ArrayList(100000000)) how do I assign values to those elements? I tried something like (list[0] = array) but I get a bunch of errors.
Thanks for your help,
Keith
You get errors because you try to assign values to items that does not exist.
If you specify a number when creating an ArrayList, that is only the initial capacity of the list. The list is still empty, you only tell it how large you expect the list to get, so that it can allocate that amount from the start.
Use the Add method to add the items to the ArrayList, just as I showed in the example.
If you read 100480507 items and use arrays with 1000 items, your ArrayList will only contain 100481 items.
---
b { font-weight: normal; }
|
|
|
|
|
What are you planning to do with the data read from the file?
Determine the top 5 entries in terms of rating, or something?
If you know what you want to do with the items, you can optimize for that by reading one record at a time in memory, this greatly reduces the memory footprint although it will be slower.
WM.
What about weapons of mass-construction?
|
|
|
|
|
Hi folks,
today I was trying to do some stuff with WMI.
I was trying to solve a problem many people seem to have who drag their laptop from some place to another (eg from home to university). Maybe at home you have one configuration for your networking needs and university makes you use completely different network settings. So basically what i was trying to do is a little application that allows you to save several profiles with configurations (such as IP Addresses, Subnet etc) and assign them when needed.
the actual problem appeared when i was trying to utilize WMI to assign an address to a certain network adapter. the code looks pretty much like in this article:
http://www.codeproject.com/cs/system/cstcpipwmi.asp
The second part of the article contains the assignment of an IPAddress to the adapter.
For me this thing isn't working. I dont get any Exceptions, everything just works fine, but nothing happens. I do the InvokeMethod thing (Win32_NetworkAdapterConfiguration.EnableStatic) and that is just it. I dont get a result, the Address stays as it is.
the wmi works fine for other stuff such as simply retrieving the adapters to see which ones can be used or are enabled or to see the MAC or whatever.
Did anyone of you have problems with wmi? Is maybe even my attempt to solve my original problem wrong (should I even use WMI to do this?) Did i forget to do some security clearance within the framework or something?
I use Visual Studio 2005 Professional Edition (VS Version 8.0.50727.42) on a Windows Professional. .NET 2.0 installed etc.
Thanx in advance
|
|
|
|
|
Does this[^] give you any clues?
Dave Kreskowiak
Microsoft MVP - Visual Basic
|
|
|
|
|
How do I do a rollover on a subitem in a listview so it displays all the test when the column is sized less than the text (i.e. the user can't see all the test in the grid so a rollover shows the full text).
Thanks.
|
|
|
|
|
Hello
What you are talking about is ListView.LabelWrap property. Set it to true. Yet it only works on the icons view.
For other views I guess you'd have to make a custom ListView or something...
Regards
|
|
|
|
|
Thanks for your response. I'm trying to do it on the details view.
|
|
|
|
|
I am creating a card game on C# . I am using a button to display cards and on the same button to change unwanted cards. But sometimes when i click the button to change cards the GUI Freezes and i have to exit the application with (END NOW) or from the windows task manager; and some times it runs normally without freezing... I donno if it is memory related stuff but i really need some help to go on with the project... Any Help Guys ?!
|
|
|
|
|
You'll have to log information from your application to narrow down where it's freezing. Then step through your code at that point to see what's going on, and what it would take to break your code at that point. It's also possible that there is a condition in your code that will cause an infitite loop, which will hang your app like this.
Dave Kreskowiak
Microsoft MVP - Visual Basic
|
|
|
|
|
Attach the processor to the Visual Studio Debugger and then hit Debug->Break All. Ctrl+Alt+H should give you the list of threads in the process. You can now go look at each of them (just double click the list item) and see why/how a thread hung. Typically, the UI thread would be the first thread in the list.
Regards
Senthil
|
|
|
|
|
Hi,
Im making a calculator.
It should have 4 buttons. Each button has a 'sign' like '+', '-', '/', 'x'.
Now suppose i press the '+', than it should display calculations like this:
1+4=
4+8=
23+12=
But how can i make it move to a next window with these calculations?
Thanks in advance!
|
|
|
|
|
Yustme wrote: But how can i make it move to a next window with these calculations?
You haven't specified what windows?? Why do you make multiple windows in a simple calculator?! Also, how/where the user is inputting the numbers?
Regards
|
|
|
|
|
Hi,
A window like the form itself where you can put controls in it.
In the beginning, only those 4 buttons are in sight. But after pressing one, it should display those sums or how ever they are called.
This program is suppose to help the kids to compute those sums.
Thanks in advance!
|
|
|
|
|
overload the constructor of next window and pass result in that constructor. now in form2_load event set those valuse
class Result
{
private double calculated;
result()
{
// inintalizetion code
}
result(double result)
{
// inintalizetion code
this.calculated=result;
}
private void Result_load (object ... ,event ...)
{
TextBox.text = calculated.toString();
}
}
Hope it'll solve your problem 
sameer
|
|
|
|
|
Hi sameer,
Thanks for your reply.
But how would i do that graphically? Where can i put a second window?
|
|
|
|
|
I have created a windows form application...
the problem is that when the keyboard keystroke combination ALT-F4 was press my application quits.
How can i stop the user from hitting the ALT-F4 key to terminate my application?
thanks in advance
|
|
|
|
|
You need to override the ProcessDialogKey method of the form. The follow code will allow you to block ALT-F4 from closing your application.
protected override bool ProcessDialogKey(Keys keyData)
{
Keys altf4 = Keys.Alt | Keys.F4;
if (keyData == altf4)
{
return true;
}
return base.ProcessDialogKey(keyData);
}
|
|
|
|
|
It works like a charm! Excellent
Thanks a lot for the help, Nick
|
|
|
|
|
I am trying to implement a structure that starts with a root class which points to many different nodes. Each node then points to two unique classes. I would like to be able to add nodes and hence the two classes associated with each node. Any help on this would be greatly appreciated.
cheers
mike
|
|
|
|
|
So you just want a class represenative of your root that contains an array of each unique class?
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
