|
inho, another case (see my post) that illustrates why a requirement the OP search CP for resources prior to posting would be valuable. And, yes, I know that would be "iffy" to implement.
«Where is the Life we have lost in living? Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information?» T. S. Elliot
|
|
|
|
|
CodeProject is your friend: "C# ZKTeco Biometric Device Getting Started" by Ozesh Thapa, 2019: [^]
«Where is the Life we have lost in living? Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information?» T. S. Elliot
|
|
|
|
|
I am joining two list<t> type and each list has more than 800000 records.
may be this huge records causing this issue. my pc is 64 bit and 8GB RAM. i have windows 7 OS installed.
var QCViewAllHistValue1 = (from frmlst in cfList
join viewalllst in QCViewAllBrokerList1
on new
{
val = String.IsNullOrEmpty(frmlst.Section) ? "" : frmlst.Section.Trim().ToUpper(),
val1 = String.IsNullOrEmpty(frmlst.xFundCode) ? "" : frmlst.xFundCode.Trim().ToUpper(),
val2 = String.IsNullOrEmpty(frmlst.Period) ? "" : frmlst.Period.Replace("A", "").Replace("E", "").Trim().ToUpper(),
val3 = String.IsNullOrEmpty(frmlst.Broker) ? "" : frmlst.Broker.Trim().ToUpper()
}
equals new
{
val = String.IsNullOrEmpty(viewalllst.ViewAllSection) ? "" : viewalllst.ViewAllSection.Trim().ToUpper(),
val1 = String.IsNullOrEmpty(viewalllst.xFundCode) ? "" : viewalllst.xFundCode.Trim().ToUpper(),
val2 = String.IsNullOrEmpty(viewalllst.ViewAllPeriod) ? "" : viewalllst.ViewAllPeriod.Replace("A", "").Replace("E", "").Trim().ToUpper(),
val3 = String.IsNullOrEmpty(viewalllst.ViewAllBroker) ? "" : viewalllst.ViewAllBroker.Trim().ToUpper()
}
select new QCHelper()
{
Value = viewalllst == null ? string.Empty : (viewalllst.Value == null ? string.Empty : viewalllst.Value),
}).ToList<QCHelper>();
cfList & QCViewAllBrokerList1 these are two List<t> type. i am joining these two list and getting out of memory exception
i follow these two below links but there suggestion does not work for me.
https://stackoverflow.com/a/17322112/10839668
https://stackoverflow.com/a/53424048/10839668
please guide me how could i get rid of out of memory exception. thanks
|
|
|
|
|
That query is such a dog breakfast that I'm not surprised it fails. You are probably creating the equivalent of an outer join and getting 800k * 800k records.
I would ask a couple of question of the requirements, why do you need 800k records in the client and how do you expect to use the collection when you have it.
If this was my problem I would create a view in the database to validate the joins and then select on the view with a filter (you cannot consume 800k record in the client).
Never underestimate the power of human stupidity -
RAH
I'm old. I know stuff - JSOP
|
|
|
|
|
My question is .Net can not handle big data when run on 64 bit processor?
so many company work with huge data...how they handle it. as per our client requirement i have to work with xml file. we store data in xml not in db. this is client requirement.
any good suggestion please.
|
|
|
|
|
Up until .NET 4.5, there was a hard limit on the size of any single object, which was 2GB. With .NET 4.5 and above, that limit ins't hard set any more. You can set an option in your apps App.Config file to remove that limit.
See With .NET 4.5, 10 years memory limit of 2 GB is over | Bahrudin Hrnjica Blog[^] for a short explanation.
But, you still have a problem. There is only ever so much memory in a machine. If your .XML file grows (and do NOT make the mistake of saying it never will!) you will eventually run out of memory, even on a 64-bit machine.
|
|
|
|
|
Mou_kol wrote: so many company work with huge data...how they handle it.
By reading and updating parts of it.
Mou_kol wrote: this is client requirement.
The client might want that as an endresult, but doesn't care for intermediate stages. You can still mirror a copy on a database for quering 
Bastard Programmer from Hell 
If you can't read my code, try converting it here[^]
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
Yes; it can handle it.
It's the "size of the data"; not the number of records that matters.
But since you never once mentioned "bytes", I would not assume that "number of records" is the issue.
Pre-process / pre-select / stage the input files (instead of in the join) if you want to gain any insight at all into "your data".
You gain nothing by trying to get one statement to do everything. LINQ is not "A.I."
The Master said, 'Am I indeed possessed of knowledge? I am not knowing. But if a mean person, who appears quite empty-like, ask anything of me, I set it forth from one end to the other, and exhaust it.'
― Confucian Analects
|
|
|
|
|
Mou_kol wrote: any good suggestion please.
Either use a database like everyone else or get a new client.
Companies that have even small volumes of data almost all use a database, XML is a transport medium NOT a storage tool and even that has been replaced by JSON.
As others suggested if you must use linq then pre filter the records so you are working with a smaller data set.
Never underestimate the power of human stupidity -
RAH
I'm old. I know stuff - JSOP
|
|
|
|
|
You're doing a "new" projection into two objects, doing all kinds of string manipulations which create new string objects, all to do a comparison across 800,000 records IN MEMORY, not on the SQL server, and you wonder why you get OutOfMemory exceptions?
I agree, that query is complete garbage and doesn't scale at all. You have to create a view in the database for this kind of work.
If you're doing a IsNullOrEmpty and Trim's and ToUpper's, your data is garbage to begin with and needs to be cleaned up before it ever gets into the database.
.
|
|
|
|
|
As others have stated, doing this in a database is probably better. Databases are not magic though. Set it up wrong (lack of indexes, asking it to join with queries written as bad in SQL as these are written in Linq) and it will be crazy slow.
If it isn't possible to use a database I would do load the second lists into a lookup (if multiple records can have the same key) or dictionary (if you are sure they are unique.
Now the first list can do a "manual" join. Assuming you used a lookup (use .ToLookup or load directly to your own ILookup implementation) it will be something like (just written from memory, you will need to adapt it):
from leftItem in list1
let discustingKey = CalculateYourUglyKey(leftItem)
where list2Lookup.ContainsKey(discustingKey)
from rightItem in list2Lookup[discustingKey]
select WhateverYouWantToCreateFromLeftAndRightItem(leftItem, rightItem)
Sorry, I can't say how to format it exactly so it gives you the current output, because frankly your code is so ugly I can't read it - but this should give you an idea on how to reduce your key calculation to once per item in each list.
Instead of using Linq to make the "kind of a join", you can even do it from within an XML reader as each element is read. Then only the second list and the result needs to be in memory at the same time.
|
|
|
|
|
your sample code not clear to me.can you please post a working small sample code using lookup what you have show me as sample. thanks
|
|
|
|
|
Sorry, I am on holiday and I am not going to spend time on this. Try to google: linq tolookup
|
|
|
|
|
hello,
I have this while loop
while (true)
{
Thread.Sleep(40);
if (Control.MouseButtons == MouseButtons.Left)
{
MessageBox.Show("Left");
}
if (Control.MouseButtons == MouseButtons.Right)
{
MessageBox.Show("Right");
}
I use this on threading to monitor the mouse button state any time without focus on particular
form.
I need make the same monitor to the mouse wheel, how I can do this in the same loop?
Thanks
|
|
|
|
|
0: disclaimer: I am not informed about how the use of threading may come into play here.
1: don't use a 'while loop: it's going to degrade performance: what you need in this case is a 'hook to get all Mouse Events ... either for a Form (WndProc), or for your app (App) ... or, for every running process (global).
2: why you need a 'hook: only the Control with Focus gets the WM_MOUSEWHEEL message; you can define an EventHandler to get the 'MouseWheel status:
private void Form1_Load(object sender, EventArgs e)
{
this.MouseWheel += OnMouseWheel;
}
private int lastMWDelta;
private int lastMWClicks;
public Point lastMWPt;
public Control lastMWActiveControl;
private void OnMouseWheel(object sender, MouseEventArgs e)
{
lastMWDelta = e.Delta;
lastMWClicks = e.Clicks;
lastMWPt = e.Location;
lastMWActiveControl = sender as Control;
Console.WriteLine($"wheel: control: {lastMWActiveControl.Name} delta: {lastMWDelta}");
}If there are only specific Controls you want to get wheel events for: define the same type of EventHandler for those.
3: how do you implement a 'hook ?
a: first decide what type of 'hook is required.
b: read the articles here on CodeProject about 'hooks ... like: [^], [^], [^], [^]
Come back with code, and specific issues, or questions.
«Where is the Life we have lost in living? Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information?» T. S. Elliot
|
|
|
|
|
Can anyone provide an example, or point me to a walkthrough?
I need clarification on how to make a sql table(SQL Express) using code first approach in entity framework model and:(add, edit, save & delete) in that table, and to do that using MVVM pattern, in a WPF application.
Specifically I am facing a problem in trying to create the database using the c# code in the wpf application without having to access the SQl Server Management Studio to create the database.
I would really appreciate any advice.
|
|
|
|
|
You're not going to find a walkthrough with all of those things, you need to break the problem down. If your issue is creating a DB using code first then google for a solution to that. That solution will work regardless of if it is a console app, a website or a WPF application.
create database ef code first - Google Search[^]
|
|
|
|
|
i know the basic method but i am having problems in finding exact syntax for my requirement .i have done extensive searches in google.I was hoping for more specific resources
|
|
|
|
|
Quote: but i am having problems in finding exact syntax for my requirement
Still sounds like: give me the codez.
The Master said, 'Am I indeed possessed of knowledge? I am not knowing. But if a mean person, who appears quite empty-like, ask anything of me, I set it forth from one end to the other, and exhaust it.'
― Confucian Analects
|
|
|
|
|
Member 14520635 wrote: i am having problems in finding exact syntax
Post what you have and we can then advise you if there are better/correct methods of getting where you need to be.
Never underestimate the power of human stupidity -
RAH
I'm old. I know stuff - JSOP
|
|
|
|
|
CP has many useful examples. Get to (home)work.
«Where is the Life we have lost in living? Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information?» T. S. Elliot
|
|
|
|
|
Console.WriteLine("-----Enter at least 10 numbers-----");
int[] num = new int[10];
int input = 0;
while (input <= 10)
{
Console.Write("Enter a whole number: ");
num[input] = Convert.ToInt32(Console.ReadLine());
input++;
}
Console.Write("\nThis are the list of Even Numbers: ");
int Even = 0;
while (Even < 10)
{
if (num[input] % 2 == 0)
Console.Write(num[input]);
Even++;
}
Console.Write("\nThis are the list of your Odd Numbers: ");
int Odd = 0;
while (Odd < 10)
{
if (num[input] % 2 == 1)
Console.Write(num[input]);
Odd++;
}
Console.ReadLine();
|
|
|
|
|
num array has 10 elements with indices from 0 to 9.
Change the line
while (input <= 10) to
while (input < 10)
This you could have easily spotted if you had launched a debug session.
|
|
|
|
|
A debug session. Are you crazy? Then he would have to press F5. Much easier to post all the source code here, and press F5 to see if there is a repl... ohh.. wait. Never mind.
|
|
|
|
|
Phil.o is exactly right with what your problem is: array indexes in c# (and most modern languages) run from zero to N-1 where N is the number of elements. An attempt to use index N or any negative number is automatically outside the array and will generate a error.
But ... a couple of things:
0) It's a very good idea to always use curly brackets round even single lines inside an if:
if (num[input] % 2 == 0)
{
Console.Write(num[input]);
}
Even++;
If you don't, then you can get problems because the indentation is just to make teh code more readable to us, it means nothing to the compiler. If you add a line which is meant to be inside the if without curly brackets, it looks like it should work, but it won't:
if (num[input] % 2 == 0)
Console.Write(num[input]);
num[input] = 0;
Even++;
If you always use the brackets, it's obvious and you can't get it wrong:
if (num[input] % 2 == 0)
{
Console.Write(num[input]);
num[input] = 0;
}
Even++;
1) Don't comment what the instruction is doing:
int[] num = new int[10]; We know it's declaring an array, because we can read the code: it's declaring an integer array variable called num and assigning a new instance of 10 elements.
Your comment tells us nothing useful; it add nothing to the code. In fact it tells us less than the code does, and it's more likely to get "out of step" with the code if you change the array to a List (which is a kind of array that you can add and remove elements from - you'll meet them later).
Pretty much every comment in there does the same thing: document what the code is doing, not why you are doing it.
A better comment in this case is no comment at all, but use a different name for the variable:
int[] userInputs = new int[10]; That tells you what the array actually contains, without needing a comment.
This is called "self documentation" and it has the major advantage that it's much, much more likely to be kept up to date when code changes.
2) Don't use Convert to parse user inputs: users make mistakes, and Convert will cause your app to fail if they do. While this is only ten numbers, it's still going to be very annoying to type nine of them successfully and then miskey on the tenth and have the app crash out as a result!
Instead, use the TryParse methods:
int[] userInputs = new int[10];
int index = 0;
while (index < userInputs.Length)
{
Console.Write("Enter a whole number: ");
string input = Console.ReadLine();
if (int.TryParse(input, out userInputs[index]))
{
index++;
}
else
{
Console.WriteLine($"\"{input}\" is not an integer!");
}
}
3) Are you sure you want to use input in your final pair of loops?
Sent from my Amstrad PC 1640
Never throw anything away, Griff
Bad command or file name. Bad, bad command! Sit! Stay! Staaaay...
AntiTwitter: @DalekDave is now a follower!
modified 4-Jul-19 3:08am.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
