|
Well, my guess is that the sheer number of records your processing in-memory is just flat out insane. :P
The problem is most likely not a row count limitation of SqlDataReader...but rather a memory limitation. You are trying to create a massive List of Record objects...the overhead for that is going to be fairly high compared to just displaying rows of text in, say, Sql Management Studio. Depending on exactly how many columns you are setting on your Record object, the data types of each, and the lengths of strings that may exist in your result set...you could be looking at GIGS of memory usage here.
I am not exactly sure what your doing or what your needs are...but you should look into batching your work. Either chunk it up into significantly smaller data sets (say, 10000 records each) and process them one at a time...or distribute those chunks out to a server farm to process them in parallel. If you absolutely need to process all 8 million records at once...then you are probably doing something that calls for LAPACK or some other matrix or vector processor that can efficiently handle large data sets.
|
|
|
|
|
Hi Jon,
Thanks for your reply, and the suggestions. Could I point you to my reply to Luc for a discussion of this out-of-memory limitation? And, I have been very careful to limit the number of fields I use and don't have any string columns, so although I take your point that 8 million rows is a beast, in SQL management studio the size of comma delimited output file is less than 1.5gig - I've got a 32 bit machine with maximum memory so I am skating on thin ice, but still I expected it to work out memory wise.
I'll look into LAPACK - thanks for the suggestion.
Ben
|
|
|
|
|
Ben Cocker wrote: SQL management studio the size of comma delimited output file is less than 1.5gig
So what? That's a file on disk that doesn't have the overhead of a storage class managing the data. THere is no limit on the Reader, since it only reads a single record at a time and returns the fields in that record. The real question is where is this data going?? Is it going into an ever-expanding collection?? If so, then you're running the machine out of memory.
|
|
|
|
|
Well, maybe you are confused about the datareader and dataset in ADO.net. DataReader object just go forward for ONE record each time. Generally, it will not take you too much memory for only one record, right?
I Love KongFu~
|
|
|
|
|
Ben - what could you possibly do with this many records? You certainly couldn't present this to a user - are you meant to be passing this to some other process (say to store to a file).
I suspect you've got two problems - one, the record at position 726,301 is invalid in some way which you aren't catching in your code and two, the number of records is too great. BTW - it's never a good idea to just consume exceptions. You should actually do something with it, even if it's only log the exception. I normally use a variation of this function to parse the items (I'm typing this from memory so it may need a bit of tidying up):
public static T ParseValue(this SqlDataReader reader, string fieldName)
{
int ordinal = reader.GetOrdinal(fieldName);
if (reader.IsDBNull(ordinal))
return default(T);
return (T)reader.GetValue(ordinal);
}Then you use this as
NewRecord.PAR1 = Reader.ParseValue("PAR1");
"WPF has many lovers. It's a veritable porn star!" - Josh Smith As Braveheart once said, "You can take our freedom but you'll never take our Hobnobs!" - Martin Hughes.
My blog | My articles | MoXAML PowerToys
|
|
|
|
|
Hi All,
i'm try to take the VB script and Dll in this article http://support.microsoft.com/kb/935635
and make the functionality work with C#.
i can add the ExIpSec.dll type library and with the using statement (using EXIPSECLib) i can expose the Interfaces and i can some of them to work. but when i try to retrieve the list if IP addresses i just get System.object[]. i have tried converting this to a string[] but i get nothing back.
can anyone tell how to correct call this dll? - DllImport does seem to work either!!
Many Thanks,
Phil
|
|
|
|
|
You may want to start at the beginning to understand how COM Interop [^]works
Yusuf
|
|
|
|
|
Well, without more info, we can not do anything with it. But all you did so far seems correct.
In fact, .Net will wrap up the COM for you(if you add COM dll from the Visual Studio), we do not need to worry about the interface. I am intended to say maybe you miss something in your code, such as initialize certain parameter.
I Love KongFu~
|
|
|
|
|
the lines in the VB script are:
Set objDsIpSec = CreateObject("ExIpSec.ExIpSecurity")
objDsIpSec.BindToSmtpVsi "someSMTPserver", 1, "someDC"
objDsIpSec.GetRelayIpList
listToDisplayIp = objDsIpSec.IpGrant
listToDisplayIp appears to be a string array.
the lines in using in C# are:
ExIpSecurity ipsec = new ExIpSecurity();
string[] s;
ipsec.BindToSmtpVsi("someSMTPserver", 1, "someDC");
ipsec.GetRelayIpList();
s = ipsec.IPGrant.ToString().Split(' ');
foreach (string i in s)
{
MessageBox.Show(i.ToString());
}
if i run the VB script i get a list of IP address but if i run the C# code i get one popup box saying System.Object[]
what am i doing wrong?
|
|
|
|
|
lane0p2 wrote: ipsec.BindToSmtpVsi("someSMTPserver", 1, "someDC");
ipsec.GetRelayIpList();
s = ipsec.IPGrant.ToString().Split(' ');
Did you debug the code and check what the value of IPGrant is?
I Love KongFu~
|
|
|
|
|
hello,
done it!
when debugging i could see all the entries in IPGrant, the problem was converting the object array into a string array, i managed this with:
ExIpSecurity ipsec = new ExIpSecurity();
ipsec.BindToSmtpVsi(someSMTPserver, 1, "someDC");
ipsec.GetRelayIpList();
object[] o = (object[])ipsec.IPGrant;
string[] s = new string[o.Length];
Array arr = s;
o.CopyTo(arr, 0);
s = (string[])arr;
foreach (string i in s)
{
sw.WriteLine(i);
}
thanks for your help mate,
have a good weekend.
Phil
|
|
|
|
|
Hi all, can someone help with the following issue I'm having.
I am returning a dataset with a list of names of screen objects for a form.
I am trying to loop round the dataset, pick up the name then get the type of the control.
As follows...
foreach (DataRow row in dataSet_Groups.sp091_load_user_screen_components_layout.Rows)
{
strObjectName = Convert.ToString(row["ScreenObjectName"]);
strXMLText = Convert.ToString(row["XMLText"]);
intX = Convert.ToInt32(row["X"]);
intY = Convert.ToInt32(row["Y"]);
intWidth = Convert.ToInt32(row["Width"]);
intHeight = Convert.ToInt32(row["Weight"]);
intPosition = Convert.ToInt32(row["Position"]);
ctrlOnForm = frmToProcess.strObjectName;
switch (ctrlOnForm.GetType().FullName)
{
case "textEdit":
break;
}
}
frmToProcess is the form in question. How do I get the the value out of strObjectName and link it onto the the form name so I can check it's properties (see bold part in code). It's almost as if I need to perform Macro substitution (from my old FoxPro days)!
Sorry if my description is confusing.
Thanks.
|
|
|
|
|
this.Controls.Find(strObjectName, true);
TVMU^P[[IGIOQHG^JSH`A#@`RFJ\c^JPL>;"[,*/|+&WLEZGc`AFXc!L
%^]*IRXD#@GKCQ`R\^SF_WcHbORY87֦ʻ6ϣN8ȤBcRAV\Z^&SU~%CSWQ@#2
W_AD`EPABIKRDFVS)EVLQK)JKSQXUFYK[M`UKs*$GwU#(QDXBER@CBN%
Rs0~53%eYrd8mt^7Z6]iTF+(EWfJ9zaK-iTV.C\y<pjxsg-b$f4ia>
--------------------------------------------------------
128 bit encrypted signature, crack if you can
|
|
|
|
|
|
Hi all, I need help
I have a IE toolbar menu button: Generate
My flow is
After i click the Generate button, the link on my current website will appear in a:
Form1 > textBoxWebPageURL
Form the Form1.Designer.cs
this.textBoxWebPageURL.Location = new System.Drawing.Point(238, 13);
this.textBoxWebPageURL.Name = "textBoxWebPageURL";
this.textBoxWebPageURL.Size = new System.Drawing.Size(328, 20);
this.textBoxWebPageURL.TabIndex = 5;
this.textBoxWebPageURL.Text = "http://www.codeproject.com"; <-- Whereby i have to hardcode it
How do i make it auto capture the current website that i am browsing and the textbox will display the current website ?
Thank You
|
|
|
|
|
Hi, Here is below code as i am trying to populate two combobox for different values but getting OledbException while running:
I have two tables called
Packing and Products
Packing=id,name
Products:id,prodname,price,packing [defined by number]
I have setup relation between them in MSAccess which is done..
Product.Packing=Number
Packing.PackingName=Text
Code:
private void Form1_Load(object sender, EventArgs e)
{
string sql ="SELECT distinct ProductName FROM Products";
string sql2 = "SELECT pac.PackingName FROM Products prod,Packing pac where prod.Packing=pac.PackingName ";
DataSet ds=new DataSet();
try
{
database = new OleDbConnection(connectionString);
database.Open();
}
catch (Exception x)
{
MessageBox.Show(x.Message);
}
OleDbDataAdapter adp=new OleDbDataAdapter(sql,database);
adp = new OleDbDataAdapter(sql2,database);
adp.Fill(ds,"ProductName");
adp.Fill(ds,"PackingName");
foreach (DataRow dr in ds.Tables[0].Rows)
{
comboBox3.Items.Add(dr[0].ToString()) ;
comboBox1.Items.Add(dr[1].ToString());
}
}
Any help please...Thanks,
|
|
|
|
|
shah123 wrote: OleDbDataAdapter adp=new OleDbDataAdapter(sql,database); adp = new OleDbDataAdapter(sql2,database);
you are using the same data adapter for box queries. Is this what you want to do?
You need to process the first data adapter before reusing it or use different data adapter, even better, refactor your code and call it as many times as you want, for example
FillComboBox(comboBox, sql)
{
DataSet ds = new DataSet();
try
{
database = new OleDbConnection(connectionString);
database.Open();
OleDbDataAdapter adp = new OleDbDataAdapter(sql, database);
adp.Fill(ds, "tableName");
foreach (DataRow dr in ds.Tables[0].Rows)
{
comboBox.Items.Add(dr[0].ToString());
}
}
catch (Exception x)
{
MessageBox.Show(x.Message);
}
}
Yusuf
|
|
|
|
|
Thanks Yusuf, But i have one different issue then just oledb exception...
I have two tables
Products=ProdId,ProdName,Price,Packing(Number as datatype)
Packing=packingId,PackingName (Text as datatype)
Now below code is in C# ..first check pls if "sql2" is correct query and then
when i execute an application i get an error "Expression Mismatch" on line:
adp.Fill(ds,"PackingName");
Thanks,
regards,
|
|
|
|
|
I got the solution of how to tackle and did it by doing SQL query again to be bit corrected..and now no exception...
private void Form1_Load(object sender, EventArgs e)
{
string sql ="SELECT distinct ProductName FROM Products";
string sql2 = "SELECT DISTINCT Packing.PackingName FROM Packing INNER JOIN Products ON Packing.PackingID=Products.Packing";
DataSet ds=new DataSet();
try
{
database = new OleDbConnection(connectionString);
database.Open();
}
catch (Exception x)
{
MessageBox.Show(x.Message);
}
OleDbDataAdapter adp = new OleDbDataAdapter(sql, database);
adp.Fill(ds, "ProductName");
adp = new OleDbDataAdapter(sql2, database);
adp.Fill(ds,"PackingName");
foreach (DataRow dr in ds.Tables[0].Rows)
{
comboBox3.Items.Add(dr[0].ToString()) ;
comboBox1.Items.Add(dr[1].ToString());
}
But now in combobox1 i get the same values as in combobox3? 
I think so pb in foreach loop?
Any advise please...
|
|
|
|
|
Hi,
I have a problem with connecting Lotus Notes via OLE. The connector itself is working and I can generate an E-Mail. But I cannot access the property ConvertMIME.
I have played around a little bit and found out:
Type NotesSessionType = Type.GetTypeFromProgID("Notes.NotesSession");
object NotesSession = Activator.CreateInstance(NotesSessionType);
bool obj = (bool)NotesSessionType.InvokeMember("ConvertMIME", BindingFlags.DeclaredOnly | BindingFlags.Public | BindingFlags.NonPublic |
BindingFlags.Instance | BindingFlags.GetProperty, null, NotesSession, null);
Works fine. But:
NotesSessionType.InvokeMember("ConvertMIME", BindingFlags.DeclaredOnly | BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.SetProperty, null, NotesSession, new object[] {false});
Crashes with „TargetInvocationException“. HRESULT: 0x8000FFFF (E_UNEXPECTED). What I’m doing wrong?
|
|
|
|
|
Hi folks,
it seems to be a serious problem. I would like to thank all peoples who read the thread and thinking about it. I have solved this issue with a VB.DLL I wrote and call in C#. If you have another suggestion please post.
Regards
gremgiz
|
|
|
|
|
Hi
i'm looking for some tips and tricks about how to insert into relational data (Master/Detail) at the same time.
for example, user want to create new Customer, open Customer form to set it's values and insert some Orders for that Customer (which did not inserted into database yet) in Orders section, then save all together at the same time to database.
can anybody help me and has experience for this issue ?
thanks
|
|
|
|
|
A transaction could be used to wrap both insert statements into an atomic operation that would "behave" like a single statement.
I are troll
|
|
|
|
|
Thanks Eddy
I know this which run these operations under a single transaction, but my problem is that how to insert foreign key (from Customers) in to order record (because new customer has not been inserted yet, hence it does not a valid Primary key). i'm looking for some tips and tricks about this operation.
|
|
|
|
|
I think I don't understand exactly what you want/need. Would this help, or am I missing something in your question?
BEGIN TRANSACTION
INSERT INTO Customers (CustName) VALUES ('John') -- Id field is identity, and therefore not named in the fieldlist.
DECLARE @custId AS BIGINT
SELECT @custId = @@IDENTITY -- put new custId key into "@custId" var
INSERT INTO ORDER (OrderId, CustomerId) VALUES (1, @custId)
COMMIT TRANSACTION
I are troll
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
