Click here to Skip to main content
15,900,461 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
Hi,

I have a Dataset in which I am fetching a very large number of rows i.e. around 1 million.

Now, I am sending the rows fetched in this Dataset into a method in batch sizes of 100 each.

My code is as below :


C#
List<Zsfpdate> myDataList = null;
foreach (DataRow dRow in ds.Tables[0].Rows)
{

	if ((count++ % 100) == 0)
	{
		myDataList = new List<Zsfpdate>();
		copyTable = new DataTable();
		ds1 = new DataSet();
		copyTable = ds.Tables[0].Clone();
		ds1.Tables.Add(copyTable);

	}

	copyTable.ImportRow(dRow);

	if (copyTable.Rows.Count == 100)
	{
		Program obj = new Program();
		obj.SendForProcessing(copyTable, myDataList);
	}

}



Now the problem is this condition :


C#
if (copyTable.Rows.Count == 100) i.e. everytime I need to check this condition & then send the rows to the method for processing.

However, in this way we are not able to catch the remaining rows i.e. Lets say we have 503 rows in the Dataset.
Now, the batch will run 5 times each with a 100 rows sent to the method each time. But the last 3 rows will not get captured in this way.

Can anybody suggest a workaround?

Thanks,
Aaakar
Posted
Updated 30-Mar-16 3:52am
v3

Try something like this:
C#
int count = 0;
DataTable copyTable = null;
List<Zsfpdate> myDataList = null;

foreach (DataRow dRow in ds.Tables[0].Rows)
{
    if ((count % 100) == 0)
    {
        if (copyTable != null && copyTable.Rows.Count != 0)
        {
            Program obj = new Program();
            obj.SendForProcessing(copyTable, myDataList);
        }
        
        myDataList = new List<Szfpdate>();
        copyTable = ds.Tables[0].Clone();
        ds1 = new DataSet();
        ds1.Tables.Add(copyTable);
    }
    
    copyTable.ImportRow(dRow);
    count++;
}

if (copyTable != null && copyTable.Rows.Count != 0)
{
    Program obj = new Program();
    obj.SendForProcessing(copyTable, myDataList);
}
 
Share this answer
 
You can just check below the foreach-loop if the copyTable contains any left-over rows.

Also, it appears you're doing some unneccessary stuff in the first if-block: You're double-initializing copyTable and you never use the DataSet ds1 for anything. And I'm wondering what the myDataList is good for because you're sending it empty into SendForProcessing(..) and even if that method inserts something into it, after that you don't use it here. So it seems redundant to me, at least here - if you need one in SendForProcessing(..) then you could create it there. (Unless you've left out code here.)

I've commented out the lines which I think aren't required or which I have replaced:

C#
//List<Zsfpdate> myDataList = null;

copyTable = ds.Tables[0].Clone();

foreach (DataRow dRow in ds.Tables[0].Rows)
{
    //if ((count++ % 100) == 0)
    //{
        //myDataList = new List<Zsfpdate>();
        //copyTable = new DataTable();
        //ds1 = new DataSet();
        //copyTable = ds.Tables[0].Clone();
        //ds1.Tables.Add(copyTable);
    //}
 
    copyTable.ImportRow(dRow);
 
    if (copyTable.Rows.Count == 100)
    {
        //Program obj = new Program();
        //obj.SendForProcessing(copyTable, myDataList);
        new Program().SendForProcessing(copyTable);
        copyTable = ds.Tables[0].Clone();
    }
}

if (copyTable.Rows.Count > 0)
{
    new Program().SendForProcessing(copyTable);
}
 
Share this answer
 
v2
You can also try another kind of loop:
C#
for(first=0; first< ds.Tables[0].Rows.Count ; first+=100)
{
    last= min(first+100, ds.Tables[0].Rows.Count);

    myDataList = new List<zsfpdate>();
    copyTable = new DataTable();
    ds1 = new DataSet();
    copyTable = ds.Tables[0].Clone();
    ds1.Tables.Add(copyTable);   

    for cnt= first; cnt < last; cnt++)
    {
        copyTable.ImportRow(ds.Tables[0].Rows[cnt]);
    }

    //Program obj = new Program();
    //obj.SendForProcessing(copyTable, myDataList);
    new Program().SendForProcessing(copyTable);
    copyTable = null;
}

Untested
The principle, you make an outer loop that tell you which blog to process.
Nota: your original loop look a little complicated.
 
Share this answer
 
v5

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900