Click here to Skip to main content
15,867,594 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
Problem Definition:

1. Read all the records from Azure storage.
2. Loop through each row and generate another row.
3. Insert a newly generated row into the same table.

I have around 50000 records in the Azure table. So my question is How can read data in batches for such a big operation in c# for eg... Read 100 records process them then read another 100 and process them and so on?

What I have tried:

Nothing yet, Still trying to figure out how I can do batch processing of records.
Posted
Updated 7-Jun-20 11:50am
v2

A quick how to read Azure table - Google Search[^] turns up the following very helpful articles (with code):

* Working with Azure Storage Tables using C# - CodeProject[^]
* Get started with Azure Table storage using .NET | Microsoft Docs[^]
* Azure Storage Table Design Guide | Microsoft Docs[^]
* How to write to and read from Windows Azure tables within Web Applications – Bruno Terkaly – Developer Evangelist – bterkaly@microsoft.com[^]
* and much much more! :)

UPDATE For "Batching" Operations, Troy Hunt: Working with 154 million records on Azure Table Storage – the story of “Have I been pwned?”[^] talks about a TableBatchOperation method call. Also a section in the article called "Importing 22,500 rows per second into Azure Table Storage". This sounds like what you are looking for.
 
Share this answer
 
v2
Comments
deepakdynamite 1-Aug-17 2:43am    
Thanks for your reply. I'm familiar with Azure SDK. What I'm seeking is a way to read data in batches from Azure storage as it's not a good practice to read 100,000 rows in memory and then loop through it ...
C#
CloudStorageAccount cloudStorageAccount = CloudStorageAccount.Parse(configuration.GetConnectionString("Storage"));

            CloudTableClient tableClient = cloudStorageAccount.CreateCloudTableClient();
            CloudTable myEntityTable = tableClient.GetTableReference(configuration.GetSection("myEntityTable").Value);

            var continuationToken = default(TableContinuationToken);
            var query = new TableQuery<MyEntity>();
            var batchNumber = 1;

            do
            {
                var myEntityTableBatchResult = await myEntityTable .ExecuteQuerySegmentedAsync(query, continuationToken).ConfigureAwait(false);
                continuationToken = myEntityTableBatchResult.ContinuationToken;

                if (continuationToken != null && myEntityTableBatchResult != null && myEntityTableBatchResult.Results.Count > 0)
                {
                    Console.WriteLine($"Processing Batch Number {batchNumber} for My Entity table.");
                    
                    foreach (var item in myEntityTableBatchResult.Results)
                    {
                        // DO you processing.
                    }
                    Console.WriteLine($"Processing for Batch Number {batchNumber} in MyEntity table completed.");
                    batchNumber++;
                }
            }
            while (continuationToken != null);
 
Share this answer
 
Comments
[no name] 18-Jun-19 8:21am    
thanks!

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900