Click here to Skip to main content
15,904,828 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
I have a requirement to download 10000+ files asynchronously from Sharepoint online.
But there are some limitations on the framework, it has to be coded using.NET 4.0 framework only. Hence cant advanced TPL library. I wanted to use Task-based Asynchronous Pattern (TAP): "Asynchronous programming - C# | Microsoft Docs[^]

How do I download multiple files using async way?

the below code method runs for each file in a synchronous manner, is it possible that I make the call to this method async using batch wise logic?

What I have tried:

private bool WritetoDisk(ClientContext context, Microsoft.SharePoint.Client.File file, string ListName)
      {
          try
          {


              var sw = new Stopwatch();
              sw.Start();

              string Destination = @"c:\\temp";
              string destinationfolder = Destination + "/" + ListName;

              Stream fs = Microsoft.SharePoint.Client.File.OpenBinaryDirect(context, file.ServerRelativeUrl).Stream;

              byte[] binary = ReadFully(fs);


              if (!Directory.Exists(destinationfolder))
              {
                  Directory.CreateDirectory(destinationfolder);
              }

              FileStream stream = new FileStream(destinationfolder + "/" + file.Name, FileMode.Create);

              BinaryWriter writer = new BinaryWriter(stream);

              writer.Write(binary);

              writer.Close();
              sw.Stop();


          }
          catch (Exception e)
          {
          }
          return true;
      }
Posted
Updated 25-Jun-19 19:47pm
v2
Comments
[no name] 25-Jun-19 8:21am    
Ha ha ha ha ha ha ha. You funny.
Richard MacCutchan 25-Jun-19 8:26am    
You could just run 10000+ copies of your app.
Afzaal Ahmad Zeeshan 25-Jun-19 12:14pm    
On 10,000 machines, with dedicated network bandwidth. :laugh:

Don't be silly.
What you are trying to do (I suspect) is run 10,000+ downloads simultaneously, not asynchronously - and that won't improve the download speed in any way, shape, or form: it may make it slower (it almost certainly will) and it will most likely crash the app or possibly one or other of the systems.

Why?
Well ... have you ever heard of DDoS? Distributed Denial of Service? That works by flooding a server with requests so that it is so busy it crashes or is otherwise unable to process new (legitimate) requests.
And that's what you are trying to do, without the "Distributed" part.

No matter how much memory you have, you will exhaust it: and then yoru system will go into grind mode as it starts paging to disk in order to free up memory to add more downloads, which are already using so much bandwidth that you internet connection slows to a crawl ...

No, you can't do it.
 
Share this answer
 
Comments
karekarmahadev 26-Jun-19 1:38am    
@OriginalGriff, Thanks for your comments!!. Yes, you are right, I had tried to use Multithreading approach, but the application went into "Not Responding " mode.
OriginalGriff 26-Jun-19 1:44am    
You're welcome!
Quote:
10000+ files asynchronously from Sharepoint online
Then why don't you use the SharePoint SDKs for this? They will take care of how to order the download jobs, download and provide you with the 10,000+ files on your machine.

Welcome to the Microsoft SharePoint 2010 SDK | Microsoft Docs[^]
Quote:
But there are some limitations on the framework, it has to be coded using.NET 4.0 framework only. Hence cant use TPL library.
But you can use TPL in .NET framework 4.0. Don't take my word for this,
Starting with the .NET Framework 4, the TPL is the preferred way to write multithreaded and parallel code.
So basically, you can do both the things without any problem—read OriginalGriff's answer on what limitations you will be hitting in your own machine as well as on Microsoft's servers. A good way to achieve this requirement would be to first zip the folder (or the files, wherever they are placed) and then download that single zip file.

Remember, with asynchronous approach you are not improving the performance or download time rather (at the scale of 10,000+ tasks) you are degrading the performance at a big scale. Each task would require internet, as well as memory and CPU resources, and this might cause starvation in your tasks... Many bad things will happen, so to say.
 
Share this answer
 
Comments
karekarmahadev 26-Jun-19 0:39am    
"Then why don't you use the SharePoint SDKs for this? They will take care of how to order the download jobs, download and provide you with the 10,000+ files on your machine."

Yes, I am using SharePoint SDKs, which allows me to download 10000+ files without any issues. What I wanted to do is speed up this operation, so was thinking about asynchronous way. But as per some expert comments above looks like this is not a good approach.
Afzaal Ahmad Zeeshan 26-Jun-19 5:25am    
Improvement in the download speed doesn't need any programming (apart from some fine tunings) but in real it requires a good internet connection.
karekarmahadev 26-Jun-19 1:32am    
"But you can use TPL in .NET framework 4.0. Don't take my word for this, "
>> Agree, what I wanted to use was Aync/Task, My bad I thought it was part of TPL.
Task-based Asynchronous Pattern (TAP). https://docs.microsoft.com/en-us/dotnet/csharp/async

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900