vb.net console application to allow multhithreading process to copy files

Question

2.33/5 (3 votes)

See more:

Hi There,

I would like to know if it is possible to have a multithread application that does a copy process of about 20GB of files from a source folder to a destination folder. At the moment I have a copy process that copies 1 file at a time and need my process to be faster. Can anyone explian how this is done and illustrate with some code? I would really be thankfull if this could be showed to me.

Posted 17-Sep-13 23:20pm

Kats2512

Add a Solution

Comments

CPallini 19-Sep-13 7:48am

Probably multithreading will just slow down your copy operations.

3 solutions

Solution 2

"I would like to know if it is possible to have a multithread application that does a copy process of about 20GB of files from a source folder to a destination folder"

- Yes. I actually took the time to write one... tweak it so that people could use it to create batch jobs... adjust how many threads each copy operation would use... and then the guys at Microsoft made my solution worthless by including this functionality naively in Windows 8. You can also use Robocopy, which is a multithreaded command line tool for copying large amounts of files, and is available in earlier versions of windows.

- Pete

Posted 18-Sep-13 3:17am

pdoxtader

Solution 4

What noone here has mentions is that multi-threading your copy operation may be a complete waste of time depending on your disk configuration.

If you're running a disk array then the copy operation might speed up a bit (don't expect orders of magnitude performance increases!) because individual drives in the array can be read/written at the same time.

But, if this is a single disk, the drive can only read or write one section of a single file at a time. Having multiple threads trying to read different parts of the drive at the same time will only reasult in n-1 of those threads waiting for 1 thread to complete its operation.

Your problem is I/O bound, not compute bound. Throwing more threads at an I/O operation does little to improve performance.

A bigger jump in performance is gained by higher-performance drives.

Posted 19-Sep-13 6:07am

Dave Kreskowiak

Comments

phil.o 19-Sep-13 13:17pm

5'ed!

pdoxtader 19-Sep-13 16:40pm

Yes, although I found that you could get a performance increase while copying multiple SMALL files from and to the same drive, as long as you kept the number of threads you were using small (10 or 15 or so). If you're copying large files, then copying them sequentially and un-buffered is the way to go.

But this whole subject is pointless now... I don't think the OP (or anyone else) should pursue this anymore. That's the point I was trying to make in my solution... Microsoft has thoroughly covered it now in their current implementation of windows.

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Pheonyx · Accepted Answer · 2013-09-17T23:50:00

There are various approaches you could take to do this,

The first would be to implement the "Parallel.For" or "Parallel.ForEach" approach which will use .Net's parallel class to distribute the required operation.

This link gives an example of the Parallel.ForEach
Parallel.ForEach Example[^]

There is also an example of the Parallel.For look just here[^]

If that does not do your task, then you will want to look at your own implementation using threading, where you create threads (up to a controlled limit), and each one is responsible for copying a file.

Now, I have my own concern with this, and that is that the more copy/paste processes you have going on the faster it might be only to a point. This point is the point at which the Disk I/O speed or the Memory space (as in RAM) start to bottle neck.

When you are in windows and you start copying lots of files individually but at the same time, you should notice that all the copies "appear" to slow down. Just something to be aware of.