Best method to copy a lot of files to a remote storage?

Question

1.00/5 (2 votes)

See more:

hi
i need to copy a lot of image files (more that a thousand image, each of which is maximum 1 MB), which are stored on different machines on intranet, to a central NAS storage.
i will write a C# program for that..
what is the most efficient method to do the job? is (File.Copy) enough, or...?

This process will be a daily job,
and it's NOT JUST a copying process.. files have to be organized in a specific hierarchy and their information has to be updated in database, so i think synchronization applications won't work for me

Since i need to update file status on database, then i need to know when the copying process of the file is finished, and whether it succeeded or failed (in in general, sometimes files are corrupted for some reason when copying fails)

What I have tried:

i have tried File.Copy for little number of files.. but haven't tried it for a large number.

Posted 8-Nov-17 6:57am

nina4ever

Updated 8-Nov-17 22:05pm

OriginalGriff

v2

Add a Solution

Comments

phil.o 8-Nov-17 15:52pm

Most efficient in which terms? Clearly your requirements go further the simple file copy: you have to get the source file, link it to its record in the database, categorize it on criteria we don't know, then copy it to its destination, maybe verify by a checksum it has not been altered in the process, then, if check has succeeded, delete the source file and update the database record with the new path.
File.Copy can do its job, but it's only a little part of the whole process.

nina4ever 9-Nov-17 3:50am

you are right. it is a whole process, and what you mentioned is correct.
but in my question i am only concerned about efficiency of the (copying) itself, since i have a large number of files, and i need a way to make sure they have been copied correctly, without corruption for example, and of course i need to know when file copy has finished (as i recall, by File.Copy there is no notification when copy is finished)..

phil.o 9-Nov-17 4:16am

If you want to have an idea of the progress of a copy operation, you can go for streams. You will find plenty of examples of file copy with streams using delegates to signal the current progress. File.Copy does not provide this feature.

nina4ever 9-Nov-17 5:33am

thanks for your help. i will give it a try

nina4ever 9-Nov-17 5:40am

i found these answers. are they good?

https://stackoverflow.com/questions/39591579/how-to-use-filestream-for-copying-files-in-c-sharp

https://stackoverflow.com/questions/3427185/using-filestream-classes-to-copy-files-why-does-output-not-match-input

phil.o 9-Nov-17 6:33am

Yes they are. There is also an article on CP: File Copy Control with Progress Indication[^]

nina4ever 13-Nov-17 2:51am

i know how to use credentials to copy files from a remote machine to a local path, and vice versa.. but.. how to do that when both source and destination are remote machines and both need credentials?

phil.o 13-Nov-17 6:06am

Then I guess you need to provide credentials for both of them.

nina4ever 13-Nov-17 6:33am

sure.. i usually use this class
to get access to a remote machine using its IP and credentials :

https://gist.github.com/AlanBarber/92db36339a129b94b7dd

but i can't switch the connection to the destination IP in the middle of copy !!
do i have to copy files from source to temp local folder first, and then gain access to destination and re-copy the file??? or is there a way to get access permission to 2 IPs in the same time ??

phil.o 13-Nov-17 6:42am

You need two distinct instances of the NetworkConnection class, each with its own credentials.

nina4ever 13-Nov-17 7:26am

thanks

Sinisa Hajnal 9-Nov-17 6:14am

1000 files is not too much on any reasonable system. Focus on the rest of the functionality.

nina4ever 9-Nov-17 13:58pm

the problem is that some links in the company intranet between remote machines and the central storage are SLOW.. that's why the copy process isn't that trivial

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)