Concatenate large number of binary files

Question

2.00/5 (1 vote)

See more:

I have a large number (~1800) of small (~2-6Kb) files that i wish to concatenate into one large file.

I tried using a PowerShell script using Get-Content and Add-Content, but that was agonisingly slow. Does anyone know of a relatively fast way of doing this using either a PowerShell script, some other system commands, or a small application?

I don't want to download and install some bloat-ware just to do this simple operation, but a single stand-alone executable would be fine. I guess i could whack together a small C++ application myself, but i don't know if that would be any faster.

So, any suggestions?

(EDIT: I should mention that i did use the -ReadCount flag on Get-Content, to increase the input buffer, but that didn't seem to help. And, of course, the -Encoding was set to byte. So i'm not sure if the bottle-neck is I/O or something PowerShell is doing internally.)

Posted 22-Jan-12 12:12pm

XTAL256

Updated 13-May-20 1:12am

v3

Add a Solution

Comments

Sergey Alexandrovich Kryukov 22-Jan-12 18:17pm

Tag your platform and language. (Yes, I see C++ and PowerShell, but tag it! You are interested in it, not us.)
The problem is very simple. Where is your difficulty?
--SA

XTAL256 22-Jan-12 18:24pm

Hmm, i know i am/could use C++ and PowerShell, but my question was not limited to them. But i will use those tags at your request.

Sergey Alexandrovich Kryukov 22-Jan-12 18:19pm

As to the "faster". Why do you think anyone is more interested in investing her or his time in this boring task then you do? Why C++, by the way? The only thing you know or what? Why PowerShell?
--SA

Peter_in_2780 22-Jan-12 18:43pm

What's wrong with the wildcard form of copy command?
copy /b sourcesdir\*.* destfile
or something very similar

XTAL256 22-Jan-12 21:22pm

I have seen this method mentioned in other forums, when i was searching for the PowerShell way the other day. But those people just gave an example of two or more files. I didn't think of using a wildcard.
I doubt it would be much faster, but i could give it a go.

EDIT: Wow, that's super fast! I don't have access to the actual files at the moment, so i just tried it on some dummy data, but the operation took only a few seconds for 1000 files of 3Kb each.

thanks

Peter_in_2780 22-Jan-12 22:59pm

Seeing you like it, I'm going to turn my comment into an answer. Then you can accept it. :)
Peter

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Peter_in_2780 · Accepted Answer · 2012-01-22T17:03:00

What's wrong with the wildcard form of copy command?

copy /b sourcesdir\*.* destfile

or something very similar

Comment: The actual copying of the files is nothing compared to finding, opening, closing them, etc. That explains why wildcards is the big speedup. Not a line of code to write, either! :)

Cheers,
Peter