Algorithms

7-Oct-13 22:35

The current thinking is to calculate hash for every file ,and then delete them.
Now we can use MD5 or SHA1 to calculate hash.From the speed point of view, MD4 is a better choice when MD4 is faster but more unsafer.But after experiment, i found even md4 is so slow that it would cost a lot of time when i dealt with large files.
So any algorithm to work faster?

jschell8-Oct-13 10:28

jschell

8-Oct-13 10:28

If the name and size are the same, then delete them.

happy liu8-Oct-13 14:51

8-Oct-13 14:51

yeah ,it is a good idea.
Another situation is that -The contents of file is the same but names are different.
And i want to know which kind of hash algorithm has the best speed?
or any other way to speed up?

Manfred Rudolf Bihy9-Oct-13 0:14

Manfred Rudolf Bihy

9-Oct-13 0:14

You can considerably speed up the process: If you have mostly big files you can use a two step hash comparison. In the first step you take only say 1 - 5 KB from the beginning of the file to build your hash (I'd recommend SHA2[^]). You only perform a hash calculation for the complete file if the hashes for two files match.

Best,

Manfred

"I had the right to remain silent, but I didn't have the ability!"

Ron White, Comedian

happy liu9-Oct-13 16:44

9-Oct-13 16:44

Thanks for your advice.
However i wonder why SHA2 should be a better choice which is slower than MD5 or MD4?

Manfred Rudolf Bihy9-Oct-13 16:51

Manfred Rudolf Bihy

9-Oct-13 16:51

It's been developed to be more robust, i.e. the probability to have collisions is reduced.
Just look it up, there are plenty of good explanations on the internet on what the actual improvements over its predecessors are.

Cheers!

"I had the right to remain silent, but I didn't have the ability!"

Ron White, Comedian

happy liu9-Oct-13 16:52

9-Oct-13 16:52

Thanks! ^_^

saephoed8-Jan-14 12:50

saephoed

8-Jan-14 12:50

i found a similar solution when i had to recover 4.5TB of files mostly between .5 - 15GB where around 1-2% of them were early terminated during a copy process, but left with correct filesize filled up with zeroes.

it adds to your suggestion to scan all big files with big fixed and some random jumps up to their ends.

Kornfeld Eliyahu Peter9-Oct-13 0:50

Kornfeld Eliyahu Peter

9-Oct-13 0:50

It is almost sure that the problem not your hash algorithm but I/O. All (most) hash algorithms are short and fast enough to not blame them...

If you have large number of files you have to rethink your approach.

1. use file system's FileInfo - name, size, creation, last modified and so
2. if you can't you may consider to hash only the first block (4K) of every file and go forward only for those found the same.

I'm not questioning your powers of observation; I'm merely remarking upon the paradox of asking a masked man who he is (V).

happy liu9-Oct-13 16:42

9-Oct-13 16:42

Thank you V. ^_^
Your advice is good. In the beginning i had not a clear understanding of this question.
Now i agree I/O should be the key point, and i wish i can find a way to resolve my problem.
Thanks!

Kornfeld Eliyahu Peter10-Oct-13 1:10

Kornfeld Eliyahu Peter

10-Oct-13 1:10

Yes I/O is a key issue...
If it is in your power you may move the storage to a more capable one, like SCSI or flash, that should speed up I/O...

I'm not questioning your powers of observation; I'm merely remarking upon the paradox of asking a masked man who he is (V).

image processing toolkits | batch image processing

C^hris L^osinger10-Oct-13 5:05

C^hris L^osinger

10-Oct-13 5:05

i wrote a little app to do this for my own use, and i used multiple tests:

for every file to compare, find:
1. size - if they aren't the same size, they aren't the same file.
2. a copy of the first 100 bytes - if the first 100 bytes don't match, they aren't the same file. this is easy and fast to compare, requires no computation, and doesn't take much to store.

that stage is very fast. it generally takes far longer to just get the list of files than it does to do all of those tests.

after that,
calculate the hash (i used SHA1) of the first few KB for each of the remaining files. compare the hashes.

after those two stages, you will have eliminated most of the non-duplicates. then, you can do a full hash of the remaining files to find any definite duplicates.

Kornfeld Eliyahu Peter10-Oct-13 5:42

Kornfeld Eliyahu Peter

10-Oct-13 5:42

You may give a polish to that code and publish it here as a tip or article...

I'm not questioning your powers of observation; I'm merely remarking upon the paradox of asking a masked man who he is (V).

Manoj Kumar Rai13-Oct-13 9:26

Manoj Kumar Rai

13-Oct-13 9:26

I agree one can have mupltiple type of algorithms based on suitability:

1. check for size if does not matches then those files are not same.

2. Compare 2% character from begining and then from end if those does not matches then files are not same.

3. Finally it can be once option to simply compare the remaining part of both files or use CRC, checksum, MD SHA. If MD and SHA or not already calculated and stored then those will be a bit costly.

Manoj
Never Gives up

Estimating RGB function blackbox

theafien3-Dec-13 3:58

theafien

3-Dec-13 3:58

store information as bytes size and simple crc code.
performe loop and initially check bytes size if equal performe crc or hash check.
Or too do hash checking only when bytes then equal storing hash for future use.

codetowns27-Sep-13 23:04

codetowns

27-Sep-13 23:04

Hi,

I have a 2D color map (without a color bar depth key) that depicts depth in various colors. Ideally I would like to get an estimate of all relative depths from a function of the RGB values of a color.

What I do know is the colors of the highest, middle and lowest points. So I have something like:

Fn(R,G,B) = Depth
Fn(255,255,0) = 1000
Fn(125,125,0) = 500
Fn(22, 125, 20) = 0

I realise that there won't be an exact solution and some sort of linear assumption would have to be made, but can anyone suggest a good way to approach this? (assuming it is do-able!)

I guess the images I'm talking about would look similar to this one but have various non-rainbow color changes and also have no depth key:

Similar map link

Many thanks for any assistance.

Alan Balkany30-Sep-13 4:16

Alan Balkany

30-Sep-13 4:16

A simple, naive approach is to collect as many (R, G, B, Depth) tuples as you can get, then do multiple linear regression with Depth as the dependent variable.

If the linear assumption doesn't give satisfactory results, you can try multiple polynomial regression, or splitting the data set into two partitions, and using regression separately on each.

codetowns1-Oct-13 5:26

codetowns

1-Oct-13 5:26

Many thanks for the help.

I'm pretty sure that linear regression won't work because I have an image that has large Green values at both high and low depth.

If I understand it right, Polynomial regression for R, G or B may sometimes work but I think that often the depth can be an unknown function of all three colours.

I wonder if it makes sense to combine the best Polynomial regression fit for each one somehow?

Alan Balkany1-Oct-13 5:29

Alan Balkany

1-Oct-13 5:29

"...I have an image that has large Green values at both high and low depth."

I suspected this, which is why I suggested dividing the space into separate partitions and solving them independently. Statisticians call this "elaboration".