|
i found a similar solution when i had to recover 4.5TB of files mostly between .5 - 15GB where around 1-2% of them were early terminated during a copy process, but left with correct filesize filled up with zeroes.
it adds to your suggestion to scan all big files with big fixed and some random jumps up to their ends.
|
|
|
|
|
It is almost sure that the problem not your hash algorithm but I/O. All (most) hash algorithms are short and fast enough to not blame them...
If you have large number of files you have to rethink your approach.
1. use file system's FileInfo - name, size, creation, last modified and so
2. if you can't you may consider to hash only the first block (4K) of every file and go forward only for those found the same.
I'm not questioning your powers of observation; I'm merely remarking upon the paradox of asking a masked man who he is (V).
|
|
|
|
|
Thank you V. ^_^
Your advice is good. In the beginning i had not a clear understanding of this question.
Now i agree I/O should be the key point, and i wish i can find a way to resolve my problem.
Thanks!
|
|
|
|
|
Yes I/O is a key issue...
If it is in your power you may move the storage to a more capable one, like SCSI or flash, that should speed up I/O...
I'm not questioning your powers of observation; I'm merely remarking upon the paradox of asking a masked man who he is (V).
|
|
|
|
|
i wrote a little app to do this for my own use, and i used multiple tests:
for every file to compare, find:
1. size - if they aren't the same size, they aren't the same file.
2. a copy of the first 100 bytes - if the first 100 bytes don't match, they aren't the same file. this is easy and fast to compare, requires no computation, and doesn't take much to store.
that stage is very fast. it generally takes far longer to just get the list of files than it does to do all of those tests.
after that,
calculate the hash (i used SHA1) of the first few KB for each of the remaining files. compare the hashes.
after those two stages, you will have eliminated most of the non-duplicates. then, you can do a full hash of the remaining files to find any definite duplicates.
|
|
|
|
|
You may give a polish to that code and publish it here as a tip or article...
I'm not questioning your powers of observation; I'm merely remarking upon the paradox of asking a masked man who he is (V).
|
|
|
|
|
I agree one can have mupltiple type of algorithms based on suitability:
1. check for size if does not matches then those files are not same.
2. Compare 2% character from begining and then from end if those does not matches then files are not same.
3. Finally it can be once option to simply compare the remaining part of both files or use CRC, checksum, MD SHA. If MD and SHA or not already calculated and stored then those will be a bit costly.
Manoj
Never Gives up
|
|
|
|
|
store information as bytes size and simple crc code.
performe loop and initially check bytes size if equal performe crc or hash check.
Or too do hash checking only when bytes then equal storing hash for future use.
|
|
|
|
|
Hi,
I have a 2D color map (without a color bar depth key) that depicts depth in various colors. Ideally I would like to get an estimate of all relative depths from a function of the RGB values of a color.
What I do know is the colors of the highest, middle and lowest points. So I have something like:
Fn(R,G,B) = Depth
Fn(255,255,0) = 1000
Fn(125,125,0) = 500
Fn(22, 125, 20) = 0
I realise that there won't be an exact solution and some sort of linear assumption would have to be made, but can anyone suggest a good way to approach this? (assuming it is do-able!)
I guess the images I'm talking about would look similar to this one but have various non-rainbow color changes and also have no depth key:
Similar map link
Many thanks for any assistance.
|
|
|
|
|
A simple, naive approach is to collect as many (R, G, B, Depth) tuples as you can get, then do multiple linear regression with Depth as the dependent variable.
If the linear assumption doesn't give satisfactory results, you can try multiple polynomial regression, or splitting the data set into two partitions, and using regression separately on each.
|
|
|
|
|
Many thanks for the help.
I'm pretty sure that linear regression won't work because I have an image that has large Green values at both high and low depth.
If I understand it right, Polynomial regression for R, G or B may sometimes work but I think that often the depth can be an unknown function of all three colours.
I wonder if it makes sense to combine the best Polynomial regression fit for each one somehow?
|
|
|
|
|
"...I have an image that has large Green values at both high and low depth."
I suspected this, which is why I suggested dividing the space into separate partitions and solving them independently. Statisticians call this "elaboration".
|
|
|
|
|
Is it possible that converting from RGB to HLS color representation might give values mre amenable to regression? Just a thought.
|
|
|
|
|
1-How to make a sandbox that sandboxing and preventing Malicous Java script at URL address bar with such permissions like never change the title page , never change the images or flash or any other multimedia from outside in the page and never excute a code from outside ?
2-after detecting and preventing through Sandbox , how to take the piece of code which causes the problem from a big malicious script ? e.g. eval() is the responsible for making the problem so i want the program to catch for the 3rd step
3-if for example eval() is the bad function which causes the trouble then how to REPLACE it -mapping- with a safe function? -lets say only a mapping for eval() ,execScript , SetTimeOut , SetTimeInterval
innerHTML , outerHTML document.cookies, location.replace , location.assign , script.src href , src , Iframe -
I red many articles but i didnt find safe mapping for unsafe functions
What are the safe mapping for those unsafe functions or properties??
I red this it was helpful but not complete:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/eval[^]
Note:
Though I hope to make this Algorithm implemented with c# -I posted this question there- but any help to solve this queston with any language will be welcomed.
|
|
|
|
|
|
Am working on a robotic arm, and I intend using an algorithm to generate the shortest route to a given point in3D space. Its kind of like optimization .Stuff for shortest route . etc....Am to use C# and genetic .Algorithm. Please help out . .
|
|
|
|
|
You need to define classes for Route (which will contain an array of genes), and Population (which will contain an array of Routes. Then write an evaluation function for class Route that gives the "value" of the Route's gene list; the length of the route seems appropriate.
Initialize the genes of each Route in the Population to small random values. Then:
Evaluate each Route. Delete the Routes with the highest scores (e.g. keep the 10% with the shortest routes). Refill the population with combinations of genes from pairs of Routes in the shortest 10%, with random "mutations" (altering one or more genes).
Repeat until a short enough route evolves.
|
|
|
|
|
|
Hi,
I need to write a pesudocode about InvertRelation function for 2-level Binary Decision diagrams.
InvertRelation takes as input a 2L-level quasi-reduced BDD rooted at r encoding a relation R : B^L → 2^{B^L} and returns the 2L-level quasi-reduced BDD rooted at s encoding the relation R^{−1} :B^L →2^{B^L},that is, j ∈ R(i) iff i ∈ R^{−1}(j).
the input BDD r uses the variable order x′1, x1, ..., x′L, xL and that the result BDD s uses the variable order x1,x′1,...,xL,x′L. Thus r is at level L and its children are at level L′, while s is at level L′ and its children are at level L.
Any help is appreciated.
compengr
|
|
|
|
|
Funny, I was just looking for that.
I couldn't find anything though.
yes this post is useless..
|
|
|
|
|
If you want some help, try explaining what you need without cryptic terms you haven't defined.
|
|
|
|
|
I'm currently studying cryptography implementation in depth. To that end I'm reading various RFCs and trying to implement the algorithms so I can get a better understanding of them and hopefully make my applications more secure when cryptography is used. Right now I'm reading the TLS RFC 5246.
The first algorithm in the document is a pseudorandom function (PRF) that takes a secret, a seed, and a label and produces an output of a specified length. This is Section 5 (Page 14 in the PDF) of the document. It defines a function called P_hash(secret, data) that uses a single hash function to expand a secret and seed to an arbitrary length:
Pseudo-code (Page 15):
P_hash(secret, seed) = HMAC_hash(secret, A(1) + seed) +
HMAC_hash(secret, A(2) + seed) +
HMAC_hash(secret, A(3) + seed) + ...
where + indicates concatenation.
A() is defined as:
A(0) = seed
A(i) = HMAC_hash(secret, A(i-1))
P_hash is iterated as many times as necessary to produce the required length. The example given in the RFC is if P_SHA256 is being used to create 80 bytes, it will be iterated three times (through A(3)), creating 96 bytes of data where the last 16 bytes of the final iteration is discarded to leave the needed 80 bytes.
The PRF is created by applying P_hash to the secret as follows:
PRF(secret, label, seed) = P_<hash>(secret, label + seed)
I've defined the two functions (PRF & P_hash) but added two additional parameters; the first if reqLength to set the output length and HmacLength which is an enum that will restrict the allowed hash function to SHA256, SHA384, or SHA512.
The code that follows I believe follows the RFC correctly, however I feel it is inefficient because of how I'm going back and forth between List(Of Byte) and Byte()'s, but I can't figure out how to simplify it; possibly because I've been programming all day or maybe because it is 1:30 AM. Either way I was hoping that someone could help simplify the code because everything I tried (like eliminating some of the loops) resulted in the code not compiling for various reasons.
The code I have so far is this:
Public Class PRF
Public Enum P_SHA
HMAC_256
HMAC_384
HMAC_512
End Enum
Public Function PRF(secret As Byte(), label As Byte(), seed As Byte(), reqLength As Integer, Optional HmacLength As P_SHA = P_SHA.HMAC_512)
Dim temporaryArray As New List(Of Byte)
temporaryArray.AddRange(label)
temporaryArray.AddRange(seed)
Return P_hash(secret, temporaryArray.ToArray, reqLength, HmacLength)
End Function
Private Function P_hash(secret As Byte(), seed As Byte(), reqLength As Integer, Optional HmacLength As P_SHA = P_SHA.HMAC_512) As Byte()
Dim data As New List(Of Byte)
Dim HMAC_hash As HMAC
Select Case HmacLength
Case P_SHA.HMAC_256
HMAC_hash = New HMACSHA256(secret)
Case P_SHA.HMAC_384
HMAC_hash = New HMACSHA384(secret)
Case Else
HMAC_hash = New HMACSHA512(secret)
End Select
Dim i As Integer = 1
Dim A As New List(Of Byte())
A.Add(seed)
Dim concatenateByte As New List(Of Byte())
Dim temporaryArray As New List(Of Byte)
Do Until data.Count >= reqLength
concatenateByte.Clear()
concatenateByte.Add(A(i - 1))
concatenateByte.Add(seed)
For Each byt As Byte() In concatenateByte
For Each b As Byte In byt
temporaryArray.Add(b)
Next
Next
data.AddRange(HMAC_hash.ComputeHash(temporaryArray.ToArray))
Loop
If data.Count = reqLength Then
Return data.ToArray
Else
Return data.GetRange(0, reqLength).ToArray
End If
End Function
End Class
The arrows indicate the parts of the code that I imagine there is a way to eliminate although I can't see it.
Any suggestions or advice would be greatly appreciated. Thanks in advance.
|
|
|
|
|
I pass Event mechanism in MS, locking send / publish notification .I do not pass Concurrent Collection yet.
Consider, as definition ask, Events as a certain particular type of Delegates.When invoke declared Delegate,the correspondent method is invoked.Assume that invocation is made for Annonymous Methods, but - code for method is recommanded to be expressed much more as Lambda Expression, Closures and Functors.- .So,I consider evaluating Closures with SystemReflection namespace.Intuitvly seems that
SystemReflection offer all need stuff for this.Can Closures be somehow different evaluated?
modified 10-Aug-13 7:51am.
|
|
|
|
|
Does anyone know of a paid or free C# implementation of GARCH?
I have thumbed thru few Excel spreadsheet samples and C++ code in Boost as well as statistics based library source ... but that implementation seems too complicated because I want something for educational purposes. May eventually move to the same industrial versions found in those library but it is difficult to start from there.
GARCH stands for Generalized AutoRegressive Conditional Heteroskedasticity. It is a statistical algorithm
|
|
|
|
|
 How to find the shortest path between two nodes in a undigraph,but the constraints of this path is that it should via several given intermediate nodes. This uidigraph is a sparse graph and may not a connected graph.
The weight of every edge in the graph is the same. The number of nodes in the graph is about several thousand.
Do you have some ideas about it? Looking forward to your reply!
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
