Click here to Skip to main content
15,868,016 members
Please Sign up or sign in to vote.
1.00/5 (1 vote)
See more:
for example i have 0.093 its output should be 0.092. another example if i have 0.15 it should be 0.14 and so on for others. i try this code: but the result for 0.093 is -0.007

What I have tried:

dim result as integer
result = val(orig.text) - .1
Posted
Updated 16-Oct-17 21:19pm

That's not going to be as simple as your think: reducing the least significant digit by one is ok for integer values, but floating point values are not stored in a way which it totally compatible with that: since they are stored as binary, they only "approximate" to the value you see, rather than genuinely holding the actual value.

And the code you show has problems: integers do not hold the factional part of a number at all, so your two example numbers would both give a result of zero.

But ... you are trying to reduce the least significant digit of a user entered string, which actually makes the process easier!
Work with the string directly, and you can do it:
1) Identify the index of least the significant digit: initially that is the length of the string minus one.
2) Convert the string to a char array
3) Now loop.
3.1) Look at the character at the index. Is it zero?
3.2) If it is, make it a nine, decrement the index by one, and continue with the loop.
3.3) Otherwise, is it a period?
3.3.1) If it is, decrement the index by one, and continue with the loop.
3.3.2) Otherwise, reduce the character buy one, and exit the loop.
4) After the loop, convert the char array back to a string, and convert it to a Double
 
Share this answer
 
Comments
GKP1992 17-Oct-17 4:44am    
I was considering the point you make in step 3.2, but I did not go with it since a fraction with a 0 in the end, did not make sense. Since it is a user entered string, it doesn't have to.
OriginalGriff 17-Oct-17 5:16am    
I thought about it, but figured that if the user typed 'em, he probably had a reason...
But it's simple to get rid of them, just trim the string:
input = input.TrimEnd("0"C)
GKP1992 17-Oct-17 6:55am    
I think trimming will give a wrong result if I use my solution, 0.60 - 0.01 = 0.50, whereas 0.6 - 0.1 = 0.5. My vote of 5.
GKP1992 17-Oct-17 7:01am    
Just tested my solution with trailing zeroes, it works :-D, even though I did not intend it to.
From what I understand, you need to subtract 1 from the face value of the digit that occurs at the end of a fraction.

To do this, first, you'll have to find the position of the decimal in the fraction, then subtract 1/10^positionOfDecimal from the number.

Example code.
Dim result As Interger
Dim indexOfDecimalPoint As Integer = orig.text.IndexOf(".")
Dim numberOfDecimals As Integer =  orig.text.Substring(indexOfDecimalPoint + 1).Length

result = Val(orig.text) - 1/(10^numberOfDecimals)


The above code should do what you're trying.
 
Share this answer
 
v3

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900