Magic binary numbers
Jul. 19th, 2004 09:57 am(or why floating point numbers are bad)
Now in theory 65.49*100-6549 should equal zero. In practice...
(same happens in perl as well)
I guess 65.49 is an irrational binary. What fun!
I hit this 'cos my money tracking programs told me one credit card had "-0.00" on it. Hmmph.
For anyone who cares the solution is:
I haven't had to do this for 20 years. I guess I don't deal with floating point numbers much!
Now in theory 65.49*100-6549 should equal zero. In practice...
% echo | awk '{ a=65.49 ; b=100*a-6549; print b }'
-9.09495e-13
(same happens in perl as well)
I guess 65.49 is an irrational binary. What fun!
I hit this 'cos my money tracking programs told me one credit card had "-0.00" on it. Hmmph.
For anyone who cares the solution is:
% echo | awk '{ a=65.49 ; b=int(100*a+0.5)-6549; print b }'
0
I haven't had to do this for 20 years. I guess I don't deal with floating point numbers much!