sweh: (Default)
[personal profile] sweh
(or why floating point numbers are bad)

Now in theory 65.49*100-6549 should equal zero. In practice...

  % echo | awk '{ a=65.49 ; b=100*a-6549; print b }'
  -9.09495e-13

(same happens in perl as well)

I guess 65.49 is an irrational binary. What fun!

I hit this 'cos my money tracking programs told me one credit card had "-0.00" on it. Hmmph.

For anyone who cares the solution is:

  % echo | awk '{ a=65.49 ; b=int(100*a+0.5)-6549; print b }'
  0


I haven't had to do this for 20 years. I guess I don't deal with floating point numbers much!
If you don't have an account you can create one now.
HTML doesn't work in the subject.
More info about formatting

January 2026

S M T W T F S
    123
45678910
11121314151617
181920212223 24
25262728293031

Style Credit

Expand Cut Tags

No cut tags
Page generated Feb. 5th, 2026 02:18 am
Powered by Dreamwidth Studios