[Ti] interesting calculator behavior!?

[Steve Wozniak]

At 3:46 PM +0200 2003.08.26, Eduard Hoenkamp wrote:

>I happened upon the following oddity using the OSX calculator (the brushed metal one under Applications). This happens in 'advanced' mode.
>
>112*46 ->    5152
>but:
>112*4.6 ->   515.1999999999999
>which should have been 515.2 obviously (and which it does produce in 'basic' mode). It reminded me of the Intel processor bug, and I hope it tickles someone's curiosity and/or imagination. Eduard.

Your problem is that 4.6 (decimal) does not store perfectly in binary. It repeats forever, like 1/3 does in decimal (.3333333...). 6/10 in binary is .10011001100... At some point you have to clip off a dwindling amount, and perfect precision calculations from that point on have "roundoff" errors.

There have been computer systems that kept track of fractions but not common ones.

These days computers do have hardware and software techniques to maintain extreme accuracy (it gets worse with transcendentals like sin and cosine). But the trick still becomes one of rounding answers to display fewer digits.

A lot of scientific types prefer metric to English units because of a base 10 commonality. Actually, nature has dictated that the only bases which would avoid roundoff errors due to base conversion on computers would be binary bases, like base 8 or base 16.
-- 

Regards,

Steve  (is tv wake zone?)