**De vergankelijkheid - double-precision**

2005/06/13 | J. Lester Novros II

For years I wondered why the calculator on my computer did not work correctly even though I paid lots for the machine ity came on. However, I just read this very enlightening comment on Slashdot and now, finally, I know. It cannot be done. Heh, who would've thought that.>>

This isn't a bug in the arithmetic; it is an artifact of the design. The calculator is almost certainly using the hardware's double-precision floating-point numbers for calculation. The implementation obeys the IEEE 754 standard for floating-point arithmetic. It represents a number with a sign, 11 bits for an exponent of two, and 53 bits for the significand (the "fraction part" in some sense and including an implicit one bit).

The mathematical number 9533.24 cannot be represented exactly as a double-precision number, because 9533.24 expressed in binary has a repeating string that goes on forever. It is 10010100111101.00111101011100001010001111010111000 01010001111... When you round it after 53 bits, you have 10010100111101.00111101011100001010001111010111000 0101, or 81889908046875/8589934592 or about 9533.2399999999997817.

Similarly, 215.10 is 11010111.00011001100110011001100110011001100110011 00110011001... Rounded to 53 digits, that is 11010111.00011001100110011001100110011001100110011 0011 or 7568158436307763/35184372088832 or about 215.09999999999999432.

The difference is exactly 327852904935829005/35184372088832 or 10010001100110.00100011110101110000101000111101011 1000001101 or about 9318.1399999999997874. However, you cannot represent the difference in double-precision, because it requires too many bits. The result of a subtraction instruction is rounded, and you get 640337704952791/68719476736 or 10010001100110.00100011110101110000101000111101011 1 or about 9318.1399999999994179.

(Caveat: I produced the above numbers with some quick Maple commands. They could be off a bit, but the concepts are correct.)

It might be nice if calculators intended for the general public used decimal arithmetic internally. (But it still would not be able to exactly calculate 1/3 * 3. There will always be limits to mathematical correctness.) But that is an issue of application design; it has nothing to do with correct floating-point results, as mentioned in the post you responded to. The floating-point arithmetic here is correct.

edp (Eric Postpischil)