undefined | Better HN

0 pointscanucker20161y ago0 comments

Acording to https://en.wikipedia.org/wiki/Windows_Calculator#Windows_9x_..., the Win98 version of calc.exe got an arbitrary-precision math library for basic math functionality to fix this sort of problem.

also mentioned by Raymond Chen, https://devblogs.microsoft.com/oldnewthing/20040525-00/?p=39... :

"Today, Calc’s internal computations are done with infinite precision for basic operations (addition, subtraction, multiplication, division) and 32 digits of precision for advanced operations (square root, transcendental operators)."

And they fixed the square-root-of-a-perfect-square bug a few years ago, https://www.reddit.com/r/Windows10/comments/89s53g/microsoft...

0 comments

zelphirkalt1y ago

Can this be true? Does the calculator have any irrational number? What if I enter PI * 2? Infinite precision, wouldn't that mean running until your RAM is full? Or would it simply work, because no irrational number is calculated to infinite (as many as the device could) digits and is therefore finite and can easily be multiplied?

Or another even simpler case: If division is infinite precision, and I enter (2/3) * X, does the calculator internally work with fractions? Otherwise it would have infinite digits to compute.

canucker2016OP1y ago

For PI, calc.exe, in scientific mode, returns 3.1415926535897932384626433832795 which is 32 digits of precision (33 chars if you include the decimal pt).

Khoth1y ago

For 2/3, it does indeed work with fractions internally. For pi, it uses some floating point representation I think, you get about 50 digits.

j / k navigate · click thread line to collapse