Re: Possible math bug?
Re: Possible math bug?
- Subject: Re: Possible math bug?
- From: Emmanuel <email@hidden>
- Date: Fri, 18 Feb 2005 09:11:26 +0100
At 8:26 PM -0800 2/17/05, Paul Berkowitz wrote:
On 2/17/05 8:20 PM, "Hanaan Rosenthal" <email@hidden> wrote:
Can anyone explain this:
1.000088E+4 - (round 1.000088E+4)
result:
-0.120000000001
The translated version of this is:
10000.88 - (round 10000.88)
and the result should be (I think) -0.12
A typical floating point imprecision. Computers are built on powers
of 2, not powers of 10. Calculations using extreme decimal fractions
almost always have errors of this sort.
To elaborate on Paul's reply and help you predict what happens,
remember that AppleScript computes in double precision: 8 bytes,
including 2 for the signs and the exponent. This means, a relative
precision of 2-48, or 1E-15. When you pass to AppleScript a number
around 1E4, and unless you're specially lucky (you're passing a sum
of powers of 2), AppleScript may store it exact up to 1E4 x 1E-15 =
1E-11. You have been observing an error of 1E-12, which is fair, thus.
Emmanuel
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Applescript-users mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden