• Open Menu Close Menu
  • Apple
  • Shopping Bag
  • Apple
  • Mac
  • iPad
  • iPhone
  • Watch
  • TV
  • Music
  • Support
  • Search apple.com
  • Shopping Bag

Lists

Open Menu Close Menu
  • Terms and Conditions
  • Lists hosted on this site
  • Email the Postmaster
  • Tips for posting to public mailing lists
Re: Possible math bug?
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Possible math bug?


  • Subject: Re: Possible math bug?
  • From: Emmanuel <email@hidden>
  • Date: Fri, 18 Feb 2005 09:11:26 +0100

At 8:26 PM -0800 2/17/05, Paul Berkowitz wrote:
On 2/17/05 8:20 PM, "Hanaan Rosenthal" <email@hidden> wrote:

Can anyone explain this:

1.000088E+4 - (round 1.000088E+4) result:
-0.120000000001


The translated version of this is:
10000.88 - (round 10000.88) and the result should be (I think) -0.12




A typical floating point imprecision. Computers are built on powers of 2, not powers of 10. Calculations using extreme decimal fractions almost always have errors of this sort.

To elaborate on Paul's reply and help you predict what happens, remember that AppleScript computes in double precision: 8 bytes, including 2 for the signs and the exponent. This means, a relative precision of 2-48, or 1E-15. When you pass to AppleScript a number around 1E4, and unless you're specially lucky (you're passing a sum of powers of 2), AppleScript may store it exact up to 1E4 x 1E-15 = 1E-11. You have been observing an error of 1E-12, which is fair, thus.


Emmanuel
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Applescript-users mailing list      (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden


References: 
 >Re: Possible math bug? (From: Paul Berkowitz <email@hidden>)

  • Prev by Date: Re: Quark 5 'every text' problem
  • Next by Date: Re: Getting list of connected users
  • Previous by thread: Re: Possible math bug?
  • Next by thread: make new finder window with properties {target:alias} falls short?
  • Index(es):
    • Date
    • Thread