• Open Menu Close Menu
  • Apple
  • Shopping Bag
  • Apple
  • Mac
  • iPad
  • iPhone
  • Watch
  • TV
  • Music
  • Support
  • Search apple.com
  • Shopping Bag

Lists

Open Menu Close Menu
  • Terms and Conditions
  • Lists hosted on this site
  • Email the Postmaster
  • Tips for posting to public mailing lists
Re: Possible math bug?
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Possible math bug?

  • Subject: Re: Possible math bug?
  • From: Paul Berkowitz <email@hidden>
  • Date: Thu, 17 Feb 2005 20:26:06 -0800
Title: Re: Possible math bug?
On 2/17/05 8:20 PM, "Hanaan Rosenthal" <email@hidden> wrote:

Can anyone explain this:

1.000088E+4 - (round 1.000088E+4)  
result:
-0.120000000001

The translated version of this is:
10000.88 - (round 10000.88)  
and the result should be (I think) -0.12


A typical floating point imprecision. Computers are built on powers of 2, not powers of 10. Calculations using extreme decimal fractions almost always have errors of this sort.

--
Paul Berkowitz
 
 _______________________________________________
Do not post admin requests to the list. They will be ignored.
Applescript-users mailing list      (email@hidden)
Help/Unsubscribe/Update your Subscription:

This email sent to email@hidden
  • Follow-Ups:
    • Re: Possible math bug?
      • From: Emmanuel <email@hidden>
References: 
 >Possible math bug? (From: Hanaan Rosenthal <email@hidden>)

  • Prev by Date: Possible math bug?
  • Next by Date: Re: Viewing a jpeg
  • Previous by thread: Possible math bug?
  • Next by thread: Re: Possible math bug?
  • Index(es):
    • Date
    • Thread