Re: number with one decimal
Re: number with one decimal
- Subject: Re: number with one decimal
- From: Doug McNutt <email@hidden>
- Date: Mon, 18 Jul 2011 14:01:37 -0600
At 21:26 +0200 7/18/11, Willem wrote:
>So applescript thinks that 82 / 10 = 8,1999999999
I can't help you with formatting but what you're seeing is a much deeper situation that is involved with the way computers handle decimal fractions in IEEE floating point binary arithmetic.
1/10 base ten converted to binary is a repeating number:
0.000110011001100110011..... forever
It's just like 1/3 base ten is .3333333333333... forever
But the 64 bit floating format only allows 52 bits of the fraction. When it gets truncated you get the result you see. About 15 decimal digits is the limit of accuracy for floats.
What you need is some rounding and formatting designed for the purpose.
Smile? C standard library? Excel? Numbers? But I think not in standard AppleScript.
--
--> From the U S of A, the only socialist country that refuses to admit it. <--
_______________________________________________
Do not post admin requests to the list. They will be ignored.
AppleScript-Users mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
Archives: http://lists.apple.com/archives/applescript-users
This email sent to email@hidden