re: integers which just won't be round
re: integers which just won't be round
- Subject: re: integers which just won't be round
- From: John W Baxter <email@hidden>
- Date: Tue, 30 Apr 2002 14:32:55 -0700
At 16:28 +0100 4/29/2002, Charles Arthur wrote:
>
On Sun, 28 Apr 2002 11:30:15 +0100, under another topic line, Nigel Garvey
>
<email@hidden> wrote:
>
> set aardvark to (1234.56 + 7248.83)
>
>
>
> realToString(aardvark)
>
> --> "8483.389999999999"
>
>
>
>The best fix for your version at the moment would seem to be:
>
>
>
> on realToString(theReal)
>
> set oldTIDs to applescript's text item delimiters
>
> set applescript's text item delimiters to {""}
>
> set str to numToStr(theReal)
>
> set applescript's text item delimiters to oldTIDs
>
> return str
>
> end realToString
>
>
>
> realToString(aardvark)
>
> --> "8483.39"
>
>
Could someone explain why this happens, and what the easy cure is? I'm
>
doing something at the moment where I create columns in Applescript Studio.
>
I do a sum on some numbers (turn them into integers and then divide by 10
>
to get 1 dec place).
>
>
Run in SE, the numbers come up as expected, with 1 dp.
>
Run in AStudio, I get stuff like 80.09999999999.
>
>
Why? More importantly, how not?
Why does it happen? Because neither 1234.56 nor 7248.83 can be represented
exactly in the form of binary floating point representation which is being
used. In fact, only 4% of the numbers with two decimal digits (and
reasonable magnitude) can be: xxx.00, xxx.25, xxx.50, and xxx.75 The
other 96 are approximated. So you add two approximations together and get
an approximation. Which happens in the example to be 8483.389999999999
when represented as a string.
You need code which rounds off that number (in the course of making it a
string: .39 is one of the fractions which won't represent exactly). I
would think that Studio offers such a routine, but I've put Studio aside
for lack of time.
--John
--
John Baxter email@hidden Port Ludlow, WA, USA
_______________________________________________
applescript-users mailing list | email@hidden
Help/Unsubscribe/Archives:
http://www.lists.apple.com/mailman/listinfo/applescript-users
Do not post admin requests to the list. They will be ignored.