Re: how to format decimal precision of a real?
Re: how to format decimal precision of a real?
- Subject: Re: how to format decimal precision of a real?
- From: John W Baxter <email@hidden>
- Date: Sun, 3 Mar 2002 09:23:06 -0800
At 23:46 -0800 3/2/2002, Bill Hoffman wrote:
>
> The simplest answer is: 324.234234234 * 100 div 1 / 100
>
>
This works fine for my needs, just what I was looking for. Thanks to all
>
who replied.
It works for Bill's needs...but it still produces an approximation to
324.23 (which AppleScript then "displays properly" (or "lies about"
depending on one's viewpoint).
It's easy to produce a different approximation to 324.23:
(324.234234234 * 100 div 1 / 100) is equal to (324 + 0.2 + 0.03)
--> false
which will look the same when displayed. 324.23 can't be represented
exactly by the kind of floating point numbers which are being used.
Also, the suggested calculation isn't rounding, it is truncating. Changing
the problem slightly:
324.235234234 * 100 div 1 / 100
--> 324.23 (still). One might want 324.24 here:
(324.235234234 * 100 + 0.5) div 1 / 100 -- parentheses necessary
--> 324.24
[This sort of calculation will round incorrectly down for
(324.235 * 100 + .05) div 1 / 100
--> 324.23
which is wrong both "as taught to some of us in school" and according to
IEEE (and the way I was taught in school in the dark ages).
To deal with this, Apple provides in (recent?) Standard Additions:
round: Round number to integer (defined in: StandardAdditions.osax)
round [real] -- the number to round
[rounding up/down/toward zero/to nearest/as taught in
school] -- the rounding direction; if omitted, rounds to nearest. "to
nearest" rounds .5 cases to the nearest even integer in order to decrease
cumulative errors. To always round .5 away from zero, use "as taught in
school."
Result: integer -- the rounded value
So: try
(round (324.234234234 * 100)) / 100
instead (if on a recent enough Standard Additions version). Slower than
the computed form, but less often the producer of unexpected results.
--John
--
John Baxter, Port Ludlow, WA USA
The floating point is the natural enemy of the equals.
The bowling ball is the natural enemy of the egg.
_______________________________________________
applescript-users mailing list | email@hidden
Help/Unsubscribe/Archives:
http://www.lists.apple.com/mailman/listinfo/applescript-users
Do not post admin requests to the list. They will be ignored.