Potential floating-point problems sought
Potential floating-point problems sought
- Subject: Potential floating-point problems sought
- From: Nigel Garvey <email@hidden>
- Date: Sun, 17 Mar 2002 20:58:15 +0000
At the beginning of the month, Emmanuel pointed out a floating-point
calculation error that occurs when the number 324.21 is multiplied by 100
and the result is divved by 1. The final result is 32420. As far as I've
tested, a similar problem occurs with any positive or negative number
that is a multiple of 324.21 and a positive power of 2. Multiplying it by
100 and divving the result by 1 produces a result that's 1 closer to zero
than it should be.
This has implications when truncating numbers to a certain number of
decimal places. One fix is to coerce the result of the multiplication to
string, as 'has' has done in the truncation/rounding handlers he
announced the other day. This irons out the tiny internal inaccuracy
before the 'div' operation takes place.
I've been looking for a fix to suit the hyper-fast handlers I posted to
ScriptBuilders last year and have come up with a modification to the
arithmetic which seems to do the trick: simply multiply by 1000 and div
by 10 instead. The truncation formula then becomes:
n * (10 ^ (d + 1)) div 10 / (10 ^ d)
... rather than:
n * (10 ^ d) div 1 / (10 ^ d)
... where n is the original number and d the required precision.
This works with everything I've tried so far, including the
324.21-related numbers. But presumably there must be other potentially
problematic numbers out there. With some, one or both of the fixes may
not work. Others might conceivably become problematic simply *because* of
the fixes.
Is there any way of anticipating what numbers might cause problems, so
that they can be tested against the fixes?
NG
_______________________________________________
applescript-users mailing list | email@hidden
Help/Unsubscribe/Archives:
http://www.lists.apple.com/mailman/listinfo/applescript-users
Do not post admin requests to the list. They will be ignored.