Re: precision problem or my error?
Re: precision problem or my error?
- Subject: Re: precision problem or my error?
- From: Steve Bird <email@hidden>
- Date: Fri, 7 Apr 2006 17:31:15 -0400
On Apr 7, 2006, at 3:54 PM, Ivan Kourtev wrote:
Hi,
I have a problem that is driving me crazy and I cannot identify the
cause of it. I have a very simple function that computes 100
equally spaced portions of an int number (essentially the values of
1%, 2%, 3%, ..., 100% of a number). The numbers aren't very big,
about 20,000,000 to 50,000,000 tops, yet I seem to be running out
of precision as if int's were 2-bytes long (and not 4). This is
happening on both a DC G5 Powermac and a G4 PB, with or without 64-
bit arithmetic checked.
a[i] = ( j * maxVal ) / parts;
--- I don't know about 64-bit arithmetic, and whether you're really
using it, but this is a problem in the 32 bit world.
When "maxVal" = 43287250, and "j" = 50, then take a look at what
you're doing.
FIRST, you multiply 43287250 * 50. That equals 2164362500, In hex,
that's 0x81018D04 - a NEGATIVE NUMBER (since the sign bit is set).
Only THEN do you divide by 100: -2130604796 / 100 = -21306047.
The way you're doing it is most precise, assuming that you have the
numeric headroom to do it. You don't.
You have choices to make:
1... You can change the order of operations: Use a[i] = j *
(maxVal / parts); If your "maxVal" is small, then you will lose
precision.
2... You can use 64-bit arithmetic. It's plain that you're not
really using it.
3... You can use floating-point arithmetic. Calculate the fraction
you want in floating point, THEN multiply by your maxVal. Convert
back to an int if you need to.
----------------------------------------------------------------
Steve Bird
Culverson Software - Elegant software that is a pleasure to use.
www.Culverson.com (toll free) 1-877-676-8175
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Cocoa-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden