Re: Accurate decimal numbers?
Re: Accurate decimal numbers?
- Subject: Re: Accurate decimal numbers?
- From: Ricky Sharp <email@hidden>
- Date: Fri, 31 Dec 2004 20:35:21 -0600
On Dec 31, 2004, at 7:32 PM, mark wrote:
In my case, I need to convert the value to a base (long integer) unit
(i.e.,
1.5m has to be converted to 1500 mm, if "mm" is the base unit). The
issue
isn't displaying, but converting to the base unit. The user would
notice if
they type in 1.5m and I convert it to 1499mm as that wasn't what they
typed
in.
Maybe I just should go back to scanDouble and add in an estimation bit
at
the limit of my accuracy, to force the rounding that I want (just
using the
rint() that you had mentioned in the 2nd part)
Don't know if this will suit your needs, but...
First ask yourself what the highest resolution unit will be (mm? cm?
etc.). This would be your normalized unit. You can then pick an
integer container to store such normalized values. You'd never have to
deal with rounding errors or attempting int to float comparisons.
For display-purposes only, you can convert to floats and do rounding to
display a certain number of decimal digits.
Finally, I highly recommend either a course or book on "Numerical
Analysis". I had such a course back during my CS degree and it was
extremely helpful when dealing with floating point values.
___________________________________________________________
Ricky A. Sharp mailto:email@hidden
Instant Interactive(tm) http://www.instantinteractive.com
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Cocoa-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden