• Open Menu Close Menu
  • Apple
  • Shopping Bag
  • Apple
  • Mac
  • iPad
  • iPhone
  • Watch
  • TV
  • Music
  • Support
  • Search apple.com
  • Shopping Bag

Lists

Open Menu Close Menu
  • Terms and Conditions
  • Lists hosted on this site
  • Email the Postmaster
  • Tips for posting to public mailing lists
Re: Accurate decimal numbers?
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Accurate decimal numbers?


  • Subject: Re: Accurate decimal numbers?
  • From: Brendan Younger <email@hidden>
  • Date: Fri, 31 Dec 2004 18:42:17 -0600


On Dec 31, 2004, at 6:08 PM, Mark Dawson wrote:

I have a couple of questions concerning how to use NSDecimal (or even if that's the correct way to go):

(1)
I'd like to convert decimal strings to unsigned longs; I'm not sure the best Cocoa way. I know that scanFloat, scanDouble doesn't work, because of conversion errors (i.e., "1.55"-->1.54999). In my case, its very important to convert the string accurately, as I'd be converting something like 1.555m --> 1555mm.

Just to be clear: scanFloat and scanDouble are not the issue, they're doing their job correctly. The fact that you're forced to represent a fraction in binary is the problem. In other words, the value you get from scanFloat is correct in the sense that you cannot tell the difference between 1.55 and 1.549999999..., they are the same number. (Just like 1.0 and 0.99999... are the *same* number.)


Now, in your case, what you really want to do is keep track of the significant digits. I'd suggest something like:

typedef struct {
double value;
uint32_t significantDigits;
} PreciseValue;
(How to actually figure out how many values to the right of the decimal point there are is an exercise for the reader. In other words, I really don't feel like coding it right now; I'm going to go celebrate 2005 instead.)


Then you know exactly how many digits your final answer will have. You can then do something like printf("%.[sigDigitsHere]f", x.value); which will do what you want.

It looks like NSDecimal (scanDecimal) is the way to go, but I'm not sure the best way to use it. I know I can use NSDecimalMultiplyByPowerOf10 to convert the NSDecimal returned from scanDecimal. What's the best way to convert an NSDecimal to an unsigned long? I wasn't sure if I needed to create an NSDecimalNumber, then do a [num double] to do a forced conversion.

(2)
Also, what's the best way to determine whether a number is a whole number? (i.e. the fractional portion is 0-->1.0). I've usually done something along the lines of


if (x != (float)((int)x))

where x is a float or double; is there an easier way to do this using an NSDecimal number?

Avoiding int<->float conversions is always best. I'd suggest if(0.0 == fabs(x - rint(x)), but I don't claim to be the expert on this.


Brendan Younger

_______________________________________________
Do not post admin requests to the list. They will be ignored.
Cocoa-dev mailing list      (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden


  • Follow-Ups:
    • Re: Accurate decimal numbers?
      • From: mark <email@hidden>
    • Re: Accurate decimal numbers?
      • From: John Stiles <email@hidden>
References: 
 >Accurate decimal numbers? (From: Mark Dawson <email@hidden>)

  • Prev by Date: makeFirstResponder and Enter
  • Next by Date: Re: How to add a URL-tile to the Dock from an Installer?
  • Previous by thread: Accurate decimal numbers?
  • Next by thread: Re: Accurate decimal numbers?
  • Index(es):
    • Date
    • Thread