Re: NSDecimalNumber and division precision
Re: NSDecimalNumber and division precision
- Subject: Re: NSDecimalNumber and division precision
- From: "Marcus S. Zarra" <email@hidden>
- Date: Fri, 20 Jul 2007 15:36:19 -0600
On Jul 20, 2007, at 2:50 PM, Eric MORAND wrote:
Once the calculations are done, derivedUnitPriceHT is
83.612040133779264214046822742474916. But if I do the same
calculation using MacOS X Calculator, it returns the exact value
83.61204013378 !
What am I doing wrong ? I thought that using NSDecimalNumber was
the solution to obtain perfect precision but it looks like I am wrong.
When I ran 100/1.196 through PCalc with 14 decimal places I got
83.6120401337793 which is a rounding of what you are getting from
NSDecimalNumber. So it looks like NSDecimalNumber is giving you the
right number but OS X Calculator is helpfully rounding it for you.
If you want the result rounded to a certain decimal place then use -
decimalNumberByDividingBy:withHandler: instead. With a handler you
can control the exact precision of the resulting NSDecimalNumber.
Marcus S. Zarra
Zarra Studios LLC
Simply Elegant Software for OS X
www.zarrastudios.com
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________
Cocoa-dev mailing list (email@hidden)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden