Re: 1.0 divided by 10 = 0.10000000000000001?
Re: 1.0 divided by 10 = 0.10000000000000001?
- Subject: Re: 1.0 divided by 10 = 0.10000000000000001?
- From: Derrick Bass <email@hidden>
- Date: Mon, 27 Mar 2006 16:24:05 -0600
On Mar 25, 2006, at 1:03 AM, Axman wrote:
from what i can figure out, i'm not doing anything wrong, but when i
debug it, AXdecimal is "0.10000000000000001", whis is not 1
divided by
10 is it? the result is that i end up with
123.5599975585938 when it should be 123.56. any ideas? Thanks in
Others have mentioned that this is due to the finite precision of
floating point types. Also important is that floats and doubles are
internally represented in binary rather than decimal and 0.1 cannot
be exactly represented as a finite binary digit string. Just like 2/3
cannot be represented as a finite length decimal sequence.
For more than you ever wanted to know, see the excellent article in
the link that Nick Zitmann provided. Then investigate NSDecimalNumber
if exact base-10 arithmetic is important in your application.
Derrick
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Cocoa-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden