Re: number formatting
Re: number formatting
- Subject: Re: number formatting
- From: Donald Brown <email@hidden>
- Date: Mon, 04 Mar 2002 06:56:43 -0600
The way I've been getting the format I want is
[msEntryStr appendFormat:@"%0.14g", mfEntry];
Donald
on 3/3/02 10:43 PM, Graeme Hiebert at email@hidden wrote:
>
The arbitrary precision calculator "bc" tells me that the last 52 bits
>
represent 1.3999999999999999111821580299874767661094665527343750. The
>
first twelve bits represent an exponent of -2, so the value of the
>
number is 1.3999999999999999111821580299874767661094665527343750 * 2^-2,
>
or 0.3499999999999999777955395074968691915273666381835937 .
>
>
True, it looks like this number should perhaps have been rounded to 0.35
>
(unless it's showing enough precision that those sevens are also being
>
displayed), but in general comparing the results of what NSLog gives you
>
to what gets put in the NSTextField is not valid. Consider John
>
McGruer's example just last night. In his case, the number "8.54" is
>
being stored as 8.5399999999999991473487170878797769546508789062500000.
>
If you round this to 15 decimal places, you will in fact get
>
8.539999999999999, and not 8.54. If you round to between 2 and 14
>
decimal places, you will get 8.54.
--
Donald Brown
email@hidden
http://www.eamontales.com
We have met the enemy and he is us - Pogo
_______________________________________________
cocoa-dev mailing list | email@hidden
Help/Unsubscribe/Archives:
http://www.lists.apple.com/mailman/listinfo/cocoa-dev
Do not post admin requests to the list. They will be ignored.