Re: number formatting
Re: number formatting
- Subject: Re: number formatting
- From: Graeme Hiebert <email@hidden>
- Date: Sun, 3 Mar 2002 20:43:36 -0800
On Sunday, March 3, 2002, at 12:48 AM, Michael Gersten wrote:
I must disagree with this.
.34999999 is a truncated, not rounded, value. It is the truncation of
.349999999 (notice one more digit on this line).
If you ROUND instead of truncating, then you get .35
In general, if you are using n bits to represent a float, you'll get
about x digits of base 10 percision. Let the default behavior be to
round to x-1 digits for display.
That should solve 99.9% (:-) of the problem. If you need something
other than rounded as close as the computer can deal with, then you
probably need a custom formatter anyways.
If you look at the binary representation, you'll see that for 0.35 it is:
hexadecimal: 3fd6 6666 6666 6666
binary: 0011 1111 1101 0110 0110 0110 0110 0110 0110 0110 0110
0110 0110 0110 0110 0110
The first twelve bits give us the sign and exponent (power of 2), and
the remaining 52 bits represent the mantissa, with an implied "1"
preceding it all.
The arbitrary precision calculator "bc" tells me that the last 52 bits
represent 1.3999999999999999111821580299874767661094665527343750. The
first twelve bits represent an exponent of -2, so the value of the
number is 1.3999999999999999111821580299874767661094665527343750 * 2^-2,
or 0.3499999999999999777955395074968691915273666381835937 .
True, it looks like this number should perhaps have been rounded to 0.35
(unless it's showing enough precision that those sevens are also being
displayed), but in general comparing the results of what NSLog gives you
to what gets put in the NSTextField is not valid. Consider John
McGruer's example just last night. In his case, the number "8.54" is
being stored as 8.5399999999999991473487170878797769546508789062500000.
If you round this to 15 decimal places, you will in fact get
8.539999999999999, and not 8.54. If you round to between 2 and 14
decimal places, you will get 8.54.
So, the problem is that NSTextField is showing about fifteen digits of
precision. I would think that it would be somewhat arbitrary as to
where one would want to round, especially when it is so easy for
developers to attach a numeric formatter to the field in the first place.
-g
_______________________________________________
cocoa-dev mailing list | email@hidden
Help/Unsubscribe/Archives:
http://www.lists.apple.com/mailman/listinfo/cocoa-dev
Do not post admin requests to the list. They will be ignored.