Re: number formatting
Re: number formatting
- Subject: Re: number formatting
- From: Michael Gersten <email@hidden>
- Date: Mon, 11 Mar 2002 23:00:55 -0800
If the 32 or 64 bits of data storing the number represent
.3499999999999999, then rounding off the one last digit gives .35.
That last nine rounds the previous one from 9 to 10; the carry then
goes up to .35.
That's not rounding to 2 digits, that's rounding to 14 digits on a
number that has 15 digits.
Now, someone else mentioned that if you go to 52 digits, you get more
-- after all those 9's, you get some 7's, and then others.
That's not relevent.
>
If you look at the binary representation, you'll see that for 0.35 it is:
>
>
hexadecimal: 3fd6 6666 6666 6666
>
binary: 0011 1111 1101 0110 0110 0110 0110 0110 0110 0110 0110
>
0110 0110 0110 0110 0110
All those 0110 sets are repeating; that's a good indicator that the
translation into another number system will also repeat.
As you extract additional decimal digits, going from .349 to .3499,
you use up more of those binary digits. Around 14 or 15 of these,
depending on the number, you'll use up all of those digits, and all
that will be left is the remainder.
When you say that 3fd6 6666 6666 6666 has a value of
3499999999999999777955395074968691915273666381835937, you are saying
that
3fd6 6666 6666 6666 0000 0000 0000 0000 0000 0000 ...
has that value.
Once you've gone through all the digits available -- roughly 6 or 7
for single, 14 or 15 for double -- then going any farther just
generates significant roundoff errors, like pentium math bugs. Stop at
that point, round off the last digit that was generated, and you've
got the final answer.
Put it another way: which is the better value for 2/3:
.666666666666666666660000000000000000000000
.666666666666666666670000000000000000000000
Those that say "truncate" will go for the first.
Those that say "round at the last digit that has source data" (me)
will go for the second.
Those that say "bc gives me a very different result when I ask for
huge amounts of meaningless data" will give yet another result. (Ok,
what does BC say when you ask it to translate the binary value of
2/3?)
Again: the current default is to truncate when running out of digits;
it makes more sense to round that last digit when you run out instead
of truncating.
Jim Correia wrote:
>
>
On Sunday, March 3, 2002, at 03:48 AM, Michael Gersten wrote:
>
>
>>> Do I design the default
>
>>> behavior for the computer (i.e.: 0.4 shows up as .39999 or whatever
>
>>> truncation effect it is) or do I design it for a human (who would most
>
>>> likely want 0.4 to show up as 0.4)?
>
>>
>
>> Lance, have you read what people have been telling you? Floating point
>
>> values don't have exact representations inside the machine.
>
>>
>
>> Run this code and watch it in the debugger.
>
>>
>
>> float value = 0.35;
>
>>
>
>> printf("floating point value: %f\n", value);
>
>>
>
>> The value is represented in the machine as 0.34999...
>
>>
>
>> To force it to show up as 0.35 in a text field the text field would have
>
>> to force the precision to 2 places. Then you'd be still complaining about
>
>> the cases where you wanted more or less precision.
>
>
>
> I must disagree with this.
>
>
>
> .34999999 is a truncated, not rounded, value. It is the truncation of
>
> .349999999 (notice one more digit on this line).
>
>
>
> If you ROUND instead of truncating, then you get .35
>
>
You miss the point. If the framework ROUNDS to 2 places, it will be
>
appropriate for you, but not for someone else. There is no way to look at
>
the number and empirically determine the number of significant digits the
>
programmer/user wants displayed. ROUNDing to any number of places doesn't
>
solve the problem, it just pushes the problem around a bit, doesn't solve
>
it. To solve it, use a formatter.
_______________________________________________
cocoa-dev mailing list | email@hidden
Help/Unsubscribe/Archives:
http://www.lists.apple.com/mailman/listinfo/cocoa-dev
Do not post admin requests to the list. They will be ignored.