Re: number formatting
Re: number formatting
- Subject: Re: number formatting
- From: Michael Gersten <email@hidden>
- Date: Sun, 03 Mar 2002 00:48:05 -0800
>
> Do I design the default
>
> behavior for the computer (i.e.: 0.4 shows up as .39999 or whatever
>
> truncation effect it is) or do I design it for a human (who would most
>
> likely want 0.4 to show up as 0.4)?
>
>
Lance, have you read what people have been telling you? Floating point
>
values don't have exact representations inside the machine.
>
>
Run this code and watch it in the debugger.
>
>
float value = 0.35;
>
>
printf("floating point value: %f\n", value);
>
>
The value is represented in the machine as 0.34999...
>
>
To force it to show up as 0.35 in a text field the text field would have
>
to force the precision to 2 places. Then you'd be still complaining about
>
the cases where you wanted more or less precision.
I must disagree with this.
.34999999 is a truncated, not rounded, value. It is the truncation of
.349999999 (notice one more digit on this line).
If you ROUND instead of truncating, then you get .35
In general, if you are using n bits to represent a float, you'll get
about x digits of base 10 percision. Let the default behavior be to
round to x-1 digits for display.
That should solve 99.9% (:-) of the problem. If you need something
other than rounded as close as the computer can deal with, then you
probably need a custom formatter anyways.
Michael
_______________________________________________
cocoa-dev mailing list | email@hidden
Help/Unsubscribe/Archives:
http://www.lists.apple.com/mailman/listinfo/cocoa-dev
Do not post admin requests to the list. They will be ignored.