RE: Precision
RE: Precision
- Subject: RE: Precision
- From: "Jonathan E. Jackel" <email@hidden>
- Date: Mon, 25 Mar 2002 16:49:03 -0500
Have you tried using %g in the format, as in:
[NSString stringWithFormat:@"%g", myFloat]
The %g eliminates trailing zeroes from a float. Works with a double, too.
Jonathan Jackel
>
-----Original Message-----
>
From: email@hidden
>
[mailto:email@hidden]On Behalf Of Chad Armstrong
>
Sent: Sunday, March 24, 2002 1:57 AM
>
To: email@hidden
>
Subject: Precision
>
>
>
> On Saturday, March 23, 2002, at 03:53 , Erik M. Buck wrote:
>
>
>
>> Second, precision is a Computer Science 200 level issue.
>
>> Doesn't anyone
>
>> take a numerical methods course in school anymore ?
>
>
>
> Some of us didn't do CS at college - Apple is "the computer for
>
> the rest of us" - remember !-)
>
>
>
I looked in a C book, and there are two functions which will round a
>
number up or down to the closest integer, but that doesn't help much
>
when it comes to dealing with floating-point numbers.
>
>
I might either just have to live with the small problem, or figure out a
>
way to return just a float instead of a double. As mentioned before,
>
the float doesn't seem to have this odd precision problem, but when the
>
double (twice the precision of a standard float) is introduced, then
>
there is the problem. The thing which makes this error really odd is
>
the unusual case it presents. If all numbers with a .1 or .6 were
>
affected, then perhaps the problem would be a little more general. But
>
here, the problem is only for numbers from 64 to 99. 64 is pretty
>
representative of a binary number system (2^6), but the problem ends at
>
99 (which is 110011 in binary, so numbers 100000 to 110011 in binary).
>
>
Chad Armstrong
>
email@hidden
>
_______________________________________________
>
cocoa-dev mailing list | email@hidden
>
Help/Unsubscribe/Archives:
http://www.lists.apple.com/mailman/listinfo/cocoa-dev
Do not post admin requests to the list. They will be ignored.
_______________________________________________
cocoa-dev mailing list | email@hidden
Help/Unsubscribe/Archives:
http://www.lists.apple.com/mailman/listinfo/cocoa-dev
Do not post admin requests to the list. They will be ignored.
| References: | |
| | >Precision (From: Chad Armstrong <email@hidden>) |