Precision
Precision
- Subject: Precision
- From: Chad Armstrong <email@hidden>
- Date: Sat, 23 Mar 2002 23:57:19 -0700
On Saturday, March 23, 2002, at 03:53 , Erik M. Buck wrote:
Second, precision is a Computer Science 200 level issue.
Doesn't anyone
take a numerical methods course in school anymore ?
Some of us didn't do CS at college - Apple is "the computer for
the rest of us" - remember !-)
I looked in a C book, and there are two functions which will round a
number up or down to the closest integer, but that doesn't help much
when it comes to dealing with floating-point numbers.
I might either just have to live with the small problem, or figure out a
way to return just a float instead of a double. As mentioned before,
the float doesn't seem to have this odd precision problem, but when the
double (twice the precision of a standard float) is introduced, then
there is the problem. The thing which makes this error really odd is
the unusual case it presents. If all numbers with a .1 or .6 were
affected, then perhaps the problem would be a little more general. But
here, the problem is only for numbers from 64 to 99. 64 is pretty
representative of a binary number system (2^6), but the problem ends at
99 (which is 110011 in binary, so numbers 100000 to 110011 in binary).
Chad Armstrong
email@hidden
_______________________________________________
cocoa-dev mailing list | email@hidden
Help/Unsubscribe/Archives:
http://www.lists.apple.com/mailman/listinfo/cocoa-dev
Do not post admin requests to the list. They will be ignored.