Re: 1.0 divided by 10 = 0.10000000000000001?
Re: 1.0 divided by 10 = 0.10000000000000001?
- Subject: Re: 1.0 divided by 10 = 0.10000000000000001?
- From: Boyd Collier <email@hidden>
- Date: Mon, 27 Mar 2006 14:31:38 -0800
Although I didn't study the details of your code, I'm reasonably sure
that the "problem" you ran into is the inability of computers to
represent
every decimal number exactly. It's a common issue in numerical
calculations.
Take a look at http://en.wikipedia.org/wiki/Floating_point as a
reasonable brief introduction.
Boyd
On Mar 24, 2006, at 11:03 PM, Axman wrote:
(I'm really sorry, i just sent this, but forgot to change the subject.
i had to make it more meaningful. again, sorry. Al.)
Hi all, i'm a 'noob' so if what i have to say is stupid i'm really
sorry. i've only just started programming in cocoa in the last few
days, but i am almost finished a little calculator (yay :D ) but i'm
having an annoying problem when i'm trying to use decimal numbers,
i'm
using a variable that is a double (i want my little calc to be pretty
accurate, well very) and after the "." button is pressed i have it so
that this variable (which started out as 1.0) is divided by 10 and
then the number that's pressed is added to the full number that's
being worked on times (*) this variable: here's a little look:
("//debug!" is where i'm trying to find out what value
AXdecimal is)
double AXdecimal = 1.0;
bool isDecimal = 0; (that is changed when the "." is pressed, so
isDecimal is true)
...(buttons get pressed here, and then get calculated by the next
bit)
...}else if(isDecimal == 1){ //if the AXdecimal button had been
pressed
if (isNumberOne == 1){
if (numberOne == 0){
AXdecimal = AXdecimal / 10;
numberOne = button * AXdecimal;
dec = AXdecimal; //debug!
x = numberOne;
}else if(numberOne != 0) {
AXdecimal = AXdecimal / 10;
dec = AXdecimal; //debug!
numberOne = numberOne + (button *
AXdecimal);
x = numberOne;
}
}else{
if (numberTwo == 0){
AXdecimal = AXdecimal / 10;
dec = AXdecimal; //debug!
numberTwo = button * AXdecimal;
x = numberTwo;
}else if(numberTwo != 0) {
AXdecimal = AXdecimal / 10;
dec = AXdecimal; //debug!
numberTwo = numberTwo + (button *
AXdecimal);
x = numberTwo;
}
}
....
from what i can figure out, i'm not doing anything wrong, but when i
debug it, AXdecimal is "0.10000000000000001", whis is not 1
divided by
10 is it? the result is that i end up with
123.5599975585938 when it should be 123.56. any ideas? Thanks in
advance, and if you can help, could you mail me directly at
email@hidden as well as cocoa-dev@... (because i get the daily
round up thing, so i might get it today, and also other people can
see
what the answer is :) )
P.S: Cocoa and Xcode are so amazingly simple to use, thank you
soooooo
much Apple!!!
--
Al.
☠☢✯☢☠
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Cocoa-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
40sunstroke.sdsu.edu
This email sent to email@hidden
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Cocoa-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden