Re: 1.0 divided by 10 = 0.10000000000000001?
Re: 1.0 divided by 10 = 0.10000000000000001?
- Subject: Re: 1.0 divided by 10 = 0.10000000000000001?
- From: Ryan Britton <email@hidden>
- Date: Mon, 27 Mar 2006 14:06:51 -0800
The problem is the imprecision of floating point types on computer
systems. With both double and float, there are only so many digits
guaranteed to be accurate. The rest is garbage that may or may not
be correct resulting from how the number is stored in bits.
You can read more about it here:
http://en.wikipedia.org/wiki/Floating_point
On Mar 24, 2006, at 11:03 PM, Axman wrote:
(I'm really sorry, i just sent this, but forgot to change the subject.
i had to make it more meaningful. again, sorry. Al.)
Hi all, i'm a 'noob' so if what i have to say is stupid i'm really
sorry. i've only just started programming in cocoa in the last few
days, but i am almost finished a little calculator (yay :D ) but i'm
having an annoying problem when i'm trying to use decimal numbers,
i'm
using a variable that is a double (i want my little calc to be pretty
accurate, well very) and after the "." button is pressed i have it so
that this variable (which started out as 1.0) is divided by 10 and
then the number that's pressed is added to the full number that's
being worked on times (*) this variable: here's a little look:
("//debug!" is where i'm trying to find out what value
AXdecimal is)
double AXdecimal = 1.0;
bool isDecimal = 0; (that is changed when the "." is pressed, so
isDecimal is true)
...(buttons get pressed here, and then get calculated by the next
bit)
...}else if(isDecimal == 1){ //if the AXdecimal button had been
pressed
if (isNumberOne == 1){
if (numberOne == 0){
AXdecimal = AXdecimal / 10;
numberOne = button * AXdecimal;
dec = AXdecimal; //debug!
x = numberOne;
}else if(numberOne != 0) {
AXdecimal = AXdecimal / 10;
dec = AXdecimal; //debug!
numberOne = numberOne + (button *
AXdecimal);
x = numberOne;
}
}else{
if (numberTwo == 0){
AXdecimal = AXdecimal / 10;
dec = AXdecimal; //debug!
numberTwo = button * AXdecimal;
x = numberTwo;
}else if(numberTwo != 0) {
AXdecimal = AXdecimal / 10;
dec = AXdecimal; //debug!
numberTwo = numberTwo + (button *
AXdecimal);
x = numberTwo;
}
}
....
from what i can figure out, i'm not doing anything wrong, but when i
debug it, AXdecimal is "0.10000000000000001", whis is not 1
divided by
10 is it? the result is that i end up with
123.5599975585938 when it should be 123.56. any ideas? Thanks in
advance, and if you can help, could you mail me directly at
email@hidden as well as cocoa-dev@... (because i get the daily
round up thing, so i might get it today, and also other people can
see
what the answer is :) )
P.S: Cocoa and Xcode are so amazingly simple to use, thank you
soooooo
much Apple!!!
--
Al.
☠☢✯☢☠
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Cocoa-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden
Attachment:
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Cocoa-dev mailing list (email@hidden)
Help/Unsubscribe/Update your Subscription:
This email sent to email@hidden