Coding and general discussion relating to the compiler
Moderators: David Barker, Jerry Messina
-
be80be
- Registered User
- Posts: 90
- Joined: Mon Feb 23, 2009 2:15 am
- Location: tn
Post
by be80be » Tue Dec 25, 2012 8:10 pm
If i do math like this
Code: Select all
dim X as float
X = 403
X = ((1023/X)-1) // it rounds this to a 1
X = (10000/X) // this ends up a 10000 not the 6501.950 that it should be.
What am I missing here
-
Jerry Messina
- Swordfish Developer
- Posts: 1473
- Joined: Fri Jan 30, 2009 6:27 pm
- Location: US
Post
by Jerry Messina » Tue Dec 25, 2012 9:23 pm
It works fine here. Using MPLAB sim, this is what I see...
Code: Select all
dim X as float
X = 403
X = ((1023/X)-1) // x = 1.53846145
X = (10000/X) // x = 6500.00049
MPLAB displays a few more digits than a 32-bit float can actually represent, so only about the first 6 or 7 are valid.
How are you looking at X to determine what the value is?
-
be80be
- Registered User
- Posts: 90
- Joined: Mon Feb 23, 2009 2:15 am
- Location: tn
Post
by be80be » Tue Dec 25, 2012 9:26 pm
I'm output it to a LCD with DecToStr (x) and it rounds it to 1
I need to use Dec32ToStrFormat how does that work?
-
Jerry Messina
- Swordfish Developer
- Posts: 1473
- Joined: Fri Jan 30, 2009 6:27 pm
- Location: US
Post
by Jerry Messina » Tue Dec 25, 2012 9:41 pm
Burt -
I just noticed your post on DDIY
Code: Select all
X = (1023/403)-1 //I should get 1.538 in X
x = (10000/X) // should be 6501.950
What happens here is that even though you declare X as a float, the expression
is evaluated using integer math, and THEN the result is converted to a float when it's assigned to X. That's why you get X=1
You'd get something very different if you let the compiler know you're working with floating-point when it evaluates the stuff on the right-hand side of the "="
Generates a very different result
To format a floating-point string, you can use FloatToStr() in convert.bas which lets you tell it how many digits you want (defaults to 3)
-
be80be
- Registered User
- Posts: 90
- Joined: Mon Feb 23, 2009 2:15 am
- Location: tn
Post
by be80be » Tue Dec 25, 2012 9:58 pm
Jerry Thanks this fixed it I'm now reading the Resistance and outputting to a LCD
Code: Select all
While true
ADC_in = ADC.Read(0)
X = (1023.0/ADC_in)-1 //I should get 1.538 in X
I = X
Y = (10000.0/I) // should be 6501.950
LCD.WriteAt (1,1, FloatToStr(Y))
DelayMS(100)
Wend
I really don't need the right side of the decimal only the whole numbers
how would I get that without rounding