What wrong with the way swordfish does math

be80be · Post by **be80be** » Tue Dec 25, 2012 8:10 pm

If i do math like this

dim X as float
 X = 403
 X = ((1023/X)-1) // it rounds this to a 1
 X = (10000/X) // this ends up a 10000 not the 6501.950 that it should be.

What am I missing here

Jerry Messina · Post by **Jerry Messina** » Tue Dec 25, 2012 9:23 pm

It works fine here. Using MPLAB sim, this is what I see...

Code: Select all

dim X as float
 X = 403
 X = ((1023/X)-1)  // x = 1.53846145
 X = (10000/X)      // x = 6500.00049

MPLAB displays a few more digits than a 32-bit float can actually represent, so only about the first 6 or 7 are valid.

How are you looking at X to determine what the value is?

be80be · Post by **be80be** » Tue Dec 25, 2012 9:26 pm

I'm output it to a LCD with DecToStr (x) and it rounds it to 1

I need to use Dec32ToStrFormat how does that work?

Jerry Messina · Post by **Jerry Messina** » Tue Dec 25, 2012 9:41 pm

Burt -

I just noticed your post on DDIY

Code: Select all

X = (1023/403)-1 //I should get 1.538 in X
x = (10000/X) // should be 6501.950

What happens here is that even though you declare X as a float, the expression

Code: Select all

(1023/403)-1

is evaluated using integer math, and THEN the result is converted to a float when it's assigned to X. That's why you get X=1

You'd get something very different if you let the compiler know you're working with floating-point when it evaluates the stuff on the right-hand side of the "="

Code: Select all

x = (1023.0/403)-1

Generates a very different result

To format a floating-point string, you can use FloatToStr() in convert.bas which lets you tell it how many digits you want (defaults to 3)

be80be · Post by **be80be** » Tue Dec 25, 2012 9:58 pm

Jerry Thanks this fixed it I'm now reading the Resistance and outputting to a LCD

Code: Select all

While true
  ADC_in = ADC.Read(0)
  X = (1023.0/ADC_in)-1 //I should get 1.538 in X
  I = X
  Y = (10000.0/I) // should be 6501.950

  LCD.WriteAt (1,1, FloatToStr(Y))
  DelayMS(100)
 Wend

I really don't need the right side of the decimal only the whole numbers
how would I get that without rounding