I'm wondering if someone can shed some light on the floating point functions...

I'm using flt_MUL() and it yields a slightly different number than when I use a calculator.

In this example it's doing 261 x 6.8615 and getting 1790.851440, rather than the correct answer of 1790.8515.

This yields:

I haven't yet determined if this is a significant problem for me, but was curious as to why the answer differs...

Thanks,

-Brian

I'm using flt_MUL() and it yields a slightly different number than when I use a calculator.

In this example it's doing 261 x 6.8615 and getting 1790.851440, rather than the correct answer of 1790.8515.

Code:

#constant RATIO_TARGET := "6.8615" var float1[2], float2[2], float3[2]; var valWtBrei := 261; flt_UITOF(float1, valWtBrei); flt_VAL(float2, RATIO_TARGET); flt_MUL(float3, float1, float2); gfx_MoveTo(10,600); flt_PRINT(float1, "%.6f"); putstr(" x "); flt_PRINT(float2, "%.6f"); putstr(" = "); flt_PRINT(float3, "%.6f");

Code:

261.000000 x 6.861500 = 1790.851440

I haven't yet determined if this is a significant problem for me, but was curious as to why the answer differs...

Thanks,

-Brian

## Comment