No announcement yet.

Double Precision please

  • Filter
  • Time
  • Show
Clear All
new posts

  • Double Precision please

    32-bit integers would be quite useful.
    I am building a device to display data from a sensor and sound an alarm if it exceeds a threshold. 3 decimal digits of precision are all that is required for display, but there are some calculations required that cause loss of precision when done with 16-bit integers.
    I have had to jump through hoops do the math in way so as to try and preserve the 3 significant digits, but I have to restrict the range of the variables so as to avoid overflow, which prevents using the full range of the sensor.
    A LONG INTEGER type would solve all of this.
    And, yes, I have tried using Anna's half-float package:
    123 / 10 = 12.28
    Uh.... no, thanks....