Calibration Values for Resistor Cards

Pickering non-precision resistor cards provide a nonvolatile memory store for each resistor. However, this memory store is limited to 16 bits and cannot hold a calibration value of adequate precision using normal data types.

This article describes a method of storing calibration values using a half-precision floating point data type that uses just 16 bits.

IEEE 754 describes a data type called half-precision, or sometimes binary16, which is contained in 16 bits:

  • 1 bit: sign
  • 5 bits: exponent
  • 10 bits: mantissa or significand

This is capable of storing a value between 6.10x10^-5 to 6.55x10^4. That is, it is limited to a maximum value of about 65500. So, ff resistor calibration values are stored in kOhm values, then calibration values up to 65MOhm are supported. So, 12345 Ohms is stored as 12.345, and 10 Ohms is stored as 0.01.

This data type's limitation is its limited precision. The numerical values are only accurate to about 0.05%. A modification of the data type, removing the sign bit and adding a further mantissa bit, improves the resolution to about 0.025%, as shown in the diagram below.

Accuracy of the half-precision and modified half-precision data types

Graph showing accuracy of the half-precision and modified half-precision types

Functions to convert from single and double precison values to and from half precision values are provided in a library r-card-lib

In use, the functions:

HALF floatToHalf(float)
float halfToFloat(HALF)

Convert a standard floating point value to/from a HALF (16-bit) value

The modified half-precision data type is also included, these are:

HALF floatToHalfx(float)
float halfxToFloat(HALF)

Convert a standard floating point value to/from a modified HALF (16-bit) value giving improved resolution values

How did we do?
0 out of 0 people found this helpful.
Beta

How can I help?

Hi , I'm TESS - your friendly AI agent

Press Enter to submit

TESS is a moderated, AI agent. TESS can make mistakes, please verify important information independently.


Chat history

Log in or create an account to save & continue your chats with TESS.

Log in

or create an account