Calibration Values for Resistor Cards

    Pickering non-precision resistor cards provide a non-volatile memory store for each resistor, however this memory store is of limited size, only 16 bits, and cannot hold a calibration value of adequate precision using normal data types.

    This article describes a method of storing calibration values using a half-precision floating point data type that uses just 16-bits.

    IEEE 754 describes a data type called half-precision, or sometimes binary16, which is contained in 16 bits:

    1 bit: sign
    5 bits: exponent
    10 bits: mantissa or significand

    This is capable of storing a value between 6.10x10^-5 to 6.55x10^4. That is, it is limited to a maximum value of about 65500. So, ff resistor calibration values are stored in kOhm values, then calibration values up to 65MOhm are supported. So, 12345 Ohms is stored as 12.345, and 10 Ohms is stored as 0.01.

    The limitation of this data type is with its limited precision, the numerical values are only accurate to about 0.05%. A modification of the data type, removal of the sign bit and adding a further mantissa bit, gives an improvement of the resolution to give an accuracy of about 0.025% as shown in the diagram below.

    Accuracy of the half-precision and modified half-precision data types

    Graph showing accuracy of the half-precision and modified half-precision types

    Functions to convert from single and double precison values to and from half precision values are provided in a library r-card-lib


    In use, the functions:

    HALF floatToHalf(float)
    float halfToFloat(HALF)

    Convert a standard floating point value to/from a HALF (16-bit) value

    The modified half-precision data type is also included, these are:

    HALF floatToHalfx(float)
    float halfxToFloat(HALF)

    Convert a standard floating point value to/from a modified HALF (16-bit) value giving improved resolution values

    Need help?



    Support knowledgebase

    Have questions or need some help? While we enjoy talking to you, you can find technical information for our entire product range here.

    Technical know-how 


    Long-term support

    Because we manufacture our products in-house, we are able to provide you with guaranteed long-term support and low obsolescence.

    Our support


    Resource center

    From success stories and product literature to newsletters, find the information you need about our switching and simulation solutions.

    Get the resources you need


    Worldwide presence

    We supply and support our products across the globe, with seven offices in the Americas, Europe and Asia as well as additional representation. 

    Find a local office or agent near you