# Significant Digits in Calculations and in Data Results

You are here:
• Significant Digits in Calculations and in Data Results

## Preface

Force measurement devices (load cells, strain gauges, and scales) list various performance metrics on their data sheets including repeatability, hysteresis error, full scale output, input resistance, zero balance. Through the testing process, these metrics identify the device’s acceptable specification tolerance ranges. It would be impossible, and not terribly meaningful, to include every digit provided by test equipment displays. Instead, these specification values are rounded with a method that considers the level of required accuracy of the device to some number of significant digits. This is one of the various significant digit considerations in weighing applications discussed in this document.

Load cell specifications and their tolerances define the device’s ability to produce acceptable application data results. Understanding the number of significant digits and rounding used to express them avoids inconsistencies in measurement results and data gathering. Therefore, Management Quality System documentation should incorporate a formal process for recording data results using the same significant digits and rounding methodology as the Original Equipment Manufacturer (OEM). The process should include a comparison of these specifications and the customer’s measuring intent to ensure that the produced data result meets the customer expectations.

## Definitions

1. Significant Digit – Any of the figures 0 through 9, excepting leading zeros and some trailing zeros, which is used with its place to denote a numerical quantity to some desired approximation.
2. Absolute Method – A method of applying data, where all digits in the observed or calculated data are to be considered significant to determine conformance with specifications.
3. Rounding Method – A prescribed procedure applied to observed data or a calculated data result where numbers are adjusted and truncated to the nearest unit in the designated placeholder.

## Significant Digits

In today’s electronic world, spreadsheets, calculators, digital readouts can generate many numbers associated with a measurement. However, these numbers’ precision may be much different than the actual precision of the associated specification and tolerance criteria. The intent of using significant digits is to ensure that data results do not contain more digits than the least precise datum used in the calculation.

Some basic rules for significant digits are:

1. Non-zero digits are considered significant.
2. The most significant digit in a data result is the left-most non-zero digit. For example, in the result: 865,241.5307; the 8 is the most significant digit.
3. When using a decimal point in a data result, the least significant digit of the data result is the right-most digit (i.e., it can be a zero or non-zero). For example, in the result: 865,241.5307; the 7 is the least significant digit.
4. If there is no decimal point in the data result, then the right-most non-zero digit is the least significant digit. For example, in the result: 865,240; the 4 is the least significant digit.
5. The number of significant digits is the quantity of digits between and including the most and least significant digit. For example, in the result: 865,241.5307; there are 10 significant digits.

Some additional examples are in the table below, “Significant Digits,” with reference to the above rules.

Determining Significant Digits

Applying this knowledge to data sheets for load cells, strain gauges, and scales, let us assume the following specifications are given: Non-linearity <±0.030%; Repeatability <±0.017%; Input Resistance 800Ω±20, Full Scale Output 1.3 ±0.1%. According to the above rules, the Repeatability and Input Resistance have three significant digits, and Non-linearity and Full-Scale Output have two significant digits.

## Data Handling Methods

Tacuna Systems wants our customers to be aware of the important issues that may affect data results from our products, in this case, significant digits. To give an example, let’s assume a measurement system operator records data from a 6-place electronic digital readout on a load cell. The specification for the metric is 15 tons, with a tolerance range of 0.25 tons. The employee records the tonnage as follows:

Example Data Collection Table

In this example, the customer may have problems with their quality management system’s registration auditor. The third recording can be argued to be out-of-tolerance (i.e., maximum tolerance is 15.25 tons and the recorded weight is 15.2571 tons).

This example illustrates why employing standard methods for data use, taking into account significant digits of the data, is extremely important. These methods clarify the intended meaning of the specification and tolerance when compared to the observed data or calculated data results to determine conformance with specifications. Also, by using a standardized methodology, a consumer can make a valid product purchase decision by comparing like specifications across different manufacturers’ products.

The two methods for applying data results based on significant digits are the Absolute Method and the Rounding Method. Both are explained below. The one chosen by a manufacturer is typically arbitrary and may be determined by industry convention. Therefore corporate documentation should specify the method.

### Absolute Method

This is one of two methods for determining the how many digits should be included in a data result. It considers all digits in an observed or calculated data result significant to determine conformance to the specification within its tolerance. Where applicable, the absolute method compares an observed or a calculated data result, unrounded, to the specification criteria. Conformance or non-conformance is based on this comparison. It is a good practice to clearly document the use of the absolute method in any test data report or collection process documentation.

### Rounding Method

The rounding method is the second way to express output data, given the number of significant digits. In this case, a limited number of digits in the observed or calculated data result are significant for determining conformance to a specification within its tolerance. The observed or calculated data are either “rounded up” or “rounded down” per the rules below, to the nearest unit in the designated place of digits in the specification. For example, if the specification is “100 tons”, the placeholder is the hundreds place; if “10 tons”, the placeholder is the tens unit place. Likewise if “2 tons”, the placeholder is the unit; if “0.2 tons”, the tenths unit; if “0.25 tons”, the hundredth unit, and so forth. Conformance or non-conformance to the specification is based on the rounded data result. Once again, use of the rounding method should be clearly documented.

#### Rounding Rules

The basic rules for rounding a data result are as follows:

1. If the next digit beyond the placeholder digit is less than 5, then just truncate the data point. For example, the specification 17 1.0 tons has two significant digits, and the placeholder digit is the units place. If the data result is 17.4 tons, then the data result is rounded to 17 tons.
2. If the next digit beyond the placeholder digit is greater than 5, then increase the last significant digit by 1, and then truncate the remaining digits. For example, if the specification 17 1.0 tons, a data result of 17.6 tons is rounded to 18 tons.
3. If the next digit beyond the place holder digit is a 5:
• (a) If the last placeholder digit is an even number, then just truncate the data point. For example, the specification 1,602 1.0 tons has four significant digits. Given a data result of 1,602.5 tons, the data result is rounded to 1,602 tons.
• (b) If the last placeholder digit is an odd number, then increase the last significant digit by 1 and truncate the remaining digits. For example, given the same specification 1,602 1.0 tons, a data result of 1,601.5 tons is rounded to 1,602 tons.

#### Rounding Examples

Some additional examples of rounding per the above rules are in the table below, assuming 3 significant digits in the specification. Red digits in the table indicate the decision digits needed for rounding.

Rounding When There Are 3 Significant Digits in the Comparative Specification

Note that if the specification is a range (i.e., 6 – 12 Lbs.), then a data result of 7, 8.5, 9 or 11 is an acceptable data result. If the data result is 12.01 and you record 12.01, then the door is open for interpretation by others.

### Retaining Significant Digits in Calculations and Data Results

Any approach to retaining significant digits involves some loss of information. Therefore, the planned and potential uses for measured data should influence the decision of the level of rounding. Rounding data results avoids a misleading impression of precision while preventing loss of information due to coarse resolution.

Some guidelines to preserve data results are as follows:

1. Record all digits displayed in a measurement digital readout without initial regard to significant digits. If not using digital readouts, then record all digits known with certainty plus one estimated digit. (For example if a needle indicator on an analog scale display reads approximately a third of the way between two graduations, estimate a last digit that would reflect this.)
2. Avoid rounding a calculation from a data result. Carry out the calculations with all data digits and round only the final result.
3. When adding or subtracting data, the result should contain no significant digits beyond the place of the last significant digit of any datum. This means if one figure has two significant digits right of a decimal and the other has three, the result should have only two significant digits right of the decimal.
4. When multiplying or dividing data, the result should contain no more significant digits than the factor, dividend or divisor with the smaller number of significant digits.
5. When using logarithms and exponents, the digits of ln(x) or log10(x) are significant through the nth place after the decimal when x has n significant digits. The number of significant digits of ex or 10x is equal to the place of the last significant digit in x after the decimal.
6. A number representing the exact count or a mathematical constant is treated as having an infinite number of significant digits.
7. When a specification tolerance is rounded, the precision of only the final calculation used to derive the specification must conform to that specification’s significant digits.
8. In most cases, a rounded standard deviation is rounded to two significant digits.
9. In most cases, a rounded average is rounded to the last place of significant digits determined by the needs of the application.

## Conclusion

The values of the data derived from any precise measurement device are only as good as the adherence of the user to standard data handling practices. Without the consistency of applying significant digits and rounding methods, data results may indicate out-of-tolerance conditions or excessive digits being recorded. Practicing acceptable data handling methods achieves consistency in measurement results. This enables valid decision-making with the measured data.

## References

1. ASTM Designation E29-93a, Standard Practice for Using Significant Digits in Test Data to Determine Conformance with Specifications, 1999.
2. Food and Drug Administration, Office of Regulatory Affairs, ORA Laboratory Manual Volume III Section 4, Basic Statistics and Data Presentation, Rev. 2, 08/13/2019.