Select Page

Significant Digits in Calculations and in Data Results

You are here:
< All Topics
Table of Contents

Preface

Force measurement devices, such as load cells, strain gauges, and scales, include various performance metrics on their data sheets such as repeatability, hysteresis error, full scale output, input resistance, zero balance, etc. expressed with precise significant digits and using a rounding method. These product metrics, through the testing process, identify acceptable specification tolerance ranges for the measurement device.

These specifications, along with the associated tolerance range, define the capability of the measurement device to produce acceptable application data results. Therefore, the end user must understand the number of significant digits and rounding methods used to express these quantities to avoid inconsistencies in measurement equipment output data. To achieve this goal, Management Quality System documentation should incorporate a formal process for recording data results using the same significant digits and rounding methodology as the Original Equipment Manufacturer (OEM). The process should include a comparison of these specifications and the customer’s measuring intent to ensure that the produced data result meets the customer expectations.

Definitions

  1. Significant Digit – Any of the figures 0 through 9, excepting leading zeros and some trailing zeros, which is used with its place to denote a numerical quantity to some desired approximation.
  2. Absolute Method – A method of applying data, where all digits in the observed or calculated data are to be considered significant to determine conformance with specifications.
  3. Rounding Method – A prescribed procedure applied to observed data or a calculated data result where numbers are adjusted and truncated to the nearest unit in the designated placeholder.

Data Handling Methods

The use of standard methods is important to handle data results given the number of digits in that data result that are considered significant. These methods clarify the intended meaning of the specification and tolerance with which the observed data or calculated data results are compared to determine conformance with specifications. Also, by using a standardized methodology, a consumer can make a valid product purchase decision by comparing like specifications across different manufacturers’ products.

The two methods for applying data results based on significant digits are the Absolute Method and the Rounding Method. Both are explained below. The one chosen by a manufacturer is typically arbitrary and may be industry oriented. Therefore corporate documentation should specify the method.

Tacuna Systems wants our customers to be aware of the important issues that may affect data results from our products, in this case, significant digits. For example, a customer’s employee records data from a 6-place electronic digital readout on a load cell. The specification for the metric is 15 tons with a tolerance range of \pm 0.25 tons. The employee records the tonnage as follows:

Example Data Collection Table

Recording NumberWeight (t)
115.0396
214.9775
315.2571
(…and so forth)

In this example, the customer may have problems with their quality management system’s registration auditor. The third recording can be argued to be out-of-tolerance (i.e., maximum tolerance is 15.25 tons and the recorded weight is 15.2571 tons). To resolve this problem, the system designer should establish guidelines for defining the requirements for significant digits and for rounding data results.

Significant Digits

In today’s electronic world, spreadsheets, calculators, digital readouts can generate many numbers associated with a measurement. However, the precision of these numbers may be much different than the actual precision for the associated specification and tolerance criteria. A significant digit is used to express the precision or uncertainty associated with numerical data results by approximation. The intent of using significant digits is to ensure that data results do not contain more significant digits than the least precise datum used in the calculation.

Some basic rules for significant digits are:

  1. Non-zero digits are considered significant.
  2. The most significant digit in a data result is the left-most non-zero digit. For example, in the result: 865,241.5307; the 8 is the most significant digit.
  3. When using a decimal point in a data result, the least significant digit of the data result is the right-most digit (i.e., it can be a zero or non-zero). For example, in the result: 865,241.5307; the 7 is the least significant digit.
  4. If there is no decimal point in the data result, then the right-most non-zero digit is the least significant digit. For example, in the result: 865,240; the 4 is the least significant digit.
  5. The number of significant digits is the quantity of digits between and including the most and least significant digit. For example, in the result: 865,241.5307; there are 10 significant digits.

Some additional examples are in the table below, “Significant Digits,” with reference to the above rules.

Determining Significant Digits

Data ResultSignificant Digits
0.00242
0.002403
3.141596
8702
20001
2000.05
40074

Applying this knowledge to data sheets for load cells, strain gauges, and scales, let us assume the following specifications are given: Non-linearity <±0.030%; Repeatability <±0.017%; Input Resistance 800Ω±20, Full Scale Output 1.3 ±0.1%. According to the above rules, the Repeatability and Input Resistance have three significant digits, and Non-linearity and Full-Scale Output have two significant digits.

Absolute Method

This is one of two methods for determining the how many digits should be included in a data result. It considers all digits in an observed or calculated data result significant for the purpose of determining conformance with the specification and the respective tolerance. Where applicable, the absolute method compares an observed or a calculated data result, unrounded, to the specification criteria. Conformance or non-conformance is based on this comparison. It is a good practice to clearly document the use of the absolute method in any applicable test data reports or collection processes.

Rounding Method

The rounding method is the second way to express output data, given the number of significant digits. In this case, a limited number of digits in the observed or calculated data result are significant for the purpose of determining conformance or non-conformance with the specification and the respective tolerance. The observed or calculated data are either “rounded up” or “rounded down” per the rounding rules below, to the nearest unit in the designated place of digits in the specification. For example, if the specification is “100 tons”, the placeholder is the hundreds place; in “10 tons”, the placeholder is the tens unit place; likewise in “2 tons”, the placeholder is the unit; “0.2 tons”, the tenths unit; “0.25 tons”, the hundredth unit, and so forth. Conformance or non-conformance with the specification is based on the rounded data result. Once again, use of the rounding method should be clearly documented.

Rounding Rules

The basic rules for rounding a data result are as follows:

  1. If the next digit beyond the placeholder digit is less than 5, then just truncate the data point. For example, the specification 17 \pm1.0 tons has two significant digits, and the placeholder digit is the units place. If the data result is 17.4 tons, then the data result is rounded to 17 tons.
  2. If the next digit beyond the placeholder digit is greater than 5, then increase the last significant digit by 1, and then truncate the remaining digits. For example, if the specification 17 \pm1.0 tons, a data result of 17.6 tons is rounded to 18 tons.
  3. If the next digit beyond the place holder digit is a 5:
    • (a) If the last placeholder digit is an even number, then just truncate the data point. For example, the specification 1,602 \pm1.0 tons has four significant digits. Given a data result of 1,602.5 tons, the data result is rounded to 1,602 tons.
    • (b) If the last placeholder digit is an odd number, then increase the last significant digit by 1 and truncate the remaining digits. For example, given the same specification 1,602 \pm1.0 tons, a data result of 1,601.5 tons is rounded to 1,602 tons.

Rounding Examples

Some additional examples of rounding per the above rules are in the table below, given that there are 3 significant digits in the specification in question. Red digits in the table indicate the decision digits needed for rounding.

Rounding When There Are 3 Significant Digits in the Comparative Specification

Initial Data ResultSignificant Digits +1 DigitRounded Data ResultReason
15.053615.0515.03(a)
15.007815.0015.01
14.968714.9615.02
14.942114.9414.91
14.955514.9515.03(b)

Note that if the specification is a range (i.e., 6 – 12 Lbs.), then a data result of 7, 8.5, 9 or 11 is an acceptable data result. If the data result is 12.01 and you record 12.01, then the door is open for interpretation by others.

Retaining Significant Digits in Calculations and Data Results

Any approach to retaining significant digits involves some loss of information. Therefore, the process of selecting the level of rounding should carefully consider both planned and potential uses for the data. Most importantly, the number of significant digits must be adequate for comparison against the specification and its respective tolerances. Rounding data results avoids a misleading impression of precision while preventing loss of information due to coarse resolution. Some guidelines to preserve data results are as follows:

  1. Record all digits of a measurement reading displayed in a digital readout without initial regard to significant digits. If not using digital readouts, then record all digits known with certainty plus one estimated digit. (For example if a needle indicator on an analog scale display reads approximately a third of the way between two graduations, estimate a last digit that would reflect this.)
  2. Avoid rounding a calculation from a data result. Carry out the calculations with all data digits and round only the final result.
  3. When adding or subtracting data, the result should contain no significant digits beyond the place of the last significant digit of any datum. This means if one figure has two significant digits right of a decimal and the other has three, the result should have only two significant digits right of the decimal.
  4. When multiplying or dividing data, the result shall contain no more significant digits than the factor, dividend or divisor with the smaller number of significant digits.
  5. When using logarithms and exponents, the digits of ln(x) or log10(x) are significant through the nth place after the decimal when x has n significant digits. The number of significant digits of ex or 10x is equal to the place of the last significant digit in x after the decimal.
  6. A number representing the exact count or a mathematical constant is treated as having an infinite number of significant digits.
  7. When a specification tolerance is rounded, the precision of only the end calculation used to derive the specification must conform to that specification’s significant digits.
  8. In most cases, a rounded standard deviation is rounded to two significant digits, and a rounded average is rounded to the last place of significant digits determined by the needs of the application.

Conclusion

The values of the data derived from any precise measurement device are only as good as the adherence of the user to standard data handling practices. Without the consistency of applying significant digits and rounding methods, data results may indicate out-of-tolerance conditions or excessive digits being recorded. Practicing acceptable data handling methods achieves consistency in measurement results. This enables valid decision-making with the measured data.

References

  1. ASTM Designation E29-93a, Standard Practice for Using Significant Digits in Test Data to Determine Conformance with Specifications, 1999.
  2. Food and Drug Administration, Office of Regulatory Affairs, ORA Laboratory Manual Volume III Section 4, Basic Statistics and Data Presentation, Rev. 2, 08/13/2019.
Was this article helpful?
How can we improve this article?