Every force measurement device—such as load cells, strain gauges, and scales—comes with a data sheet listing key performance metrics. These include repeatability, hysteresis error, full-scale output, input resistance, and zero balance. During testing and calibration, these metrics benchmark the device’s acceptable tolerances. However, test equipment measuring the device’s performance can display many digits beyond what is practical (or even meaningful) against these tolerances. Instead, technicians will round the test data to the level of the device’s required accuracy using significant digit principles. This document explores how significant digits impact weighing device testing/calibration and why consistency in data handling is crucial.
Why Significant Digits Matter
Why are significant digits important? There are a few reasons. First, digital measurement tools generate an output with multiple decimal places, but that measurement’s precision does not always match the precision of the specification you’re comparing it to. Rounding measurement results to significant digits ensures you’re comparing apples to apples. That is, by rounding, the measurement result more accurately matches the specification’s precision.
Secondly, having well-documented procedures for rounding data sets and determining how many digits are significant is critical to ensuring consistency in measurement results across devices. Not only should all data recording methods be consistent within an organization, but they should also follow the same significant digit rules used by the Original Equipment Manufacturer (OEM) in the published specifications. This practice increases performance assessment reliability so its formal documentation in a company’s quality management system is critical.
Key Definitions
We’ll start with a few definitions of terms used in the rest of this article.
- Significant Digit – Any figure (0-9) whose place denotes a quantity to a desired precision, except for all leading zeros and some trailing zeros.
- Absolute Method – A method where all observed or calculated digits are considered significant when determining a data set’s conformance to specifications.
- Rounding Method – A procedure where observed or calculated digits are adjusted and truncated to the nearest value having a specific number of digits.
Basic Rules for Significant Digits
How do we determine which digits are significant? Below are the accepted practices.
- All non-zero digits are significant.
- The most significant digit in a data result is the left-most non-zero digit. For example, in the result, 865,241.5307, the 8 is the most significant digit.
- The least significant digit of a data result varies:
- If the measurement result contains a decimal point, the least significant digit is the right-most one. It can be a zero or non-zero. For example, in 865,241.5307 the 7 is the least significant digit.
- If there is no decimal point in the data result, then the right-most non-zero digit is the least significant. For example, in 865,240, the 4 is the least significant digit.
- The total number of significant digits is the quantity of digits between and including the most and least significant digit. For example, in the result: 865,241.5307; there are 10 significant digits.
Applying Significant Digits in Load Cell Specifications
Now consider the following load cell specifications:
- Non-linearity: <±0.030%
- Repeatability: <±0.017%
- Input Resistance: 800Ω±20
- Full-Scale Output: 1.3 ±0.1%
Applying the significant digit rules from above:
- Non-linearity and Full-Scale Output have two significant digits.
- Repeatability and Input Resistance have three significant digits.
Data Handling Methods
As mentioned above, consistent application of significant digits is crucial for quality control. Let’s examine two common methods for handling measurement data.
Absolute Method
This method considers all observed or calculated digits significant when determining conformance to a specification. The recorded value is compared directly to the specification without rounding. This approach is useful in highly precise applications but should be clearly documented.
Rounding Method
The rounding method is the second way to express output data. In this case, the numeric data are either “rounded up” or “rounded down” to match the specification sheet’s number of significant digits. A device’s conformance or non-conformance to the specification is based on rounded test data. Once again, there should be clear documentation of the rounding method’s use and the assumed number of significant digits.
Rounding Rules
The basic rules for rounding a data result, as specified in NIST Handbook 44, Appendix A are as follows:
- If the next digit to the right of the least significant digit is less than 5, truncate the value. For example, the specification 17 ±1.0 tons has two significant digits and the least significant digit is in the units place. If tests give a measurement of 17.4 tons, then the data result is rounded to 17 tons.
- If the digit to the right of the least significant digit is greater than 5, increase the last significant digit by 1, and then truncate the remaining digits. For example, if the specification 17 ±1.0 tons, a data result of 17.6 tons is rounded to 18 tons.
- If the next digit beyond the least significant digit is a 5:
- (a) If the last significant digit is an even number, truncate the data. For example, the specification 1,602 ±1.0 tons has four significant digits. Given a data result of 1,602.5 tons, the data result is rounded to 1,602 tons.
- (b) If the last significant digit is an odd number, increase it by 1 and truncate the remaining digits. For example, given the same specification 1,602 ±1.0 tons, a data result of 1,601.5 tons is rounded to 1,602 tons.
- (a) If the last significant digit is an even number, truncate the data. For example, the specification 1,602 ±1.0 tons has four significant digits. Given a data result of 1,602.5 tons, the data result is rounded to 1,602 tons.
Rounding Examples
Some additional examples of rounding per the above rules are in the table below, assuming 3 significant digits in the specification. Red digits in the table indicate the decision digits needed for rounding.
Rounding When There Are 3 Significant Digits in the Comparative Specification
| Test Data Result | Significant Digits +1 Digit | Rounded Data Result | Reason |
|---|---|---|---|
| 15.0536 | 15.05 | 15.0 | 3(a) |
| 15.0078 | 15.00 | 15.0 | 1 |
| 14.9687 | 14.96 | 15.0 | 2 |
| 14.9421 | 14.94 | 14.9 | 1 |
| 14.9555 | 14.95 | 15.0 | 3(b) |
Note that if the specification is a range (i.e., 6 – 12 Lbs.), then a data result of 7, 8.5, 9 or 11 is an acceptable data result. If the data result is 12.01 and you record 12.01, then the door is open for interpretation by others.
Example: Recording Weigh System Test Data and Determining Compliance
Assume a weighing system operator records data displayed on a 6-place electronic digital load cell readout. The operator takes repeated measurements with a calibration weight of 15 tons, with a tolerance range of ±0.25 tons. He records the following:
Example Data Collection Table
| Reading Number | Weight (t) |
|---|---|
| 1 | 15.0396 |
| 2 | 14.9775 |
| 3 | 15.2541 |
| (…and so forth) | … |
In this case, the third reading (15.2541 tons) appears to exceed the tolerance limit of 15.25 tons. Without proper rounding, or without well-documented methods, this could create inconsistencies in data interpretation and quality system audits.
This example illustrates the importance of employing standard data handling methods in weighing applications, considering significant digits. These methods clarify the intended tolerances when determining a load cell’s conformance to them using observed or calculated data.
Guidelines for Retaining Significant Digits in Data Results
To avoid misleading precision while preserving accuracy when rounding data sets, consider the following:
- Initially record all measurement digits displayed on a digital readout disregarding significant digits. If using analog readouts, record all digits known with certainty plus one estimated digit. For example, if a needle indicator on an analog scale reads approximately a third of the way between two graduations, estimate a last digit that would reflect this.
- Perform any calculations using all data digits and round only the final result.
- When this result is used to determine conformance to a specification within its tolerances, round that final result to the specification tolerance’s significant digits.
- When adding or subtracting data, round to match the least precise datum in the operation.
- When multiplying or dividing data, the result should contain no more significant digits than the least precise factor, dividend or divisor.
- For logarithms and exponents, the digits of ln(x) or log10(x) are significant through the nth place after the decimal when x has n significant digits. The number ex or 10x rounds to the place of the last significant digit in x after the decimal.
- A mathematical constant, by convention, has an infinite number of significant digits.
- A rounded standard deviation most often rounds to two significant digits.
- A rounded average is most often rounded to the number of significant digits required by the specific weighing application.
Conclusion
Accuracy of recorded data from any precision weighing device depends on best practices including consistent data handling practices. Applying significant digit rules and standardizing rounding methods ensures valid, repeatable results. This is especially true when calibrating a measuring system and determining system compliance to tolerances. This practice helps avoid misinterpretations of data, supports quality control, and improves confidence in measurement-based decisions.
References
- ASTM Designation E29-93a, Standard Practice for Using Significant Digits in Test Data to Determine Conformance with Specifications, 1999.
- Food and Drug Administration, Office of Regulatory Affairs, ORA Laboratory Manual Volume III Section 4, Basic Statistics and Data Presentation, Rev. 2, 08/13/2019.
- National Institute of Standards and Technology (NIST) Handbook 44, Appendix A Fundamental Considerations Associated with the Enforcement of Handbook 44 Codes, Section 10. Rounding Off Numerical Values, 2025.
