High-resolution digital indicators and DAQs often display far more decimal places than a load cell’s accuracy tolerance warrants. This is particularly relevant when weighing system data is used for testing, proof of concept, or system calibration. Applying significant digit and rounding principles ensures test data matches published specification tolerances without introducing false precision or audit compliance issues.
Key Takeaways
- Precision Matching: Rounding test readings to match a specification’s significant digits ensures valid, apples-to-apples pass/fail comparisons.
- NIST Compliance: Adhering to NIST Handbook 44 Appendix A rounding standards (including the round-to-even rule) removes operator bias during quality audits.
- Intermediate Data Retention: Retain all raw display digits during mathematical calculations, rounding only the final result to the least precise input’s significant figures.
- Audit Alignment: Formalizing significant digit conventions within your Quality Management System (QMS) guarantees consistent data recording across all test stations.
Why Significant Digits Matter in Weighing Applications
Load cell datasheets specify performance tolerances using key metrics: repeatability, hysteresis, full-scale output, resistance, and zero balance. When benchmarking a sensor during calibration or system testing, raw digital readouts frequently output 6 to 8 digits. Recording unrounded raw values creates two primary issues:
- Mismatched Precision: Comparing a 6-digit test reading to a 2-digit specification tolerance creates artificial non-conformance.
- Audit Inconsistency: Without documented rounding procedures, different technicians will record different values for the same measurement, undermining Quality Management System (QMS) reliability.
Rounding measurement results to significant digits ensures you’re comparing apples to apples. Using the same significant digit rules as the Original Equipment Manufacturer ensures the apple is from the same tree. That is, by rounding with consistent rules, the measurement result more accurately matches the specification’s precision.
Key Definitions
These establish a common understanding for terms that appear in the rest of this article.
- Significant Digit – Any numerical digit (0-9) that denotes a quantity to a desired precision, excluding all leading zeros and some trailing zeros per the rules that follow.
- Absolute Method – A compliance evaluation method where all observed or calculated digits are considered, without rounding, when determining a data set’s conformance to specifications.
- Rounding Method – A procedure where observed or calculated digits are adjusted and truncated to the nearest value matching the number of significant digits of the comparative specification.
Rules for Identifying Significant Digits
Based on mathematical convention, these rules have been adapted and accepted by metrological standards bodies.
- Non-Zero Digits: All non-zero numbers are significant.
- Most Significant Digit (MSD): The leftmost non-zero digit in a number. (In the number 865,241.5307, 8 is the MSD).
- Least Significant Digit (LSD):
- With a decimal point: The rightmost digit (zero or non-zero). In 865,241.5307, 7 is the LSD.
- Without a decimal point: The rightmost non-zero digit. In 865,240, 4 is the LSD.
- Total Count of Signficant Digits: The total number of digits from the MSD through the LSD. The number 865,241.5307 contains 10 significant digits.
Applying Significant Digits in Load Cell Specifications
Consider typical manufacturer datasheet tolerances. The number of significant digits in each spec is as follows:
- Non-Linearity: \(<\pm 0.030\%\) (2 significant digits: 3, 0)
- Repeatability: \(<\pm 0.017\%\) (2 significant digits: 1, 7)
- Input Resistance: \(800\,\Omega \pm 20\) (3 significant digits in 800\(\Omega\))
- Full-Scale Output: \(1.3 \pm 0.1\%\) (2 significant digits in 1.3)
System Conformance to Specifications: Absolute vs. Rounding Method Comparisons
As mentioned above, consistent application of significant digits rules is crucial when comparing measurement data to expected results. Two methods, as defined above, are possible.
- The Absolute Method: When using this method, retain and consider all observed or calculated digits when comparing to an expected result. For example, a reading of 12.0001 lbs. compared to an expected limit of 12 lbs. is not compliant. This method is preferable for highly precise applications such as high precision physics laboratories.
- The Rounding Method: For this method, round measured results (up or down per the rules in the next section) to match the spec sheet’s precision level.
When recording data, clearly document the chosen rounding method and number of significant digits.
NIST Handbook 44 Rounding Rules
The basic rules for rounding a data result, as specified in NIST Handbook 44, Appendix A are as follows:
- If the next digit to the right of the least significant digit < 5 \(\rightarrow\) truncate the remaining digits.
- For example, the specification 17 ±1.0 tons has two significant digits and the least significant digit is in the units place. If tests give a measurement of 17.4 tons, round the data result to 17 tons.
- For example, the specification 17 ±1.0 tons has two significant digits and the least significant digit is in the units place. If tests give a measurement of 17.4 tons, round the data result to 17 tons.
- If the digit to the right of the least significant digit > 5 \(\rightarrow\) increase the LSD by 1 (round up), then truncate the remaining digits.
- For example, given a specification of 17 ±1.0 tons, round a data result of 17.6 tons to 18 tons.
- For example, given a specification of 17 ±1.0 tons, round a data result of 17.6 tons to 18 tons.
- If the next digit beyond the least significant digit = 5 \(\rightarrow\) round-to-even:
- If the last significant digit is an even number \(\rightarrow\) truncate the remaining digits.
For example, the specification 1,602 ±1.0 tons has four significant digits. Given a data result of 1,602.5 tons, round the data result to 1,602 tons. - If the last significant digit is an odd number \(\rightarrow\) increase the LSD by 1 (round up to the next even LSD) and truncate the remaining digits.
For example, given the same specification 1,602 ±1.0 tons, round a data result of 1,601.5 tons to 1,602 tons.
- If the last significant digit is an even number \(\rightarrow\) truncate the remaining digits.
The rationale for rule 3 is the upward statistical bias that comes from the mathematical convention of rounding down when the number after the LSD <5 and rounding up when it is 5 or greater. That is, when the four non-zero digits (1, 2, 3, 4) round down, while five digits (5, 6, 7, 8, 9) round up, this can artificially inflate a sum of many rounded measurements. The standards body approach of “Round-to-Even” creates a scenario where roughly half of the 5s round down and half round up, resulting in a net-zero statistical bias.
Rounding Reference Table
Some additional examples of rounding per the above rules are in the table below, assuming 3 significant digits in the specification. Red digits in the table indicate the decision digits needed for rounding.
Rounding When There Are 3 Significant Digits in the Comparative Specification
| Test Data Result | Significant Digits +1 Digit | Rounded Data Result | NIST HB44 Rule |
|---|---|---|---|
| 15.0536 | 15.05 | 15.0 | 3(a): Digit beyond LSD = 5 and LSD is even \(\rightarrow\) Truncate |
| 15.0078 | 15.00 | 15.0 | 1: Digit after LSD < 5 \(\rightarrow\) Truncate |
| 14.9687 | 14.96 | 15.0 | 2: Digit after LSD > 5 \(\rightarrow\) Round up |
| 14.9421 | 14.94 | 14.9 | 1: Digit after LSD < 5 \(\rightarrow\) Truncate |
| 14.9555 | 14.95 | 15.0 | 3(b): Digit beyond LSD = 5 and LSD is odd \(\rightarrow\) Round up |
Practical Rounding Example: Calibration and Compliance
An operator tests a scale using a 15-ton calibration weight with a tolerance of ±0.25 tons (acceptable range: 14.75 to 15.25 tons). The 6-digit indicator displays:
- Test 1: 15.0396 tons \(\rightarrow\) Rounds to 15.04 tons (PASS)
- Test 2: 14.9775 tons \(\rightarrow\)Rounds to 14.98 tons (PASS)
- Test 3: 15.2541 \(\rightarrow\) Rounds to 15.25 tons (PASS)
Without documented rounding rules, an auditor might reject Test 3 (15.2541) for exceeding 15.2500, despite the digit .0041 falling well beneath the sensor’s measurement uncertainty threshold.
This example illustrates the importance of employing standard data handling methods in weighing applications with respect to significant digits. These methods clarify the intended tolerances when determining a load cell’s conformance to them using observed or calculated data.
Guidelines for Retaining Significant Digits in Mathematical Operations and Data Results
To preserve precision without introducing rounding errors during complex calculations:
- Raw Recording: Record all displayed digits during initial data capture. On analog displays, record all known digits plus one estimated digit. For example, if a needle indicator on an analog scale reads approximately a third of the way between two graduations, estimate a last digit of 3.
- Intermediate Math: Carry all unrounded digits through multi-step equations. Round only the final result:
- Addition / Subtraction: Round the final sum or difference to match the decimal place of the least precise input value.
- Multiplication / Division: Round the final product or quotient to match the total number of significant digits of the least precise factor.
- Logarithms (\(\log_{10} x\) or \(\ln x\)): Keep \(n\) decimal places in the result if \(x\) contains \(n\) significant digits.
- Exponents (\(10^x\) or \(e^x\)): Round the output to match the number of decimal places in \(x\).
- Constants: Mathematical constants (\(\pi\), \(g\)) contain an infinite number of significant digits.
- Statistical Outputs:
- Standard Deviation: By convention, round to 2 significant digits.
- Averages: Round to the significant digits required by the specific weighing application.
Summary and Next Steps
Consistent data handling converts raw load cell signals to an audit-ready data set with common understanding among all users. This standardization of significant digits organization-wide ensures compliance with NIST standards, supports quality control, and ensures confidence in measurement-based decisions.
References
- ASTM Designation E29-93a, Standard Practice for Using Significant Digits in Test Data to Determine Conformance with Specifications, 1999.
- Food and Drug Administration, Office of Regulatory Affairs, ORA Laboratory Manual Volume III Section 4, Basic Statistics and Data Presentation, Rev. 2, 08/13/2019.
- National Institute of Standards and Technology (NIST) Handbook 44, Appendix A Fundamental Considerations Associated with the Enforcement of Handbook 44 Codes, Section 10. Rounding Off Numerical Values, 2025.
