The resolution uncertainty associated with digital indications revisited: the inclusion of the quantization effect and the impact of noise presence in the estimation process

Abstract

The indicated value on the displaying device of a digital measuring instrument is usually considered as the resultant of rounding the measurand value to the nearest indication. This hypothesis is conventional and forms the basis of estimating the resolution uncertainty associated with digital indications. However, the essential quantization process needed in converting the measurand analog signal into a digital form makes the precise estimation of the resolution uncertainty shift from the one based on the aforementioned hypothesis. In this study, the effect of the quantization process on the resolution uncertainty was analyzed, a proposed algorithm for the estimation process was presented, and finally the impact of noise presence was considered. Based on the current study, the measurement result usually has a shift from the indicated value. In the presence of noise, the resolution uncertainty of stable indications can be reduced if the peak-to-peak noise could be quantified with adequate accuracy.Also, the current study recommends the revision of the broad applicability of the conventional estimation method of the resolution uncertainty and encourages standardization bodies to specify the main requirements in digital measuring instruments that guarantee their indicating behavior.