Measurement error models and variance estimation in the presence of rounding error effects

Abstract

An approach to estimating measurement error variances for any instrument having round-off effects that might also have instrument bias is presented. Recently finite instrument resolution effects on error variances have been studied, but negligible instrument bias was assumed and the contexts were different than considered here. Our intent is to use repeated measurements on several standards to estimate the instrument’s random and systematic error variances. Recognizing that rounding impacts item bias and variance in a manner that depends on the true value, an approach is presented to estimate random error variance and instrument systematic error variance. The key finding is that item-specific bias can interfere with the estimation of overall instrument bias unless appropriate error modeling and associated inference steps are taken.