Assessing Binary Measurement Systems: A Cost-Effective Alternative to Complete Verification

Abstract

Suppose we plan to assess a binary measurement system when the misclassification probabilities vary from part to part. We consider the estimation of the average error probabilities of such a system when a gold standard (error-free) system is available to verify the status of any part. We examine plans where we first measure a sample of n parts r times each with the binary measurement system. Then we study the impact on the precision and robustness of the estimates if we use the gold-standard system to verify the true status of none, some, or all of the sampled parts. We show that a partial verification plan has comparable performance to full verification in terms of the precision and robustness of the estimates while requiring as few as 10% of parts to be verified. When the gold-standard system is expensive or time consuming, eliminating the need to verify all parts dramatically reduces the cost of the assessment study.