Evaluating risk in an abnormal world: how arbitrary probability distributions affect false accept and reject evaluation

Abstract

Risk mitigation strategies commonly use the Test Uncertainty Ratio (TUR) and End of Period Reliability (EOPR) to ensure a measurement is adequate for making acceptance decisions. In some cases, TUR and EOPR are used to determine an appropriate guardband factor, which is then used to reduce the risk of an incorrect decision. Unfortunately, the common guidance of maintaining a TUR of at least 4:1 was developed to simplify the underlying calculus in an era predating modern computing. As such, using a TUR to determine the adequacy of a measurement and set guardband limits assumes that the probability distributions describing the product and the measurement uncertainty are unbiased and normally distributed. The Guide to the Expression of Uncertainty in Measurement (GUM) and its supplements describe several situations where uncertainty in the measurement will not follow a normal distribution. Additionally, it is frequently assumed that the prior distribution of products being measured is normal and unbiased, often with little or no evidence. Sometimes a 95% EOPR is even assigned with no justification. Despite the evidence of non-normal behavior in measurements and products, risk evaluations typically assume normality in both distributions. While evaluating the Probability of False Accept (PFA) and the Probability of False Reject (PFR) is more challenging when the probability distributions are non-normal, the calculus is straightforward using either numerical integration or Monte Carlo techniques. In addition to covering methods for evaluating the actual PFA and PFR without relying on the archaic TUR metric, this work considers several case studies of risk evaluation, including both global and specific risk, when the product or the test measurement uncertainty do not follow normal distributions. Neglecting non-normal behavior can greatly affect PFA and PFR by either over- or underestimating the probabilities depending on the parameters of the distributions. A good prior knowledge of the product being measured is required for a meaningful global risk analysis.