Using a baseline with the probability of agreement to compare distribution characteristics

Abstract

Comparing the center or spread of two samples is a common problem with established solutions in the statistics literature. There exist many methods for comparing two samples, including parametric and non-parametric hypothesis tests, equivalence tests, and the probability of agreement. In this paper, we propose a new probability of agreement solution that combines the merits of the existing methods to allow flexible comparisons across a variety of distributional characteristics, the ability for the user to specify and explore what sizes of difference are practically important, with a straightforward decision rule for easy implementation. We focus on applications where interest lies in comparing a new or “comparison” sample with an existing “reference” sample. The new approach introduces a baseline summary that allows one to contextualize an observed difference between two samples by also considering the anticipated difference between two samples taken from the same distribution, accounting for uncertainty due to sample size and the context-specific equivalence threshold. The method is evaluated with a detailed simulation study and then illustrated with an example comparing two samples of powder, where the goal is to evaluate if multiple aspects of the distribution of the particle sizes for a new sample match those of the established reference sample. A Shiny app implementing the methods is provided to facilitate convenient use for new applications.