Tolerance Intervals: Alternatives to Credibility Intervals in Validity Generalization Research

Abstract

In validity generalization research, the estimated mean and variance of the true validity distribution are often used to construct a credibility interval, an inter val containing a specified proportion of the true valid ity distribution. The statistical interpretation of this in terval in the literature has varied between Bayesian and classical (frequentist) viewpoints. Credibility in tervals are here discussed from the frequentist perspec tive. These are known as "tolerance intervals" in the statistical literature. Two new methods for construct ing a credibility interval are presented. Unlike the cur rent method of constructing the credibility interval, tolerance intervals have known performance character istics across repeated applications, justifying confi dence statements. The new methods may be useful in validity generalization research involving a small or moderate number of validation studies. Index terms: Bayesian statistics, Credibility intervals, Meta- analysis, Tolerance intervals, True validity distribu tion, Validity generalization.