Criterion Measures Are Not Always Continuous: Consequential Validity Ratio for Logistic Regression

Abstract

Neither the theoretical conceptualization of validity, validation methodologies, nor the relative importance of different aspects of validity in measure development, is static over time. With the evolution of social values, validity evidence based on testing consequences has received greater attention in measurement research and practice. However, the methodological basis for generating this type of validity evidence is relatively underdeveloped compared to other psychometric properties (e.g., dimensionality, content validity, and reliability). The Consequential Validity Ratio (CVR) is a recently developed method for quantifying and representing how well test scores avoid the improper influence of participant demographics in the prediction of a validation criterion. The original CVR can only be applied when the criterion measures are continuous; however, binary criteria are also common in scale validation practice. This study theoretically proposes six candidate formulas for the CVR that can be used with binary criteria in a logistic regression framework (i.e., CVRB). A simulation analysis with a two-factor design was conducted to help determine the final formulation of the CVRB. The final formulation is conceptually aligned with the original CVR, straightforward to calculate, intuitive to interpret, and has adequate efficacy in generating evidence for the consequential basis of validity.