Research methods in performance validity testing studies: Criterion grouping approach impacts study outcomes

Abstract

Objective: Performance validity test (PVT) research studies commonly utilize a known-groups design, but the criterion grouping approaches within the design vary greatly from one study to another. At the present time, it is unclear as to what degree different criterion grouping approaches might impact PVT classification accuracy statistics. Method: To analyze this, the authors used three different criterion grouping approaches to examine how classification accuracy statistics of a PVT (Word Choice Test; WCT) would differ. The three criterion grouping approaches included: (1) failure of 2+ PVTs versus failure of 0 PVTs, (2) failure of 2+ PVTs versus failure of 0–1 PVT, and (3) failure of a stand-alone PVT versus passing of a stand-alone PVT (Test of Memory Malingering). Results: When setting specificity at ≥.90, WCT cutoff scores ranged from 41 to 44 and associated sensitivity values ranged from .64 to .88, depending on the criterion grouping approach that was utilized. Conclusions: When using a stand-alone PVT to define criterion group status, classification accuracy rates of the WCT were higher than expected, likely due to strong correlations between the reference PVT and the WCT. This held true even when considering evidence that this grouping approach results in higher rates of criterion group misclassification. Conversely, when using criterion grouping approaches that utilized failure of 2+ PVTs, accuracy rates were more consistent with expectations. These findings demonstrate that criterion grouping approaches can impact PVT classification accuracy rates and resultant cutoff scores. Strengths, weaknesses, and practical implications of each of the criterion grouping approaches are discussed.