Cross-validation of non-memory-based embedded performance validity tests for detecting invalid performance among patients with and without neurocognitive impairment

Abstract

ABSTRACT Introduction Embedded performance validity tests (PVTs) allow for continuous and economical validity assessment during neuropsychological evaluations; however, similar to their freestanding counterparts, a limitation of well-validated embedded PVTs is that the majority are memory-based. This study cross-validated several previously identified non-memory-based PVTs derived from language, processing speed, and executive functioning tests within a single mixed clinical neuropsychiatric sample with and without cognitive impairment. Method This cross-sectional study included data from 124 clinical patients who underwent outpatient neuropsychological evaluation. Validity groups were determined by four independent criterion PVTs (failing ≤1 or ≥2), resulting in 98 valid (68% cognitively impaired) and 26 invalid performances. In total, 23 previously identified embedded PVTs derived from Verbal Fluency (VF), Trail Making Test (TMT), Stroop (SCWT), and Wisconsin Card Sorting Test (WCST) were examined. Results All VF, SCWT, and TMT PVTs, along with WCST Categories, significantly differed between validity groups (ηp2 =.05-.22) with areas under the curve (AUCs) of.65-.81 and 19-54% sensitivity (≥89% specificity) at optimal cut-scores. When subdivided by impairment status, all PVTs except for WCST Failures to Maintain Set were significant (AUCs =.75–94) with 33-85% sensitivity (≥90% specificity) in the cognitively unimpaired group. Among the cognitively impaired group, most VF, TMT, and SCWT PVTs remained significant, albeit with decreased accuracy (AUCs =.65-.76) and sensitivities (19-54%) at optimal cut-scores, whereas all WCST PVTs were nonsignificant. Across groups, SCWT embedded PVTs evidenced the strongest psychometric properties. Conclusion VF, TMT, and SCWT embedded PVTs generally demonstrated moderate accuracy for identifying invalid neuropsychological performance. However, performance on these non-memory-based PVTs from processing speed and executive functioning tests are not immune to the effects of cognitive impairment, such that alternate cut-scores (with reduced sensitivity if adequate specificity is maintained) are indicated in cases where the clinical history is consistent with cognitive impairment. In contrast, WCST indices generally had poor accuracy.