Embedded Performance Validity Tests Research Articles

Replicating the classification accuracy of the Verbal Paired Associates and Visual Reproduction recognition trials as embedded performance validity tests.

This study was designed to replicate previous research on the clinical utility of the Verbal Paired Associates (VPA) and Visual Reproduction (VR) subtests of the WMS-IV as embedded performance validity tests (PVTs) and perform a critical item (CR) analysis within the VPA recognition trial. Archival data were collected from a mixed clinical sample of 119 adults (MAge = 42.5, MEducation = 13.9). Classification accuracy was computed against psychometrically defined criterion groups based on the outcome of various free-standing and embedded PVTs. Age-corrected scaled scores ≤ 6 were specific (.89-.98) but had variable sensitivity (.36-.64). A VPA recognition cutoff of ≤ 34 produced a good combination of sensitivity (.46-.56) and specificity (.92-.93), as did a VR recognition cutoff of ≤ 4 (.48-.53 sensitivity at .86-.94 specificity). Critical item analysis expanded the VPA's sensitivity by 3.5%-7.0% and specificity by 5%-8%. Negative learning curves (declining output on subsequent encoding trials) were rare but highly specific (.99-1.00) to noncredible responding. Results largely support previous reports on the clinical utility of the VPA and VR as embedded PVTs. Sample-specific fluctuations in their classification accuracy warrant further research into the generalizability of the findings. Critical item analysis offers a cost-effective method for increasing confidence in the interpretation of the VPA recognition trial as a PVT. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

The impact of race and other demographic factors on the false positive rates of five embedded Performance Validity Tests (PVTs) in a Veteran sample

ABSTRACT Introduction It is common to use normative adjustments based on race to maintain accuracy when interpreting cognitive test results during neuropsychological assessment. However, embedded performance validity tests (PVTs) do not adjust for these racial differences and may result in elevated rates of false positives in African American/Black (AA) samples compared to European American/White (EA) samples. Methods Veterans without Major Neurocognitive Disorder completed an outpatient neuropsychological assessment and were deemed to be performing in a valid manner (e.g., passing both the Test of Memory Malingering Trial 1 (TOMM1) and the Medical Symptom Validity Test (MSVT), (n = 531, EA = 473, AA = 58). Five embedded PVTs were administered to all patients: WAIS-III/IV Processing Speed Index (PSI), Brief Visuospatial Memory Test-Revised: Discrimination Index (BVMT-R), TMT-A (secs), California Verbal Learning Test-II (CVLT-II) Forced Choice, and WAIS-III/IV Digit Span Scaled Score. Individual PVT false positive rates, as well as the rate of failing two or more embedded PVTs, were calculated. Results Failure rates of two embedded PVTs (PSI, TMT-A), and the total number of PVTs failed, were higher in the AA sample. The PSI and TMT-A remained significantly impacted by race after accounting for age, education, sex, and presence of Mild Neurocognitive Disorder. There were PVT failure rates greater than 10% (and considered false positives) in both groups (AA: PSI, TMT-A, and BVMT-R, 12–24%; EA: BVMT-R, 17%). Failing 2 or more PVTs (AA = 9%, EA = 4%) was impacted by education and Mild Neurocognitive Disorder but not by race. Conclusions Individual (timed) PVTs showed higher false positive rates in the AA sample even after accounting for demographic factors and diagnosis of Mild Neurocognitive Disorder. Requiring failure on 2 or more embedded PVTs reduced false positive rates to acceptable levels across both groups (10% or less) and was not significantly influenced by race.

Cross-validation of the Trail Making Test as a non-memory-based embedded performance validity test among veterans with and without cognitive impairment

ABSTRACT Objective This study cross-validated multiple Trail Making Test (TMT) Parts A and B scores as non-memory-based embedded performance validity tests (PVTs) for detecting invalid neuropsychological performance among veterans with and without cognitive impairment. Method Data were collected from a demographically and diagnostically diverse mixed clinical sample of 100 veterans undergoing outpatient neuropsychological evaluation at a Southwestern VA Medical Center. As part of a larger battery of neuropsychological tests, all veterans completed TMT A and B and four independent criterion PVTs, which were used to classify veterans into valid (n = 75) and invalid (n = 25) groups. Among the valid group 47% (n = 35) were cognitively impaired. Results Among the overall sample, all embedded PVTs derived from TMT A and B raw and demographically corrected T-scores significantly differed between validity groups (ηp2 = .21–.31) with significant areas under the curve (AUCs) of .72–.78 and 32–48% sensitivity (≥91% specificity) at optimal cut-scores. When subdivided by cognitive impairment status (i.e., valid-unimpaired vs. invalid; valid-impaired vs. invalid), all TMT scores yielded significant AUCs of .80–.88 and 56%–72% sensitivity (≥90% specificity) at optimal cut-scores. Among veterans with cognitive impairment, neither TMT A or B raw scores were able to significantly differentiate the invalid from the valid-cognitively impaired group; however, demographically corrected T-scores were able to significantly differentiate groups but had poor classification accuracy (AUCs = .66–.68) and reduced sensitivity of 28%–44% (≥91% specificity). Conclusions Embedded PVTs derived from TMT Parts A and B raw and T-scores were able to accurately differentiate valid from invalid neuropsychological performance among veterans without cognitive impairment; however, the demographically corrected T-scores generally were more robust and consistent with prior studies compared to raw scores. By contrast, TMT embedded PVTs had poor accuracy and low sensitivity among veterans with cognitive impairment, suggesting limited utility as PVTs among populations with cognitive dysfunction.

21 A Comparison of the Memory and Non-Memory Based Performance Validity Measures for Detecting Invalid Neuropsychological Test Performance among Individuals with and without Memory Impairment

Objective:Few to no studies have directly compared the relative classification accuracies of the memory-based (Brief Visuospatial Memory Test-Revised Recognition Discrimination [BVMT-R RD] and Rey Auditory Verbal Learning Test Forced Choice [RAVLT FC]) and non-memory based (Reliable Digit Span [RDS] and Stroop Color and Word Test Word Reading trial [SCWT WR]) embedded performance validity tests (PVTs). This study’s main objective was to evaluate their relative classification accuracies head-to-head, as well as examine how their psychometric properties may vary among subgroups with and without genuine memory impairment.Participants and Methods:This cross-sectional study included 293 adult patients who were administered the BVMT-R, WAIS-IV Digit Span, RAVLT and SCWT during outpatient neuropsychological evaluation at a Midwestern academic medical center. The overall sample was 58.0% female, 36.2% non-Hispanic White, 41.3% non-Hispanic Black, 15.7% Hispanic, 4.8% Asian/Pacific Islander, and 2.0% other, with a mean age of 45.7 (SD=15.8) and a mean education of 13.9 years (SD=2.8). Three patients had missing data, resulting in a final sample size of 290. Two hundred thirty-three patients (80%) were classified as having valid neurocognitive performance and 57 (20%) as having invalid neurocognitive performance based on performance across four independent, criterion PVTs (i.e., Test of Malingering Memory Trial 1, Word Choice Test, Dot Counting Test, Medical Symptom Validity Test). Of those with valid neurocognitive performance, 76 (48%) patients were considered as having genuine memory impairment through a memory composite band score (T<37 for (RAVLT Delayed Recall T-score + BVMT-R Delay Recall T-score/2).Results:The average memory composite band score for valid neurocognitive scores was T = 49.63 as compared to T = 27.57 for genuine memory impairment individuals. Receiver operating characteristic [ROC] curve analyses yielded significant areas under the curve (AUCs=.79-.87) for all four validity indices (p’s < .001). When maintaining acceptable specificity (91%-95%), all validity indices demonstrated acceptable yet varied sensitivities (35%-65%). Among the subgroup with genuine memory impairment, ROC curve analyses yielded significantly lower AUCs (.64-.69) for three validity indices (p’s < .001), except RDS (AUC=.644). At acceptable specificity (88%-93%), they yielded significantly lower sensitivities across indices (19%-39%). In the current sample, RAVLT FC and BVMT-R RD had the largest changes in sensitivities, with 19% and 26% sensitivity/90%-92% specificity at optimal cut-scores of <10 and <2, respectively, for individuals with memory impairment, compared to 65% and 61% sensitivity/94% specificity at optimal cut-scores of <13 and <4, respectively, for those without memory impairment.Conclusions:Of the four validity scales, memory-based embedded PVTs yielded higher sensitivities while maintaining acceptable specificity compared to non-memory based embedded PVTs. However, they were also susceptible to the greatest declines in sensitivity among the subgroup with genuine memory impairment. As a result, careful consideration should be given to using memory-based embedded PVTs among individuals with clinically significant memory impairment based on other sources of information (e.g., clinical history, behavioral observation).

87 Examining the use of the Embedded Performance Validity Test in the Brief Visuospatial Memory Test-Revised Among Spanish-Speaking Patients

Objective:The Brief Visuospatial Memory Test-Revised (BVMT-R) Recognition Discrimination (RD) index has emerged as an embedded performance validity test (PVT). However, there do not appear to be any studies that have examined its utility in Spanish-speaking samples. This pilot study examined the classification accuracy of the BVMT-R RD for detecting performance invalidity in a Spanish-speaking forensic sample.Participants and Methods:This cross-sectional study utilized a sample of 89 Spanish speakers that were administered the BVMT-R during an outpatient neuropsychological evaluation. Out of the 89 Spanish speakers, 43 were subjects in litigation, 32 were neurological patients evaluated for clinical purposes, and 14 were healthy controls. The sample was 67% male/33% female, 53% South American, 33% Caribbean (Dominican, Puerto Rican, Cuban), 10% Central American, 3% North American (Mexican), and 1% Spanish, with a mean age of 44.2 years (SD = 14.2; range = 20-78) and mean education of 11 years (SD = 3.7; range = 0-20). Test administration for each patient was completed in Spanish by a fluent, Spanish-speaking examiner. In total, 64/89 (72%) were classified as valid and 25/89 (28%) as invalid based on performance across the Test of Memory Malingering (TOMM), at least one additional PVT (Rey-15 item memory test; Rey Dot Counting Test; Reliable Digit Span; WHO-AVLT recognition trial) and objective diagnostic criteria identifying invalid performance. Analyses included three univariate analyses of variance (ANOVA), with the groups (healthy vs neurological vs litigation) as independent variables and performance on BVMT-RD as the dependent variable.Results:Statistically significant differences among the groups were found F(2,86)=8.32, p < .001). Post-hoc analysis (Scheffe test) showed the mean of the litigation group to be significantly lower than the means of the other two groups (healthy and neurological), which showed no difference between them. An ANOVA with validity groups as the fixed factor and BVMT-R RD index as the dependent variable was significant F(1,85)= 21.02, p <.001). Results of a ROC curve analysis yielded statistically significant AUC (.794). The optimal cut-score was BVMT-R RD < 5 (48% sensitivity/88% specificity).Conclusions:Results of the BVMT-R RD index in this Spanish-speaking population differed by subgroup, with worse performance seen in individuals involved in litigation, compared to those who were not (healthy and neurological). Notably, the BVMT-R RD index significantly differentiated validity groups, maintaining adequate sensitivity and good specificity. Overall, results demonstrate promise for BVMT-RD as a PVT for Spanish-speaking populations.

80 Demographic Factors Affecting Sensitivity Rates of Performance Validity Tests in Spanish-Speaking Forensic Litigants.

Objective:This retrospective study compared base rates of failure on a series of standalone and embedded performance validity tests (PVTs) in a sample of Spanish-speaking forensic litigants and explored the impact of demographic factors on PVT performance.Participants and Methods:62 Spanish-speaking participants involved in litigation (primarily for work-related mTBI) underwent outpatient neuropsychological evaluation. Country of origin spanned South American (56.5%), Caribbean (22.5%), Central American (16.1%), North American (3.2%), and Spanish (1.6%) regions. Of this sample, 56 completed the Test of Memory Malingering (TOMM), 45 completed the Rey Fifteen Item Test (RFIT), and 49 completed the Dot Counting Test (DCT). Embedded validity measures, Reliable Digit Span (RDS) and the WHO-Auditory Verbal Learning Test (WHO-AVLT) were completed by 32 and 48 participants, respectively.Results:Effects of age (M=42.4, SD=11.72) and region of origin did not significantly impact overall performance on any measure. Mean scores across all standalone PVTs were below cutoffs, which have been previously suggested for use with Spanish-speaking populations (TOMM <40, RFIT total <21, DCT e-score >18). Overall base rates of failure were as follows: 52.5% TOMM (T1 M= 37.5, SD=10.7; T2 M=35.1, SD=10.6), 64.9% RFIT (M=17.8, SD=7.8) 57.6% DCT (M=18.3, SD=8.8), 51.1% RDS (M=6.1, SD=1.6), 29.4% WHO-AVLT (M=10.7, SD=3.9. Years of education (M=9.98, 3.96) was significantly correlated with RFIT total score (r(43) = .48, p<.01) and DCT e-score (r(47) = -.34, p<.05. When stratified by level of education (0-6, 7-11, and 12+), a large discrepancy in base rate of failure was observed on the RFIT, with failures in 92% of participants with less than six years of education, as compared to 52% and 59% failure in those with 7-11 and 12+ years, respectively. Variability in base rates of DCT failure across levels of education, although less extreme than on the RFIT, again demonstrated higher rates of failure in participants with less than six years of education (0-6: 71%, 7-11: 54%, 12+: 52%).Conclusions:These findings add to the existing literature surrounding measurement of suboptimal effort in Spanish-speaking populations. Base rates of PVT failure on both standalone and embedded measures were generally much higher than those reported in prior studies of forensic or compensation-seeking groups, including some with Spanish speaking participants. These high rates of failure are likely attributable, at least in part, to sample characteristics, due to the high proportion of individuals engaged in litigation associated with workplace injuries on construction sites at the study location. Such findings illustrate the importance of a thorough effort assessment for this population. Finally, results demonstrating reduced specificity of the RFIT in Spanish-speaking participants with less than six years of education, suggesting caution is warranted for its use in neuropsychological evaluations with such individuals.

A - 183 Novel Performance Validity Indicators for Frequently Used Memory Tests Using Infrequent Patterns of Memory Performance.

Determining performance validity is an imperative part of neuropsychological assessment in concussion management. Several existing embedded performance validity tests (PVTs) have been developed for popular measures of memory, Hopkins Verbal Learning Test Revised (HVLT-R) and Brief Visuospatial Memory Test Revised (BVMT-R). However, many PVTs utilize cognitive performance measures to classify performance invalidity, which increases the likelihood of false positives, especially in clinical populations. To reduce this possibility, a novel embedded PVT was developed utilizing infrequent patterns of performance in a primarily amnestic, dementia population. Neuropsychological data for 50 adult outpatients (Mean Age = 73, Range = 50-97) who met criteria for Major Neurocognitive Disorder (MND) were reviewed. Infrequent patterns of performance (IPOP) were identified on the HVLT-R and BVMT-R, including inconsistency between recall and recognition, unusually high intrusions, exceedingly poor learning, and lack of primary effect. Scaled scores were developed from the IPOP scores (0 = 0-74%, 1 = 75-94%, 2 = 95-99%). Total IPOP scaled scores were compared to existing embedded PVTs, which utilized recognition discrimination and percent retention scores, to examine classification rates. Failure rates on the embedded PVTs for the HVLT-R were 16-42% and BVMT-R 38-58% whereas elevated IPOP scaled scores were identified in less than 15% of the sample. High frequencies of embedded PVT failures were found in a clinical sample of patients with MND, which is concerning for false positive misclassification. Infrequent patterns of memory performances identified in a dementia population were less frequent and could potentially be utilized to examine performance validity in persons suspected of producing invalid memory performances.

H - 53 Performance Validity in a Mexican Population.

The aim of the present study was to cross-validate recent research on three embedded performance validity tests (PVTs) in a Spanish-speaking Mexican sample. The sample consisted of 52 male and 77 female participants residing in Mexico with a mean age of 25.26 (SD = 10.45). Participants were neurologically and psychologically healthy and completed a neuropsychological battery in Spanish. The Reliable Digit Span, Phonemic Fluency Test (i.e., FAS), and Semantic Fluency Tests were used to evaluate failure rates in our sample. Daugherty and colleagues' (2022) PVT cutoff scores were used. Chi-square analysis was used to evaluate failure rates between the groups. Chi-square analysis revealed no significant failure rate differences between gender groups on any PVTs. However, we found that 12% of the male group failed the Phonemic Fluency Test compared to 5% of the female group, X2 = 1.747, p = 0.186. Meanwhile, 2% of the male group failed the Semantic Fluency Test compared to 0% of the female group, X2 = 1.492, p = 0.222. Finally, 6% of the male group failed the Reliable Digit Span compared to 17% of the female group, X2 = 3.529, p = 0.060. Our data shows that male participants demonstrate higher failure rates on verbal fluency tasks than females. Moreover, males had lower failure rates compared to females on Reliable Digit Span. Further exploration with bigger sample sizes is needed to explore what may be influencing validity test performance between gender groups. More studies in this area will help researchers and clinicians determine if persons completing cognitive tasks in Spanish are proving adequate effort.

Logical memory, visual reproduction, and verbal paired associates are effective embedded validity indicators in patients with traumatic brain injury

Objective This study was design to evaluate the potential of the recognition trials for the Logical Memory (LM), Visual Reproduction (VR), and Verbal Paired Associates (VPA) subtests of the Wechsler Memory Scales–Fourth Edition (WMS-IV) to serve as embedded performance validity tests (PVTs). Method The classification accuracy of the three WMS-IV subtests was computed against three different criterion PVTs in a sample of 103 adults with traumatic brain injury (TBI). Results The optimal cutoffs (LM ≤ 20, VR ≤ 3, VPA ≤ 36) produced good combinations of sensitivity (.33–.87) and specificity (.92–.98). An age-corrected scaled score of ≤5 on either of the free recall trials on the VPA was specific (.91–.92) and relatively sensitive (.48–.57) to psychometrically defined invalid performance. A VR I ≤ 5 or VR II ≤ 4 had comparable specificity, but lower sensitivity (.25–.42). There was no difference in failure rate as a function of TBI severity. Conclusions In addition to LM, VR, and VPA can also function as embedded PVTs. Failing validity cutoffs on these subtests signals an increased risk of non-credible presentation and is robust to genuine neurocognitive impairment. However, they should not be used in isolation to determine the validity of an overall neurocognitive profile.

Base rates of non-credible performance in a post-secondary student sample seeking accessibility accommodations

Objective: Performance Validity Tests (PVTs) have been used to identify non-credible performance in clinical, medicolegal, forensic, and, more recently, academic settings. The inclusion of PVTs when administering psychoeducational assessments is essential given that specific accommodation such as flexible deadlines and increased writing time can provide an external incentive for students without disabilities to feign symptoms. Method: The present study used archival data to establish base rates of non-credible performance in a sample of post-secondary students (n = 1045) who underwent a comprehensive psychoeducational evaluation for the purposes of obtaining academic accommodations. In accordance with current guidelines, non-credible performance was determined by failure on two or more freestanding or embedded PVTs. Results: 9.4% of participants failed at least two of the PVTs they were administered, of which 8.5% failed two PVTs, and approximately 1% failed three PVTs. Base rates of failure for specific PVTs ranged from 25% (b Test) to 11.2% (TOVA). Conclusions: The present study found a lower base rate of non-credible performance than previously observed in comparable populations. This likely reflects the utilization of conservative criteria in detecting non-credible performance to avoid false positives. By contrast, inconsistent base rates previously found in the literature may reflect inconsistent methodologies. These results further emphasize the importance of administering multiple PVTs during psychoeducational assessments. The implications of these findings can further inform clinicians administering assessments in academic settings and aid in the appropriate utilization of PVTs in psychoeducational evaluation to determine accessibility accommodations.

New Child and Adolescent Memory Profile Embedded Performance Validity Test.

It is essential to interpret performance validity tests (PVTs) that are well-established and have strong psychometrics. This study evaluated the Child and Adolescent Memory Profile (ChAMP) Validity Indicator (VI) using a pediatric sample with traumatic brain injury (TBI). A cross-sectional sample of N = 110 youth (mean age = 15.1years, standard deviation [SD] = 2.4 range = 8-18) on average 32.7weeks (SD = 40.9) post TBI (71.8% mild/concussion; 3.6% complicated mild; 24.6% moderate-to-severe) were administered the ChAMP and two stand-alone PVTs. Criterion for valid performance was scores above cutoffs on both PVTs; criterion for invalid performance was scores below cutoffs on both PVTs. Classification statistics were used to evaluate the existing ChAMP VI and establish a new VI cutoff score if needed. There were no significant differences in demographics or time since injury between those deemed valid (n = 96) or invalid (n = 14), but all ChAMP scores were significantly lower in those deemed invalid. The original ChAMP VI cutoff score was highly specific (no false positives) but also highly insensitive (sensitivity [SN] = .07, specificity [SP] = 1.0). Based on area under the curve (AUC) analysis (0.94), a new cutoff score was established using the sum of scaled scores (VI-SS). A ChAMP VI-SS score of 32 or lower achieved strong SN (86%) and SP (92%). Using a 15% base rate, positive predictive value was 64% and negative predictive value was 97%. The originally proposed ChAMP VI has insufficient SN in pediatric TBI. However, this study yields a promising new ChAMP VI-SS, with classification metrics that exceed any other current embedded PVT in pediatrics.

Specificity of Embedded Performance Validity Tests in Elderly Veterans with Mild and Major Neurocognitive Disorder.

This study explored the specificity of four embedded performance validity tests (PVTs) derived from common neuropsychological tasks in a sample of older veterans with verified cognitive decline and whose performance was deemed valid by licensed psychologists. Participants were 180 veterans who underwent comprehensive neuropsychological evaluation, were determined to have valid performance following profile analysis/conceptualization, and were diagnosed with mild neurocognitive disorder (i.e., MCI; n = 64) or major neurocognitive disorder (i.e., Dementia; n =116). All participants completed at least one of four embedded PVTs: Reliable Digit Span (RDS), California Verbal Learning Test-2nd ed. Short Form (CVLT-II SF) Forced choice, Trails B:A, and Delis-Kaplan Executive Function System (DKEFS) Letter and Category Fluency. Adequate specificity (i.e., ≥90%) was achieved at modified cut-scores for all embedded PVTs across MCI and Dementia groups. Trails B:A demonstrated near perfect specificity at its traditional cut-score (Trails B:A < 1.5). RDS ≤ 5 and CVLT-II SF Forced Choice ≤7 led to <10% false positive classification errors across MCI and dementia groups. DKEFS Letter and Category Fluency achieved 90% specificity at extremely low normative cut-scores. RDS, Trails B:A, and CVLT-II SF Forced Choice reflect promising embedded PVTs in the context of dementia evaluations. DKEFS Letter and Category Fluency appear too sensitive to genuine neurocognitive decline and, therefore, are inappropriate PVTs in adults with MCI or dementia. Additional research into embedded PVT sensitivity (via known-groups or analogue designs) in MCI and dementia is needed.

Racial differences in positive findings on embedded performance validity tests

Introduction Embedded performance validity tests (PVTs) may show increased positive findings in racially diverse examinees. This study examined positive findings in an older adult sample of African American (AA) and European American (EA) individuals recruited as part of a study on aging and cognition. Method The project involved secondary analysis of deidentified National Alzheimer’s Coordinating Center data (N = 22,688). Exclusion criteria included diagnosis of dementia (n = 5,550), mild cognitive impairment (MCI; n = 5,160), impaired but not MCI (n = 1,126), other race (n = 864), and abnormal Mini Mental State Examination (MMSE < 25; n = 135). The initial sample included 9,853 participants (16.4% AA). Propensity score matching matched AA and EA participants on age, education, sex, and MMSE score. The final sample included 3,024 individuals with 50% of participants identifying as AA. Premorbid ability estimates were calculated based on demographics. Failure rates on five raw score and six age-adjusted scaled score PVTs were examined by race. Results Age, education, sex, MMSE, and premorbid ability estimate were not significantly different by race. Thirteen percent of AA and 3.8% of EA participants failed two or more raw score PVTs (p < .0001). On age-adjusted PVTs, 20.6% of AA and 5.9% of EA participants failed two or more (p < .0001). Conclusions PVT failure rates were significantly higher among AA participants. Findings indicate a need for cautious interpretation of embedded PVTs with underrepresented groups. Adjustments to embedded PVT cutoffs may need to be considered to improve diagnostic accuracy.

A Direct Comparison of 10 WAIS-IV Digit Span Embedded Validity Indicators among a Mixed Neuropsychiatric Sample with Varying Degrees of Cognitive Impairment.

Reliable Digit Span (RDS), RDS-Revised (RDS-R), and age-corrected scaled score (ACSS) have been previously validated as embedded performance validity tests (PVTs) from the Wechsler Adult Intelligence Scale-IV Digit Span subtest (WAIS-IV DS). However, few studies have directly compared the relative utility of these and other proposed WAIS-IV DS validity indicators within a single sample. This study compared classification accuracies of 10 WAIS-IV DS indices in a mixed neuropsychiatric sample of 227 outpatients who completed a standardized neuropsychological battery. Participants with ≤1 PVT failures of the four, freestanding criterion PVTs constituted the valid group (n = 181), whereas those with ≥2 PVT failures formed the invalid group (n = 46). Among the valid group, 113 met criteria for mild cognitive impairment (MCI). Classification accuracies for all DS indicators were statistically significant across the overall sample and subsamples with and without MCI, apart from indices derived from the Forward trial in the MCI sample. DS Sequencing ACSS, working memory RDS (wmRDS), and DS ACSS emerged as the most effective predictors of validity status, with acceptable to excellent classification accuracy for the overall sample (AUCs = 0.792-0.816; 35%-50% sensitivity/88%-96% specificity). Although most DS indices demonstrated clinical utility as embedded PVTs, DS Sequencing ACSS, wmRDS, and DS ACSS may be particularly robust to cognitive impairment, minimizing risk of false positive errors while identifying noncredible performance. Moreover, DS indices incorporating data from multiple trials (i.e., wmRDS, DS ACSS) also generally yielded greater classification accuracy than those derived from a single trial.

Embedded cognitive and emotional/affective self-reported symptom validity indices on the patient competency rating scale

ABSTRACT Objective Although there is an abundance of research on stand-alone and embedded performance validity tests and stand-alone symptom validity tests (SVTs), less emphasis has been placed on embedded SVTs. The goal of the current study was to examine the ability of embedded indicators within the Patient Competency Rating Scale (PCRS) to separately detect invalid cognitive and/or emotional/affective symptom responding. Method Participants included 299 veterans assessed in a VA medical center epilepsy monitoring unit from 2013–2017 (mean age = 48.8 years, SD = 13.5 years). Two SVT composites were created; self-reported cognitive symptom validity (SVT-C) and self-reported emotional/affective symptom validity (SVT-E). Groups were compared on PCRS total and index scores (i.e., cognitive, activities of daily living, emotional, and interpersonal competencies) using ANOVAs. Receiver operating characteristic (ROC) curve analyses assessed the classification accuracy of the PCRS total and index scores for SVT-C and SVT-E. Results In ANOVAs, SVT-C was significantly associated with all PCRS indices, while SVT-E was only significantly associated with the PCRS total, emotional, and interpersonal competency indices. Although the PCRS-T ≤ 90 had the strongest classification of SVT-C and SVT-E (specificities: .90, sensitivities: .44 to .50), PCRS index scores showed suggestive evidence of domain specificity, with PCRS-ADL ≤22, PCRS-C ≤ 20, and PCRS-CADL ≤45 best classifying SVT-C (specificities: .92, sensitivities: .33) and the PCRS-E ≤ 18 best classifying the SVT-E group (specificity: .93, sensitivity: .40). Conclusion Results suggest the PCRS may be used to obtain clinically useful information while including embedded indicators that can assess cognitive and/or emotional/affective symptom invalidity.

Cross-Cultural Performance Validity Testing: Managing False Positives in Examinees with Limited English Proficiency

ABSTRACT Base rates of failure (BR Fail ) on performance validity tests (PVTs) were examined in university students with limited English proficiency (LEP). BR Fail was calculated for several free-standing and embedded PVTs. All free-standing PVTs and certain embedded indicators were robust to LEP. However, LEP was associated with unacceptably high BR Fail (20–50%) on several embedded PVTs with high levels of verbal mediation (even multivariate models of PVT could not contain BR Fail ). In conclusion, failing free-standing/dedicated PVTs cannot be attributed to LEP. However, the elevated BR Fail on several embedded PVTs in university students suggest an unacceptably high overall risk of false positives associated with LEP.

A-231 Can Embedded Performance Validity Measures Predict Performance on a Stand-Alone Test?

Abstract Objective: Determine the extent to which a group of embedded performance validity tests (PVT) predict performance on a stand-alone test; and examine the classification accuracy of embedded PVTs. Method: Participants were 102 adult patients who underwent neuropsychological evaluation (M age = 31.85, SD = 11.79; M education = 13.82, SD = 2.50; M FSIQ = 104.72). Patients were administered the Medical Symptom Validity Test (MSVT), and 10 embedded PVT scores were extracted from their neuropsychological profiles. Logistic regression determined the extent to which scores on embedded PVTs predicted MSVT designation as credible or noncredible. Classification accuracy also was calculated for the embedded PVTs, both individually and based on number of “failures.” Results: A final regression model (Chi-square (6) = 20.06, p = .003, Nagelkerke R Square = .42) was 38% sensitive, 97% specific, and correctly predicted credible patients (negative predictive accuracy = 95%) more often than noncredible patients (positive predictive accuracy = 50%). Several embedded PVTs had comparable classification accuracy. Conservatively defining noncredible performance as failure on at least three embedded PVTs produced the best combination of sensitivity (50%) and specificity (86%), and was associated with a false positive rate of 14%. Conclusions: Embedded PVTs correctly predict credible and noncredible performance on the MSVT at a high level. Although a multivariate predictive approach may have slightly more utility than examining embedded PVT scores separately, several individual PVTs have comparable accuracy. It remains unclear which approach is superior, and more research is needed.

A-230 A Story Told in Four Parts: Embedded Validity Indicators’ Effectiveness in Predicting Word Choice Test Validity Classification

Abstract Objective: Many individual embedded performance validity tests (PVTs) have been independently cross-validated as validity indices among various clinical populations; however, research examining the additional predictive value from combining multiple embedded PVTs is more limited. The study's purpose was to examine the accuracy of eight embedded PVTs for predicting pass/failure performance on a well-validated freestanding PVT, the Advanced Clinical Solutions: Word Choice Test (WCT). Method: The study consisted of 225 adults referred for outpatient neuropsychological evaluation that included the following embedded PVTs: Rey Auditory Verbal Learning Test (RAVLT) Forced Choice (FC), RAVLT Effort Score, Brief Visual Memory Test-Revised Recognition Discrimination (BVMT-RD), Digit Span Age Corrected Scaled Score, Reliable Digit Span, Stroop Color and Word Test Word Reading T-Score, Letter Fluency T-Score, and Trail Making Test Part A T-Score. The sample was 56% female and racially diverse (40% Black, 35% White, 17% Hispanic, 5% Asian, and 2% other ethnicities) with a mean age of 45.96 (SD=16.12) and mean education of 13.96 (SD=2.73) years. Results: A forward stepwise logistic regression was performed to predict WCT pass/fail using the embedded PVTs as predictors. The model was achieved in three steps (p95% accuracy in correctly classifying WCT pass group. Conclusion: RAVLT-FC, BVMT-RD, and TMT-A combined reliably predict WCT pass/fail group. Verbal and attention based embedded PVTs were found to have the most robust predictive capacity of WCT validity status.

A-227 A Story Told in Four Parts: Embedded Validity Indicators’ Effectiveness in Predicting Dot Counting Test Validity Classification

Abstract Objective: Combining multiple embedded performance validity tests (PVTs) can produce similar classification accuracy to freestanding PVTs. However, there is a lack of research on the incremental predictive power of various combinations of PVTs. Thus, we compared eight embedded PVTs to assess those that best predict classification accuracy on the Dot Counting Test (DCT). Method: Sample included 225 patients (mean age=45.96; mean education=13.96; 56% female, 44% male; 35% White, 40% Black, 17% Hispanic, 5% Asian, 2% Other) undergoing neuropsychological evaluation that included multiple embedded PVTs, including: Brief Visuospatial Memory Test-Revised Recognition Discrimination (BVMT-R RD), Stroop Color and Word Test Word Reading T-Score (Stroop-T), Trail Making Test Part A T-Score (TMT-A), Rey Auditory Verbal Learning Test (RAVLT) Forced Choice, RAVLT Effort Score, Digit Span Age Corrected Scaled Score, Reliable Digit Span, and Letter Fluency T-Score. Patients were classified into valid/invalid groups based on four independent criterion PVTs. Results: A forward stepwise logistic regression was performed to predict DCT pass/fail using the aforementioned embedded PVTs as predictors. The model was achieved in three steps (p&amp;lt;.05); Step 1: BVMT-R RD (Classification Accuracy=87.6%; Nagelkerke R2 =.30), Step 2: BVMT-R RD + Stroop-T (Classification Accuracy=89.6%; Nagelkerke R2 =.44); Step 3: BVMT-R RD + Stroop-T + TMT-A (Classification Accuracy=90.1%; Nagelkerke R2 =.49). Conclusion: BVMT-R RD + Stroop-T + TMT-A reliably predicted the DCT pass/fail group. Thus, this combination of embedded PVTs may be reliable predictors of validity classification when time prohibits delivery of freestanding PVTs, such as the DCT.

A-229 A Story Told in Four Parts: Embedded Validity Indicators’ Effectiveness in Predicting Test of Memory Malingering Validity Classification

Abstract Objective: The continuous sampling of performance validity throughout an evaluation is essential to ensure accurate and reliable data in neuropsychological assessment. Although embedded performance validity tests (PVTs) have less sensitivity than freestanding PVTs, the combination of multiple embedded PVTs can have similar classification accuracy to freestanding PVTs. This study examined a set of common embedded PVTs to assess those that best predict classification accuracy on the Test of Memory Malingering (TOMM). Method: The sample was 56% female (n=126), and racially diverse with 40% Black, 35% White, 17% Hispanic, 5% Asian, and 2% other ethnicities. Participants underwent a comprehensive neuropsychological evaluation that included the following embedded PVTs: Rey Auditory Verbal Learning Test (RAVLT) Forced Choice (FC), RAVLT Effort Score, Brief Visual Memory Test-Revised Recognition Discrimination, Digit Span Age Corrected Scaled Score (DS ACSS), Reliable Digit Span, Stroop Color and Word Test Word Reading T-Score, Letter Fluency T-Score, and Trail Making Test Part A T-Score. Results: A forward stepwise logistic regression was performed to predict TOMM Trial 1 pass/fail using the embedded PVTs as predictors. The model was achieved in two steps (p95% accuracy in correctly classifying TOMM T1 pass group. Conclusions: Results indicate that RAVLT FC and DS ACSS together reliably predict TOMM pass/fail group. When time constraints limit use of the TOMM, RAVLT FC and DSACC may be reliable predictors of validity classification.