Standalone Performance Validity Test Research Articles

Chronic Pain: Utility of an Eye-Tracker Integrated Stand-alone Performance Validity Test

Noncredible cognitive performance among chronic pain patients garners increased clinical attention. The Word Memory Test (WMT)—a well-established stand-alone validity indicator—was recently integrated with an eye tracker, and its utility was assessed using healthy simulators. The current study expands on this earlier work by assessing the utility of the eye-tracker integrated WMT to detect noncredible cognitive performance in the context of chronic pain. Chronic pain outpatients were randomly assigned to either a simulation (i.e., patients simulating cognitive impairment; n = 22) or honest control (i.e., patients performing to the best of their ability; n = 23) conditions. They then completed the WMT’s immediate recognition (IR) subtest while their eye movements were recorded. Simulators gazed less at relevant stimuli and gazed more at irrelevant stimuli than controls. Sensitivity levels tended to be low to moderate when maintaining specificities ≥ 90%, as customary in the field. While a previously developed scale that integrates eye movement measures using a logistic regression did not adequately differentiate the groups, conjunctive rules (i.e., the participant was required to fail both the WMT's classification scheme and the eye movement measure with the strongest discriminative capacity) were associated with higher specificities than those of the WMT’s conventional classification scheme. Overall, the eye-tracker integrated WMT shows initial clinical utility for detecting noncredible cognitive performance. Decreasing costs of eye trackers and enhanced usability will hopefully encourage further research of their utility for detecting noncredible cognitive performance and integration of this novel technology with other stand-alone validity indicators.

Are correlations among behavioral decision making tasks moderated by simulated cognitive impairment?

Behavioral decision making tasks are common in research settings, with only the Iowa Gambling Task available for clinical assessments. However, correlations among these tasks are low, indicating each may assess a distinct component of decision making. In addition, it is unclear whether these tasks are sensitive to invalid performance or even simulated impairment. The present study examined relationships among decision making tasks and whether simulated impairment moderates the relationships among them. Across two studies (Study 1: n = 166, Study 2: n = 130), undergraduate student participants were asked to try their best or to simulate a specific diagnosis (Attention-Deficit/Hyperactivity Disorder; Study 1), decision making impairment (Study 2), or general cognitive impairment (Study 2). They then completed a battery of tests including embedded and standalone performance validity tests (PVTs) and three behavioral decision making tasks. Across studies, participants simulating impairment were not distinguishable from controls on any of the behavioral tasks. Few significant correlations emerged among tasks across studies and the pattern of relationships between tasks did not differ on the basis of simulator or PVT failure status. Collectively, our findings suggest that these tasks may not be vulnerable to simulated cognitive impairment, and that the tasks measure largely non-overlapping aspects of decision making.

12 Measuring effort on a continuum provides improved insight into concussion baseline cognitive assessments

Objective:Baseline assessment of cognitive performance is common practice under many concussion management protocols and is required for collegiate athletes by the NCAA. The purpose of baseline cognitive assessment is to understand an athlete’s individual uninjured cognitive performance, as opposed to using population normative data. This baseline can then serve as a reference point for recovery after concussion and can inform return-to-play decisions. However, multiple factors, including lack of effort, can contribute to misrepresentation of baseline results which raises concern for reliability during return-to-play decision-making. Measuring effort across a continuum, rather than as a dichotomous variable (good versus poor effort) may provide informative insight related to cognitive performance at baseline.Participants and Methods:Collegiate athletes (n = 231) completed the Immediate Post-Concussion Assessment and Cognitive Test (ImPACT) as part of their baseline pre-participation concussion evaluation. ImPACT creates composite scores of Verbal Memory, Visual Memory, Visual-Motor Speed, and Reaction Time. Baseline self-reported symptoms and total hours of sleep the night prior to testing are also collected through ImPACT. ImPACT has one embedded indicator within the program to assess effort, and research has identified an additional three embedded indicators. Athletes were also administered one stand-alone performance validity test, either the Medical Symptom Validity Test (n = 130) or the Rey Dot Counting Test (n = 101), to independently measure effort. Effort was estimated across a continuum (zero, one, two, or three or more failed effort indicators) with both stand-alone and embedded effort indicators. We evaluated the relationship between effort, symptoms, self-reported sleep, Reaction Time composite score and Visual-Motor Speed composite score using a linear regression model.Results:We found that 121 athletes passed all effort indicators, while 39 athletes failed only one effort indicator, 40 athletes failed two effort indicators, and 31 athletes failed three or four (three+) effort indicators. Self-reported symptoms and total hours of sleep were not related to effort, but Reaction Time and VisualMotor Speed composites were. Specifically, performance on the Visual-Motor Speed composite was significantly worse for athletes who failed two or three+ effort indicators compared to athletes who did not fail any, and performance on the Reaction Time composite was significantly worse only for athletes who failed three+ effort indicators. Additionally, athletes who failed one or more effort indicators and reported less sleep performed worse on both the Visual-Motor Speed and Reaction Time composites, compared to those who reported less sleep and did not fail any effort indicators.Conclusions:Athletes who failed one effort indicator did not perform significantly worse on Reaction Time and Visual-Motor Speed composites compared to those who passed all effort indicators. However, 31% of athletes failed two or more effort indicators and these athletes performed worse on cognitive tests, likely due to factors impacting their ability to put forth good effort. These results suggest that effort is more complex than a previously used dichotomous variable and highlights the importance of using several indicators of effort throughout baseline assessments. In addition, the importance of sleep should be emphasized during baseline assessments, especially when effort is questionable.

35 Pairwise Concurrence Rates Between Standalone and Embedded Performance Validity Tests

Objective:Determine the classification concordance between a standalone performance validity test (PVT) and embedded PVTs from multiple cognitive domains.Participants and Methods:Participants were 106 patients (49.1% female; 69% white) that underwent neuropsychological evaluation at an outpatient university doctoral clinical psychology training and research clinic (M/SD: age = 32.38/11.95; education = 13.7/2.75). A comprehensive neuropsychological battery included the Medical Symptom Validity Test (MSVT) and embedded PVTs from different cognitive domains: attention - Wechsler Adult Intelligence Scale - Fourth Edition Reliable Digit Span and Digit Span age-corrected scaled score (DS ACSS); memory - California Verbal Learning Test, 3rd Edition (CVLT-3) Forced-Choice Recognition (FCR), executive functions -Wisconsin Card Sorting Test (WCST) Failure to Maintain Set (FMS); visual-spatial/construction -Rey Complex Figure Test (RCFT) Copy raw score; language - Boston Diagnostic Aphasia Examination Complex Ideation Material (CIM); and motor functions - Finger Tapping Test (FTT). All participants were administered the MSVT but not all participants were administered all seven embedded PVTs. Credible/noncredible classification concordance rates and kappa correlations (i.e., percentage of agreement) were computed for each pairwise PVT combination.Results:Twenty-two percent (n = 23) of the sample failed at least one PVT, with 17.0% (n = 18) failing at least two. DS ACSS was the embedded PVT with the highest MSVT concordance rate at 92.4% and a fair kappa coefficient of .39; WCST FMS had the lowest concordance with MSVT at 82.9% and a slight kappa coefficient of .19. The highest concordance among embedded PVTs from different cognitive domains was CVLT-3 FCR and RCFT Copy raw score at 89.7% with a fair kappa coefficient of .35; the lowest agreement among embedded PVTs was WCST FMS and FTT at 74.0% with a kappa coefficient of -.02. More conservative kappa coefficients among all pair-wise embedded PVT combinations from different cognitive domains ranged from -.02 to .36. For all standalone and embedded PVT pairwise concordance rates, only two fell below the recommended minimum agreement of 80%: FCR vs. FMS = 79.3% and FMS vs. FTT = 74.0%.Conclusions:Embedded PVTs across various cognitive domains have high agreement with a standalone PVT to aid in classifying noncredible performance, in the 83-92% range. Embedded PVTs from different cognitive domains also have mostly high agreement classification rates amongst themselves in aiding to determine noncredible performance, in the 74-90% range, with the lowest agreement rate between executive function and motor tests at 74%. More conservative kappa-based agreements between PVT pair-wise combinations were fairly consistent with other studies, with most being in the fair range. Finally, these findings indicate about a 17% base rate of noncredible cognitive performance in an outpatient university-based clinic.

80 Demographic Factors Affecting Sensitivity Rates of Performance Validity Tests in Spanish-Speaking Forensic Litigants.

Objective:This retrospective study compared base rates of failure on a series of standalone and embedded performance validity tests (PVTs) in a sample of Spanish-speaking forensic litigants and explored the impact of demographic factors on PVT performance.Participants and Methods:62 Spanish-speaking participants involved in litigation (primarily for work-related mTBI) underwent outpatient neuropsychological evaluation. Country of origin spanned South American (56.5%), Caribbean (22.5%), Central American (16.1%), North American (3.2%), and Spanish (1.6%) regions. Of this sample, 56 completed the Test of Memory Malingering (TOMM), 45 completed the Rey Fifteen Item Test (RFIT), and 49 completed the Dot Counting Test (DCT). Embedded validity measures, Reliable Digit Span (RDS) and the WHO-Auditory Verbal Learning Test (WHO-AVLT) were completed by 32 and 48 participants, respectively.Results:Effects of age (M=42.4, SD=11.72) and region of origin did not significantly impact overall performance on any measure. Mean scores across all standalone PVTs were below cutoffs, which have been previously suggested for use with Spanish-speaking populations (TOMM <40, RFIT total <21, DCT e-score >18). Overall base rates of failure were as follows: 52.5% TOMM (T1 M= 37.5, SD=10.7; T2 M=35.1, SD=10.6), 64.9% RFIT (M=17.8, SD=7.8) 57.6% DCT (M=18.3, SD=8.8), 51.1% RDS (M=6.1, SD=1.6), 29.4% WHO-AVLT (M=10.7, SD=3.9. Years of education (M=9.98, 3.96) was significantly correlated with RFIT total score (r(43) = .48, p<.01) and DCT e-score (r(47) = -.34, p<.05. When stratified by level of education (0-6, 7-11, and 12+), a large discrepancy in base rate of failure was observed on the RFIT, with failures in 92% of participants with less than six years of education, as compared to 52% and 59% failure in those with 7-11 and 12+ years, respectively. Variability in base rates of DCT failure across levels of education, although less extreme than on the RFIT, again demonstrated higher rates of failure in participants with less than six years of education (0-6: 71%, 7-11: 54%, 12+: 52%).Conclusions:These findings add to the existing literature surrounding measurement of suboptimal effort in Spanish-speaking populations. Base rates of PVT failure on both standalone and embedded measures were generally much higher than those reported in prior studies of forensic or compensation-seeking groups, including some with Spanish speaking participants. These high rates of failure are likely attributable, at least in part, to sample characteristics, due to the high proportion of individuals engaged in litigation associated with workplace injuries on construction sites at the study location. Such findings illustrate the importance of a thorough effort assessment for this population. Finally, results demonstrating reduced specificity of the RFIT in Spanish-speaking participants with less than six years of education, suggesting caution is warranted for its use in neuropsychological evaluations with such individuals.

40 Performance validity in a presurgical epilepsy population

Objective:To examine whether suboptimal performance as determined by formal validity testing would predict neurocognitive scores in a sample of 83 pre-surgical, non-litigating epilepsy patients.Participants and Methods:Participants were 83 patients who underwent comprehensive outpatient neuropsychological testing as part as their evaluation as epilepsy surgery candidates. The sample consisted of 41 females and 42 males, with 72 patients identifying as White, 5 as Black, 2 as Hispanic, 1 as Asian, and 2 as other. Mean age was 36 (SD=12.4) mean FSIQ was 87 (SD=12.7), mean years of education of 12.9 (SD=2.1). Each patient’s assessment included a stand-alone performance validity test (PVTs)— Word Memory Test (WMT), the Test of Memory Malingering (TOMM), or the Medical Symptom Validity Test (MSVT)—as well as two embedded measures of validity—the California Verbal Learning Test Forced Choice (CVLT FC) and WAIS-IV Reliable Digit Span (RDS). Pass/fail rates were analyzed, with valid performance being defined as pass score on at least two of the completed PVTs (N=73 Pass Effort group 86.9%; N=10 Failed Effort group 11.9%). Point-biserial Pearson correlations were conducted to determine the relationship between validity pass/fail status and WAIS-IV FSIQ, VCI, and PRI scores, CVLT-II Trials 1-5 Total T scores, CVLT-II Long Delay Free Recall z scores, WMS-III Logical Memory II T scores, BVMT Total Recall T scores, BVMT Delayed Recall T scores, and Trail Making Test (TMT) B T scores.Results:Significant relationships were found between Failed Effort group and all neurocognitive scores except BVMT Total Recall. On average, the Failed Effort group obtained significantly lower FSIQ (M=76.57, SD=10.94), VCI (M=80.89, SD=16.03), PRI (M=81.00, SD=14.91), CVLT-II Trials 1-5 Total (M =34, SD=6.89), CVLT-II Long Delay Free Recall (M =-2.44, SD=1.43), WMS-IV Logical Memory II (M =4.83, SD=2.79), BVMT Delayed Recall (M=26.38, SD=6.41), and TMT B (M=29.70, SD=11.46) standard scores compared to the Pass Effort group (FSIQ M=88.09, SD=12.52; VCI M=92.13, SD=13.61; PRI M=91.14, SD=12.06; CVLT-II Trials 1-5 Total M=47.86, SD=12.02; CVLT- II Long Delay Free Recall M=-.44, SD=1.11; WMS-III Logical Memory II M=8.41, SD=3.17; BVMT Delayed Recall M=39.19, SD=12.66; TMT B M=39.34, SD=13.18). Correlation coefficients were r=-.266* (FSIQ), r=-.255* (VCI), r=-.271* (PRI), r=.361**(CVLT-II Total), r=-.474** (CVLT-II LDFR), r=-.298** (WMS-IV LM II), r=-.308** (BVMT DR), and r=-.240* (TMTB). All coefficients were significant at the .05 (*) or .01 (**) level.Conclusions:Results suggest that pass/fail status on formal validity testing predicts depressed performance on a variety of neurocognitive measures. Therefore, predicting surgical outcome of resection/ablation (e.g., compensation of contralateral hemisphere) should not be based upon neuropsychological memory performance alone when there are failures on tests of engagement as memory scores have strong correlations to pass/fail status on formal validity testing. Overall, this emphasizes the importance of routinely integrating PVTs as part of pre-epilepsy surgery neuropsychological evaluations.

92 Validation of Coin-in-Hand Procedure in a Veteran Population

Objective:Performance validity tests (PVTs) provide a methodological approach to detecting credible neurocognitive performances. This proves invaluable to the diagnostic process, as it allows neuropsychologists to objectively determine if an evaluation reflects a patient’s true neurocognitive abilities or if external factors are impacting the results. However, their addition to a testing battery can increase an already lengthy evaluation. As such, there is a need for sensitive but less time intensive PVTs. The purpose of this study is to validate the Coin-in-Hand (CIH) procedure as a quick and effective PVT within a veteran population.Participants and Methods:68 English-speaking patients were identified from an outpatient neuropsychological assessment dataset. Performances were correlated to the well- validated Reliable Digit Span (RDS), and several other soft indicators of task engagement including expanded COWAT, BVMT-False Alarms (FA), WCST Failure to Maintain Set (FTM), TOMM, and the RBANS Effort Index (EI). All participants attempted CIH and RDS, testing was discontinued if 2 or more PVTs were invalid. An AUC analysis was conducted to determine how well the CIH discriminated between valid and invalid performance and determine the tests optimal cut-off score (sensitivity > 0.90 while maintaining the highest possible specificity). Logistic Regression was conducted to determine how well the CIH predicted performance validity.Results:Subject mean(SD) age and education were 55.25 (16.06) and 13.41 (2.55) years, respectively. 17% female, 60% Caucasian, and 32% Black. Descriptive statistics for each of the other performance validity tests were gathered. The CIH demonstrated low diagnostic accuracy (AUC = .66; p >.05; CI = .51 -.81); a cut score of <8 resulted in a sensitivity of .96 and a specificty of .64. Logistic Regression showed that CIH performance significantly predicted performance validity (X2 = -0.93; df = 1; N = 68; p < .05), accounting for 18-28% of the variance in performance classification (Cox & Snell R2 = .18; Nagelkerke R2 = .28). It correctly classified 96% of valid performers, but only correctly classified 35% of invalid performers, with an overall correct prediction rate of 83%. A predicted chase in log odds (B= -.93) and odd ratio [Exp (B) =.40] indicated that every unit increase in CIH score was associated with a decrease probability of performance invalidity. Logistic regression was also used to calculate the probability of performance invalidity at each possible CIH score (Table 1).Conclusions:Results suggests that poor performance on CIH does not necessarily equate to invalid performances, but instead, should act as a screener to cue neuropsychologists working with Veterans that additional PVTs should be considered. Overall, it was determined that CIH was able to correctly predict 35% of invalid performers and 96% of valid performers, with an overall correct prediction rate of 83%, suggesting the procedure may be too simple to be an effective standalone PVT for clinical use. These results also highlight that every correct response on the CIH was associated with a decreased probability of performance invalidity. Additionally, an AUC analysis determined the tests optimal cut off score to be <8, suggesting that shortening the procedure may be as effective as giving the full 10 trials.

Review of Statistical and Methodological Issues in the Forensic Prediction of Malingering from Validity Tests: Part I: Statistical Issues.

Forensic neuropsychological examinations with determination of malingering have tremendous social, legal, and economic consequences. Thousands of studies have been published aimed at developing and validating methods to diagnose malingering in forensic settings, based largely on approximately 50 validity tests, including embedded and stand-alone performance validity tests. This is the first part of a two-part review. Part I explores three statistical issues related to the validation of validity tests as predictors of malingering, including (a) the need to report a complete set of classification accuracy statistics, (b) how to detect and handle collinearity among validity tests, and (c) how to assess the classification accuracy of algorithms for aggregating information from multiple validity tests. In the Part II companion paper, three closely related research methodological issues will be examined. Statistical issues are explored through conceptual analysis, statistical simulations, and through reanalysis of findings from prior validation studies. Findings suggest extant neuropsychological validity tests are collinear and contribute redundant information to the prediction of malingering among forensic examinees. Findings further suggest that existing diagnostic algorithms may miss diagnostic accuracy targets under most realistic conditions. The review makes several recommendations to address these concerns, including (a) reporting of full confusion table statistics with 95% confidence intervals in diagnostic trials, (b) the use of logistic regression, and (c) adoption of the consensus model on the "transparent reporting of multivariate prediction models for individual prognosis or diagnosis" (TRIPOD) in the malingering literature.

Review of Statistical and Methodological Issues in the Forensic Prediction of Malingering from Validity Tests: Part II-Methodological Issues.

Forensic neuropsychological examinations to detect malingering in patients with neurocognitive, physical, and psychological dysfunction have tremendous social, legal, and economic importance. Thousands of studies have been published to develop and validate methods to forensically detect malingering based largely on approximately 50 validity tests, including embedded and stand-alone performance and symptom validity tests. This is Part II of a two-part review of statistical and methodological issues in the forensic prediction of malingering based on validity tests. The Part I companion paper explored key statistical issues. Part II examines related methodological issues through conceptual analysis, statistical simulations, and reanalysis of findings from prior validity test validation studies. Methodological issues examined include the distinction between analog simulation and forensic studies, the effect of excluding too-close-to-call (TCTC) cases from analyses, the distinction between criterion-related and construct validation studies, and the application of the Revised Quality Assessment of Diagnostic Accuracy Studies tool (QUADAS-2) in all Test of Memory Malingering (TOMM) validation studies published within approximately the first 20years following its initial publication to assess risk of bias. Findings include that analog studies are commonly confused for forensic validation studies, and that construct validation studies are routinely presented as if they were criterion-reference validation studies. After accounting for the exclusion of TCTC cases, actual classification accuracy was found to be well below claimed levels. QUADAS-2 results revealed that extant TOMM validation studies all had a high risk of bias, with not a single TOMM validation study with low risk of bias. Recommendations include adoption of well-established guidelines from the biomedical diagnostics literature for good quality criterion-referenced validation studies and examination of implications for malingering determination practices. Design of future studies may hinge on the availability of an incontrovertible reference standard of the malingering status of examinees.

Standalone Performance Validity Tests May Be Differentially Related to Measures of Working Memory, Processing Speed, and Verbal Memory in Patients With Multiple Sclerosis.

Cognitive functioning may account for minimal levels (i.e., 5%-14%) of variance of performance validity test (PVT) scores in clinical examinees. The present study extended this research twofold: (a) by determining the variance cognitive functioning explains within three distinct PVTs (b) in a sample of patients with multiple sclerosis (pwMS). Seventy-five pwMS (Mage = 48.50, 70.6% female, 80.9% White) completed the Victoria Symptom Validity Test (VSVT), Word Choice Test (WCT), Dot Counting Test (DCT), and three objective measures of working memory, processing speed, and verbal memory as part of clinical neuropsychological assessment. Regression analyses in credible groups (ns ranged from 54 to 63) indicated that cognitive functioning explained 24% to 38% of the variance in logarithmically transformed PVT variables. Variance from cognitive testing differed across PVTs: verbal memory significantly influenced both VSVT and WCT scores; working memory influenced VSVT and DCT scores; and processing speed influenced DCT scores. The WCT appeared least related to cognitive functioning of the included PVTs. Alternative plausible explanations, including the apparent domain/modality specificity hypothesis of PVTs versus the potential sensitivity of these PVTs to neurocognitive dysfunction in pwMS were discussed. Continued psychometric investigations into factors affecting performance validity, especially in multiple sclerosis, are warranted.

New Child and Adolescent Memory Profile Embedded Performance Validity Test.

It is essential to interpret performance validity tests (PVTs) that are well-established and have strong psychometrics. This study evaluated the Child and Adolescent Memory Profile (ChAMP) Validity Indicator (VI) using a pediatric sample with traumatic brain injury (TBI). A cross-sectional sample of N = 110 youth (mean age = 15.1years, standard deviation [SD] = 2.4 range = 8-18) on average 32.7weeks (SD = 40.9) post TBI (71.8% mild/concussion; 3.6% complicated mild; 24.6% moderate-to-severe) were administered the ChAMP and two stand-alone PVTs. Criterion for valid performance was scores above cutoffs on both PVTs; criterion for invalid performance was scores below cutoffs on both PVTs. Classification statistics were used to evaluate the existing ChAMP VI and establish a new VI cutoff score if needed. There were no significant differences in demographics or time since injury between those deemed valid (n = 96) or invalid (n = 14), but all ChAMP scores were significantly lower in those deemed invalid. The original ChAMP VI cutoff score was highly specific (no false positives) but also highly insensitive (sensitivity [SN] = .07, specificity [SP] = 1.0). Based on area under the curve (AUC) analysis (0.94), a new cutoff score was established using the sum of scaled scores (VI-SS). A ChAMP VI-SS score of 32 or lower achieved strong SN (86%) and SP (92%). Using a 15% base rate, positive predictive value was 64% and negative predictive value was 97%. The originally proposed ChAMP VI has insufficient SN in pediatric TBI. However, this study yields a promising new ChAMP VI-SS, with classification metrics that exceed any other current embedded PVT in pediatrics.

Varying failure criteria on performance validity tests influences interpretation of cognitive outcomes.

The present study examined the effects of applying various performance validity tests (PVT) failure criteria on the relationship between cognitive outcomes and posttraumatic stress (PTS) symptomology. One hundred and ninety-nine veterans with a history of mild traumatic brain injury referred for clinical evaluation completed cognitive tests, PVTs, and self-report measures of PTS symptoms and symptom exaggeration. Normative T scores of select cognitive tests were averaged into memory, attention/processing speed, and executive functioning composites. Separate one way analyses of variance assessed differences among high PTS (n = 140) versus low PTS (n = 59) groups and were repeated excluding participants based on varying combinations of PVT failure criteria. When no PVTs were considered, the high PTS group demonstrated worse performance across all three cognitive domains. Excluding those who failed two or more stand-alone, or two or more embedded validity measures resulted in group differences across all cognitive composites. When participants were excluded based on failure of any one embedded and any one stand-alone PVT measure combined, the high PTS group performed worse on the executive functioning and attention/processing speed composites. The remaining three proposed methods to control for performance validity resulted in null PTS-cognition relationships. Results remained largely consistent after controlling for symptom exaggeration. Methods of defining PVT failure can greatly influence differences in cognitive function between groups defined by PTS symptom levels. Findings highlight the importance of considering performance validity when interpreting cognitive data and warrant future investigation of PVT failure criteria in other conditions. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

A-233 Less Items, Less Problems: Using TOMMe10 to Predict Performance Validity

Abstract Objective: A frequently used standalone performance validity test (PVT) is the Test of Memory Malingering (TOMM), which utilizes two or three visual recognition trials to assess for suboptimal participant effort. In situations with limited time, shortened evaluation times are preferred. One method for this is to look at the error rate on the first 10 TOMM items (TOMMe10) for predicting invalid performance. The objective of this review is to investigate the utility of TOMMe10 in adult neuropsychological evaluations, including potential cutoffs. Data Selection: A systematic review of available literature was conducted using the PubMed database. The following keywords were used during the search: “TOMM first 10 items” and “TOMMe10.” Eight out of eleven articles were found to have met the inclusion criteria for this review, which required the study to speak to cutoffs used and to the effectiveness of TOMMe10 at predicting performance validity with adults on the TOMM or other validated assessments. Data Synthesis: All eight studies demonstrated adequate sensitivity, specificity, and discriminability of TOMMe10 for invalid performance, with all but four supporting a cutoff of ≥2 errors. Three proposed a cutoff of ≥1 error, while one proposed a cutoff of ≥3 errors. Both alternate cutoffs demonstrated adequate predictive power. Conclusions: This research demonstrated that TOMMe10 could accurately predict invalid PVT performance, except in cases of severe memory impairment. Optimal TOMMe10 cutoff results were mixed but a cutoff of ≥2 errors was the most supported. Future research could assess the utility of the various suggested TOMMe10 cutoffs in specific situations.

What are the predictors of TOMM failure in clinical TBI populations? A retrospective analysis.

To determine base rates of invalid performance on the Test of Memory Malingering (TOMM) in patients with traumatic brain injury (TBI) undertaking rehabilitation who were referred for clinical assessment, and the factors contributing to TOMM failure. Retrospective file review of consecutive TBI referrals for neuropsychological assessment over seven years. TOMM failure was conventionally defined as performance <45/50 on Trial 2 or Retention Trial. Demographic, injury, financial compensation, occupational, and medical variables were collected. Four hundred and ninety one TBI cases (Median age = 40 years [IQR = 26-52], 79% male, 82%severe TBI) were identified. Overall, 48 cases (9.78%) failed the TOMM. Logistic regression analyses revealed that use of an interpreter during the assessment (adjusted odds ratio [aOR] = 8.25, 95%CI = 3.96-17.18), outpatient setting (aOR = 4.80, 95%CI = 1.87-12.31) and post-injury psychological distress (aOR = 2.77, 95%CI = 1.35-5.70) were significant multivariate predictors of TOMM failure. The TOMM failure rate for interpreter cases was 49% (21/43) in the outpatient setting vs. 7% (2/30) in the inpatient setting. By comparison, 9% (21/230) of non-interpreter outpatient cases failed the TOMM vs. 2% (4/188) of inpatient cases. TOMM failure very rarely occurs in clinical assessment of TBI patients in the inpatient rehabilitation setting. It is more common in the outpatient setting, particularly in non-English-speaking people requiring an interpreter. The findings reinforce the importance of routinely administering stand-alone performance validity tests in assessments of clinical TBI populations, particularly in outpatient settings, to ensure that neuropsychological test results can be interpreted with a high degree of confidence.

Examining the Clinical Utility of Selected Memory-Based Embedded Performance Validity Tests in Neuropsychological Assessment of Patients with Multiple Sclerosis.

Within the neuropsychological assessment, clinicians are responsible for ensuring the validity of obtained cognitive data. As such, increased attention is being paid to performance validity in patients with multiple sclerosis (pwMS). Experts have proposed batteries of neuropsychological tests for use in this population, though none contain recommendations for standalone performance validity tests (PVTs). The California Verbal Learning Test, Second Edition (CVLT-II) and Brief Visuospatial Memory Test, Revised (BVMT-R)—both of which are included in the aforementioned recommended neuropsychological batteries—include previously validated embedded PVTs (which offer some advantages, including expedience and reduced costs), with no prior work exploring their utility in pwMS. The purpose of the present study was to determine the potential clinical utility of embedded PVTs to detect the signal of non-credibility as operationally defined by below criterion standalone PVT performance. One hundred thirty-three (133) patients (M age = 48.28; 76.7% women; 85.0% White) with MS were referred for neuropsychological assessment at a large, Midwestern academic medical center. Patients were placed into “credible” (n = 100) or “noncredible” (n = 33) groups based on a standalone PVT criterion. Classification statistics for four CVLT-II and BVMT-R PVTs of interest in isolation were poor (AUCs = 0.58–0.62). Several arithmetic and logistic regression-derived multivariate formulas were calculated, all of which similarly demonstrated poor discriminability (AUCs = 0.61–0.64). Although embedded PVTs may arguably maximize efficiency and minimize test burden in pwMS, common ones in the CVLT-II and BVMT-R may not be psychometrically appropriate, sufficiently sensitive, nor substitutable for standalone PVTs in this population. Clinical neuropsychologists who evaluate such patients are encouraged to include standalone PVTs in their assessment batteries to ensure that clinical care conclusions drawn from neuropsychological data are valid.

A-183 Base Rates of Performance and Symptom Validity Test Failures in Active Duty and Veteran Samples Referred for Attention-Deficit/Hyperactivity Disorder Evaluation

Abstract Objective To evaluate base rates of abnormal MMPI-2-RF symptom validity test (SVT) and stand-alone performance validity test (PVT) scores in active duty (ad) service member and Veterans Affairs clinical samples referred for neuropsychological assessment of ADHD. Method MMPI-2-RF SVT scores and various PVTs were evaluated in two samples referred specifically for assessment of ADHD: service members (N = 82) and Veterans (N = 355). SVT and PVT outcomes were then compared to published data establishing invalid base rates in larger, more general samples. Results Elevated MMPI-2-RF overreport scales (F-r ≥ 79; FS, RBS, and FBS-r ≥ 80; &amp; Fp-r ≥ 70) ranged from 14.6% (FBS-r) to 40.0% (RBS) in the ad sample and 12.0% (FBS-r) and 28.9% (RBS) in the veteran sample. MMPI-2-RF profiles that were invalidated due to overreport (e.g., F-r = 120, all others ≥100) were under 5% on the veteran sample and no greater than 6.1% in the ad sample. Elevations on underreport scales were higher in the veteran sample (L-r = 12.9%, K-r = 2.0%) than the ad sample (L-r = 3.7%, K-r = 1.2%). In the ad sample, 21.5% failed one PVT and 6.3% failed two. In the Veteran sample, 23.4% failed one PVT. Conclusions Elevated MMPI-2-RF overreport scales were relatively common on ad and Veterans referred for ADHD evaluations. However, frequencies of elevated and invalidated scales were diminished relative to overall base rates reported in national data. Regarding PVT failure, both samples were lower than national average invalid rates. Despite the incentive inherent in ADHD exams, invalidity in both samples were consistently lower for both SVTs and PVTs.

A-39 New Cut-Score to Improve Performance of the Memory Validity Profile (MVP) in Heterogenous Clinical Populations

Abstract Objective The Memory Validity Profile (MVP) is a standalone performance validity test developed specifically for use with children. Prior research has demonstrated the MVP’s strength in its ease of administration to children with a wide range of intellectual abilities. However, it has been found to lack sensitivity in detecting noncredible performance in select clinical populations using published cut-offs. The current study examines the MVP’s performance in a diagnostically heterogeneous clinical sample and proposes a new cut-off for optimization of sensitivity and specificity. Method Archival clinical data was examined from 96 participants referred for a comprehensive neuropsychological evaluation (ages 6–18). Receiver operating characteristic analysis was used to assess the discriminative ability of MVP in detecting cases of noncredible performance defined as failures on both the Test of Memory Malingering and Reliable Digit Span. Results Using published cut-offs, the MVP demonstrated perfect specificity (100%) but suboptimal sensitivity (33.3%). Receiver operating characteristic analysis revealed strong discrimination using MVP Total score (AUC = 0.891 (p &amp;lt; 0.001) and a MVP Total cut-score of ≤30 resulted in optimal sensitivity (89%) and specificity (63%). Conclusions Our findings provide additional evidence that published MVP cut-offs may be too lenient to adequately capture instances of noncredible performance and indicate an MVP Total score cut-off of ≤30 may be more appropriate for use with heterogeneous clinical populations.

The Development and Validation of the Embedded Validity Indicator for the Neuropsychological Assessment Battery.

The purpose of this study was to develop and validate an embedded measure of performance validity within the Neuropsychological Assessment Battery (NAB). This study involved a retrospective chart review at an outpatient neuropsychology clinic. Participants were 183 adults (ages 18-70) who completed the attention and memory modules of the NAB, as well as the Word Choice Test, Green's Medical Symptom Validity Test (MSVT), and Green's Non-Verbal MSVT, as part of a clinical neuropsychological assessment (n=147) or as part of a forensic neuropsychological evaluation (n=36). Replicating methodology utilized by Silverberg etal. (2007) for the development of the Effort Index within the Repeatable Battery for the Assessment of Neuropsychological Status, an Embedded Validity Indictor (EVI) for the NAB was developed in the present study based on Digits Forward and List Learning Long Delayed Forced-Choice Recognition (list recognition) subtests. Receiver operating characteristic curve analyses indicated the newly developed NAB EVI was able to significantly differentiate between valid and invalid status on stand-alone performance-validity tests, with area under the curve values ranging from 0.797 to 0.977. Optimal cutoffs for medical, forensic, and mixed samples were identified. The newly developed NAB EVI shows promise as an embedded performance validity measure; however, due to moderate sensitivity, it should be used in combination with stand-alone performance validity tests to detect invalid performance.

New cut-score to improve performance of the Memory Validity Profile (MVP) in heterogenous clinical populations

ABSTRACT The Memory Validity Profile (MVP) is a standalone performance validity test developed specifically for use with children. Prior research has demonstrated the MVP’s strength in its ease of administration to children with a wide range of intellectual abilities. However, it has been found to lack sensitivity in detecting noncredible performance in select clinical populations using published cutoffs. The current study examines the MVP’s performance in a diagnostically heterogeneous clinical sample and proposes a new cutoff for optimization of sensitivity and specificity. Archival clinical data were examined from 96 participants referred for a comprehensive neuropsychological evaluation (ages 6–18). Receiver operating characteristic analysis was used to assess the discriminative ability of MVP in detecting cases of noncredible performance defined as failures on both the Test of Memory Malingering and Reliable Digit Span. Using published cutoffs, the MVP demonstrated perfect specificity (100%) but suboptimal sensitivity (33.3%). Receiver operating characteristic analysis revealed strong discrimination using MVP Total score (AUC = 0.891 (p < 0.001)) and a MVP Total cut-score of ≤30 resulted in optimal sensitivity (89%) and specificity (63%). Our findings provide additional evidence that published MVP cutoffs may be too lenient to adequately capture instances of noncredible performance and indicate an MVP Total score cutoff of ≤30 may be more appropriate for use with heterogeneous clinical populations.

Relationship between Standalone Performance Validity Test Failure and Emotionality among Youth/student Athletes Experiencing Prolonged Recovery following Sports-related Concussion

ABSTRACT This study documented the rate of Performance Validity Testing (PVT) failure in 81 youth athletes, aged 10–21 years, experiencing prolonged recovery following sports-related concussion, and the relationship between PVT and emotional symptoms. Neuropsychological assessments were conducted across three test sessions with a stand-alone PVT at each session. Results showed that 48% (39/81) of individuals failed at least one PVT, with an overall PVT failure rate of 26% (64/243). Those failing at least one PVT scored significantly higher on anxiety but not depression or somatization. Results illustrate the importance of including measures of emotional and behavioral functioning in testing following SRC.