Criterion Performance Validity Tests Research Articles

B - 129 Examining the Influence of Age and Other Demographics on Word Choice (WC)

Abstract Objective Performance validity tests (PVTs) often vary by examinee characteristics including age and education among other demographics. Normative data on WC are limited to adults in the 16–69 age range. We examined relationships among demographic variables and WC in a mixed clinical sample. Method Local clinical data were retrospectively examined. Cases with WC and four independent PVTs were included; cases with dementia, intellectual disability or psychosis were excluded yielding a sample (N = 362) in which 61% of participants were female and 44% identified as Hispanic/Latino. Average age and education were 48 and 13 years, respectively. Outcome measures included WC and PVT scores and dichotomous cutoffs. Results WC was associated with age (r = −0.27) but not education. WC scores were significantly different by gender with a small effect (d = 0.39) but not by ethnicity. Using logistic regression with WC as outcome, age, education, and gender were significant predictors. Significantly more males (11%) scored below the WC cutoff than females (5%) in the valid group. Among participants 70 and older, 17% scored below the standard WC cutoff, but only 7% scored below expectation on two or more criterion PVTs. Lowering the WC cutoff resulted in improved specificity in the 70 and older subset: 15% with WC ≤ 40, and 11% with WC ≤ 36. Conclusions WC scores varied by age and gender. The standard WC cutoff showed an elevated false positive rate among adults 70 and older. Consideration might be given to adjusting the cutoff if subsequent research supports these preliminary findings. Further research might also examine relationships among gender, education, and WC.

Effect of processing speed and memory performance on classification accuracy of the dot counting test in a mixed neuropsychiatric sample

ABSTRACT Objective This study examined the impact of impairment in two specific cognitive abilities, processing speed and memory, on Dot Counting Test (DCT) classification accuracy by evaluating performance validity classification accuracy across cognitively unimpaired, single-domain impairment, and multidomain impairment subgroups within a mixed clinical sample. Method Cross-sectional data were analyzed from 348 adult outpatients classified as valid (n = 284) or invalid (n = 64) based on four independent criterion performance validity tests (PVTs). Unimpaired (n = 164), single-domain processing speed impairment (n = 24), single-domain memory impairment (n = 53), and multidomain processing speed and memory impairment (n = 43) clinical subgroups were established among the valid group. Both the traditional DCT E-score and unrounded E-score were examined. Results Overall, the DCT demonstrated acceptable to excellent classification accuracy across the unimpaired (area under the curve [AUC] traditional E-score=.855; unrounded E-score=.855) and single-domain impairment groups (traditional E-score AUCs = .690–.754; unrounded E-score AUCs = .692–747). However, it did not reliably discriminate the multidomain processing speed and memory impairment group from the invalid performers (traditional and unrounded E-scores AUC = .557). Conclusions Findings support the DCT as a non-memory-based freestanding PVT for use with single-domain cognitive impairment, with traditional E-score ≥17 (unrounded E-score ≥16.95) recommended for those with memory impairment and traditional E-score ≥19 (unrounded ≥18.08) with processing speed impairment. Moreover, results replicated previously established optimal cutoffs for unimpaired groups using both the traditional (≥14) and unrounded (≥13.84) E-scores. However, the DCT did not reliably discriminate between invalid performance and multidomain cognitive impairment, indicating caution is warranted when using the DCT with patients suspected of greater cognitive impairment.

Lowered cutoffs to reduce false positives on the Word Memory Test

ABSTRACT Objective To adjust the decision criterion for the Word Memory Test (WMT, Green, 2003) to minimize the frequency of false positives. Method Archival data were combined into a database (n = 3,210) to examine the best cut score for the WMT. We compared results based on the original scoring rules and those based on adjusted scoring rules using a criterion based on 16 performance validity tests (PVTs) exclusive of the WMT. Cutoffs based on peer-reviewed publications and test manuals were used. The resulting PVT composite was considered the best estimate of validity status. We focused on a specificity of .90 with a false-positive rate of less than .10 across multiple samples. Results Each examinee was administered the WMT, as well as on average 5.5 (SD = 2.5) other PVTs. Based on the original scoring rules of the WMT, 31.8% of examinees failed. Using a single failure on the criterion PVT (C-PVT), the base rate of failure was 45.9%. When requiring two or more failures on the C-PVT, the failure rate dropped to 22.8%. Applying a contingency analysis (i.e., X2) to the two failures model on the C-PVT measure and using the original rules for the WMT resulted in only 65.3% agreement. However, using our adjusted rules for the WMT, which consisted of relying on only the IR and DR WMT subtest scores with a cutoff of 77.5%, agreement between the adjusted and the C-PVT criterion equaled 80.8%, for an improvement of 12.1% identified. The adjustmeny resulted in a 49.2% reduction in false positives while preserving a sensitivity of 53.6%. The specificity for the new rules was 88.8%, for a false positive rate of 11.2%. Conclusions Results supported lowering of the cut score for correct responding from 82.5% to 77.5% correct. We also recommend discontinuing the use of the Consistency subtest score in the determination of WMT failure.

Comparing embedded performance validity indicators within the WAIS-IV Letter-Number sequencing subtest to Reliable Digit Span among adults referred for evaluation of attention-deficit/hyperactivity disorder

Objectives: This study investigated the Wechsler Adult Intelligence Scale-Fourth Edition Letter-Number Sequencing (LNS) subtest as an embedded performance validity indicator among adults undergoing an attention-deficit/hyperactivity disorder (ADHD) evaluation, and its potential incremental value over Reliable Digit Span (RDS). Method: This cross-sectional study comprised 543 adults who underwent neuropsychological evaluation for ADHD. Patients were divided into valid (n = 480) and invalid (n = 63) groups based on multiple criterion performance validity tests. Results: LNS total raw scores, age-corrected scaled scores, and age- and education-corrected T-scores demonstrated excellent classification accuracy (area under the curve of .84, .83, and .82, respectively). The optimal cutoff for LNS raw score (≤16), age-corrected scaled score (≤7), and age- and education-corrected T-score (≤36) yielded .51 sensitivity and .94 specificity. Slightly lower sensitivity (.40) and higher specificity (.98) was associated with a more conservative T-score cutoff of ≤33. Multivariate models incorporating both LNS and RDS improved classification accuracy (area under the curve of .86), and LNS scores explained a significant but modest proportion of variance in validity status above and beyond RDS. Chaining LNS T-score of ≤33 with RDS cutoff of ≤7 increased sensitivity to .69 while maintaining ≥.90 specificity. Conclusions: Findings provide preliminary evidence for the criterion and construct validity of LNS as an embedded validity indicator in ADHD evaluations. Practitioners are encouraged to use LNS T-score cutoff of ≤33 or ≤36 to assess the validity of obtained test data. Employing either of these LNS cutoffs with RDS may enhance the detection of invalid performance.

21 A Comparison of the Memory and Non-Memory Based Performance Validity Measures for Detecting Invalid Neuropsychological Test Performance among Individuals with and without Memory Impairment

Objective:Few to no studies have directly compared the relative classification accuracies of the memory-based (Brief Visuospatial Memory Test-Revised Recognition Discrimination [BVMT-R RD] and Rey Auditory Verbal Learning Test Forced Choice [RAVLT FC]) and non-memory based (Reliable Digit Span [RDS] and Stroop Color and Word Test Word Reading trial [SCWT WR]) embedded performance validity tests (PVTs). This study’s main objective was to evaluate their relative classification accuracies head-to-head, as well as examine how their psychometric properties may vary among subgroups with and without genuine memory impairment.Participants and Methods:This cross-sectional study included 293 adult patients who were administered the BVMT-R, WAIS-IV Digit Span, RAVLT and SCWT during outpatient neuropsychological evaluation at a Midwestern academic medical center. The overall sample was 58.0% female, 36.2% non-Hispanic White, 41.3% non-Hispanic Black, 15.7% Hispanic, 4.8% Asian/Pacific Islander, and 2.0% other, with a mean age of 45.7 (SD=15.8) and a mean education of 13.9 years (SD=2.8). Three patients had missing data, resulting in a final sample size of 290. Two hundred thirty-three patients (80%) were classified as having valid neurocognitive performance and 57 (20%) as having invalid neurocognitive performance based on performance across four independent, criterion PVTs (i.e., Test of Malingering Memory Trial 1, Word Choice Test, Dot Counting Test, Medical Symptom Validity Test). Of those with valid neurocognitive performance, 76 (48%) patients were considered as having genuine memory impairment through a memory composite band score (T<37 for (RAVLT Delayed Recall T-score + BVMT-R Delay Recall T-score/2).Results:The average memory composite band score for valid neurocognitive scores was T = 49.63 as compared to T = 27.57 for genuine memory impairment individuals. Receiver operating characteristic [ROC] curve analyses yielded significant areas under the curve (AUCs=.79-.87) for all four validity indices (p’s < .001). When maintaining acceptable specificity (91%-95%), all validity indices demonstrated acceptable yet varied sensitivities (35%-65%). Among the subgroup with genuine memory impairment, ROC curve analyses yielded significantly lower AUCs (.64-.69) for three validity indices (p’s < .001), except RDS (AUC=.644). At acceptable specificity (88%-93%), they yielded significantly lower sensitivities across indices (19%-39%). In the current sample, RAVLT FC and BVMT-R RD had the largest changes in sensitivities, with 19% and 26% sensitivity/90%-92% specificity at optimal cut-scores of <10 and <2, respectively, for individuals with memory impairment, compared to 65% and 61% sensitivity/94% specificity at optimal cut-scores of <13 and <4, respectively, for those without memory impairment.Conclusions:Of the four validity scales, memory-based embedded PVTs yielded higher sensitivities while maintaining acceptable specificity compared to non-memory based embedded PVTs. However, they were also susceptible to the greatest declines in sensitivity among the subgroup with genuine memory impairment. As a result, careful consideration should be given to using memory-based embedded PVTs among individuals with clinically significant memory impairment based on other sources of information (e.g., clinical history, behavioral observation).

Logical memory, visual reproduction, and verbal paired associates are effective embedded validity indicators in patients with traumatic brain injury

Objective This study was design to evaluate the potential of the recognition trials for the Logical Memory (LM), Visual Reproduction (VR), and Verbal Paired Associates (VPA) subtests of the Wechsler Memory Scales–Fourth Edition (WMS-IV) to serve as embedded performance validity tests (PVTs). Method The classification accuracy of the three WMS-IV subtests was computed against three different criterion PVTs in a sample of 103 adults with traumatic brain injury (TBI). Results The optimal cutoffs (LM ≤ 20, VR ≤ 3, VPA ≤ 36) produced good combinations of sensitivity (.33–.87) and specificity (.92–.98). An age-corrected scaled score of ≤5 on either of the free recall trials on the VPA was specific (.91–.92) and relatively sensitive (.48–.57) to psychometrically defined invalid performance. A VR I ≤ 5 or VR II ≤ 4 had comparable specificity, but lower sensitivity (.25–.42). There was no difference in failure rate as a function of TBI severity. Conclusions In addition to LM, VR, and VPA can also function as embedded PVTs. Failing validity cutoffs on these subtests signals an increased risk of non-credible presentation and is robust to genuine neurocognitive impairment. However, they should not be used in isolation to determine the validity of an overall neurocognitive profile.

Multivariate models provide an effective psychometric solution to the variability in classification accuracy of D-KEFS Stroop performance validity cutoffs

ObjectiveThe study was designed to expand on the results of previous investigations on the D-KEFS Stroop as a performance validity test (PVT), which produced diverging conclusions. Method The classification accuracy of previously proposed validity cutoffs on the D-KEFS Stroop was computed against four different criterion PVTs in two independent samples: patients with uncomplicated mild TBI (n = 68) and disability benefit applicants (n = 49). Results Age-corrected scaled scores (ACSSs) ≤6 on individual subtests often fell short of specificity standards. Making the cutoffs more conservative improved specificity, but at a significant cost to sensitivity. In contrast, multivariate models (≥3 failures at ACSS ≤6 or ≥2 failures at ACSS ≤5 on the four subtests) produced good combinations of sensitivity (.39-.79) and specificity (.85-1.00), correctly classifying 74.6-90.6% of the sample. A novel validity scale, the D-KEFS Stroop Index correctly classified between 78.7% and 93.3% of the sample. Conclusions A multivariate approach to performance validity assessment provides a methodological safeguard against sample- and instrument-specific fluctuations in classification accuracy, strikes a reasonable balance between sensitivity and specificity, and mitigates the invalid before impaired paradox.

Comparing the Independent and Aggregated Accuracy of Trial 1 and the First 10 TOMM Items for Detecting Invalid Neuropsychological Test Performance Across Civilian and Veteran Clinical Samples.

Previous studies support using two abbreviated tests of the Test of Memory Malingering (TOMM), including (a) Trial 1 (T1) and (b) the number of errors on the first 10 items of T1 (T1e10), as performance validity tests (PVTs). In this study, we examined the independent and aggregated predictive utility of TOMM T1 and T1e10 for identifying invalid neuropsychological test performance across two clinical samples. We employed cross-sectional research to examine two independent and demographically diverse mixed samples of military veterans and civilians (VA = 108; academic medical center = 234) of patients who underwent neuropsychological evaluations. We determined validity groups by patient performance on four independent criterion PVTs. We established concordances between passing/failing the TOMM T1e10 and T1, followed by logistic regression to determine individual and aggregated accuracy of T1e10 and T1 for predicting validity group membership. Concordance between passing T1e10 and T1 was high, as was overall validity (87-98%) across samples. By contrast, T1e10 failure was more highly concordant with T1 failure (69-77%) than with overall invalidity status (59-60%) per criterion PVTs, whereas T1 failure was more highly concordant with invalidity status (72-88%) per criterion PVTs. Logistic regression analyses demonstrated similar results, with T1 accounting for more variance than T1e10. However, combining T1e10 and T1 accounted for the most variance of any model, with T1e10 and T1 each emerging as significant predictors. TOMM T1 and, to a lesser extent, T1e10 were significant predictors of independent criterion-derived validity status across two distinct clinical samples, but they did not offer improved classification accuracy when aggregated.

The impact of visual memory impairment on Victoria Symptom Validity Test performance: A known-groups analysis

Objective We assessed the effect of visual learning and recall impairment on Victoria Symptom Validity Test (VSVT) accuracy and response latency for Easy, Difficult, and Total Items. Method A sample of 163 adult patients administered the VSVT and Brief Visuospatial Memory Test-Revised were classified as valid (114/163) or invalid (49/163) groups via independent criterion performance validity tests (PVTs). Classification accuracies for all VSVT indices were examined for the overall sample, and separately for subgroups based on visual memory functioning. Results In the overall sample, all indices produced acceptable classification accuracy (areas under the curve [AUCs] ≥ 0.79). When stratified by visual learning/recall impairment, accuracy indices yielded acceptable classification for both the unimpaired (AUCs ≥0.79) and impaired subsamples (AUCs ≥0.75). Latency indices had acceptable classification accuracy for the unimpaired subsample (AUCs ≥0.74), but accuracy and sensitivity dropped for the impaired sample (AUCs ≥0.67). Conclusions VSVT accuracy and response latency yielded acceptable classification accuracies in the overall sample, and this effect was maintained in those with and without visual learning/recall impairment for the accuracy indices. Findings indicate that the VSVT is a psychometrically robust PVT with largely invariant cut-scores, even in the presence of bona fide visual learning/recall impairment.

Examining Traditional and Novel Validity Indicators from the Medical Symptom Validity Test Across Levels of Verbal and Visual Memory Impairment.

This cross-sectional study examined accuracy of traditional Medical Symptom Validity Test (MSVT) validity indicators, including immediate recognition (IR), delayed recognition (DR), and consistency (CNS), as well as a novel indicator derived from the mean performance on IR, DR, and CNS across verbal, visual, and combined learning and memory impairment bands. A sample of 180 adult outpatients was divided into valid (n = 150) and invalid (n = 30) groups based on results of four independent criterion performance validity tests. Verbal and visual learning and recall were classified as indicative of no impairment, mild impairment, or severe impairment based on performance on the Rey Auditory Verbal Learning Test and Brief Visuospatial Memory Test-Revised, respectively. In general, individual MSVT subtests were able to accurately classify performance as valid or invalid, even in the context of severe learning and memory deficits. However, as verbal and visual memory impairment increased, optimal MSVT cut-scores diverged from manual-specified cutoffs such that DR and CNS required cut-scores to be lowered to maintain adequate specificity. By contrast, the newly proposed scoring algorithm generally showed more robust psychometric properties across the memory impairment bands. The mean performance index, a novel scoring algorithm using the mean of the three primary MSVT subtests, may be a more robust validity indicator than the individual MSVT subtests in the context of bona fide memory impairment.

Flipping the Script: Measuring Both Performance Validity and Cognitive Ability with the Forced Choice Recognition Trial of the RCFT.

In this study we attempted to replicate the classification accuracy of the newly introduced Forced Choice Recognition trial (FCR) of the Rey Complex Figure Test (RCFT) in a clinical sample. We administered the RCFT FCR and the earlier Yes/No Recognition trial from the RCFT to 52 clinically referred patients as part of a comprehensive neuropsychological test battery and incentivized a separate control group of 83 university students to perform well on these measures. We then computed the classification accuracies of both measures against criterion performance validity tests (PVTs) and compared results between the two samples. At previously published validity cutoffs (≤16 & ≤17), the RCFT FCR remained specific (.84–1.00) to psychometrically defined non-credible responding. Simultaneously, the RCFT FCR was more sensitive to examinees’ natural variability in visual-perceptual and verbal memory skills than the Yes/No Recognition trial. Even after being reduced to a seven-point scale (18-24) by the validity cutoffs, both RCFT recognition scores continued to provide clinically useful information on visual memory. This is the first study to validate the RCFT FCR as a PVT in a clinical sample. Our data also support its use for measuring cognitive ability. Replication studies with more diverse samples and different criterion measures are still needed before large-scale clinical application of this scale.

All of the accuracy in half of the time: Assessing abbreviated versions of the Test of Memory Malingering in the context of verbal and visual memory impairment

Objective The Test of Memory Malingering (TOMM) Trial 1 (T1) and errors on the first 10 items of T1 (T1-e10) were developed as briefer versions of the TOMM to minimize evaluation time and burden, although the effect of genuine memory impairment on these indices is not well established. This study examined whether increasing material-specific verbal and visual memory impairment affected T1 and T1-e10 performance and accuracy for detecting invalidity. Method: Data from 155 neuropsychiatric patients administered the TOMM, Rey Auditory Verbal Learning Test (RAVLT), and Brief Visuospatial Memory Test-Revised (BVMT-R) during outpatient evaluation were examined. Valid (N = 125) and invalid (N = 30) groups were established by four independent criterion performance validity tests. Verbal/visual memory impairment was classified as ≥37 T (normal memory); 30 T-36T (mild impairment); and ≤29 T (severe impairment). Results: Overall, T1 had outstanding accuracy, with 77% sensitivity/90% specificity. T1-e10 was less accurate but had excellent discriminability, with 60% sensitivity/87% specificity. T1 maintained excellent accuracy regardless of memory impairment severity, with 77% sensitivity/≥88% specificity and a relatively invariant cut-score even among those with severe verbal/visual memory impairment. T1-e10 had excellent classification accuracy among those with normal memory and mild impairment, but accuracy and sensitivity dropped with severe impairment and the optimal cut-score had to be increased to maintain adequate specificity. Conclusion: TOMM T1 is an effective performance validity test with strong psychometric properties regardless of material-specificity and severity of memory impairment. By contrast, T1-e10 functions relatively well in the context of mild memory impairment but has reduced discriminability with severe memory impairment.

Assessment of differential neurocognitive performance based on the number of performance validity tests failures: A cross-validation study across multiple mixed clinical samples

Objective: This cross-sectional study examined the effect of number of Performance Validity Test (PVT) failures on neuropsychological test performance among a demographically diverse Veteran (VA) sample (n = 76) and academic medical sample (AMC; n = 128). A secondary goal was to investigate the psychometric implications of including versus excluding those with one PVT failure when cross-validating a series of embedded PVTs. Method: All patients completed the same six criterion PVTs, with the AMC sample completing three additional embedded PVTs. Neurocognitive test performance differences were examined based on number of PVT failures (0, 1, 2+) for both samples, and effect of number of criterion failures on embedded PVT performance was analyzed among the AMC sample. Results: Both groups with 0 or 1 PVT failures performed better than those with ≥2 PVT failures across most cognitive tests. There were nonsignificant differences between those with 0 or 1 PVT failures except for one test in the AMC sample. Receiver operator characteristic curve analyses found no differences in optimal cut score based on number of PVT failures when retaining/excluding one PVT failure. Conclusion: Findings support the use of ≥2 PVT failures as indicative of performance invalidity. These findings strongly support including those with one PVT failure with those with zero PVT failures in diagnostic accuracy studies, given that their inclusion reflects actual clinical practice, does not reduce sample sizes, and does not artificially deflate neurocognitive test results or inflate PVT classification accuracy statistics.

A Known-Groups Validation of the Medical Symptom Validity Test and Analysis of the Genuine Memory Impairment Profile.

This study cross-validated the Medical Symptom Validity Test (MSVT) in a mixed neuropsychiatric sample and examined its accuracy for identifying invalid neuropsychological performance using a known-groups design. Cross-sectional data from 129 clinical patients who completed the MSVT were examined. Validity groups were established using six, independent criterion performance validity tests, which yielded 98 patients in the valid group and 31 in the invalid group. All MSVT subtest scores were significantly lower in the invalid group (ηp2=.22-.39). Using published cut-scores, sensitivities of 42% to 71% were found among the primary effort subtests, and 74% sensitivity/90% specificity was observed for the overall MSVT. Among this sample, the MSVT component validity scales produced areas under the curve of .78-.86, suggesting moderate classification accuracy. At optimal cut-scores, the MSVT primary effort validity scales demonstrated 55% to 71% sensitivity/91% to 93% specificity, with the Consistency subtest exhibiting the strongest psychometric properties. The MSVT exhibited relatively robust sensitivity and specificity, supporting its utility as a briefer freestanding performance validity test to its predecessor, the Word Memory Test. Finally, the Genuine Memory Impairment Profile appears promising for patients with Major Neurocognitive Disorder, but is cautioned against for those without significant functional decline in activities of daily living at this time.

The Divergent Roles of Symptom and Performance Validity in the Assessment of ADHD

Objective: This study examined concordance between symptom and performance validity among clinically-referred patients undergoing neuropsychological evaluation for Attention-Deficit/Hyperactivity Disorder (ADHD). Method: Data from 203 patients who completed the WAIS-IV Working Memory Index, the Clinical Assessment of Attention Deficit-Adult (CAT-A), and ≥4 criterion performance validity tests (PVTs) were analyzed. Results: Symptom and performance validity were concordant in 76% of cases, with the majority being valid performance. Of the remaining 24% of cases with divergent validity findings, patients were more likely to exhibit symptom invalidity (15%) than performance invalidity (9%). Patients demonstrating symptom invalidity endorsed significantly more ADHD symptoms than those with credible symptom reporting (ηp2 = .06–.15), but comparable working memory test performance, whereas patients with performance invalidity had significantly worse working memory performance than those with valid PVT performance (ηp2 = .18). Conclusion: Symptom and performance invalidity represent dissociable constructs in patients undergoing neuropsychological evaluation of ADHD and should be evaluated independently.

A-219 Cross-validation of a Delis-Kaplan Executive Function System Color Word Interference Test embedded performance validity test

Abstract Objective Eglit and colleagues (2019) found that the sum of the four age-corrected scaled scores from the Color Word Interference Test (CWIT) of the Delis-Kaplan Executive Function System (D-KEFS) could act as a useful embedded performance validity test (PVT). As such, we attempted to cross-validate their findings. Method Patients included 312 individuals who completed neuropsychological evaluations. Individuals were excluded if they were not administered CWIT, were not administered at least 4 criterion PVTs, had diagnoses of dementia or intellectual disability, or had indeterminate validity results (i.e., failure of one PVT). Valid performers (n = 231) were those who passed all criterion PVTs while invalid performers (n = 81) failed two or more criterion PVTs. A receiver operating characteristic curve was conducted for the CWIT embedded PVT. Results Area under the curve (AUC) was .804. At a cutoff of < 26, specificity was 90% and sensitivity was 48%. At a cutoff of < 20 (the first cutoff at which 90% specificity was found by Eglit et al.), specificity was 95% and sensitivity was 32%. At an even more conservative cutoff of < 18 (the cutoff recommended by Eglit et al.), specificity was 96% and sensitivity was 22%. Conclusions These results cross-validate Eglit et al.’s findings, indicating that the sum of age-corrected scaled scores across the four CWIT trials can effectively serve as an embedded PVT. A more liberal cutoff was able to be applied in our sample but, even at conservative cutoffs documented in Eglit et al., sensitivity rates were adequate enough to warrant use of the index as an embedded PVT.

A-210 Does Rounding Matter? Cross-Validation of E-Score Calculation Method for the Dot Counting Test

Abstract Objective Recent evidence suggested the traditional “rounding” procedure used to calculate Dot Counting Test (DCT) E-score cut-offs provides little advantage and may inadvertently lower test sensitivity. This study examined whether DCT psychometric properties differ when E-score values are rounded. Method This cross-sectional study included 132 mixed neuropsychiatric patients who completed the DCT during outpatient evaluation. The sample was 55% female/45% male and 36% Caucasian/35% African American/20% Hispanic/7% Asian/2% other, with mean age of 44.4 (SD=16.1), and mean education of 14.0 years (SD=2.5). In total, 105 (80%) had valid neuropsychological test performance and 27 (20%) had invalid performance based on 4 independent criterion performance validity tests. Results In the overall sample, receiver operating characteristic (ROC) curve analyses yielded significant areas under the curve (.802-.817) for both rounded and unrounded E-score values with respective optimal cut-scores of ≥19 and ≥19.73, both producing 44% sensitivity/93% specificity. Among cognitively impaired patients, ROC curve analyses yielded significant AUCs (.756-.764), and suggested the same cut-scores and sensitivities, albeit with minimally reduced specificity (traditional: 91%; unrounded: 92%). In contrast, more liberal cut-scores of ≥13 (traditional) and ≥13.745 (unrounded) were indicated among cognitively unimpaired patients (AUCs: .880-.906), and sensitivity was notably improved (traditional: 74%; unrounded: 67%) with equivalent specificity (90%). Conclusions Findings from the overall sample suggested marginally better classification accuracy for the traditional E-score, though both methods demonstrated comparable psychometric properties. The optimal cut-score for cognitively unimpaired patients replicated findings from prior literature, but a higher cut-score was indicated for cognitively impaired patients.

Cross-validation of the Invalid Forgetting Frequency Index (IFFI) from the Test of Memory Malingering.

To increase sensitivity of the Test of Memory Malingering (TOMM), adjustments have been proposed, including adding consistency indices. The Invalid Forgetting Frequency Index (IFFI) is the most recently developed consistency index. While strong classification accuracy rates were originally reported, it currently lacks cross-validation. A sample of 184 outpatients was utilized. Valid performers passed all criterion performance validity tests (PVTs) and invalid performers failed two or more PVTs. Classification accuracy statistics were calculated. AUC for the IFFI was 0.80, demonstrating adequate discrimination between valid and invalid groups. A score of 3 or more inconsistent responses resulted in sensitivity and specificity rates of 63% and 92%, respectively. This is the first article to cross-validate the IFFI. In both the original IFFI study and the current study, the same cut-off was found to maintain at least 90% specificity while producing higher sensitivity rates than those achieved by traditional TOMM indices.

Impact of criterion measures on the classification accuracy of TOMM-1

This study was designed to examine the effect of various criterion measures on the classification accuracy of Trial 1 of the Test of Memory Malingering (TOMM-1), a free-standing performance validity test (PVT). Archival data were collected from a case sequence of 91 (M Age = 42.2 years; M Education = 12.7) patients clinically referred for neuropsychological assessment. Trials 2 and Retention of the TOMM, the Word Choice Test, and three validity composites were used as criterion PVTs. Classification accuracy varied systematically as a function of criterion PVT. TOMM-1 ≤ 43 emerged as the optimal cutoff, resulting in a wide range of sensitivity (.47–1.00), with perfect overall specificity. Failing the TOMM-1 was unrelated to age, education or gender, but was associated with elevated self-reported depression. Results support the utility of TOMM-1 as an independent, free-standing, single-trial PVT. Consistent with previous reports, the choice of criterion measure influences parameter estimates of the PVT being calibrated. The methodological implications of modality specificity to PVT research and clinical/forensic practice should be considered when evaluating cutoffs or interpreting scores in the failing range.

Classification accuracy rates of four TOMM validity indices when examined independently and jointly

Objective: This study investigated sensitivity and specificity rates of four Test of Memory Malingering (TOMM) indices (Trial 1, Trial 2, Retention, and Albany Consistency Index (ACI)) and examined how classification accuracy rates change when utilizing these indices in various combinations.Method: A sample of 202 neuropsychological outpatients was utilized. Patients were categorized as valid performers if they passed all criterion performance validity tests (PVTs) and were determined to be invalid performers if they failed two or more criterion PVTs. Classification accuracy statistics were obtained for individual TOMM indices as well as combinations of TOMM indices.Results: When using only Trial 1 as a validity indicator, the TOMM identified 57% of invalidly performing individuals. When all TOMM indices were examined, the ACI demonstrated the highest sensitivity value (63%) but it also demonstrated the lowest specificity value (91%). Allowing for failure of any of the four TOMM indices provided the best overall sensitivity value (67%) while maintaining adequate specificity (90%). Finally, it was determined that failure of three or more TOMM validity indices resulted in a specificity rate of 97% and failure of four of more TOMM validity indices resulted in a specificity rate of 98%.Conclusions: Classification accuracy of TOMM validity indices are discussed in relation to positive and negative predictive values. Results suggest that clinicians can examine all four TOMM validity indices concurrently, particularly in settings where high base rates of invalidity occur.