Performance Validity Tests Research Articles

Chronic Pain: Utility of an Eye-Tracker Integrated Stand-alone Performance Validity Test

Noncredible cognitive performance among chronic pain patients garners increased clinical attention. The Word Memory Test (WMT)—a well-established stand-alone validity indicator—was recently integrated with an eye tracker, and its utility was assessed using healthy simulators. The current study expands on this earlier work by assessing the utility of the eye-tracker integrated WMT to detect noncredible cognitive performance in the context of chronic pain. Chronic pain outpatients were randomly assigned to either a simulation (i.e., patients simulating cognitive impairment; n = 22) or honest control (i.e., patients performing to the best of their ability; n = 23) conditions. They then completed the WMT’s immediate recognition (IR) subtest while their eye movements were recorded. Simulators gazed less at relevant stimuli and gazed more at irrelevant stimuli than controls. Sensitivity levels tended to be low to moderate when maintaining specificities ≥ 90%, as customary in the field. While a previously developed scale that integrates eye movement measures using a logistic regression did not adequately differentiate the groups, conjunctive rules (i.e., the participant was required to fail both the WMT's classification scheme and the eye movement measure with the strongest discriminative capacity) were associated with higher specificities than those of the WMT’s conventional classification scheme. Overall, the eye-tracker integrated WMT shows initial clinical utility for detecting noncredible cognitive performance. Decreasing costs of eye trackers and enhanced usability will hopefully encourage further research of their utility for detecting noncredible cognitive performance and integration of this novel technology with other stand-alone validity indicators.

Forensic Prediction of Malingering from Performance Validity Tests: Review of Leonhard (2023, a, b, c)

Forensic Prediction of Malingering from Performance Validity Tests: Review of Leonhard (2023, a, b, c)

Performance validity testing in individuals with alcohol-related cognitive impairments.

Performance validity testing in individuals with alcohol-related cognitive impairments.

Clinical validation of a blood-based liquid biopsy test integrating cell-free DNA quantification and next-generation sequencing for cancer screening in dogs.

To validate the performance of a novel, integrated test for canine cancer screening that combines cell-free DNA quantification with next-generation sequencing (NGS) analysis. Retrospective data from a total of 1,947 cancer-diagnosed and presumably cancer-free dogs were used to validate test performance for the detection of 7 predefined cancer types (lymphoma, hemangiosarcoma, osteosarcoma, leukemia, histiocytic sarcoma, primary lung tumors, and urothelial carcinoma), using independent training and testing sets. Cell-free DNA quantification data from all samples were analyzed using a proprietary machine learning algorithm to determine a Cancer Probability Index (High, Moderate, or Low). High and Low Probability of Cancer were final result classifications. Moderate cases were additionally analyzed by NGS to arrive at a final classification of High Probability of Cancer (Cancer Signal Detected) or Low Probability of Cancer (Cancer Signal Not Detected). Of the 595 dogs in the testing set, 89% (n = 530) received a High or Low Probability result based on the machine learning algorithm; 11% (65) were Moderate Probability, and NGS results were used to assign a final classification. Overall, 87 of 122 dogs with the 7 predefined cancer types were classified as High Probability and 467 of 473 presumably cancer-free dogs were classified as Low Probability, corresponding to a sensitivity of 71.3% for the predefined cancer types at a specificity of 98.7%. This integrated test offers a novel option to screen for cancer types that may be difficult to detect by physical examination at a dog's wellness visit.

Validity of the Groningen Effort Test in patients with suspected chronic solvent-induced encephalopathy.

The use of performance validity tests (PVTs) in a neuropsychological assessment to determine indications of invalid performance has been a common practice for over a decade. Most PVTs are memory-based; therefore, the Groningen Effort Test (GET), a non-memory-based PVT, has been developed. This study aimed to validate the GET in patients with suspected chronic solvent-induced encephalopathy (CSE) using the criterion standard of 2PVTs. A second goal was to determine diagnostic accuracy for GET. Sixty patients with suspected CSE referred for NPA were included. The GET was compared to the criterion standard of 2PVTs based on the Test of Memory Malingering and the Amsterdam Short Term Memory Test. The frequency of invalid performance using the GET was significantly higher compared to the criterion of 2PVTs (51.7% vs. 20.0% respectively; p < 0.001). For the GET index, the sensitivity was 75% and the specificity was 54%, with a Youden's Index of 27. The GET showed significantly more invalid performance compared to the 2PVTs criterion suggesting a high number of false positives. The general accepted minimum norm of specificity for PVTs of >90% was not met. Therefore, the GET is of limited use in clinical practice with suspected CSE patients.

Using the MMPI-2-RF, IOP-29, IOP-M, and FIT in the In-Person and Remote Administration Formats: A Simulation Study on Feigned mTBI.

Our study compared the impact of administering Symptom Validity Tests (SVTs) and Performance Validity Tests (PVTs) in in-person versus remote formats and assessed different approaches to combining validity test results. Using the MMPI-2-RF, IOP-29, IOP-M, and FIT, we assessed 164 adults, with half instructed to feign mild traumatic brain injury (mTBI) and half to respond honestly. Within each subgroup, half completed the tests in person, and the other half completed them online via videoconferencing. Results from 2 ×2 analyses of variance showed no significant effects of administration format on SVT and PVT scores. When comparing feigners to controls, the MMPI-2-RF RBS exhibited the largest effect size (d = 3.05) among all examined measures. Accordingly, we conducted a series of two-step hierarchical logistic regression models by entering the MMPI-2-RF RBS first, followed by each other SVT and PVT individually. We found that the IOP-29 and IOP-M were the only measures that yielded incremental validity beyond the effects of the MMPI-2-RF RBS in predicting group membership. Taken together, these findings suggest that administering these SVTs and PVTs in-person or remotely yields similar results, and the combination of MMPI and IOP indexes might be particularly effective in identifying feigned mTBI.

Measures of performance validity for Hispanic/Latin American adults: A PRISMA systematic review.

Use of performance validity tests (PVTs) to assess suboptimal performance has become a recommended feature in neuropsychological evaluations. However, most of the literature has examined the utility of PVTs in primarily non-Hispanic White, monolingual English-speaking samples. Given the continual growth of culturally and linguistically diverse (CALD) populations, especially Hispanic/Latin Americans (H/Ls), it is necessary to provide an updated review on the available literature involving the use of PVTs in H/L adults. We conducted a systematic review that examined the available literature surrounding use of performance validity measures in H/L adults. This review identified evidence for the utility of PVTs in assessing invalid performance in H/Ls and compiled psychometric properties related to these measures, specifically rates of specificity and sensitivity when available. We identified six standalone and four embedded performance validity measures that have been examined in H/Ls. The Test of Memory Malingering (TOMM) was identified as the PVT most studied in various H/L heritage groups. Across the studies reviewed, adjustments to cutoff scores are generally recommended for the majority of PVTs discussed. Within H/Ls, specificity rates tend to be bolstered when sociocultural factors, like education, are considered. Recommended cutoff scores across the different PVTs reviewed are not typically suitable for use in heritage groups of H/L adults, especially in groups that include adults with lower levels of education and literacy. Therefore, education may need to be considered when applying cutoffs to these populations.

Replicating the classification accuracy of the Verbal Paired Associates and Visual Reproduction recognition trials as embedded performance validity tests.

This study was designed to replicate previous research on the clinical utility of the Verbal Paired Associates (VPA) and Visual Reproduction (VR) subtests of the WMS-IV as embedded performance validity tests (PVTs) and perform a critical item (CR) analysis within the VPA recognition trial. Archival data were collected from a mixed clinical sample of 119 adults (MAge = 42.5, MEducation = 13.9). Classification accuracy was computed against psychometrically defined criterion groups based on the outcome of various free-standing and embedded PVTs. Age-corrected scaled scores ≤ 6 were specific (.89-.98) but had variable sensitivity (.36-.64). A VPA recognition cutoff of ≤ 34 produced a good combination of sensitivity (.46-.56) and specificity (.92-.93), as did a VR recognition cutoff of ≤ 4 (.48-.53 sensitivity at .86-.94 specificity). Critical item analysis expanded the VPA's sensitivity by 3.5%-7.0% and specificity by 5%-8%. Negative learning curves (declining output on subsequent encoding trials) were rare but highly specific (.99-1.00) to noncredible responding. Results largely support previous reports on the clinical utility of the VPA and VR as embedded PVTs. Sample-specific fluctuations in their classification accuracy warrant further research into the generalizability of the findings. Critical item analysis offers a cost-effective method for increasing confidence in the interpretation of the VPA recognition trial as a PVT. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

The Impact of Adverse Childhood Experiences on Symptom and Performance Validity Tests Among a Multiracial Sample Presenting for ADHD Evaluation.

Adverse childhood experiences (ACEs) are commonly reported in individuals presenting for attention-deficit hyperactivity disorder (ADHD) evaluation. Performance validity tests (PVTs) and symptom validity tests (SVTs) are essential to ADHD evaluations in young adults, but extant research suggests that those who report ACEs may be inaccurately classified as invalid on these measures. The current study aimed to assess the degree to which ACE exposure differentiated PVT and SVT performance and ADHD symptom reporting in a multi-racial sample of adults presenting for ADHD evaluation. This study included 170 adults referred for outpatient neuropsychological ADHD evaluation who completed the ACE Checklist and a neurocognitive battery that included multiple PVTs and SVTs. Analysis of variance was used to examine differences in PVT and SVT performance among those with high (≥4) and low (≤3) reported ACEs. Main effects of the ACE group were observed, such that high ACE group reporting demonstrated higher scores on SVTs assessing ADHD symptom over-reporting and infrequent psychiatric and somatic symptoms on the Minnesota Multiphasic Personality Inventory-2-Restructured Form. Conversely, no significant differences emerged in total PVT failures across ACE groups. Those with high ACE exposure were more likely to have higher scores on SVTs assessing over-reporting and infrequent responses. In contrast, ACE exposure did not affect PVT performance. Thus, ACE exposure should be considered specifically when evaluating SVT performance in the context of ADHD evaluations, and more work is needed to understand factors that contribute to different patterns of symptom reporting as a function of ACE exposure.

Use of perceptual memory as a performance validity indicator: initial validation with simulated mild traumatic brain injury

ABSTRACT Introduction Many commonly employed performance validity tests (PVTs) are several decades old and vulnerable to compromise, leading to a need for novel instruments. Because implicit/non-declarative memory may be robust to brain damage, tasks that rely upon such memory may serve as an effective PVT. Using a simulation design, this experiment evaluated whether novel tasks that rely upon perceptual memory hold promise as PVTs. Method Sixty healthy participants were provided instructions to simulate symptoms of mild traumatic brain injury (TBI), and they were compared to a group of 20 honest responding individuals. Simulator groups received varying levels of information concerning TBI symptoms, resulting in naïve, sophisticated, and test-coached groups. The Word Memory Test, Test of Memory Malingering, and California Verbal Learning Test-II Forced Choice Recognition Test were administered. To assess perceptual memory, selected images from the Gollin Incomplete Figures and Mooney Closure Test were presented as visual perception tasks. After brief delays, memory for the images was assessed. Results No group differences emerged on the perception trials of the Gollin and Mooney figures, but simulators remembered fewer images than the honest responders. Simulator groups differed on the standard PVTs, but they performed equivalently on the Gollin and Mooney figures, implying robustness to coaching. Relying upon a criterion of 90% specificity, the Gollin and Mooney figures achieved at least 90% sensitivity, comparing favorably to the standard PVTs. Conclusions The Gollin and Mooney figures hold promise as novel PVTs. As perceptual memory tests, they may be relatively robust to brain damage, but future research involving clinical samples is necessary to substantiate this assertion.

The use of an integrated framework combining eco-evolutionary data and species distribution models to predict range shifts of species under changing climates

Species distribution models (SDMs) are powerful tools that can predict potential distributions of species under climate change. However, traditional SDMs that rely on current species occurrences may underestimate their climatic tolerances and potential distributions. To address this limitation, we developed an integrated framework that incorporates eco-evolutionary data into SDMs. In our approach, the fundamental niches of species are constructed by their realized niches in different periods, and those fundamental niches are used to predict potential distributions of species. Our framework includes multiple phylogenetic analyses, such as niche evolution rate estimation and ancestral area reconstruction. These analyses provide deeper insights into the responses of species to climate change. We applied our approach to the Chrysanthemum zawadskii species complex to evaluate its efficacy through comprehensive performance evaluations and validation tests. Our framework can be applied broadly to species with available phylogenetic data and occurrence records, making it a valuable tool for understanding species adaptation in a rapidly changing world.•Integrating the niches of species in different periods estimates more complete climatic envelopes for them.•Combining eco-evolutionary data with SDMs predicts more comprehensive potential distributions of species under climate change.•Our framework provides a general procedure for species with phylogenetic data and occurrence records.

Cross validation of the response bias scale and the response bias scale-19 in active-duty personnel: use on the MMPI-2-RF and MMPI-3

ABSTRACT The Response Bias Scale (RBS) is the central measure of cognitive over-reporting in the MMPI-family of instruments. Relative to other clinical populations, the research evaluating the detection of over-reporting is more limited in Veteran and Active-Duty personnel, which has produced some psychometric variability across studies. Some have suggested that the original scale construction methods resulted in items which negatively impact classification accuracy and in response crafted an abbreviated version of the RBS (RBS-19; Ratcliffe et al., 2022; Spencer et al., 2022). In addition, the most recent edition of the MMPI is based on new normative data, which impacts the ability to use existing literature to determine effective cut-scores for the RBS (despite all items having been retained across MMPI versions). To date, no published research exists for the MMPI-3 RBS. The current study examined the utility of the RBS and the RBS-19 in a sample of Active-Duty personnel (n = 186) referred for neuropsychological evaluation. Using performance validity tests as the study criterion, we found that the RBS-19 was generally equitably to RBS in classification. Correlations with other MMPI-2-RF over- and under-reporting symptom validity tests were slightly stronger for RBS-19. Implications and directions for research and practice with RBS/RBS-19 are discussed, along with implications for neuropsychological assessment and response validity theory.

Comparing embedded performance validity indicators within the WAIS-IV Letter-Number sequencing subtest to Reliable Digit Span among adults referred for evaluation of attention-deficit/hyperactivity disorder

Objectives: This study investigated the Wechsler Adult Intelligence Scale-Fourth Edition Letter-Number Sequencing (LNS) subtest as an embedded performance validity indicator among adults undergoing an attention-deficit/hyperactivity disorder (ADHD) evaluation, and its potential incremental value over Reliable Digit Span (RDS). Method: This cross-sectional study comprised 543 adults who underwent neuropsychological evaluation for ADHD. Patients were divided into valid (n = 480) and invalid (n = 63) groups based on multiple criterion performance validity tests. Results: LNS total raw scores, age-corrected scaled scores, and age- and education-corrected T-scores demonstrated excellent classification accuracy (area under the curve of .84, .83, and .82, respectively). The optimal cutoff for LNS raw score (≤16), age-corrected scaled score (≤7), and age- and education-corrected T-score (≤36) yielded .51 sensitivity and .94 specificity. Slightly lower sensitivity (.40) and higher specificity (.98) was associated with a more conservative T-score cutoff of ≤33. Multivariate models incorporating both LNS and RDS improved classification accuracy (area under the curve of .86), and LNS scores explained a significant but modest proportion of variance in validity status above and beyond RDS. Chaining LNS T-score of ≤33 with RDS cutoff of ≤7 increased sensitivity to .69 while maintaining ≥.90 specificity. Conclusions: Findings provide preliminary evidence for the criterion and construct validity of LNS as an embedded validity indicator in ADHD evaluations. Practitioners are encouraged to use LNS T-score cutoff of ≤33 or ≤36 to assess the validity of obtained test data. Employing either of these LNS cutoffs with RDS may enhance the detection of invalid performance.

Performance and symptom validity indicators among children undergoing cognitive surveillance following treatment for craniopharyngioma.

Performance validity tests (PVTs) and symptom validity tests (SVTs) are essential to neuropsychological evaluations, helping ensure findings reflect true abilities or concerns. It is unclear how PVTs and SVTs perform in children who received radiotherapy for brain tumors. Accordingly, we investigated the rate of noncredible performance on validity indicators as well as associations with fatigue and lower intellectual functioning. Embedded PVTs and SVTs were investigated in 98 patients with pediatric craniopharyngioma undergoing proton radiotherapy (PRT). The contribution of fatigue, sleepiness, and lower intellectual functioning to embedded PVT performance was examined. Further, we investigated PVTs and SVTs in relation to cognitive performance at pre-PRT baseline and change over time. SVTs on parent measures were not an area of concern. PVTs identified 0-31% of the cohort as demonstrating possible noncredible performance at baseline, with stable findings 1 year following PRT. Reliable digit span (RDS) noted the highest PVT failure rate; RDS has been criticized for false positives in pediatric populations, especially children with neurological impairment. Objective sleepiness was strongly associated with PVT failure, stressing need to consider arousal level when interpreting cognitive performance in children with craniopharyngioma. Lower intellectual functioning also needs to be considered when interpreting task engagement indices as it was strongly associated with PVT failure. Embedded PVTs should be used with caution in pediatric craniopharyngioma patients who have received PRT. Future research should investigate different cut-off scores and validity indicator combinations to best differentiate noncredible performance due to task engagement versus variable arousal and/or lower intellectual functioning.

Generating Mobile Virtual Tour Using UAV and 360 Degree Panorama for Geography-Environmental Learning in Higher Educationw

Nowadays, the limitations of media in geo-environmental learning present a significant challenge, as media plays an indispensable role in fostering students’ comprehensive understanding. Employing technology capable of delivering comprehensive landscape information is crucial for effective environmental education. Utilizing Unmanned Aerial Vehicles (UAVs) and 360-degree panoramic cameras represents a judicious technological choice, which can be seamlessly integrated into virtual tours. This research aims to develop a Mobile Virtual Tour (Movie-Tour) as a medium to support learning, especially for materials regarding environmental geography. The research and development (RnD) method is used in research with the ADDIE model (Analysis, Design, Development, Implementation and Evaluation). Data collection for the development of the product involved conducting a need assessment, performing validation tests and conducting trials with students. To gather field data for creating environmental geography materials for the Movie-Tour product, we utilized an Unmanned Aerial Vehicle (UAV) and a 360-degree stereoscopic camera. The results demonstrate that Movie- Tour is an educational medium capable of delivering an immersive learning experience and comprehensive materials, allowing for the virtual visualization of real-world conditions in the field. This capability enhances students’ engagement in exploring physical geography conditions, fosters independent knowledge acquisition, and nurtures their innate curiosity. Movie-Tour stands as a potent and practical educational tool, offering an effective and secure learning experience for students, eliminating the need for direct field visits.

Two validation studies of a performance validity test for autistic adults

In two studies we examined the potential of a simple emotion recognition task, the Morel Emotional Numbing Test (MENT), as a performance validity test (PVT) for autism-related cognitive difficulties in adulthood. The aim of a PVT is to indicate non-credible performance, which can aid the interpretation of psychological assessments. There are currently no validated PVTs for autism-related difficulties in adulthood. In Study 1, non-autistic university students (aged 18–46 years) were instructed to simulate that they were autistic during a psychological assessment (simulation condition; n = 26). These students made more errors on the MENT than those instructed to do their best (control condition; n = 26). In Study 2, we tested how well autistic adults performed on the MENT. We found that clinically diagnosed autistic adults and non-autistic adults (both n = 25; 27–57 years; IQ > 80) performed equally well on the MENT. Moreover, autistic adults made significantly fewer errors than the instructed simulators in Study 1. The MENT reached a specificity of ≥98% (identifying 100% of non-simulators as non-simulator in Study 1 and 98% in Study 2) and a sensitivity of 96% (identifying 96% of simulators as simulator). Together these findings provide the first empirical evidence for the validity of the MENT as a potential PVT for autism-related cognitive difficulties.

Single-cell transcriptome sequencing reveals aberrantly activated inter-tumor cell signaling pathways in the development of clear cell renal cell carcinoma

BackgroundAberrant intracellular or intercellular signaling pathways are important mechanisms that contribute to the development and progression of cancer. However, the intercellular communication associated with the development of ccRCC is currently unknown. The purpose of this study was to examine the aberrant tumor cell-to-cell communication signals during the development of ccRCC.MethodsWe conducted an analysis on the scRNA-seq data of 6 ccRCC and 6 normal kidney tissues. This analysis included sub clustering, CNV analysis, single-cell trajectory analysis, cell–cell communication analysis, and transcription factor analysis. Moreover, we performed validation tests on clinical samples using multiplex immunofluorescence.ResultsThis study identified eleven aberrantly activated intercellular signaling pathways in tumor clusters from ccRCC samples. Among these, two of the majors signaling molecules, MIF and SPP1, were mainly secreted by a subpopulation of cancer stem cells. This subpopulation demonstrated high expression levels of the cancer stem cell markers POU5F1 and CD44 (POU5F1hiCD44hiE.T), with the transcription factor POU5F1 regulating the expression of SPP1. Further research demonstrated that SPP1 binds to integrin receptors on the surface of target cells and promotes ccRCC development and progression by activating potential signaling mechanisms such as ILK and JAK/STAT.ConclusionAberrantly activated tumor intercellular signaling pathways promote the development and progression of ccRCC. The cancer stem cell subpopulation (POU5F1hiCD44hiE.T) promotes malignant transformation and the development of a malignant phenotype by releasing aberrant signaling molecules and interacting with other tumor cells.

The relationship between performance validity testing, external incentives, and cognitive functioning in long COVID

ABSTRACT Introduction Performance validity test (PVT) failures occur in clinical practice and at higher rates with external incentives. However, little PVT research has been applied to the Long COVID population. This study aims to address this gap. Methods Participants were 247 consecutive individuals with Long COVID seen for neuropsychological evaluation who completed 4 PVTs and a standardized neuropsychological battery. The sample was 84.2% White and 66% female. The mean age was 51.16 years and mean education was 14.75 years. Medical records were searched for external incentive (e.g., disability claims). Three groups were created based on PVT failures (Pass [no failures], Intermediate [1 failure], and Fail [2+ failures]). Results A total of 8.9% participants failed 2+ PVTs, 6.4% failed one PVT, and 85% passed PVTs. From the full sample, 25.1% were identified with external incentive. However, there was a significant difference between the rates of external incentives in the Fail group (54.5%) compared to the Pass (22.1%) and Intermediate (20%) groups. Further, the Fail group had lower cognitive scores and higher frequency of impaired range scores, consistent with PVT research in other populations. External incentives were uncorrelated with cognitive performance. Conclusions Consistent with other populations, results suggest Long COVID cases are not immune to PVT failure and external incentives are associated with PVT failure. Results indicated that individuals in the Pass and Intermediate groups showed no evidence for significant cognitive deficits, but the Fail group had significantly poorer cognitive performance. Thus, PVTs should be routinely administered in Long COVID cases and research.

The impact of race and other demographic factors on the false positive rates of five embedded Performance Validity Tests (PVTs) in a Veteran sample

ABSTRACT Introduction It is common to use normative adjustments based on race to maintain accuracy when interpreting cognitive test results during neuropsychological assessment. However, embedded performance validity tests (PVTs) do not adjust for these racial differences and may result in elevated rates of false positives in African American/Black (AA) samples compared to European American/White (EA) samples. Methods Veterans without Major Neurocognitive Disorder completed an outpatient neuropsychological assessment and were deemed to be performing in a valid manner (e.g., passing both the Test of Memory Malingering Trial 1 (TOMM1) and the Medical Symptom Validity Test (MSVT), (n = 531, EA = 473, AA = 58). Five embedded PVTs were administered to all patients: WAIS-III/IV Processing Speed Index (PSI), Brief Visuospatial Memory Test-Revised: Discrimination Index (BVMT-R), TMT-A (secs), California Verbal Learning Test-II (CVLT-II) Forced Choice, and WAIS-III/IV Digit Span Scaled Score. Individual PVT false positive rates, as well as the rate of failing two or more embedded PVTs, were calculated. Results Failure rates of two embedded PVTs (PSI, TMT-A), and the total number of PVTs failed, were higher in the AA sample. The PSI and TMT-A remained significantly impacted by race after accounting for age, education, sex, and presence of Mild Neurocognitive Disorder. There were PVT failure rates greater than 10% (and considered false positives) in both groups (AA: PSI, TMT-A, and BVMT-R, 12–24%; EA: BVMT-R, 17%). Failing 2 or more PVTs (AA = 9%, EA = 4%) was impacted by education and Mild Neurocognitive Disorder but not by race. Conclusions Individual (timed) PVTs showed higher false positive rates in the AA sample even after accounting for demographic factors and diagnosis of Mild Neurocognitive Disorder. Requiring failure on 2 or more embedded PVTs reduced false positive rates to acceptable levels across both groups (10% or less) and was not significantly influenced by race.

Lowered cutoffs to reduce false positives on the Word Memory Test

ABSTRACT Objective To adjust the decision criterion for the Word Memory Test (WMT, Green, 2003) to minimize the frequency of false positives. Method Archival data were combined into a database (n = 3,210) to examine the best cut score for the WMT. We compared results based on the original scoring rules and those based on adjusted scoring rules using a criterion based on 16 performance validity tests (PVTs) exclusive of the WMT. Cutoffs based on peer-reviewed publications and test manuals were used. The resulting PVT composite was considered the best estimate of validity status. We focused on a specificity of .90 with a false-positive rate of less than .10 across multiple samples. Results Each examinee was administered the WMT, as well as on average 5.5 (SD = 2.5) other PVTs. Based on the original scoring rules of the WMT, 31.8% of examinees failed. Using a single failure on the criterion PVT (C-PVT), the base rate of failure was 45.9%. When requiring two or more failures on the C-PVT, the failure rate dropped to 22.8%. Applying a contingency analysis (i.e., X2) to the two failures model on the C-PVT measure and using the original rules for the WMT resulted in only 65.3% agreement. However, using our adjusted rules for the WMT, which consisted of relying on only the IR and DR WMT subtest scores with a cutoff of 77.5%, agreement between the adjusted and the C-PVT criterion equaled 80.8%, for an improvement of 12.1% identified. The adjustmeny resulted in a 49.2% reduction in false positives while preserving a sensitivity of 53.6%. The specificity for the new rules was 88.8%, for a false positive rate of 11.2%. Conclusions Results supported lowering of the cut score for correct responding from 82.5% to 77.5% correct. We also recommend discontinuing the use of the Consistency subtest score in the determination of WMT failure.