Embedded Performance Validity Tests Research Articles

Further examination of embedded performance validity indicators for the Conners’ Continuous Performance Test and Brief Test of Attention in a large outpatient clinical sample

Objective: Assessing test performance validity is a standard clinical practice and although studies have examined the utility of cognitive/memory measures, few have examined attention measures as indicators of performance validity beyond the Reliable Digit Span. The current study further investigates the classification probability of embedded Performance Validity Tests (PVTs) within the Brief Test of Attention (BTA) and the Conners’ Continuous Performance Test (CPT-II), in a large clinical sample. Method: This was a retrospective study of 615 patients consecutively referred for comprehensive outpatient neuropsychological evaluation. Non-credible performance was defined two ways: failure on one or more PVTs and failure on two or more PVTs. Classification probability of the BTA and CPT-II into non-credible groups was assessed. Sensitivity, specificity, positive predictive value, and negative predictive value were derived to identify clinically relevant cut-off scores. Results: When using failure on two or more PVTs as the indicator for non-credible responding compared to failure on one or more PVTs, highest classification probability, or area under the curve (AUC), was achieved by the BTA (AUC = .87 vs. .79). CPT-II Omission, Commission, and Total Errors exhibited higher classification probability as well. Conclusion: Overall, these findings corroborate previous findings, extending them to a large clinical sample. BTA and CPT-II are useful embedded performance validity indicators within a clinical battery but should not be used in isolation without other performance validity indicators.

Embedded performance validity testing in neuropsychological assessment: Potential clinical tools

ABSTRACTThe article aims to suggest clinically-useful tools in neuropsychological assessment for efficient use of embedded measures of performance validity. To accomplish this, we integrated available validity-related and statistical research from the literature, consensus statements, and survey-based data from practicing neuropsychologists. We provide recommendations for use of 1) Cutoffs for embedded performance validity tests including Reliable Digit Span, California Verbal Learning Test (Second Edition) Forced Choice Recognition, Rey-Osterrieth Complex Figure Test Combination Score, Wisconsin Card Sorting Test Failure to Maintain Set, and the Finger Tapping Test; 2) Selecting number of performance validity measures to administer in an assessment; and 3) Hypothetical clinical decision-making models for use of performance validity testing in a neuropsychological assessment collectively considering behavior, patient reporting, and data indicating invalid or noncredible performance. Performance validity testing helps inform the clinician about an individual’s general approach to tasks: response to failure, task engagement and persistence, compliance with task demands. Data-driven clinical suggestions provide a resource to clinicians and to instigate conversation within the field to make more uniform, testable decisions to further the discussion, and guide future research in this area.

Embedded performance validity tests within the Hopkins Verbal Learning Test – Revised and the Brief Visuospatial Memory Test – Revised

Objective: Various research studies and neuropsychology practice organizations have reiterated the importance of developing embedded performance validity tests (PVTs) to detect potentially invalid neurocognitive test data. This study investigated whether measures within the Hopkins Verbal Learning Test – Revised (HVLT-R) and the Brief Visuospatial Memory Test – Revised (BVMT-R) could accurately classify individuals who fail two or more PVTs during routine clinical assessment.Method: The present sample of 109 United States military veterans (Mean age = 52.4, SD = 13.3), all consisted of clinically referred patients and received a battery of neuropsychological tests. Based on performance validity findings, veterans were assigned to valid (n = 86) or invalid (n = 23) groups. Of the 109 patients in the overall sample, 77 were administered the HLVT-R and 75 were administered the BVMT-R, which were examined for classification accuracy.Results: The HVLT-R Recognition Discrimination Index and the BVMT-R Retention Percentage showed good to adequate discrimination with an area under the curve of .78 and .70, respectively. The HVLT-R Recognition Discrimination Index showed sensitivity of .53 with specificity of .93. The BVMT-R Retention Percentage demonstrated sensitivity of .31 with specificity of .92.Conclusions: When used in conjunction with other PVTs, these new embedded PVTs may be effective in the detection of invalid test data, although they are not intended for use in patients with dementia.

CVLT-II Forced Choice Recognition Trial as an Embedded Validity Indicator: A Systematic Review of the Evidence.

The Forced Choice Recognition (FCR) trial of the California Verbal Learning Test, 2nd edition, was designed as an embedded performance validity test (PVT). To our knowledge, this is the first systematic review of classification accuracy against reference PVTs. Results from peer-reviewed studies with FCR data published since 2002 encompassing a variety of clinical, research, and forensic samples were summarized, including 37 studies with FCR failure rates (N=7575) and 17 with concordance rates with established PVTs (N=4432). All healthy controls scored >14 on FCR. On average, 16.9% of the entire sample scored ≤14, while 25.9% failed reference PVTs. Presence or absence of external incentives to appear impaired (as identified by researchers) resulted in different failure rates (13.6% vs. 3.5%), as did failing or passing reference PVTs (49.0% vs. 6.4%). FCR ≤14 produced an overall classification accuracy of 72%, demonstrating higher specificity (.93) than sensitivity (.50) to invalid performance. Failure rates increased with the severity of cognitive impairment. In the absence of serious neurocognitive disorder, FCR ≤14 is highly specific, but only moderately sensitive to invalid responding. Passing FCR does not rule out a non-credible presentation, but failing FCR rules it in with high accuracy. The heterogeneity in sample characteristics and reference PVTs, as well as the quality of the criterion measure across studies, is a major limitation of this review and the basic methodology of PVT research in general. (JINS, 2016, 22, 851-858).

False-Positive Error Rates for Reliable Digit Span and Auditory Verbal Learning Test Performance Validity Measures in Amnestic Mild Cognitive Impairment and Early Alzheimer Disease

The objective is to examine failure on three embedded performance validity tests [Reliable Digit Span (RDS), Auditory Verbal Learning Test (AVLT) logistic regression, and AVLT recognition memory] in early Alzheimer disease (AD; n = 178), amnestic mild cognitive impairment (MCI; n = 365), and cognitively intact age-matched controls (n = 206). Neuropsychological tests scores were obtained from subjects participating in the Alzheimer's Disease Neuroimaging Initiative (ADNI). RDS failure using a ≤7 RDS threshold was 60/178 (34%) for early AD, 52/365 (14%) for MCI, and 17/206 (8%) for controls. A ≤6 RDS criterion reduced this rate to 24/178 (13%) for early AD, 15/365 (4%) for MCI, and 7/206 (3%) for controls. AVLT logistic regression probability of ≥.76 yielded unacceptably high false-positive rates in both clinical groups [early AD = 149/178 (79%); MCI = 159/365 (44%)] but not cognitively intact controls (13/206, 6%). AVLT recognition criterion of ≤9/15 classified 125/178 (70%) of early AD, 155/365 (42%) of MCI, and 18/206 (9%) of control scores as invalid, which decreased to 66/178 (37%) for early AD, 46/365 (13%) for MCI, and 10/206 (5%) for controls when applying a ≤5/15 criterion. Despite high false-positive rates across individual measures and thresholds, combining RDS ≤ 6 and AVLT recognition ≤9/15 classified only 9/178 (5%) of early AD and 4/365 (1%) of MCI patients as invalid performers. Embedded validity cutoffs derived from mixed clinical groups produce unacceptably high false-positive rates in MCI and early AD. Combining embedded PVT indicators lowers the false-positive rate.

Clinical utility of embedded performance validity tests on the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) following mild traumatic brain injury

ABSTRACTThis study evaluated the clinical utility of two embedded performance validity tests (PVTs) developed for the Repeatable Battery for the Assessment of Neuropsychological Status: the Effort Index (EI) and the Effort Scale (ES) in mild traumatic brain injury (TBI) patients. Participants were 250 military service members (94.0% male; Age: M = 28.4, SD = 7.6) evaluated following mild TBI on average 7.4 months (SD = 15.6) post-injury. Participants were divided into two groups based on their performance on the Test of Memory Malingering: PVT-Pass, n = 193; PVT-Fail, n = 57. For the EI, recommended cut-offs for extremely probable, highly probable, and probable poor effort were established. A cut-off score of >3 resulted in low sensitivity (.14), high specificity (.99) and positive predictive power (.94), and moderate negative predictive power (.68) and is recommended for identifying highly probable poor effort. For both the EI and ES, cut-offs for probable poor effort were identified; however, classification accuracy was not much improved relative to simply using the sum of the List Recognition and Digit Span raw scores to classify poor effort. It is acknowledged that the use of a different criterion would likely have resulted in different findings. Nevertheless, findings support the use of the EI and the ES as a “red flag” for possible poor effort in mild TBI patients, but suggest that, in most cases, additional PVTs would be required to accurately rule poor effort in or out.

Evaluating the Medical Symptom Validity Test (MSVT) in a Sample of Veterans Between the Ages of 18 to 64

ABSTRACTThe purpose of the current study was to compare three potential profiles of the Medical Symptom Validity Test (MSVT; Pass, Genuine Memory Impairment Profile [GMIP], and Fail) on other freestanding and embedded performance validity tests (PVTs). Notably, a quantitatively computed version of the GMIP was utilized in this investigation. Data obtained from veterans referred for a neuropsychological evaluation in a metropolitan Veteran Affairs medical center were included (N = 494). Individuals age 65 and older were not included to exclude individuals with dementia from this investigation. The sample revealed 222 (45%) in the Pass group. Of the 272 who failed the easy subtests of the MSVT, 221 (81%) met quantitative criteria for the GMIP and 51 (19%) were classified as Fail. The Pass group failed fewer freestanding and embedded PVTs and obtained higher raw scores on all PVTs than both GMIP and Fail groups. The differences in performances of the GMIP and Fail groups were minimal. Specifically, GMIP protocols failed fewer freestanding PVTs than the Fail group; failure on embedded PVTs did not differ between GMIP and Fail. The MSVT GMIP incorporates the presence of clinical correlates of disability to assist with this distinction, but future research should consider performances on other freestanding measures of performance validity to differentiate cognitive impairment from invalidity.

Language-based embedded performance validity measures in traumatic brain injury

No studies to date have investigated the Boston Naming Test (BNT) as an embedded performance validity test (PVT). This study investigated the classification accuracy of the Boston Naming Test (BNT) and the Verbal Fluency Test (FAS and Animal Fluency), as embedded PVTs in a compensation-seeking mild traumatic brain injury (MTBI) sample (N = 57) compared to a non-compensation-seeking moderate-to-severe TBI (STBI) sample (N = 61). Participants in the MTBI sample who failed two or more PVTs were included, as were STBI participants who passed all PVTs. The classification accuracy of the individual tests and a logistically derived combined (LANGPVT) measure were studied. Results showed significant group differences (p < .05) on BNT, Animal Fluency, and LANGPVT between the MTBI and STBI groups. However, receiver operating characteristic (ROC) analyses indicated that only LANGPVT had acceptable classification accuracy (area under the curve > .70). Setting specificity at approximately .90, the recommended LANGPVT cutoff scores had sensitivity of .26. Results indicated that, similar to other embedded PVTs, these measures had low sensitivity when adequate specificity levels were maintained. However, extremely low scores on these measures are unlikely to occur in non-compensation-seeking, non-language-impaired, STBI cases.

Performance Validity Classification Accuracy of Single-, Pairwise-, and Triple-Failure Models Using the Halstead-Reitan Neuropsychological Battery for Adults

Embedded performance validity tests (PVTs) have been criticized for their poor specificity and sensitivity. Aggregated models of embedded PVTs have been proposed to improve their classification accuracy; however, limitations to aggregation-based improvement of PVTs have yet to be explored. The current study evaluated the classification accuracy of 3 types of models of embedded PVTs in the Halstead-Reitan Neuropsychological Battery for Adults (HRNB): a single-, a pairwise-, and a triple-failure model. In addition, this study evaluated the impact of aggregating between 1 and 6 embedded PVTs in each of these 3 types of models. Analyzing only the 2, 4, and 6 most discriminating embedded PVTs in the single-, pairwise-, and triple-failure models maximized classification accuracy, respectively. Comparisons across these models indicated that the single-failure model including only the two most discriminating embedded PVTs had the best classification accuracy; however, classification accuracy was only minimally improved in this model relative to analyzing just Reliable Digit Span. These results suggest that aggregation of embedded PVTs from the HRNB does not substantially improve their classification accuracy and that the benefits of aggregating PVTs may only emerge when the PVTs entered into the aggregated models have sufficient classification accuracy on their own.

Systemically Treated Breast Cancer Patients and Controls: An Evaluation of the Presence of Noncredible Performance

This study sought to define the frequency of noncredible performance in breast cancer patients before, during and after completion of systemic treatment, as well as predictors of noncredible performance. We examined six datasets investigating the cognitive effects of chemotherapy and/or endocrine therapy. Embedded performance validity test (PVT) measures were identified and used to evaluate the datasets. One dataset included a standalone PVT. Possible noncredible performance was operationally defined as performance below criterion on three or more PVT indices. This was undertaken as cancer patients have been observed clinically to fail PVTs both in the context of external gain and independent of such motivators. A total of 534 breast cancer patients and 214 healthy controls were included in the analysis. Percentages of patients performing below cutoff on one or more PVT varied from 0% to 21.2%. Only 1 patient met the criterion of noncredible performance. Calculation of post-test probabilities indicated a more than 90% chance to detect noncredible performance. There is no evidence to suggest noncredible performance in breast cancer patients and healthy controls who choose to participate in research studies examining cognitive function. Thus, the observational data showing that non-central nervous system (CNS) cancer and therapies not targeting the CNS can have untoward effects on cognitive function are unlikely to be due to noncredible performance.