Performance Validity Tests Research Articles

B - 136 A Validity Study of an Embedded Performance Validity Measure of Wide Range Assessment of Memory and Learning, Third Edition

Abstract Objective This study investigated chance levels of responding on the Recognition Memory component of the performance validity test 2 (PVT2) measure embedded within the Wide Range Assessment of Memory and Learning, Third Edition (WRAML3; Adams & Sheslow, 2021). While standardized norms are provided, the WRAML3 has yet to examine PVT2 scores obtained through random responding. Given the impact of suboptimal effort and performance on test scores, providing scores obtained through guesstimation can help differentiate low scores due to impairment from chance responding. Methods Participants included 50 volunteer undergraduates and graduate students. To assess levels of random responding on PVT2 items, participants responded to the four Recognition memory subtests of the WRAML3 without the usual prior exposure to immediate and delayed subtest content. One-sample t-tests were conducted to compare participant performance scores to statistical chance levels of responding and the established PVT2 cut-off scores. Results Analyses revealed that the mean total PVT2 score was significantly lower than the established WRAML3 PVT2 cut-off score of 17 (t(49) = −26.88, p < 0.001). Regarding chance levels of responding, participants’ random responses were significantly different from the hypothesized chance mean of 5 (t(49) = 6.24, p < 0.001). Conclusions Findings support the overall utility of the available WRAML-3 embedded PVT2 cutoff scores while indicating individuals who rely on guesstimation techniques may be able to answer better than chance. These results establish a new standard of random response on the WRAML3, which may be utilized to determine whether scores below the PVT2 cutoff are invalid or may warrant further investigation.

Visual tests, touch responses: Computer-based neuropsychological tools

Objective: Investigate three new computer tests of visual abilities relative to traditional paper-and-pencil (P&P) tests on groups with and without cerebral neurological impairment (CNI, Non-CNI) based on MRI and EEG criteria. The new tests employ an intuitive interface with audio instructions, touch responses, non-response prompts, and videography of test behavior. The Presidents Test was designed as an achievement-related measure of visual knowledge; the Railroad Test – visual perception and delayed recognition; and the Swamp Test – visual attention. Attitudes toward testing were assessed with an original Testing Experience Questionnaire (TEQ). Method: Of the 129 participants, 84 were women, 73 identified as non-White, average age 45.5 and education 14.3 years. Procedures included the new computer tests and a selection of standard neuropsychological measures including performance validity tests (PVT). Participants who failed two or three PVTs or had missing PVT data were excluded from main analyses, resulting in N = 115. Results: The new computer tests demonstrated adequate reliability. Correlations and factor analyses confirmed the computer tests as functioning in accordance with design. The Presidents Test was associated with academic achievement. The Railroad and Swamp Tests were linked to visual perception and visual attention. Correlations between computer total test duration time and traditional speed of processing tasks were modest. Computer and traditional tests demonstrated similar discriminability between CNI and Non-CNI groups. TEQ indicated positive attitudes toward testing in general, and computer testing in particular. Conclusions: The new computer tests evaluated in this study were found to be reliable, functioned to assess the designed cognitive domains, and discriminated between CNI and Non-CNI participants similarly to the traditional neuropsychological measures. Attitudes toward computer testing were favorable.

Performance validity tests with low verbal mediation can be robust in culturally diverse mixed psychiatric populations.

Performance validity tests with low verbal mediation can be robust in culturally diverse mixed psychiatric populations.

Valid, invalid, or somewhere in between? Baseline Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT) and stand-alone performance validity testing in collegiate athletes.

Valid, invalid, or somewhere in between? Baseline Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT) and stand-alone performance validity testing in collegiate athletes.

Performance validity testing: the need for digital technology and where to go from here.

Performance validity testing: the need for digital technology and where to go from here.

Performance validity in a presurgical epilepsy population

Objective: This study examined the performance validity test (PVT) pass/fail rate in a sample of presurgical epilepsy candidates; assessed whether performance validity was associated with reduced performance across cognitive domains; investigated the relationship between performance validity and self-report mood questionnaires; and assessed whether PVT performance was associated with demographic or clinical factors (i.e. sex, race/ethnicity, age, years of education, reported history of special education, seizure longevity, and number of anti-seizure medications). Methods: One hundred and eighty-three presurgical epilepsy candidates were examined. Each patient’s assessment battery included a stand-alone performance validity measure and two embedded validity measures. Results: PVT failure rate in this sample (10%) was associated with reduced performance on all neurocognitive measures: Full Scale IQ (FSIQ; r = −0.26), CVLT-II Total Learning (r = −0.36) and Long Delay Free Recall (LDFR; r = −0.38), BVMT-R Delayed Recall (r = −0.28), and Wisconsin Card Sorting Test (Categories Completed; r = −0.32). In addition, PVT failure rate was associated with elevated scores on the Beck Anxiety Inventory (r = .22) but not on the Beck Depression Inventory (BDI-II; r = .14). Correlations that were significant at the α = 0.05 level maintained significance following post hoc Bonferroni correction. The valid and invalid groups did not differ significantly in sex, race/ethnicity, age, years of education, reported history of special education, seizure longevity, and number of anti-seizure medications. Conclusions: Results from this study suggest that PVT performance was not impacted by demographic or clinical factors and therefore may be a reliable indicator of performance validity in a presurgical epilepsy sample.

A preliminary investigation of the utility of the Word Memory Test Immediate Recognition trial as a screener for noncredible performance

The assessment of performance validity is an important consideration to the interpretation of neuropsychological data. However, commonly used performance validity tests such as the Test of Memory Malingering (TOMM) and Word Memory Test (WMT) have lengthy administration times (20–30 minutes). Alternatively, utilizing a screener of performance validity (e.g., the TOMM T1 or TOMMe10) has proven to be an effective method of assessing performance validity while conserving time. The present study investigates the use of the WMT Immediate Recognition (IR) Trial scores as a screening measure for performance validity using an archival mTBI polytrauma sample (n = 48). Results show that the WMT IR demonstrates a high degree of accuracy in predicting WMT Delayed Recognition (DR) Trial performance across a range of base rates suggesting that the WMT IR is a useful screening measure for noncredible performance. Clinical implications and selection of optimal cutoff are discussed.

Design and Implementation of a Photonic-Based Electronic Warfare System

Nowadays, the Radio frequency (RF) Photonics become an inevitable candidate to address several military related potential applications including electronic warfare (EW), photonic signal processing (PSP), photonic-based RF transportation, and photonic communications. As part of this emerging technology development/requirement, we designed a photonic-based EW systems (PEWS) in the optisystem environment (later implemented using optoelectronic components) to extract various radar waveforms such as continuous wave (CW), pulsed wave, and frequency/phase modulated wave. All these radar waveforms are transmitted and captured by the proposed/implemented PEW system, and then processed to construct the radar signatures from its electromagnetic (EM) spectrum/signals. The key parameters of various waveforms generated and processed in our research work are varied, over the time, during the performance validation tests. The values of key parameters of the waveforms: RF CW signal frequencies, pulse repetition times (PRTs), pulse-widths, Barker code phase modulations, Frank code poly-phase modulations, sweep frequencies, and RF power levels; are 100 Hz through 8 GHz, 10 ms through 2 µs, 750 ns through 2 ns, 0o /80o, 0o /90o /180o /270o, and -84 dBm through 30 dBm, respectively. The details of PEWS design approaches, their implementation methodologies, and different performance validation experimental results are reported, and analyzed. The limitations, and possible immediate research contributions/requirements are also listed.

Base rates of failure on various performance validity tests as a function of age in adults referred for neuropsychological assessment.

This study aimed to investigate the relationship between age and base rates of failure (BRFail) on various performance validity tests (PVTs) administered in medical-legal settings. Archival data were analyzed from 3,297 adults (Mage = 42.3 years; Meducation = 11.2) referred for psychological or neuropsychological assessments in a medical-legal or forensic civil disability context who passed the Word Memory Test. BRFail on 10 PVTs (three freestanding and seven embedded) were reported at multiple cutoffs across five age groups ranging from 16 years to 69 years. BRFail increased with age on most embedded PVTs, with a couple of notable exceptions. Reliable Digit Span was unrelated to age at ≤6 but produced elevated BRFail among older examinees at ≤7. Within freestanding PVTs, a positive relationship emerged between age and BRFail on most instruments/cutoffs. Older age is associated with an increased risk of false positive errors on many embedded PVTs that rely on raw scores. Although freestanding PVTs tend to be more resistant to the effects of age, several commonly used cutoffs may still produce increased false positive rates in older examinees. Taken together, results suggest that PVT scores should be interpreted in the context of patient characteristics, in an evidence-based manner, rather than by rigidly applying omnibus cutoffs. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Multivariable utility of the Memory Integrated Language and Making Change Test

Recent reports indicate that the Memory Integrated Language Test (MIL) and Making Change Test Abbreviated Index (MCT-AI), two web-based performance validity tests (PVTs), have good sensitivity and specificity when used independently. This study investigated whether using these PVTs together could improve the detection of invalid performance in a mixed neuropsychiatric sample. Participants were 129 adult outpatients who underwent a neuropsychological evaluation and were classified into valid (n = 104) or invalid (n = 25) performance groups based on several commonly used PVTs. Using cut scores of ≤41 on the MIL and ≥1.05 on the MCT-AI together enhanced classification accuracy, yielding an area under the curve of .84 (95% CI: .75, .93). As compared to using the MIL and MCT-AI independently, the combined use increased the sensitivity from .10-.31 to.70 while maintaining ≥.90 specificity. Findings also indicated that failing either the MIL or MCT-AI was associated with somewhat lower cognitive test scores, but failing both was associated with markedly lower scores. Overall, using the MIL and MCT-AI together may be an effective way to identify invalid test performance during a neuropsychological evaluation. Furthermore, pairing these tests is consistent with current practice guidelines to include multiple PVTs in a neuropsychological test battery.

A CNN- and Self-Attention-Based Maize Growth Stage Recognition Method and Platform from UAV Orthophoto Images

The accurate recognition of maize growth stages is crucial for effective farmland management strategies. In order to overcome the difficulty of quickly obtaining precise information about maize growth stage in complex farmland scenarios, this study proposes a Maize Hybrid Vision Transformer (MaizeHT) that combines a convolutional algorithmic structure with self-attention for maize growth stage recognition. The MaizeHT model utilizes a ResNet34 convolutional neural network to extract image features to self-attention, which are then transformed into sequence vectors (tokens) using Patch Embedding. It simultaneously inserts category information and location information as a token. A Transformer architecture with multi-head self-attention is employed to extract token features and predict maize growth stage categories using a linear layer. In addition, the MaizeHT model is standardized and encapsulated, and a prototype platform for intelligent maize growth stage recognition is developed for deployment on a website. Finally, the performance validation test of MaizeHT was carried out. To be specific, MaizeHT has an accuracy of 97.71% when the input image resolution is 224 × 224 and 98.71% when the input image resolution is 512 × 512 on the self-built dataset, the number of parameters is 15.446 M, and the floating-point operations are 4.148 G. The proposed maize growth stage recognition method could provide computational support for maize farm intelligence.

Providing a brief corrective statement does not improve test performance in patients invalidating testing: A multisite, single-blind randomized controlled trial

Objective: Performance below the actual abilities of the examinee can be measured using performance validity tests (PVTs). PVT failure negatively impacts the quality of the neuropsychological assessment. In our study, we addressed this issue by providing a brief corrective statement regarding invalidity to improve test-taking behavior. Methods: This study is a multisite single-blind randomized controlled trial in a consecutive sample of clinically referred adult patients (N = 196) in a general hospital setting. Patients who failed a PVT (n = 71) were randomly allocated to a corrective statement approach (CS; n = 39), in which a brief verbal corrective statement was given by the technician, or received no corrective statement upon indications of invalid performance (NO-CS; n = 32). Both groups (CS and NO-CS) were provided with the same subsequently repeated and newly administered tests. Results: There were no group (CS vs. NO-CS) differences on both the repeated and single-administered PVTs and standard cognitive tests. Furthermore, invalid performing participants benefited significantly less from the repeated test administration compared to the valid performing group. Conclusions: Our study found that a brief corrective within-session statement, to address PVT failure and improve test-taking behavior, did not improve consequent test performance. These results suggest limited value of a brief verbal corrective statement to influence performance below best of capabilities. It highlights the need for more research to identify more effective approaches that can enhance patients test-taking behavior. Ultimately, such efforts are critical in ensuring accurate diagnosis and effective treatment recommendations for patients.

Does generative artificial intelligence pose a risk to performance validity test security?

Objective We examined the performance validity test (PVT) security risk presented by artificial intelligence (AI) chatbots asking questions about neuropsychological evaluation and PVTs on two popular generative AI sites. Method In 2023 and 2024, multiple questions were posed to ChatGPT-3 and Bard (now Gemini). One set started generally and refined follow-up questions based on AI responses. A second set asked how to feign, fake, or cheat. Responses were aggregated and independently rated for inaccuracy and threat. Responses not identified as inaccurate were assigned a four-level threat rating (no, mild, moderate, or high threat). Combined inaccuracy and threat ratings were examined cross-sectionally and longitudinally. Results Combined inaccuracy rating percentages were 35 to 42% in 2023 and 16 to 28% in 2024. Combined moderate/high threat ratings were observed in 24 to 41% of responses in 2023 and in 17 to 31% of responses in 2024. More ChatGPT-3 responses were rated moderate or high threat compared to Bard/Gemini responses. Over time, ChatGPT-3 responses became more accurate with a similar threat level, but Bard/Gemini responses did not change in accuracy or threat. Responses to how to feign queries demonstrated ethical opposition to feigning. Responses to similar queries in 2024 showed even stronger ethical opposition. Conclusions AI chatbots are a threat to PVT test security. A proportion of responses were rated as moderate or high threat. Although ethical opposition to feigning guidance increased over time, the natural language interface and the volume of AI chatbot responses represent a potentially greater threat than traditional search engines.

Does neuropsychological intraindividual variability index cognitive dysfunction, an invalid presentation, or both? Preliminary findings from a mixed clinical older adult veteran sample

ABSTRACT Introduction Intraindividual variability across a battery of neuropsychological tests (IIV-dispersion) can reflect normal variation in scores or arise from cognitive impairment. An alternate interpretation is IIV-dispersion reflects reduced engagement/invalid test data, although extant research addressing this interpretation is significantly limited. Method We used a sample of 97 older adult (mean age: 69.92), predominantly White (57%) or Black/African American (34%), and predominantly cis-gender male (87%) veterans. Examinees completed a comprehensive neuropsychological battery, including measures of reduced engagement/invalid test data (a symptom validity test [SVT], multiple performance validity tests [PVTs]), as part of a clinical evaluation. IIV-dispersion was indexed using the coefficient of variance (CoV). We tested 1) the relationships of raw scores and “failures” on SVT/PVTs with IIV-dispersion, 2) the relationship between IIV-dispersion and validity/neurocognitive disorder status, and 3) whether IIV-dispersion discriminated the validity/neurocognitive disorder groups using receiver operating characteristic (ROC) curves. Results IIV-dispersion was significantly and independently associated with a selection of PVTs, with small to very large effect sizes. Participants with invalid profiles and cognitively impaired participants with valid profiles exhibited medium to large (d = .55–1.09) elevations in IIV-dispersion compared to cognitively unimpaired participants with valid profiles. A non-significant but small to medium (d = .35–.60) elevation in IIV-dispersion was observed for participants with invalid profiles compared to those with a neurocognitive disorder. IIV-dispersion was largely accurate at differentiating participants without a neurocognitive disorder from invalid participants and those with a neurocognitive disorder (areas under the Curve [AUCs]=.69–.83), while accuracy was low for differentiating invalid participants from those with a neurocognitive disorder (AUCs=.58–.65). Conclusions These preliminary data suggest IIV-dispersion may be sensitive to both neurocognitive disorders and compromised engagement. Clinicians and researchers should exercise due diligence and consider test validity (e.g. PVTs, behavioral signs of engagement) as an alternate explanation prior to interpretation of intraindividual variability as an indicator of cognitive impairment.

Lithium-Ion Battery Health State Estimation Based on Feature Reconstruction and Optimized Least Squares Support Vector Machine

Abstract The state of health (SOH) of a battery is the main indicator of battery life. In order to improve the SOH estimation accuracy, a model framework for lithium-ion battery health state estimation with feature reconstruction and improved least squares support vector machine is proposed. First, the indirect health features (HF) are obtained by processing multiple health features extracted from the charging and discharging phases through principal component analysis to remove the information redundancy among multiple features. Subsequently, multiple smooth component subsequences of different frequencies are obtained by using variational modal decomposition to efficiently capture the overall downtrend and regeneration fluctuations of the data. Then, use the sparrow search algorithm to optimize the least squares support vector machine to build an estimation model, predict and superimpose the reconstructed fusion features of multiple feature subsequences. Finally, use the mapping relationship between the reconstructed HF and the SOH for the estimation. The NASA battery dataset and the University of Maryland battery dataset (CACLE) are used to perform validation tests on multiple batteries with different cycle intervals. The results show that the mean absolute error and root mean square error are less than 1% and the method has high-estimation accuracy and robustness.

The classification accuracy of Warrington’s recognition memory test (words) as a performance validity Test in a neurorehabilitation setting

This study was designed to evaluate the classification accuracy of the Warrington’s Recognition Memory Test (RMT) in 167 patients (97 or 58.1% men; M Age = 40.4; M Education= 13.8) medically referred for neuropsychological evaluation against five psychometrically defined criterion groups. At the optimal cutoff (≤42), the RMT produced an acceptable combination of sensitivity (.36-.60) and specificity (.85-.95), correctly classifying 68.4-83.3% of the sample. Making the cutoff more conservative (≤41) improved specificity (.88-.95) at the expense of sensitivity (.30-.60). Lowering the cutoff to ≤40 achieved uniformly high specificity (.91-.95) but diminished sensitivity (.27-.48). RMT scores were unrelated to lateral dominance, education, or gender. The RMT was sensitive to a three-way classification of performance validity (Pass/Borderline/Fail), further demonstrating its discriminant power. Despite a notable decline in research studies focused on its classification accuracy within the last decade, the RMT remains an effective free-standing PVT that is robust to demographic variables. Relatively low sensitivity is its main liability. Further research is needed on its cross-cultural validity (sensitivity to limited English proficiency).

An Examination of WAIS-IV Digit Span Performance Inconsistency as a Novel Embedded Performance Validity Test Among Adults Clinically Referred for Attention-Deficit/Hyperactivity Disorder

Embedded performance validity tests (PVTs), like Digit Span PVTs from Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV), offer a valuable means of evaluating validity without extending administration time. This study investigated the utility of novel indices of performance inconsistency for WAIS-IV Digit Span (DS IRs) in the detection of invalid performance among 705 adults referred for ADHD evaluation. Results showed DS IR indices were inadequate in classifying overall validity status (areas under the curve = 0.52–0.59). Predictably, four established Digit Span PVTs effectively distinguished between valid and invalid performance score groups (areas under the curve = 0.74–0.78) with 32–49% sensitivity and 86–93% specificity at optimal cut-scores. Overall, individuals with noncredible performance scores did not differ significantly from those with valid scores regarding performance inconsistency on WAIS-IV Digit Span.

The Impact of Increasing Material-Specific Verbal and Visual Memory Impairment Severity on Embedded Performance Validity Tests in the Rey Auditory Verbal Learning Test and Brief Visuospatial Memory Test-Revised

The Impact of Increasing Material-Specific Verbal and Visual Memory Impairment Severity on Embedded Performance Validity Tests in the Rey Auditory Verbal Learning Test and Brief Visuospatial Memory Test-Revised

Effect of processing speed and memory performance on classification accuracy of the dot counting test in a mixed neuropsychiatric sample

ABSTRACT Objective This study examined the impact of impairment in two specific cognitive abilities, processing speed and memory, on Dot Counting Test (DCT) classification accuracy by evaluating performance validity classification accuracy across cognitively unimpaired, single-domain impairment, and multidomain impairment subgroups within a mixed clinical sample. Method Cross-sectional data were analyzed from 348 adult outpatients classified as valid (n = 284) or invalid (n = 64) based on four independent criterion performance validity tests (PVTs). Unimpaired (n = 164), single-domain processing speed impairment (n = 24), single-domain memory impairment (n = 53), and multidomain processing speed and memory impairment (n = 43) clinical subgroups were established among the valid group. Both the traditional DCT E-score and unrounded E-score were examined. Results Overall, the DCT demonstrated acceptable to excellent classification accuracy across the unimpaired (area under the curve [AUC] traditional E-score=.855; unrounded E-score=.855) and single-domain impairment groups (traditional E-score AUCs = .690–.754; unrounded E-score AUCs = .692–747). However, it did not reliably discriminate the multidomain processing speed and memory impairment group from the invalid performers (traditional and unrounded E-scores AUC = .557). Conclusions Findings support the DCT as a non-memory-based freestanding PVT for use with single-domain cognitive impairment, with traditional E-score ≥17 (unrounded E-score ≥16.95) recommended for those with memory impairment and traditional E-score ≥19 (unrounded ≥18.08) with processing speed impairment. Moreover, results replicated previously established optimal cutoffs for unimpaired groups using both the traditional (≥14) and unrounded (≥13.84) E-scores. However, the DCT did not reliably discriminate between invalid performance and multidomain cognitive impairment, indicating caution is warranted when using the DCT with patients suspected of greater cognitive impairment.

The Making Change Test: Initial validation of a novel digitized performance validity test for tele-neuropsychology.

The Making Change Test (MCT) is a brief, digitized freestanding performance validity test (PVT) designed for tele-neuropsychology (TeleNP). The objective of this study was to report the initial validation of the MCT in a mixed neuropsychiatric sample referred for neuropsychological evaluation using a known-groups design. The sample consisted of 136 adult outpatients who underwent a neuropsychological evaluation. Patients were classified as valid (n=115) or invalid (n=21) based on several established PVTs. Two validity indicators were calculated and assessed, including an Accuracy Response-Score and an Abbreviated Index. The Accuracy Response-Score incorporated both response time and errors. The Abbreviated Index aggregated response time and errors across the most sensitive test items in terms of predicting performance validity status. Correlational analyses indicated that the MCT Accuracy Response-Score and Abbreviated Index were more similar to non-memory-based PVTs than memory-based PVTs. Both the MCT Accuracy Response-Score and Abbreviated Index indicated acceptable classification accuracy (area under the curve of .77). The optimal cut score for the MCT Accuracy Response-Score (≥24) yielded a sensitivity of .38 and specificity of .90. The optimal cut score associated with the Abbreviated Index yielded slightly better operating characteristics, with a sensitivity of .50 and specificity of .90. Initial findings provide support for the criterion and construct validity of the MCT and suggest a promising TeleNP future for this performance validity tool. However, additional support is necessary before the MCT can be used clinically.