77 The Shell Game Task: Pilot Data Using a Simulator-Design Study to Evaluate a Novel Attentional Performance Validity Test

Abstract

Objective:We developed the Shell Game Task (SGT) as a novel Performance Validity Test (PVT). While most PVTs use a forced-choice paradigm with “memory” as the primary domain being assessed, the SGT is a face-valid measure of attention and working memory. We explored the accuracy of the SGT to detect noncredible performance using a simulatordesign study.Participants and Methods:Ninety-four university students were randomly assigned to either best effort (CON) (n=49) or simulating traumatic brain injury (TBI) (SIM) (n=45) conditions. Participants completed a full battery of neuropsychological tests to simulate an actual evaluation, including the Test of Memory Malingering (TOMM) and the SGT. The SGT involves three cups and a red ball shown on the screen. Participants watch as the ball is placed under one of the three cups. Cups are then shuffled. Participants are asked to track the cup that contains the ball and correctly identify its location. We created two difficulty levels (easy vs hard, 20 trials each) by changing the number of times the cups were shuffled. Participants were given feedback (correct vs incorrect) after each trial. At the conclusion of the study, participants were asked about adherence to study directions they were given.Results:Participants with missing data (CON=1; SIM=2) or who reported non-adherence to study directions (CON=2; SIM=1) were removed from analyses. Twenty-five percent in SIM and 0% in CON failed TOMMTrial 2 (<45) suggesting adequate manipulation of groups. Groups were not different in age, gender, ethnicity, or education (all p’s>.05). There were 9 participants in each group with concussion/TBI history. TBI history was not significantly related to performance on the SGT in either group, although participants with TBI history tended to do better. Average performances on TOMM Trial 1 (36.62 vs 47.91, p<.001) and TOMM Trial 2 (37.50 vs 49.71, p<.001) were significantly lower in the SIM group. Performance on SGT was also significantly lower in the SIM group across SGT Total Correct (20.17 vs 24.65 of 40, p=.008), SGT Easy (10.60 vs 13.52 of 20, p=.002), and SGT Hard (9.57 vs 11.13 of 20, p=.068). Mixed ANOVA showed a trend towards significant group by SGT difficulty interaction (F(1,86)=3.41, p=.052, np2=.043). There was steeper decline in performance on SGT Hard compared to SGT Easy for CON. ROC analyses suggested adequate but not ideal sensitivity/specificity: scores <8 on SGT Easy (sensitivity=26%; false positive=11%), <7 on SGT Hard (sensitivity=26%; false positive=7%), and <15 on SGT Total (sensitivity=24%; false positive=9%).Conclusions:These preliminary data indicate the SGT may be able to detect malingered TBI. However, additional development of this measure is necessary. Further refinement of difficulty level may improve sensitivity/specificity (e.g., CON mean performance for SGT Easy trails was 13.52, suggesting the items may be too difficult). This study was limited to an online administration due to COVID, which could have affected results; future studies should test inperson administration of the SGT. In addition, performance in clinical control groups (larger samples of individuals with mild TBI, ADHD) should be tested to better determine specificity for these preliminary cutoffs.