Inter-Rater Reliability of EEG-Based Encephalopathy Grading (S30.008)

Abstract

<h3>Objective:</h3> To measure inter-rater reliability of experts in assessing encephalopathy severity using the VE-CAM-S grading system. <h3>Background:</h3> The VE-CAM-S (Visual EEG Confusion Assessment Method – Severity) scale quantifies encephalopathy severity based on EEG features. However, inter-rater reliability of experts using the scale has yet to be assessed. <h3>Design/Methods:</h3> We created an online test with thirty-two 15-second EEG samples. Each question asked users to indicate the presence/absence of each of 29 EEG features, 11 of which were used in the VE-CAM-S. Gold-standard was based on consensus of 3 authors (IS, FN, MBW). Ten experts from 6 institutions participated. We quantified performance by average spearman correlation of VE-CAM-S scores with the gold standard, and average sensitivity/specificity. We performed a qualitative analysis to identify errors in recognizing EEG features that most affected VE-CAM-S scores. <h3>Results:</h3> The average [95%CI] correlation between VE-CAM-S scores with the gold standard was 0.73 [0.59–0.86]. Specificity was very high (&gt;90%) for all but generalized delta (77%). Sensitivity was high (&gt;70%) for all but brief generalized attenuations (69%), generalized periodic discharges (67%), generalized theta (63%), BIRDS (57%), generalized alpha (57%), extreme delta brushes/EDB (50%), and generalized beta (50%). Probable reasons for errors were subtlety of some findings; confusing some findings (e.g., generalized beta vs. myogenic artifact, burst suppression vs. brief generalized attenuations); failure to correctly recognize BIRDs (mislabeled as focal IEDs) and EDB (mislabeled as GRDA). The largest errors occurred when experts missed or falsely identified features that carry higher weight in the VE-CAM-S scoring rubric. <h3>Conclusions:</h3> Expert agreement in VE-CAM-S scoring is high. Error analysis identified several ways to improve future versions, including breaking high-stakes features into smaller parts; creating a “cheat sheet” with scored examples to allow scorers to choose the closest match; and designing teaching materials to help scorers recognize subtle variations of high-stakes patterns. <b>Disclosure:</b> Ms. Zahoor has nothing to disclose. Ms. Banks has nothing to disclose. Mr. Tesh has nothing to disclose. Dr. Eckhardt has stock in Moderna. Dr. Sun has nothing to disclose. Dr. Greenblatt has nothing to disclose. Dr. Herlopian has nothing to disclose. Dr. Karakis has nothing to disclose. Dr. Katyal has nothing to disclose. Dr. Nayak has nothing to disclose. The institution of Dr. Ng has received personal compensation in the range of $500-$4,999 for serving as a Consultant for Eisai Canada. The institution of Dr. Ng has received personal compensation in the range of $500-$4,999 for serving as a Consultant for Paladin Canada. The institution of Dr. Ng has received personal compensation in the range of $500-$4,999 for serving on a Speakers Bureau for Eisai Canada. The institution of Dr. Ng has received personal compensation in the range of $500-$4,999 for serving on a Speakers Bureau for UCB Canada. Dr. Ng has received publishing royalties from a publication relating to health care. Dr. Williams has nothing to disclose. Dr. Sheikh has nothing to disclose. Dr. Nascimento has received personal compensation in the range of $500-$4,999 for serving as a Consultant for Zogenix. The institution of Dr. Nascimento has received personal compensation in the range of $500-$4,999 for serving on a Scientific Advisory or Data Safety Monitoring board for Biocodex. Dr. Nascimento has received personal compensation in the range of $500-$4,999 for serving as an Editor, Associate Editor, or Editorial Advisory Board Member for AAN. The institution of Dr. Nascimento has received research support from Zogenix. The institution of Dr. Nascimento has received research support from Biocodex. The institution of Dr. Nascimento has received research support from Greenwich. Dr. Nascimento has received publishing royalties from a publication relating to health care. Dr. Nascimento has a non-compensated relationship as a Editorial Team Member with Neurology RFS that is relevant to AAN interests or activities. Dr. Nascimento has a non-compensated relationship as a Production Team with Neurology Podcast that is relevant to AAN interests or activities. Dr. Westover has stock in Beacon Biosignals. The institution of Dr. Westover has received research support from NIH. Dr. Westover has received publishing royalties from a publication relating to health care. Dr. Westover has a non-compensated relationship as a cofounder with Beacon Biosignals that is relevant to AAN interests or activities.