High Interrater Variability in Intraoperative Language Testing and Interpretation in Awake Brain Mapping Among Neurosurgeons or Neuropsychologists: An Emerging Need for Standardization

Abstract

Brain mapping with direct electric stimulation is considered the gold standard for maximum safe resection of tumors affecting eloquent regions. However, no consensus exists in selection and interpretation of intraoperative testing for language and other cognitive domains. Our aim was to capture and statistically analyze variability in practices in intraoperative language testing among neurosurgeons and neuropsychologists in the United States, Europe, and the rest of the world. An electronic questionnaire was developed by a multidisciplinary team at Queen Square, London, and distributed internationally through selected organized societies. The survey included 2 domains: terminology and common understanding of clinical deficits; and selection of intraoperative tests used per specific brain region. Participants were stratified by specialty, years of experience, and monthly caseload. Data were analyzed using Krippendorff α, Wilcoxon rank sum test, and Kruskal-Wallis analysis of variance. A total of 137 specialists participated. A low agreement was recorded for each of the 20 questions (Krippendorff α=-0.023 to 0.312). Further subgroup analysis revealed low interrater reliability independent of specialism (neurosurgeons, α= 0.013-0.318 compared with nonneurosurgeons, α=-0.021 to 0.398; P= 0.808) and years of experience (<1 years, α=-0.003 to 0.282; 2-5 years, α= 0.009-0.327; 6-10 years, α= 0.003-0.234; and >10 years, α=-0.003 to 0.372; P= 0.200). The current study documents high interrater variability, regardless of specialism and years of experience in the cohort of neurosurgeons and language specialists surveyed and may be applicable to a wider group of specialists, indicating the need to reduce interobserver, interinstitutional and interspecialty variability, reach consensus, and increase the validity, interpretation, and predictive power of intraoperative mapping.