Platform Session – Electromyography: Large inter-rater variation on revised El Escorial and Awaji diagnostic criteria for amyotrophic lateral sclerosis

Abstract

Introduction The revised El Escorial Criteria (rEEC) from 2000 and their amendment, the Awaji Criteria (AC) from 2008, are widely used diagnostic criteria for amyotrophic lateral sclerosis (ALS) and their sensitivity and specificity have been evaluated in several studies, however, reproducibility among different raters has not been studied. This study was undertaken to assess inter-rater agreement on rEEC and AC. Methods Eight experienced physicians from seven different countries independently classified 399 patients referred for ALS according to the rEEC and the AC. Clinical and electrophysiological data were presented in standardised forms. Agreement was evaluated by Kappa coefficients. Diagnostic sensitivity and specificity were calculated from the majority diagnosis of the eight physicians for 349 cases with follow-up data. Results Kappa coefficients differed among the diagnostic categories for rEEC (“Definite”: 0.50, CI:0.39–0.60, “Probable”: 0.36, CI:0.31–0.41, “Probable Laboratory Supported”: 0.25,CI:0.18–0.34, “Possible”: 0.14, CI:0.09–0.20, and “Not-ALS”: 0.59, CI:0.51–0.65) as well as for AC (“Definite”: 0.44, CI:0.37–0.51, “Probable”: 0.34, CI:0.29–0.39, “Possible”: 0.33, CI:0.26–0.40, and “Not-ALS”: 0.65, CI:0.58–0.72). The only significant difference in agreement between rEEC and AC was for the category “Possible”. A large fraction of cases classified as “Probable” using the rEEC were upgraded to “Definite” using the AC, however, many cases classified as “Probable Laboratory Supported” by rEEC were downgraded to “Possible” by the AC. Sensitivity for “Definite/Probable” versus “Possible/Not-ALS” did not differ between rEEC (0.64, CI:0.58–0.69) and AC (0.63, CI:0.58–0.69). Similarly, there was no difference in specificity between rEEC (0.91, CI:0.74–0.98) and AC (0.85, CI:0.67–0.94). Conclusion There is a large inter-rater variation on rEEC and AC most pronounced for the “Probable Laboratory Supported” and “Possible” categories. The sensitivity was rather low compared to other studies and was not significantly increased using AC, probably due to omission of the “Probable Laboratory Supported” category. However, in trials including patients with possible disease AC may be advantageous. Inter-rater variation may be due to (1) a high complexity of the rEEC, which is also inherited in the AC, including definition of four body regions, interpretation of clinical findings (upper/lower motor neuron signs), significance of rostral/caudal signs, and significance of EMG findings (acute/chronic signs), and (2) the need to apply rEEC together with AC. However, pre-study training and quality control at individual laboratories may reduce variation and improve sensitivity and specificity. The results indicate that there is a need for initiatives to develop more simple and reproducible criteria.