Large Scale Assessment of Consistency in Sleep Stage Scoring Rules Among Multiple Sleep Centers Using an Interpretable Machine Learning Algorithm

Abstract

Background: Polysomnography is the gold standard in identifying sleep stages; however, there are discrepancies in the standards used by technicians. Because organizing meetings to reach a consensus among multiple sleep centers is time consuming, we developed an artificial intelligence (AI) system to evaluate the reliability and consistency of sleep scoring. Methods: An interpretable machine learning algorithm was used to evaluate interrater reliability (IRR) among sleep centers. Intra-center and inter-center assessments were conducted on 679 subjects without sleep apnea in six sleep centers in Taiwan. IRR was estimated based on prediction outcomes. Findings: In the intra-center assessment, the median accuracy of the databases ranged from 78·8% to 81·9% with the exception of one hospital (designated E) with an accuracy of 72·5%. In the inter-center assessment, the median accuracy ranged from 74·4% to 79·9% when hospital E was excluded from testing and training. The performance of the proposed method was higher for N2, awake, and REM, compared to N1 and N3. There was a significant difference in the prediction models learned from hospital E and others. Interpretation: The proposed AI system proved highly effective in assessing IRR. Increasing the interrater agreement rate would lead to further improvements in the accuracy of the proposed sleep stage annotation system. Funding: This research was supported by grants from the Ministry of Science and Technology, Taiwan (MOST-109-2119-M-002-014), and the Chang Gung Medical Research Program (CMRPG3K0201). Declaration of Interest: The authors declare no competing interests. Ethical Approval: The study protocol was approved by the Institutional Review Board of each hospital (Chang Gung Memorial Hospital’s IRB No: 201800609B0; MacKay Memorial Hospital’s IRB No: 18MMHIS142e; Shuanh-Ho Hospital’s IRB No: N201911007, N201903142; Taipei Tzu Chi Hospital’s IRB No: 07-XD-083; Taichung Tzu Chi Hospital’s IRB No: REC107-37, and Taipei Veterans General Hospital’s IRB No: 2018-12-009AC).