Hidden Test Set Research Articles

A novel multi-scale 2D CNN with weighted focal loss for arrhythmias detection on varying-dimensional ECGs

Objective. The ECG is a standard diagnostic tool for identifying many arrhythmias. Accurate diagnosis and early intervention for arrhythmias are of great significance to the prevention and treatment of cardiovascular disease. Our objective is to develop an algorithm that can automatically identify 30 arrhythmias by using varying-dimensional ECG signals. Approach. In this paper, we firstly proposed a novel multi-scale 2D CNN that can effectively capture pathological information from small-scale to large-scale from ECG signals to identify 30 arrhythmias from 12-lead, 6-lead, 4-lead, 3-lead, and 2-lead ECGs. Secondly, we explored the effects of varying convolution kernels sizes and branch subnetworks on the model’s performance for each arrhythmia. Thirdly, we introduced the weighted focal loss to alleviate the positive-negative class imbalance problem in the multi-label arrhythmias classification. Fourthly, we explored the utility of reduced-lead ECGs in detecting arrhythmias by comparing the performances of models on varying-dimensional ECGs. Main results. As a follow-up entry after the PhysioNet/Computing in Cardiology Challenge (2021), our proposed approach achieved the official test scores of 0.52, 0.47, 0.53, 0.51, and 0.50 for the 12-lead, 6-lead, 4-lead, 3-lead, and 2-lead ECGs on the hidden test set (comparable to that of 6th, 11th, 4th, 5th, and 7th out of 39 teams in the Challenge). Significance. A multi-scale framework capable of detecting 30 arrhythmias from varying-dimensional ECGs was proposed in our work. We preliminarily verified that the multi-scale perception fields may be necessary to capture more comprehensive pathological information for arrhythmias detection. Besides, we also verified that the weighted focal loss may alleviate the positive–negative class imbalance and improve the model’s generalization performance on the cross-dataset. In addition, we observed that some reduced-lead models, such as the 4-lead and 3-lead models, can even achieve performance that is almost comparable to that of the 12-lead model. The excellent performance of our proposed framework demonstrates its great potential in detecting a wide range of arrhythmias.

Classification of multi-lead ECG with deep residual convolutional neural networks

Objective. Automatic electrocardiogram (ECG) interpretation based on deep learning methods is attracting increasing attention. In this study, we propose a novel method to accurately classify multi-lead ECGs using deep residual neural networks. Approach. ECG recordings from seven different open databases were provided by PhysioNet/Computing in Cardiology Challenge 2021. All the ECGs were pre-processed to obtain the same sampling rate. The label inconsistency among the databases was corrected by adding or removing specific labels. A label mask was created to filter out potentially incorrectly labelled data. Five models based on deep residual convolutional neural networks were optimized using an asymmetric loss function to classify multi-lead ECGs. Main results. The proposed method achieved an official challenge score of 0.54, 0.52, 0.50, 0.51, and 0.50 on twelve-lead, six-lead, four-lead, three-lead, and two-lead ECG test sets, respectively. These scores were ranked 5th, 3rd, 7th, 5th and 7th, respectively, in the challenge. Significance. The proposed method can correct the differential labeling tendency of databases from different sources and exhibits good generalization for classifying multi-lead ECGs in the hidden test set. The proposed models have the potential for clinical applications.

Robust PVC Identification by Fusing Expert System and Deep Learning.

Premature ventricular contraction (PVC) is one of the common ventricular arrhythmias, which may cause stroke or sudden cardiac death. Automatic long-term electrocardiogram (ECG) analysis algorithms could provide diagnosis suggestion and even early warning for physicians. However, they are mutually exclusive in terms of robustness, generalization and low complexity. In this study, a novel PVC recognition algorithm that combines deep learning-based heartbeat template clusterer and expert system-based heartbeat classifier is proposed. A long short-term memory-based auto-encoder (LSTM-AE) network was used to extract features from ECG heartbeats for K-means clustering. Thus, the templates were constructed and determined based on clustering results. Finally, the PVC heartbeats were recognized based on a combination of multiple rules, including template matching and rhythm characteristics. Three quantitative parameters, sensitivity (Se), positive predictive value (P+) and accuracy (ACC), were used to evaluate the performances of the proposed method on the MIT-BIH Arrhythmia database and the St. Petersburg Institute of Cardiological Technics database. Se on the two test databases was 87.51% and 87.92%, respectively; P+ was 92.47% and 93.18%, respectively; and ACC was 98.63% and 97.89%, respectively. The PVC scores on the third China Physiological Signal Challenge 2020 training set and hidden test set were 36,256 and 46,706, respectively, which could rank first in the open-source codes. The results showed that the combination strategy of expert system and deep learning can provide new insights for robust and generalized PVC identification from long-term single-lead ECG recordings.

An accurate and efficient method to train classifiers for atrial fibrillation detection in ECGs: Learning by asking better questions

BackgroundAn increasing number of wearables are capable of measuring electrocardiograms (ECGs), which may help in early detection of atrial fibrillation (AF). Therefore, many studies focus on automated detection of AF in ECGs. A major obstacle is the required amount of manually labelled data. This study aimed to provide an efficient and reliable method to train a classifier for AF detection using large datasets of real-life ECGs. MethodHuman-controlled semi-supervised learning was applied, consisting of two phases: the pre-training phase and the semi-automated training phase. During pre-training, an initial classifier was trained, which was used to predict the classes of new ECG segments in the semi-automated training phase. Based on the degree of certainty, segments were added to the training dataset automatically or after human validation. Thereafter, the classifier was retrained and this procedure was repeated. To test the model performance, a real-life telemetry dataset containing 3,846,564 30-s ECG segments of hospitalized patients (n = 476) and the CinC Challenge 2017 database were used. ResultsAfter pre-training, the average F1-score on a hidden testing dataset was 89.0%. Furthermore, after the pre-training phase 68.0% of all segments in the hidden test set could be classified with an estimated probability of successful classification of 99%, providing an F1-score of 97.9% for these segments. During the semi-automated training phase, this F1-score showed little variation (97.3%–97.9% in the hidden test set), whilst the number of segments which could be automatically classified increased from 68.0% to 75.8% due to the enhanced training dataset. At the same time, the overall F1-score increased from 89.0% to 91.4%. ConclusionsHuman-validated semi-supervised learning makes training a classifier more time efficient without compromising on accuracy, hence this method might be valuable in the automated detection of AF in real-life ECGs.

Two-Step Imputation and AdaBoost-Based Classification for Early Prediction of Sepsis on Imbalanced Clinical Data.

Sepsis is a life-threatening response to infection that causes tissue damage, organ failure, and death. Effective early prediction of sepsis would improve patients' diagnosis and reduce the cost associated with late-stage sepsis infection by applying appropriate early intervention. However, effective early prediction is challenging because sepsis biomarkers are neither obvious nor definitive, and sepsis datasets are heavily imbalanced against positive diagnosis of sepsis while containing significant missing values. Early prediction of sepsis in ICUs using clinical data is the objective of the PhysioNet/Computing in Cardiology Challenge 2019. In this article, we proposed a machine learning algorithm to aid in the early detection of sepsis. We applied linear interpolation and implemented a sample weighted AdaBoost model to predict sepsis 6 hours before clinical diagnosis. Medical data contains more than 40,000 patients gathered from three geographically distinct U.S. hospital systems that consisted of a combination of hourly vital sign, lab values, and static patient descriptions. The challenge metric, however, did not directly reward models for their generalizability across institutions. The article is evaluated using a new metric called Utility Score that is defined as Official scoring criteria. Our approach was among the top 10% of entries to the Challenge on a hidden test set. Herein, we demonstrate that our proposed approach was the most effective of the Challenge entrants when such generalizability is explicitly accounted for in model evaluation.

Reduction of false alarms in the intensive care unit using an optimized machine learning based approach

This work attempts to reduce the number of false alarms generated by bedside monitors in the intensive care unit (ICU), as a majority of current alarms are false. In this study, we applied methods that can be categorized into three stages: signal processing, feature extraction, and optimized machine learning. At the stage of signal processing, we ensured that the heartbeats were properly annotated. During feature extraction, besides extracting features that are relevant to the arrhythmic alarms, we also extracted a set of signal quality indices (SQIs), which we used to distinguish noise/artifact from normal physiological signals. When applying a machine learning algorithm (Random Forest), we performed feature selection in order to reduce the complexity of the models and improve the efficiency of the algorithm. The dataset used is from Reducing False Arrhythmia Alarms in the ICU: the PhysioNet/Computing in Cardiology Challenge 2015. Using the performance metric “score” from the Challenge, we achieved a score of 83.08 in the real-time category on the hidden test set, which is the highest in all published work.

You Snooze, You Win: the PhysioNet/Computing in Cardiology Challenge 2018.

The PhysioNet/Computing in Cardiology Challenge 2018 focused on the use of various physiological signals (EEG, EOG, EMG, ECG, SaO2) collected during polysomnographic sleep studies to detect sources of arousal (non-apnea) during sleep. A total of 1,983 polysomnographic recordings were made available to the entrants. The arousal labels for 994 of the recordings were made available in a public training set while 989 labels were retained in a hidden test set. Challengers were asked to develop an algorithm that could label the presence of arousals within the hidden test set. The performance metric used to assess entrants was the area under the precision-recall curve. A total of twenty-two independent teams entered the Challenge, deploying a variety of methods from generalized linear models to deep neural networks.

Parallel use of a convolutional neural network and bagged tree ensemble for the classification of Holter ECG

The automated detection of arrhythmia in a Holter ECG signal is a challenging task due to its complex clinical content and data quantity. It is also challenging due to the fact that Holter ECG is usually affected by noise. Such noise may be the result of the regular activity of patients using the Holter ECG—partially unplugged electrodes, short-time disconnections due to movement, or disturbances caused by electric devices or infrastructure. Furthermore, regular patient activities such as movement also affect the ECG signals and, in connection with artificial noise, may render the ECG non-readable or may lead to misinterpretation of the ECG.Objective: In accordance with the PhysioNet/CinC Challenge 2017, we propose a method for automated classification of 1-lead Holter ECG recordings. Approach: The proposed method classifies a tested record into one of four classes—‘normal’, ‘atrial fibrillation’, ‘other arrhythmia’ or ‘too noisy to classify’. It uses two machine learning methods in parallel. The first—a bagged tree ensemble (BTE)—processes a set of 43 features based on QRS detection and PQRS morphology. The second—a convolutional neural network connected to a shallow neural network (CNN/NN)—uses ECG filtered by nine different filters (8× envelograms, 1× band-pass). If the output of CNN/NN reaches a specific level of certainty, its output is used. Otherwise, the BTE output is preferred.Main results: The proposed method was trained using a reduced version of the public PhysioNet/CinC Challenge 2017 dataset (8183 records) and remotely tested on the hidden dataset on PhysioNet servers (3658 records). The method achieved F1 test scores of 0.92, 0.82 and 0.74 for normal recordings, atrial fibrillation and recordings containing other arrhythmias, respectively. The overall F1 score measured on the hidden test-set was 0.83. Significance: This F1 score led to shared rank #2 in the follow-up PhysioNet/CinC Challenge 2017 ranking.

Detecting atrial fibrillation from short single lead ECGs using statistical and morphological features

Objective: Point of care ECG devices can improve the early detection of atrial fibrillation (AF). The efficiency of such devices depends on the capability of automatic AF detection against normal sinus rhythm and other arrhythmias from a short single lead ECG record in the presence of noise and artifacts. The objective of this study was to develop an algorithm that classifies a short single lead ECG record into ‘Normal’, ‘AF’, ‘Other’ and ‘Noisy’ classes, and identify the challenges in developing such algorithms and potential mitigation steps. Approach: Rule-based identification was used to detect lead inversion and records too noisy to be of immediate use. A set of statistical and morphological features describing the rhythm was then extracted, and support vector machine classifiers were used to classify records into three classes: ‘Normal’, ‘AF’ or ‘Other’. The algorithm was trained and tested using 12 186 short single lead ECGs recorded on a point of care device made available via the Computing in Cardiology Challenge 2017. Main results: The algorithm achieved a sensitivity of 77.5%, a specificity of 97.9% and an accuracy of 96.1% in the detection of AF from a non-AF rhythm in a five-fold cross validation. It achieved F1 measures of 89%, 78% and 67% for ‘Normal’, ‘AF’ and ‘Other’ classes, respectively, when evaluated with a hidden test set. The overall challenge score was 78%. Significance: Most existing algorithms can distinguish the AF rhythm from the normal sinus rhythm when ECG recordings are clean and are obtained with multi-lead systems, while their ability to discriminate against other arrhythmias and noise remains largely unknown. This study proposes an algorithm that classifies a short single lead ECG record from point of care devices into ‘Normal’, ‘AF’, ‘Other’ and ‘Noisy’ classes and discusses computational approaches to mitigate any unique challenges such as lead inversion, low amplitude signals, noise and artifacts.

The PhysioNet/Computing in Cardiology Challenge 2015: Reducing False Arrhythmia Alarms in the ICU.

High false alarm rates in the ICU decrease quality of care by slowing staff response times while increasing patient delirium through noise pollution. The 2015 Physio-Net/Computing in Cardiology Challenge provides a set of 1,250 multi-parameter ICU data segments associated with critical arrhythmia alarms, and challenges the general research community to address the issue of false alarm suppression using all available signals. Each data segment was 5 minutes long (for real time analysis), ending at the time of the alarm. For retrospective analysis, we provided a further 30 seconds of data after the alarm was triggered. A collection of 750 data segments was made available for training and a set of 500 was held back for testing. Each alarm was reviewed by expert annotators, at least two of whom agreed that the alarm was either true or false. Challenge participants were invited to submit a complete, working algorithm to distinguish true from false alarms, and received a score based on their program's performance on the hidden test set. This score was based on the percentage of alarms correct, but with a penalty that weights the suppression of true alarms five times more heavily than acceptance of false alarms. We provided three example entries based on well-known, open source signal processing algorithms, to serve as a basis for comparison and as a starting point for participants to develop their own code. A total of 38 teams submitted a total of 215 entries in this year's Challenge.