Human Expert Annotations Research Articles

An Immunofluorescence-Guided Segmentation Model in Hematoxylin and Eosin Images Is Enabled by Tissue Artifact Correction Using a Cycle-Consistent Generative Adversarial Network

Despite recent advances, the adoption of computer vision methods into clinical and commercial applications has been hampered by the limited availability of accurate ground truth tissue annotations required to train robust supervised models. Generating such ground truth can be accelerated by annotating tissue molecularly using immunofluorescence (IF) staining and mapping these annotations to a post-IF hematoxylin and eosin (H&E) (terminal H&E) stain. Mapping the annotations between IF and terminal H&E increases both the scale and accuracy by which ground truth could be generated. However, discrepancies between terminal H&E and conventional H&E caused by IF tissue processing have limited this implementation. We sought to overcome this challenge and achieve compatibility between these parallel modalities using synthetic image generation, in which a cycle-consistent generative adversarial network was applied to transfer the appearance of conventional H&E such that it emulates terminal H&E. These synthetic emulations allowed us to train a deep learning model for the segmentation of epithelium in terminal H&E that could be validated against the IF staining of epithelial-based cytokeratins. The combination of this segmentation model with the cycle-consistent generative adversarial network stain transfer model enabled performative epithelium segmentation in conventional H&E images. The approach demonstrates that the training of accurate segmentation models for the breadth of conventional H&E data can be executed free of human expert annotations by leveraging molecular annotation strategies such as IF, so long as the tissue impacts of the molecular annotation protocol are captured by generative models that can be deployed prior to the segmentation process.

Practical Sentiment Analysis for Education: The Power of Student Crowdsourcing

Sentiment analysis provides a promising tool to automatically assess the emotions voiced in written student feedback such as periodically collected unit-of-study reflections. The commonly used dictionary-based approaches are limited to major languages and fail to capture contextual differences. Pretrained large language models have been shown to be biased and online versions raise privacy concerns. Hence, we resort to traditional supervised machine learning (ML) approaches which are designed to overcome these issues by learning from domain-specific labeled data. However, these labels are hard to come by -- in our case manually annotating student feedback is prone to bias and time-consuming, especially in high-enrollment courses. In this work, we investigate the use of student crowdsourced labels for supervised sentiment analysis for education. Specifically, we compare crowdsourced and student self-reported labels with human expert annotations and use them in various ML approaches to evaluate the performance on predicting emotions of written student feedback collected from large computer science classes. We find that the random forest model trained with student-crowdsourced labels tremendously improves the identification of reflections with negative sentiment. In addition to our quantitative study, we describe our crowdsourcing experiment which was intentionally designed to be an educational activity in an introduction to data science course.

Utilizing 2D-region-based CNNs for automatic dendritic spine detection in 3D live cell imaging

Dendritic spines are considered a morphological proxy for excitatory synapses, rendering them a target of many different lines of research. Over recent years, it has become possible to simultaneously image large numbers of dendritic spines in 3D volumes of neural tissue. In contrast, currently no automated method for 3D spine detection exists that comes close to the detection performance reached by human experts. However, exploiting such datasets requires new tools for the fully automated detection and analysis of large numbers of spines. Here, we developed an efficient analysis pipeline to detect large numbers of dendritic spines in volumetric fluorescence imaging data acquired by two-photon imaging in vivo. The core of our pipeline is a deep convolutional neural network that was pretrained on a general-purpose image library and then optimized on the spine detection task. This transfer learning approach is data efficient while achieving a high detection precision. To train and validate the model we generated a labeled dataset using five human expert annotators to account for the variability in human spine detection. The pipeline enables fully automated dendritic spine detection reaching a performance slightly below that of the human experts. Our method for spine detection is fast, accurate and robust, and thus well suited for large-scale datasets with thousands of spines. The code is easily applicable to new datasets, achieving high detection performance, even without any retraining or adjustment of model parameters.

SHAMSUL: Systematic Holistic Analysis to investigate Medical Significance Utilizing Local interpretability methods in deep learning for chest radiography pathology prediction

The interpretability of deep neural networks has become a subject of great interest within the medical and healthcare domain. This attention stems from concerns regarding transparency, legal and ethical considerations, and the medical significance of predictions generated by these deep neural networks in clinical decision support systems. To address this matter, our study delves into the application of four well-established interpretability methods: Local Interpretable Model-agnostic Explanations (LIME), Shapley Additive exPlanations (SHAP), Gradient-weighted Class Activation Mapping (Grad-CAM), and Layer-wise Relevance Propagation (LRP). Leveraging the approach of transfer learning with a multi-label-multi-class chest radiography dataset, we aim to interpret predictions pertaining to specific pathology classes. Our analysis encompasses both single-label and multi-label predictions, providing a comprehensive and unbiased assessment through quantitative and qualitative investigations, which are compared against human expert annotation. Notably, Grad-CAM demonstrates the most favorable performance in quantitative evaluation, while the LIME heatmap score segmentation visualization exhibits the highest level of medical significance. Our research underscores both the outcomes and the challenges faced in the holistic approach adopted for assessing these interpretability methods and suggests that a multimodal-based approach, incorporating diverse sources of information beyond chest radiography images, could offer additional insights for enhancing interpretability in the medical domain.

Deep Learning–Enabled Assessment of Left Heart Structure and Function Predicts Cardiovascular Outcomes

BackgroundDeep learning interpretation of echocardiographic images may facilitate automated assessment of cardiac structure and function. ObjectivesWe developed a deep learning model to interpret echocardiograms and examined the association of deep learning–derived echocardiographic measures with incident outcomes. MethodsWe trained and validated a 3-dimensional convolutional neural network model for echocardiographic view classification and quantification of left atrial dimension, left ventricular wall thickness, chamber diameter, and ejection fraction. The training sample comprised 64,028 echocardiograms (n = 27,135) from a retrospective multi-institutional ambulatory cardiology electronic health record sample. Validation was performed in a separate longitudinal primary care sample and an external health care system data set. Cox models evaluated the association of model-derived left heart measures with incident outcomes. ResultsDeep learning discriminated echocardiographic views (area under the receiver operating curve >0.97 for parasternal long axis, apical 4-chamber, and apical 2-chamber views vs human expert annotation) and quantified standard left heart measures (R2 range = 0.53 to 0.91 vs study report values). Model performance was similar in 2 external validation samples. Model-derived left heart measures predicted incident heart failure, atrial fibrillation, myocardial infarction, and death. A 1-SD lower model-left ventricular ejection fraction was associated with 43% greater risk of heart failure (HR: 1.43; 95% CI: 1.23-1.66) and 17% greater risk of death (HR: 1.17; 95% CI: 1.06-1.30). Similar results were observed for other model-derived left heart measures. ConclusionsDeep learning echocardiographic interpretation accurately quantified standard measures of left heart structure and function, which in turn were associated with future clinical outcomes. Deep learning may enable automated echocardiogram interpretation and disease prediction at scale.

Comparison of a Deep Learning Model with Human Expert Annotations for Segmentation of Kidneys, Tumors, and Cysts in Routine CT Imaging Exams

Comparison of a Deep Learning Model with Human Expert Annotations for Segmentation of Kidneys, Tumors, and Cysts in Routine CT Imaging Exams

Annotation-Efficient COVID-19 Pneumonia Lesion Segmentation Using Error-Aware Unified Semisupervised and Active Learning

As the coronavirusdisease 2019 (COVID-19) pandemic continues, fast and automatic COVID-19-related pneumonia lesion segmentation method is in an urgent need. The current state-of-the-art methods for segmentation generally require sufficient amounts of annotated data for training. However, human expert annotation of such lesion on chest computed tomography (CT) scans is time-consuming and labor-intensive due to its heterogeneous appearance, ambiguous boundary, and large number of slices in 3-D CT images. Therefore, the purpose of this study is to present a novel annotation-efficient learning method for COVID-19 pneumonia lesion segmentation on CT. To make the best use of limited human expert annotation resources, we propose an error-aware unified semisupervised and active learning method. A novel error estimation network is proposed to estimate a voxelwise segmentation loss map, which is used to guide learning from unlabeled data for semisupervised learning and choose the most informative images to annotate next for active learning. Validation is carried out on segmenting pneumonia lesions in 110 chest CT scans. The experimental result demonstrates that the proposed method significantly boosts the segmentation accuracy given limited amount of human annotation, compared with a conventional fully supervised baseline (60.9% Dice to 72.0% at 30% labeled data). The performance is also competitive compared with other state-of-the-art annotation-efficient segmentation methods. The proposed method can significantly reduce the annotation effort needed to achieve accurate COVID-19 pneumonia lesion segmentation.

Dual center validation of deep learning for automated multi-label segmentation of thoracic anatomy in bedside chest radiographs

Background and ObjectivesBedside chest radiographs (CXRs) are challenging to interpret but important for monitoring cardiothoracic disease and invasive therapy devices in critical care and emergency medicine. Taking surrounding anatomy into account is likely to improve the diagnostic accuracy of artificial intelligence and bring its performance closer to that of a radiologist. Therefore, we aimed to develop a deep convolutional neural network for efficient automatic anatomy segmentation of bedside CXRs. MethodsTo improve the efficiency of the segmentation process, we introduced a "human-in-the-loop" segmentation workflow with an active learning approach, looking at five major anatomical structures in the chest (heart, lungs, mediastinum, trachea, and clavicles). This allowed us to decrease the time needed for segmentation by 32% and select the most complex cases to utilize human expert annotators efficiently. After annotation of 2,000 CXRs from different Level 1 medical centers at Charité – University Hospital Berlin, there was no relevant improvement in model performance, and the annotation process was stopped. A 5-layer U-ResNet was trained for 150 epochs using a combined soft Dice similarity coefficient (DSC) and cross-entropy as a loss function. DSC, Jaccard index (JI), Hausdorff distance (HD) in mm, and average symmetric surface distance (ASSD) in mm were used to assess model performance. External validation was performed using an independent external test dataset from Aachen University Hospital (n = 20). ResultsThe final training, validation, and testing dataset consisted of 1900/50/50 segmentation masks for each anatomical structure. Our model achieved a mean DSC/JI/HD/ASSD of 0.93/0.88/32.1/5.8 for the lung, 0.92/0.86/21.65/4.85 for the mediastinum, 0.91/0.84/11.83/1.35 for the clavicles, 0.9/0.85/9.6/2.19 for the trachea, and 0.88/0.8/31.74/8.73 for the heart. Validation using the external dataset showed an overall robust performance of our algorithm. ConclusionsUsing an efficient computer-aided segmentation method with active learning, our anatomy-based model achieves comparable performance to state-of-the-art approaches. Instead of only segmenting the non-overlapping portions of the organs, as previous studies did, a closer approximation to actual anatomy is achieved by segmenting along the natural anatomical borders. This novel anatomy approach could be useful for developing pathology models for accurate and quantifiable diagnosis.

A convolutional neural network-based decision support system for neonatal quiet sleep detection.

Sleep plays an important role in neonatal brain and physical development, making its detection and characterization important for assessing early-stage development. In this study, we propose an automatic and computationally efficient algorithm to detect neonatal quiet sleep (QS) using a convolutional neural network (CNN). Our study used 38-hours of electroencephalography (EEG) recordings, collected from 19 neonates at Fudan Children's Hospital in Shanghai, China (Approval No. (2020) 22). To train and test the CNN, we extracted 12 prominent time and frequency domain features from 9 bipolar EEG channels. The CNN architecture comprised two convolutional layers with pooling and rectified linear unit (ReLU) activation. Additionally, a smoothing filter was applied to hold the sleep stage for 3 minutes. Through performance testing, our proposed method achieved impressive results, with 94.07% accuracy, 89.70% sensitivity, 94.40% specificity, 79.82% F1-score and a 0.74 kappa coefficient when compared to human expert annotations. A notable advantage of our approach is its computational efficiency, with the entire training and testing process requiring only 7.97 seconds. The proposed algorithm has been validated using leave one subject out (LOSO) validation, which demonstrates its consistent performance across a diverse range of neonates. Our findings highlight the potential of our algorithm for real-time neonatal sleep stage classification, offering a fast and cost-effective solution. This research opens avenues for further investigations in early-stage development monitoring and the assessment of neonatal health.

Part-of-Speech and Morphological Tagging of Algerian Judeo-Arabic

Most linguistic studies of Judeo-Arabic, the ensemble of dialects spoken and written by Jews in Arab lands, are qualitative in nature and rely on laborious manual annotation work, and are therefore limited in scale. In this work, we develop automatic methods for morpho-syntactic tagging of Algerian Judeo-Arabic texts published by Algerian Jews in the 19th--20th centuries, based on a linguistically tagged corpus. First, we describe our semi-automatic approach for preprocessing these texts. Then, we experiment with both an off-the-shelf morphological tagger and several specially designed neural network taggers. Finally, we perform a real-world evaluation of new texts that were never tagged before in comparison with human expert annotators. Our experimental results demonstrate that these methods can dramatically speed up and improve the linguistic research pipeline, enabling linguists to study these dialects on a much greater scale.

Deep learning-assisted high resolution mapping of vulnerable habitats within the Capbreton Canyon System, Bay of Biscay

The Capbreton Canyon System is an area currently under study for its proposal as a Site of Community Importance under the EU Habitats Directive in the context of the LIFE IP INTEMARES project. Identifying and mapping benthic Vulnerable Marine Ecosystems (VMEs) plays a key role in this process. Although obtaining information on species distribution in deep sea rocky habitats has traditionally been a complicated task, the development of underwater remote sensing techniques resulted in a massive increase in the collection of digital imagery; however, processing all this information has led to another bottleneck due to the time-consuming nature of biota manual annotation. At this point, the use of computer vision and deep learning to automate image processing has substantial benefits but has rarely been adopted within the field of marine ecology. This study presents the integration of deep learning techniques for benthic fauna identification, high resolution multibeam echosounder (MBES) data and Species Distribution Models (SDMs), to map the potential habitat of the yellow coral Dendrophyllia cornigera, a representative species of the VME 1170 Reef habitat, on the circalitoral area of the Capbreton Canyon System. The localization and identification of the coral colonies was based on more than 7500 photographs taken during the INTEMARES-CapBreton 0619 and 0620 surveys using the photogrammetric ROTV Politolana. For the automatic annotation of the image set a deep learning based framework was developed by testing two different deep neural networks architectures; a FasterRCNN+Resnet101 model, accomplishing a precision of 100% over human expert annotation for presence/absence discrimination, was selected. Environmental data included different quantitative terrain attributes derived from high resolution MBES bathymetry data. A presence-only species distribution model, Maximum Entropy (MaxEnt), was used to infer the spatial distribution of D. cornigera over the study area. Predicted occurrences corresponded mainly to relevant topographic structures with significant slope, mainly associated to the edge of the continental shelf. These results are consistent with the ecological knowledge on the species and validate the use of deep learning tools to assist in the identification and mapping of VME for management and conservation purposes. This study provides a baseline for the protection of vulnerable habitats of the Capbreton Canyon System in the context of the Natura 2000 Network.

Validating an SVM-based neonatal seizure detection algorithm for generalizability, non-inferiority and clinical efficacy

Neonatal seizure detection algorithms (SDA) are approaching the benchmark of human expert annotation. Measures of algorithm generalizability and non-inferiority as well as measures of clinical efficacy are needed to assess the full scope of neonatal SDA performance. We validated our neonatal SDA on an independent data set of 28 neonates. Generalizability was tested by comparing the performance of the original training set (cross-validation) to its performance on the validation set. Non-inferiority was tested by assessing inter-observer agreement between combinations of SDA and two human expert annotations. Clinical efficacy was tested by comparing how the SDA and human experts quantified seizure burden and identified clinically significant periods of seizure activity in the EEG. Algorithm performance was consistent between training and validation sets with no significant worsening in AUC (p > 0.05, n = 28). SDA output was inferior to the annotation of the human expert, however, re-training with an increased diversity of data resulted in non-inferior performance (Δκ = 0.077, 95% CI: −0.002-0.232, n = 18). The SDA assessment of seizure burden had an accuracy ranging from 89 to 93%, and 87% for identifying periods of clinical interest. The proposed SDA is approaching human equivalence and provides a clinically relevant interpretation of the EEG.

Statistical Atlases and Automatic Labeling Strategies to Accelerate the Analysis of Social Insect Brain Evolution

Current methods used to quantify brain size and compartmental scaling relationships in studies of social insect brain evolution involve manual annotations of images from histological samples, confocal microscopy or other sources. This process is susceptible to human bias and error and requires time-consuming effort by expert annotators. Standardized brain atlases, constructed through 3D registration and automatic segmentation, surmount these issues while increasing throughput to robustly sample diverse morphological and behavioral phenotypes. Here we design and evaluate three strategies to construct statistical brain atlases, or templates, using ants as a model taxon. The first technique creates a template by registering multiple brains of the same species. Brain regions are manually annotated on the template, and the labels are transformed back to each individual brain to obtain an automatic annotation, or to any other brain aligned with the template. The second strategy also creates a template from multiple brain images but obtains labels as a consensus from multiple manual annotations of individual brains comprising the template. The third technique is based on a template comprising brains from multiple species and the consensus of their labels. We used volume similarity as a metric to evaluate the automatic segmentation produced by each method against the inter- and intra-individual variability of human expert annotators. We found that automatic and manual methods are equivalent in volume accuracy, making the template technique an extraordinary tool to accelerate data collection and reduce human bias in the study of the evolutionary neurobiology of ants and other insects.

Towards human-level performance on automatic pose estimation of infant spontaneous movements.

Assessment of spontaneous movements can predict the long-term developmental disorders in high-risk infants. In order to develop algorithms for automated prediction of later disorders, highly precise localization of segments and joints by infant pose estimation is required. Four types of convolutional neural networks were trained and evaluated on a novel infant pose dataset, covering the large variation in 1424 videos from a clinical international community. The localization performance of the networks was evaluated as the deviation between the estimated keypoint positions and human expert annotations. The computational efficiency was also assessed to determine the feasibility of the neural networks in clinical practice. The best performing neural network had a similar localization error to the inter-rater spread of human expert annotations, while still operating efficiently. Overall, the results of our study show that pose estimation of infant spontaneous movements has a great potential to support research initiatives on early detection of developmental disorders in children with perinatal brain injuries by quantifying infant movements from video recordings with human-level performance.

Transformer based Generative Adversarial Network for Liver Segmentation.

Automated liver segmentation from radiology scans (CT, MRI) can improve surgery and therapy planning and follow-up assessment in addition to conventional use for diagnosis and prognosis. Although convolutional neural networks (CNNs) have became the standard image segmentation tasks, more recently this has started to change towards Transformers based architectures because Transformers are taking advantage of capturing long range dependence modeling capability in signals, so called attention mechanism. In this study, we propose a new segmentation approach using a hybrid approach combining the Transformer(s) with the Generative Adversarial Network (GAN) approach. The premise behind this choice is that the self-attention mechanism of the Transformers allows the network to aggregate the high dimensional feature and provide global information modeling. This mechanism provides better segmentation performance compared with traditional methods. Furthermore, we encode this generator into the GAN based architecture so that the discriminator network in the GAN can classify the credibility of the generated segmentation masks compared with the real masks coming from human (expert) annotations. This allows us to extract the high dimensional topology information in the mask for biomedical image segmentation and provide more reliable segmentation results. Our model achieved a high dice coefficient of 0.9433, recall of 0.9515, and precision of 0.9376 and outperformed other Transformer based approaches. The implementation details of the proposed architecture can be found at https://github.com/UgurDemir/tranformer_liver_segmentation.

SPAKT: A Self-Supervised Pre-TrAining Method for Knowledge Tracing

Knowledge tracing (KT) is the core task of computer-aided education systems, and it aims at predicting whether a student can answer the next exercise (i.e., question) correctly based on his/her historical answer records. In recent years, deep neural network-based approaches have been widely developed in KT and achieved promising results. More recently, several researches further boost these KT models via exploiting plentiful relationships including exercise-skill relations (E-S), the exercise similarity (E-E) as well as skill similarity (S-S). However, these relationship information are frequently absent in many real-world educational applications, and it is a labor-intensive work for human experts to label it. Inspired by recent advances in natural language processing domain, we propose a novel pre-training approach, namely as SPAKT, and utilize self-supervised learning to pre-train exercise embedding representation without the need for expensive human-expert annotations in this paper. Contrary to existing pre-training methods that highly rely on manually labeling knowledge about the E-E, S-S, or E-S relationships, the core idea of the proposed SPAKT is to design three self-attention modules to model the E-S, E-E, and S-S relationships, respectively, and all of these three modules can be trained in the self-supervised setting. As a pre-training approach, our SPAKT can be effortlessly incorporated into existing deep neural network-based KT frameworks. We experimentally show that, even without using expensive annotations about the aforementioned three kinds of relationships, our model achieves competitive performance compared with state-of-the-arts. Our algorithm implementations have been made publicly available at https://github.com/Vinci-hp/pretrainKT.

Accurate Identification of the Trabecular Meshwork under Gonioscopic View in Real Time Using Deep Learning

Accurate identification of iridocorneal structures on gonioscopy is difficult to master, and errors can lead to grave surgical complications. This study aimed to develop and train convolutional neural networks (CNNs) to accurately identify the trabecular meshwork (TM) in gonioscopic videos in real time for eventual clinical integrations. Cross-sectional study. Adult patients with open angle were identified in academic glaucoma clinics in both Taipei, Taiwan, and Irvine, California. Neural Encoder-Decoder CNNs (U-nets) were trained to predict a curve marking the TM using an expert-annotated data set of 378 gonioscopy images. The model was trained and evaluated with stratified cross-validation grouped by patients to ensure uncorrelated training and testing sets, as well as on a separate test set and 3 intraoperative gonioscopic videos of ab interno trabeculotomy with Trabectome (totaling 90 seconds long, 30 frames per second). We also evaluated our model's performance by comparing its accuracy against ophthalmologists. Successful development of real-time-capable CNNs that are accurate in predicting and marking the TM's position in video frames of gonioscopic views. Models were evaluated in comparison with human expert annotations of static images and video data. The best CNN model produced test set predictions with a median deviation of 0.8% of the video frame's height (15.25 μm) from the human experts' annotations. This error is less than the average vertical height of the TM. The worst test frame prediction of this model had an average deviation of 4% of the frame height (76.28 μm), which is still considered a successful prediction. When challenged with unseen images, the CNN model scored greater than 2 standard deviations above the mean performance of the surveyed general ophthalmologists. Our CNN model can identify the TM in gonioscopy videos in real time with remarkable accuracy, allowing it to be used in connection with a video camera intraoperatively. This model can have applications in surgical training, automated screenings, and intraoperative guidance. The dataset developed in this study is one of the first publicly available gonioscopy image banks (https://lin.hs.uci.edu/research), which may encourage future investigations in this topic.

Abstract 12669: Precision Phenotyping of Left Ventricular Hypertrophy With Echocardiographic Deep Learning

Introduction: Left ventricular hypertrophy (LVH) results from chronic remodeling caused by broad range of systemic and cardiovascular disease including hypertension, aortic stenosis, hypertrophic cardiomyopathy, and cardiac amyloidosis. Early detection and characterization of LVH can significantly impact patient care but is limited by under-recognition of hypertrophy, measurement error and variability, and difficulty differentiating etiologies of LVH. Methods: To overcome this challenge, we present EchoNet-LVH, a deep learning workflow that quantifies ventricular hypertrophy precision equal to human experts, identifies patients with LVH, and predict etiology of hypertrophy. Trained on 28,201 echocardiogram videos, our model was implemented in Pytorch and evaluated on an independent dataset from an international healthcare system. Human expert annotations of anatomic structures in parasternal long axis echocardiogram videos were used as labels for supervised training. Results: EchoNet-LVH accurately measures ventricular dimensions with an R 2 of 0.97 compared to expert annotations. EchoNet-LVH accurately evaluated intraventricular wall thickness (mean absolute error 0.14cm), left ventricular diameter (MAE 0.24cm), posterior wall thickness (MAE 0.12cm) and classifies cardiac amyloidosis (area under the curve of 0.83 on held-out test data) from other etiologies of LVH. In an external dataset of 1,791 echocardiogram studies, EchoNet-LVH reliably quantified ventricular parameters with R 2 of 0.90 and MAE 0.22cm, 0.36cm, and 0.17cm respectively for IVS, LVID, and LVPW. Conclusions: Leveraging measurements across multiple heart beats, our model can rapidly identify subtle changes in LVH, is more reproducible than human evaluation, and lays the foundation for precision diagnosis of cardiovascular disease.

Refining epileptogenic high-frequency oscillations using deep learning: a reverse engineering approach.

Intracranially recorded interictal high-frequency oscillations have been proposed as a promising spatial biomarker of the epileptogenic zone. However, its visual verification is time-consuming and exhibits poor inter-rater reliability. Furthermore, no method is currently available to distinguish high-frequency oscillations generated from the epileptogenic zone (epileptogenic high-frequency oscillations) from those generated from other areas (non-epileptogenic high-frequency oscillations). To address these issues, we constructed a deep learning-based algorithm using chronic intracranial EEG data via subdural grids from 19 children with medication-resistant neocortical epilepsy to: (i) replicate human expert annotation of artefacts and high-frequency oscillations with or without spikes, and (ii) discover epileptogenic high-frequency oscillations by designing a novel weakly supervised model. The ‘purification power’ of deep learning is then used to automatically relabel the high-frequency oscillations to distill epileptogenic high-frequency oscillations. Using 12 958 annotated high-frequency oscillation events from 19 patients, the model achieved 96.3% accuracy on artefact detection (F1 score = 96.8%) and 86.5% accuracy on classifying high-frequency oscillations with or without spikes (F1 score = 80.8%) using patient-wise cross-validation. Based on the algorithm trained from 84 602 high-frequency oscillation events from nine patients who achieved seizure-freedom after resection, the majority of such discovered epileptogenic high-frequency oscillations were found to be ones with spikes (78.6%, P < 0.001). While the resection ratio of detected high-frequency oscillations (number of resected events/number of detected events) did not correlate significantly with post-operative seizure freedom (the area under the curve = 0.76, P = 0.06), the resection ratio of epileptogenic high-frequency oscillations positively correlated with post-operative seizure freedom (the area under the curve = 0.87, P = 0.01). We discovered that epileptogenic high-frequency oscillations had a higher signal intensity associated with ripple (80–250 Hz) and fast ripple (250–500 Hz) bands at the high-frequency oscillation onset and with a lower frequency band throughout the event time window (the inverted T-shaped), compared to non-epileptogenic high-frequency oscillations. We then designed perturbations on the input of the trained model for non-epileptogenic high-frequency oscillations to determine the model’s decision-making logic. The model confidence significantly increased towards epileptogenic high-frequency oscillations by the artificial introduction of the inverted T-shaped signal template (mean probability increase: 0.285, P < 0.001), and by the artificial insertion of spike-like signals into the time domain (mean probability increase: 0.452, P < 0.001). With this deep learning-based framework, we reliably replicated high-frequency oscillation classification tasks by human experts. Using a reverse engineering technique, we distinguished epileptogenic high-frequency oscillations from others and identified its salient features that aligned with current knowledge.

Evaluation of artificial intelligence systems for assisting neurologists with fast and accurate annotations of scalp electroencephalography data

BackgroundAssistive automatic seizure detection can empower human annotators to shorten patient monitoring data review times. We present a proof-of-concept for a seizure detection system that is sensitive, automated, patient-specific, and tunable to maximise sensitivity while minimizing human annotation times. The system uses custom data preparation methods, deep learning analytics and electroencephalography (EEG) data. MethodsScalp EEG data of 365 patients containing 171,745 s ictal and 2,185,864 s interictal samples obtained from clinical monitoring systems were analysed as part of a crowdsourced artificial intelligence (AI) challenge. Participants were tasked to develop an ictal/interictal classifier with high sensitivity and low false alarm rates. We built a challenge platform that prevented participants from downloading or directly accessing the data while allowing crowdsourced model development. FindingsThe automatic detection system achieved tunable sensitivities between 75.00% and 91.60% allowing a reduction in the amount of raw EEG data to be reviewed by a human annotator by factors between 142x, and 22x respectively. The algorithm enables instantaneous reviewer-managed optimization of the balance between sensitivity and the amount of raw EEG data to be reviewed. InterpretationThis study demonstrates the utility of deep learning for patient-specific seizure detection in EEG data. Furthermore, deep learning in combination with a human reviewer can provide the basis for an assistive data labelling system lowering the time of manual review while maintaining human expert annotation performance. FundingIBM employed all IBM Research authors. Temple University employed all Temple University authors. The Icahn School of Medicine at Mount Sinai employed Eren Ahsen. The corresponding authors Stefan Harrer and Gustavo Stolovitzky declare that they had full access to all the data in the study and that they had final responsibility for the decision to submit for publication.