Consensus Labeling Research Articles

An automated artifact detection and rejection system for body surface gastric mapping.

Body surface gastric mapping (BSGM) is a new clinical tool for gastric motility diagnostics, providing high-resolution data on gastric myoelectrical activity. Artifact contamination was a key challenge to reliable test interpretation in traditional electrogastrography. This study aimed to introduce and validate an automated artifact detection and rejection system for clinical BSGM applications. Ten patients with chronic gastric symptoms generated a variety of artifacts according to a standardized protocol (176 recordings) using a commercial BSGM system (Alimetry, New Zealand). An automated artifact detection and rejection algorithm was developed, and its performance was compared with a reference standard comprising consensus labeling by 3 analysis experts, followed by comparison with 6 clinicians (3 untrained and 3 trained in artifact detection). Inter-rater reliability was calculated using Fleiss' kappa. Inter-rater reliability was 0.84 (95% CI:0.77-0.90) among experts, 0.76 (95% CI:0.68-0.83) among untrained clinicians, and 0.71 (95% CI:0.62-0.79) among trained clinicians. The sensitivity and specificity of the algorithm against experts was 96% (95% CI:91%-100%) and 95% (95% CI:90%-99%), respectively, vs 77% (95% CI:68%-85%) and 99% (95% CI:96%-100%) against untrained clinicians, and 97% (95% CI:92%-100%) and 88% (95% CI:82%-94%) against trained clinicians. An automated artifact detection and rejection algorithm was developed showing >95% sensitivity and specificity vs expert markers. This algorithm overcomes an important challenge in the clinical translation of BSGM and is now being routinely implemented in patient test interpretations.

107 Real-Time Computer Vision TTE Analytics using Existing Ultrasound Workflows

Real time ‘AI’ applications utilizing point-of-care ultrasound are often commercialized, expensive and do not work with existing ultrasound machines and workflows, hampering the reach of effective computer vision techniques to assist at the bedside for diagnostic and teaching purposes. The purpose of this study is to evaluate whether formal transthoracic echocardiogram studies can be used for transfer learning on ED echocardiogram views and to build a portable model for real-time cardiac function categorization at the bedside that works with existing ultrasound equipment. The ultimate goal of the project is to create a portable hardware and software solution that works with existing ED ultrasound machines and workflows. A previously described three-dimensional convolutional neural network (CNN), trained on 10, 030 formal transthoracic echocardiogram apical 4-chamber (A4C), was adapted for use in the ED. Expert sonographer tracings were used through weak supervision to generate frame-level semantic segmentations of the left ventricle to output video of cardiac function over time. 1123 ED A4C videos from July 2020 to December 2020 were intermixed by labeling them into four categories (‘normal,’ ‘slightly reduced,’ ‘moderately reduced,’ ‘severely reduced’) by board certified emergency physicians and consensus labeling was done by an ultrasound division faculty member. Given variability of ED A4C videos and to assist with CNN training, an interpretability label (‘yes,’ ‘partial,’ ‘no’) was also obtained corresponding to all, some, and none of the beats complete and clinically interpretable. These labeled videos were intermixed with the formal echocardiogram video set and the CNN was retrained to provide binary output (normal vs hypo- contractile) and segmented video of left ventricular cardiac function over time. Receiver operating characteristic curves (ROC) were generated to assess diagnostic ability. Finally, the video output port of an ultrasound machine was used to generate model outputs (label and segmented video) available to providers in near real-time (<0.2 seconds to create model output) using inexpensive hardware at the bedside. An initial set of 133 A4C videos were evaluated by the model before any retraining was done with a ROC of 0.81 (95% CI: 0.77 - 0.85) for a binary classifier. After intermixing the remaining ED videos and retraining the CNN, an improved ROC of 0.91 (95% CI: 0.89 - 0.94) was observed for a binary classifier. For test set A4C videos that the model deemed interpretable, segmented videos accurately demonstrated contractility by shading left ventricular cardiac function. A three-dimensional CNN trained on a cardiology echocardiogram dataset can be retrained using ED A4C videos to provide high discriminatory value and be adapted for existing ED ultrasound machines and workflows providing an inexpensive tool for real-time feedback at the bedside. Along with categorical labeling, segmented video output can help with uptake of and understanding model output, potentially lowering the barrier to entry to ED echocardiography for providers lacking formal ultrasound training and providing guidance to trainees.

Big data requirements for artificial intelligence.

To summarize how big data and artificial intelligence technologies have evolved, their current state, and next steps to enable future generations of artificial intelligence for ophthalmology. Big data in health care is ever increasing in volume and variety, enabled by the widespread adoption of electronic health records (EHRs) and standards for health data information exchange, such as Digital Imaging and Communications in Medicine and Fast Healthcare Interoperability Resources. Simultaneously, the development of powerful cloud-based storage and computing architectures supports a fertile environment for big data and artificial intelligence in health care. The high volume and velocity of imaging and structured data in ophthalmology and is one of the reasons why ophthalmology is at the forefront of artificial intelligence research. Still needed are consensus labeling conventions for performing supervised learning on big data, promotion of data sharing and reuse, standards for sharing artificial intelligence model architectures, and access to artificial intelligence models through open application program interfaces (APIs). Future requirements for big data and artificial intelligence include fostering reproducible science, continuing open innovation, and supporting the clinical use of artificial intelligence by promoting standards for data labels, data sharing, artificial intelligence model architecture sharing, and accessible code and APIs.

Deep convolutional neural networks with multiplane consensus labeling for lung function quantification using UTE proton MRI.

Ultrashort echo time (UTE) proton MRI has gained popularity for assessing lung structure and function in pulmonary imaging; however, the development of rapid biomarker extraction and regional quantification has lagged behind due to labor-intensive lung segmentation. To evaluate a deep learning (DL) approach for automated lung segmentation to extract image-based biomarkers from functional lung imaging using 3D radial UTE oxygen-enhanced (OE) MRI. Retrospective study aimed to evaluate a technical development. Forty-five human subjects, including 16 healthy volunteers, 5 asthma, and 24 patients with cystic fibrosis. 1.5T MRI, 3D radial UTE (TE = 0.08 msec) sequence. Two 3D radial UTE volumes were acquired sequentially under normoxic (21% O2 ) and hyperoxic (100% O2 ) conditions. Automated segmentation of the lungs using 2D convolutional encoder-decoder based DL method, and the subsequent functional quantification via adaptive K-means were compared with the results obtained from the reference method, supervised region growing. Relative to the reference method, the performance of DL on volumetric quantification was assessed using Dice coefficient with 95% confidence interval (CI) for accuracy, two-sided Wilcoxon signed-rank test for computation time, and Bland-Altman analysis on the functional measure derived from the OE images. The DL method produced strong agreement with supervised region growing for the right (Dice: 0.97; 95% CI = [0.96, 0.97]; P < 0.001) and left lungs (Dice: 0.96; 95% CI = [0.96, 0.97]; P < 0.001). The DL method averaged 46 seconds to generate the automatic segmentations in contrast to 1.93 hours using the reference method (P < 0.001). Bland-Altman analysis showed nonsignificant intermethod differences of volumetric (P ≥ 0.12) and functional measurements (P ≥ 0.34) in the left and right lungs. DL provides rapid, automated, and robust lung segmentation for quantification of regional lung function using UTE proton MRI. 2 Technical Efficacy: Stage 1 J. Magn. Reson. Imaging 2019;50:1169-1181.

Deep neural networks for A-line-based plaque classification in coronary intravascular optical coherence tomography images.

.We develop neural-network-based methods for classifying plaque types in clinical intravascular optical coherence tomography (IVOCT) images of coronary arteries. A single IVOCT pullback can consist of microscopic-resolution images, creating both a challenge for physician interpretation during an interventional procedure and an opportunity for automated analysis. In the proposed method, we classify each A-line, a datum element that better captures physics and pathophysiology than a voxel, as a fibrous layer followed by calcification (fibrocalcific), a fibrous layer followed by a lipidous deposit (fibrolipidic), or other. For A-line classification, the usefulness of a convolutional neural network (CNN) is compared with that of a fully connected artificial neural network (ANN). A total of 4469 image frames across 48 pullbacks that are manually labeled using consensus labeling from two experts are used for training, evaluation, and testing. A 10-fold cross-validation using held-out pullbacks is applied to assess classifier performance. Noisy A-line classifications are cleaned by applying a conditional random field (CRF) and morphological processing to pullbacks in the en-face view. With CNN (ANN) approaches, we achieve an accuracy of () for fibrocalcific, () for fibrolipidic, and () for other, across all folds following CRF noise cleaning. The results without CRF cleaning are typically reduced by 10% to 15%. The enhanced performance of the CNN was likely due to spatial invariance of the convolution operation over the input A-line. The predicted en-face classification maps of entire pullbacks agree favorably to the annotated counterparts. In some instances, small error regions are actually hard to call when re-examined by human experts. Even in worst-case pullbacks, it can be argued that the results will not negatively impact usage by physicians, as there is a preponderance of correct calls.

Convolutional Neural Networks with Template-Based Data Augmentation for Functional Lung Image Quantification

We propose an automated segmentation pipeline based on deep learning for proton lung MRI segmentation and ventilation-based quantification which improves on our previously reported methodologies in terms of computational efficiency while demonstrating accuracy and robustness. The large data requirement for the proposed framework is made possible by a novel template-based data augmentation strategy. Supporting this work is the open-source ANTsRNet-a growing repository of well-known deep learning architectures first introduced here. Deep convolutional neural network (CNN) models were constructed and trained using a custom multilabel Dice metric loss function and a novel template-based data augmentation strategy. Training (including template generation and data augmentation) employed 205 proton MR images and 73 functional lung MRI. Evaluation was performed using data sets of size 63 and 40 images, respectively. Accuracy for CNN-based proton lung MRI segmentation (in terms of Dice overlap) was left lung: 0.93 ± 0.03, right lung: 0.94 ± 0.02, and whole lung: 0.94 ± 0.02. Although slightly less accurate than our previously reported joint label fusion approach (left lung: 0.95 ± 0.02, right lung: 0.96 ± 0.01, and whole lung: 0.96 ± 0.01), processing time is <1 second per subject for the proposed approach versus ∼30 minutes per subject using joint label fusion. Accuracy for quantifying ventilation defects was determined based on a consensus labeling where average accuracy (Dice multilabel overlap of ventilation defect regions plus normal region) was 0.94 for the CNN method; 0.92 for our previously reported method; and 0.90, 0.92, and 0.94 for expert readers. The proposed framework yields accurate automated quantification in near real time. CNNs drastically reduce processing time after offline model construction and demonstrate significant future potential for facilitating quantitative analysis of functional lung MRI.

Longitudinally and inter-site consistent multi-atlas based parcellation of brain anatomy using harmonized atlases

As longitudinal and multi-site studies become increasingly frequent in neuroimaging, maintaining longitudinal and inter-scanner consistency of brain parcellation has become a major challenge due to variation in scanner models and/or image acquisition protocols across scanners and sites. We present a new automated segmentation method specifically designed to achieve a consistent parcellation of anatomical brain structures in such heterogeneous datasets. Our method combines a site-specific atlas creation strategy with a state-of-the-art multi-atlas anatomical label fusion framework. Site-specific atlases are computed such that they preserve image intensity characteristics of each site's scanner and acquisition protocol, while atlas pairs share anatomical labels in a way consistent with inter-scanner acquisition variations. This harmonization of atlases improves inter-study and longitudinal consistency of segmentations in the subsequent consensus labeling step. We tested this approach on a large sample of older adults from the Baltimore Longitudinal Study of Aging (BLSA) who had longitudinal scans acquired using two scanners that vary with respect to vendor and image acquisition protocol. We compared the proposed method to standard multi-atlas segmentation for both cross-sectional and longitudinal analyses. The harmonization significantly reduced scanner-related differences in the age trends of ROI volumes, improved longitudinal consistency of segmentations, and resulted in higher across-scanner intra-class correlations, particularly in the white matter.

Crowdsourcing Without a Crowd

We present an incremental Bayesian model that resolves key issues of crowd size and data quality for consensus labeling. We evaluate our method using data collected from a real-world citizen science program, B ee W atch , which invites members of the public in the United Kingdom to classify (label) photographs of bumblebees as one of 22 possible species. The biological recording domain poses two key and hitherto unaddressed challenges for consensus models of crowdsourcing: (1) the large number of potential species makes classification difficult, and (2) this is compounded by limited crowd availability, stemming from both the inherent difficulty of the task and the lack of relevant skills among the general public. We demonstrate that consensus labels can be reliably found in such circumstances with very small crowd sizes of around three to five users (i.e., through group sourcing). Our incremental Bayesian model, which minimizes crowd size by re-evaluating the quality of the consensus label following each species identification solicited from the crowd, is competitive with a Bayesian approach that uses a larger but fixed crowd size and outperforms majority voting. These results have important ecological applicability: biological recording programs such as B ee W atch can sustain themselves when resources such as taxonomic experts to confirm identifications by photo submitters are scarce (as is typically the case), and feedback can be provided to submitters in a timely fashion. More generally, our model provides benefits to any crowdsourced consensus labeling task where there is a cost (financial or otherwise) associated with soliciting a label.

MUSE: MUlti-atlas region Segmentation utilizing Ensembles of registration algorithms and parameters, and locally optimal atlas selection

Atlas-based automated anatomical labeling is a fundamental tool in medical image segmentation, as it defines regions of interest for subsequent analysis of structural and functional image data. The extensive investigation of multi-atlas warping and fusion techniques over the past 5 or more years has clearly demonstrated the advantages of consensus-based segmentation. However, the common approach is to use multiple atlases with a single registration method and parameter set, which is not necessarily optimal for every individual scan, anatomical region, and problem/data-type. Different registration criteria and parameter sets yield different solutions, each providing complementary information. Herein, we present a consensus labeling framework that generates a broad ensemble of labeled atlases in target image space via the use of several warping algorithms, regularization parameters, and atlases. The label fusion integrates two complementary sources of information: a local similarity ranking to select locally optimal atlases and a boundary modulation term to refine the segmentation consistently with the target image's intensity profile. The ensemble approach consistently outperforms segmentations using individual warping methods alone, achieving high accuracy on several benchmark datasets. The MUSE methodology has been used for processing thousands of scans from various datasets, producing robust and consistent results. MUSE is publicly available both as a downloadable software package, and as an application that can be run on the CBICA Image Processing Portal (https://ipp.cbica.upenn.edu), a web based platform for remote processing of medical images.

An Optimization Framework for Combining Ensembles of Classifiers and Clusterers with Applications to Nontransductive Semisupervised Learning and Transfer Learning

Unsupervised models can provide supplementary soft constraints to help classify new “target” data because similar instances in the target set are more likely to share the same class label. Such models can also help detect possible differences between training and target distributions, which is useful in applications where concept drift may take place, as in transfer learning settings. This article describes a general optimization framework that takes as input class membership estimates from existing classifiers learned on previously encountered “source” (or training) data, as well as a similarity matrix from a cluster ensemble operating solely on the target (or test) data to be classified, and yields a consensus labeling of the target data. More precisely, the application settings considered are nontransductive semisupervised and transfer learning scenarios where the training data are used only to build an ensemble of classifiers and are subsequently discarded before classifying the target data. The framework admits a wide range of loss functions and classification/clustering methods. It exploits properties of Bregman divergences in conjunction with Legendre duality to yield a principled and scalable approach. A variety of experiments show that the proposed framework can yield results substantially superior to those provided by naïvely applying classifiers learned on the original task to the target data. In addition, we show that the proposed approach, even not being conceptually transductive, can provide better results compared to some popular transductive learning techniques.

“Blood Petroleum”: True Blood, the BP Oil Spill, and Fictions of Energy/Culture

The third season of the HBO series True Blood, set in fictional Bon Temps, Louisiana, aired in a mediascape shaped by coverage of the BP oil spill that wreaked economic and ecological havoc on the US Gulf South during the summer of 2010. A retrospective examination of the series in this context, and against the grain of critical consensus labeling it mere escapism, demonstrates that taking True Blood seriously can yield compelling insights into the US Gulf South as a site in which convergences of the global and the local, of reality and representation, and of energy and cultural production result in the formation of a hybrid: energy/culture. Analysis of the storyline featuring the Vampire King of Mississippi shows how True Blood extends the long-standing cultural practice of making vampires screens for projecting collective desires and anxieties. Through a “camp aesthetic” that weaves into the Vampire King's maniacal pursuit of blood in various forms dire warnings about excessive consumption and environmental apocalypse, True Blood offers fictional ways to make meaning of the actual conditions and consequences of energy production and consumption brought to the surface with great urgency by the BP oil spill.

Extended Input Space Support Vector Machine

In some applications, the probability of error of a given classifier is too high for its practical application, but we are allowed to gather more independent test samples from the same class to reduce the probability of error of the final decision. From the point of view of hypothesis testing, the solution is given by the Neyman-Pearson lemma. However, there is no equivalent result to the Neyman-Pearson lemma when the likelihoods are unknown, and we are given a training dataset. In this brief, we explore two alternatives. First, we combine the soft (probabilistic) outputs of a given classifier to produce a consensus labeling for K test samples. In the second approach, we build a new classifier that directly computes the label for K test samples. For this second approach, we need to define an extended input space training set and incorporate the known symmetries in the classifier. This latter approach gives more accurate results, as it only requires an accurate classification boundary, while the former needs an accurate posterior probability estimate for the whole input space. We illustrate our results with well-known databases.

A high-resolution computational atlas of the human hippocampus from postmortem magnetic resonance imaging at 9.4 T

This paper describes the construction of a computational anatomical atlas of the human hippocampus. The atlas is derived from high-resolution 9.4 Tesla MRI of postmortem samples. The main subfields of the hippocampus (cornu ammonis fields CA1, CA2/3; the dentate gyrus; and the vestigial hippocampal sulcus) are labeled in the images manually using a combination of distinguishable image features and geometrical features. A synthetic average image is derived from the MRI of the samples using shape and intensity averaging in the diffeomorphic non-linear registration framework, and a consensus labeling of the template is generated. The agreement of the consensus labeling with manual labeling of each sample is measured, and the effect of aiding registration with landmarks and manually generated mask images is evaluated. The atlas is provided as an online resource with the aim of supporting subfield segmentation in emerging hippocampus imaging and image analysis techniques. An example application examining subfield-level hippocampal atrophy in temporal lobe epilepsy demonstrates the application of the atlas to in vivo studies.

Plasma Oxidized Low-Density Lipoprotein, a Strong Predictor for Acute Coronary Heart Disease Events in Apparently Healthy, Middle-Aged Men From the General Population

We propose an automated segmentation pipeline based on deep learning for proton lung MRI segmentation and ventilation-based quantification which improves on our previously reported methodologies in terms of computational efficiency while demonstrating accuracy and robustness. The large data requirement for the proposed framework is made possible by a novel template-based data augmentation strategy. Supporting this work is the open-source ANTsRNet—a growing repository of well-known deep learning architectures first introduced here.Deep convolutional neural network (CNN) models were constructed and trained using a custom multilabel Dice metric loss function and a novel template-based data augmentation strategy. Training (including template generation and data augmentation) employed 205 proton MR images and 73 functional lung MRI. Evaluation was performed using data sets of size 63 and 40 images, respectively.Accuracy for CNN-based proton lung MRI segmentation (in terms of Dice overlap) was left lung: 0.93 ± 0.03, right lung: 0.94 ± 0.02, and whole lung: 0.94 ± 0.02. Although slightly less accurate than our previously reported joint label fusion approach (left lung: 0.95 ± 0.02, right lung: 0.96 ± 0.01, and whole lung: 0.96 ± 0.01), processing time is <1 second per subject for the proposed approach versus ∼30 minutes per subject using joint label fusion. Accuracy for quantifying ventilation defects was determined based on a consensus labeling where average accuracy (Dice multilabel overlap of ventilation defect regions plus normal region) was 0.94 for the CNN method; 0.92 for our previously reported method; and 0.90, 0.92, and 0.94 for expert readers.The proposed framework yields accurate automated quantification in near real time. CNNs drastically reduce processing time after offline model construction and demonstrate significant future potential for facilitating quantitative analysis of functional lung MRI.

Use of Lyophilized Calibrant Plasmas for Simplified International Normalized Ratio Determination with a Human Tissue Factor Thromboplastin Reagent Derived from Cultured Human Cells

For monitoring of treatment with oral anticoagulants, the clotting time obtained in the prothrombin time (PT) test is transformed to the International Normalized Ratio (INR) with use of a system-specific International Sensitivity Index (ISI). The calibrant plasma procedure (CPP) is an alternative approach to INR calculation based on the use of a set of lyophilized plasmas with assigned INRs. With the CPP, a linear relationship is established between log(PT) and log(INR), using orthogonal regression. CPP was validated for Simplastin HTF, a new human tissue factor reagent derived from cultured human cells. CPP precision was assessed as the CV of the slope of the regression line. The accuracy of the CPP was determined by comparing the INR obtained with the CPP with that obtained with the established ISI-based reference method. INRs of the calibrants were assigned by different routes: by manufacturer (consensus labeling) or by use of Simplastin HTF or International Reference Preparations (IRPs; rTF/95 or RBT/90). The mean CV of the CPP regression slope ranged from 1.0% (Simplastin HTF reagent-specific INR) to 2.4% (INR assigned with rTF/95). INRs calculated with the CPP were similar to those obtained with the reference method, but when the routes for assigning INRs to the calibrant plasmas were compared, the mean difference in INR between CPP and the reference method was smaller with Simplastin HTF reagent-specific values. In several (but not all) cases, this difference was significant (P <0.05, t-test). CPP can be used for local INR determination, but better precision and accuracy are obtained with reagent-specific INRs compared with INR assignment by consensus labeling or IRP.