Standard Annotation Research Articles

Contextualized race and ethnicity annotations for clinical text from MIMIC-III

Observational health research often relies on accurate and complete race and ethnicity (RE) patient information, such as characterizing cohorts, assessing quality/performance metrics of hospitals and health systems, and identifying health disparities. While the electronic health record contains structured data such as accessible patient-level RE data, it is often missing, inaccurate, or lacking granular details. Natural language processing models can be trained to identify RE in clinical text which can supplement missing RE data in clinical data repositories. Here we describe the Contextualized Race and Ethnicity Annotations for Clinical Text (C-REACT) Dataset, which comprises 12,000 patients and 17,281 sentences from their clinical notes in the MIMIC-III dataset. Using these sentences, two sets of reference standard annotations for RE data are made available with annotation guidelines. The first set of annotations comprise highly granular information related to RE, such as preferred language and country of origin, while the second set contains RE labels annotated by physicians. This dataset can support health systems’ ability to use RE data to serve health equity goals.

Accuracy of endocardial abnormal electrogram characterization using a 17-AHA-segmental late boundary and voltage-dependent sensitivity annotation for ventricular tachycardia ablation

Abstract Background Substrate-based catheter ablation of ventricular tachycardia (VT) has become the mainstay therapy for patients with scar-related VT. Correct identification and timing of abnormal near-field EGMs and discerning it from far-field EGMs, is key for procedural success. The novel CARTO™ Late Activation Mapping (LAM) module provides fast and accurate annotation of near-field EGMs including late abnormal ventricular activation. For the algorithm to succeed, a late boundary (LB) annotation line should be positioned immediately behind the normal near-field EGM, however, since the timing of normal local near-field activation varies greatly throughout the endocardium, there is currently no consensus for the optimal LB positioning. We hypothesized that 17-AHA segmental annotation of the endocardial LB followed by scar-dependent sensitivity settings, will improve the accuracy of the near-field EGM characterization. Purpose To evaluate the accuracy of endocardial abnormal EGM characterization using a 17-AHA-segmental late boundary and voltage-dependent sensitivity annotation as compared to conventional mid-QRS LB positioning. Methods Five high-resolution substrate voltage maps were obtained from 5 patients (80% male, 68±7 years old, 80% ischemic cardiomyopathy) that underwent a catheter-based ablation for scar-related VT. A personalized CT-derived anatomical reconstruction of the 17-segment AHA model was co-registered offline with the electroanatomic voltage map, using ADAS3D software. Per segment, individual LB timings were set at the highest sensitivity. A gold standard annotation map was created by two independent experts after manual scrutiny of individual EGMs. Next, the expert map was compared to automatic maps created from segmental LB and mid-QRS LB positions at normal, high, highest or mixed (normal/high or normal/highest) sensitivities, focusing on EGM voltage and activation accuracy. Results A mean of 2041±1028 points per map were analyzed. Correct correlation of EGM annotation was found on average in 91% (activation) and 81% (voltage) for all segmental LB positioned maps (segLB), and in 83% (activation) and 77% (voltage) for all mid-QRS LB positioned maps (mid-QRSLB). The highest correlation with expert map for both activation and voltage annotation was found in the mixed sensitivity normal/highest map with segmental LB positioning (activation 98%, voltage 88%). The correlation on activation was significantly superior in the segLB map when compared to the normal sensitivity maps (p=0.0008 for mid-QRSLB and p=0.02 for segLB). For correlation on voltage, the segLBmap with mixed sensitivity was significantly superior compared to three mid-QRSLB maps at normal (p=0.04), high and highest (p=0.01) sensitivity. Conclusion Individualized, 17-AHA-segmental late boundary and voltage-dependent sensitivity annotation provides a possible accurate and clinically-relevant framework for substrate assessment. Prospective validation is needed.

Multi-stage semi-supervised learning enhances white matter hyperintensity segmentation.

White matter hyperintensities (WMHs) are frequently observed on magnetic resonance (MR) images in older adults, commonly appearing as areas of high signal intensity on fluid-attenuated inversion recovery (FLAIR) MR scans. Elevated WMH volumes are associated with a greater risk of dementia and stroke, even after accounting for vascular risk factors. Manual segmentation, while considered the ground truth, is both labor-intensive and time-consuming, limiting the generation of annotated WMH datasets. Un-annotated data are relatively available; however, the requirement of annotated data poses a challenge for developing supervised machine learning models. To address this challenge, we implemented a multi-stage semi-supervised learning (M3SL) approach that first uses un-annotated data segmented by traditional processing methods ("bronze" and "silver" quality data) and then uses a smaller number of "gold"-standard annotations for model refinement. The M3SL approach enabled fine-tuning of the model weights with the gold-standard annotations. This approach was integrated into the training of a U-Net model for WMH segmentation. We used data from three scanner vendors (over more than five scanners) and from both cognitively normal (CN) adult and patients cohorts [with mild cognitive impairment and Alzheimer's disease (AD)]. An analysis of WMH segmentation performance across both scanner and clinical stage (CN, MCI, AD) factors was conducted. We compared our results to both conventional and transfer-learning deep learning methods and observed better generalization with M3SL across different datasets. We evaluated several metrics (F-measure, IoU, and Hausdorff distance) and found significant improvements with our method compared to conventional (p < 0.001) and transfer-learning (p < 0.001). These findings suggest that automated, non-machine learning, tools have a role in a multi-stage learning framework and can reduce the impact of limited annotated data and, thus, enhance model performance.

BIBSNet: A Deep Learning Baby Image Brain Segmentation Network for MRI Scans.

Brain segmentation of infant magnetic resonance (MR) images is vitally important in studying developmental mental health and disease. The infant brain undergoes many changes throughout the first years of postnatal life, making tissue segmentation difficult for most existing algorithms. Here, we introduce a deep neural network BIBSNet (Baby and Infant Brain Segmentation Neural Network), an open-source, community-driven model that relies on data augmentation and a large sample size of manually annotated images to facilitate the production of robust and generalizable brain segmentations. Included in model training and testing were MR brain images on 84 participants with an age range of 0-8 months (median postmenstrual ages of 13.57 months). Using manually annotated real and synthetic segmentation images, the model was trained using a 10-fold cross-validation procedure. Testing occurred on MRI data processed with the DCAN labs infant-ABCD-BIDS processing pipeline using segmentations produced from gold standard manual annotation, joint-label fusion (JLF), and BIBSNet to assess model performance. Using group analyses, results suggest that cortical metrics produced using BIBSNet segmentations outperforms JLF segmentations. Additionally, when analyzing individual differences, BIBSNet segmentations perform even better. BIBSNet segmentation shows marked improvement over JLF segmentations across all age groups analyzed. The BIBSNet model is 600x faster compared to JLF and can be easily included in other processing pipelines.

Comparing Commercial and Open-Source Large Language Models for Labeling Chest Radiograph Reports.

Background Rapid advances in large language models (LLMs) have led to the development of numerous commercial and open-source models. While recent publications have explored OpenAI's GPT-4 to extract information of interest from radiology reports, there has not been a real-world comparison of GPT-4 to leading open-source models. Purpose To compare different leading open-source LLMs to GPT-4 on the task of extracting relevant findings from chest radiograph reports. Materials and Methods Two independent datasets of free-text radiology reports from chest radiograph examinations were used in this retrospective study performed between February 2, 2024, and February 14, 2024. The first dataset consisted of reports from the ImaGenome dataset, providing reference standard annotations from the MIMIC-CXR database acquired between 2011 and 2016. The second dataset consisted of randomly selected reports created at the Massachusetts General Hospital between July 2019 and July 2021. In both datasets, the commercial models GPT-3.5 Turbo and GPT-4 were compared with open-source models that included Mistral-7B and Mixtral-8 × 7B (Mistral AI), Llama 2-13B and Llama 2-70B (Meta), and Qwen1.5-72B (Alibaba Group), as well as CheXbert and CheXpert-labeler (Stanford ML Group), in their ability to accurately label the presence of multiple findings in radiograph text reports using zero-shot and few-shot prompting. The McNemar test was used to compare F1 scores between models. Results On the ImaGenome dataset (n = 450), the open-source model with the highest score, Llama 2-70B, achieved micro F1 scores of 0.97 and 0.97 for zero-shot and few-shot prompting, respectively, compared with the GPT-4 F1 scores of 0.98 and 0.98 (P > .99 and < .001 for superiority of GPT-4). On the institutional dataset (n = 500), the open-source model with the highest score, an ensemble model, achieved micro F1 scores of 0.96 and 0.97 for zero-shot and few-shot prompting, respectively, compared with the GPT-4 F1 scores of 0.98 and 0.97 (P < .001 and > .99 for superiority of GPT-4). Conclusion Although GPT-4 was superior to open-source models in zero-shot report labeling, few-shot prompting with a small number of example reports closely matched the performance of GPT-4. The benefit of few-shot prompting varied across datasets and models. © RSNA, 2024 Supplemental material is available for this article.

An analysis of FRE @ BC8 SympTEMIST track: named entity recognition.

This paper is a more in-depth analysis of the approaches used in our submission (Martínez A, García-Santa N. (2023) FRE @ BC8 SympTEMIST track: Named Entity Recognition Zenodo.) to the 'SympTEMIST' Named Entity Recognition (NER) shared subtask at 'BioCreative 2023'. We participated on the challenge submitting two systems based on a RoBERTa architecture LLM trained on Spanish-language clinical data available at 'HuggingFace' model repository. Before choosing the systems that would be submitted, we tried different combinations of the techniques described here: Conditional Random Fields and Byte-Pair Encoding dropout. In the second system we also included Sub-Subword feature based embeddings (SSW). The test set used in the challenge has now been released (López SL, Sánchez LG, Farré E etal. (2024) SympTEMIST Corpus: Gold Standard annotations for clinical symptoms, signs and findings information extraction. Zenodo), allowing us to analyze more in depth our methods, as well as measuring the impact of introducing data from CARMEN-I (Lima-López S, Farré-Maduell E, Krallinger M. (2023) CARMEN-I: Clinical Entities Annotation Guidelines in Spanish. Zenodo) corpus. Our experiments show the moderate effect of using the Sub-Subword feature based embeddings and the impact of including the symptom NER data from the CARMEN-I dataset. Database URL: https://physionet.org/content/carmen-i/1.0/.

NnU-Net-based Segmentation of Tumor Subcompartments in Pediatric Medulloblastoma Using Multiparametric MRI: A Multi-institutional Study.

Purpose To evaluate nnU-Net-based segmentation models for automated delineation of medulloblastoma tumors on multi-institutional MRI scans. Materials and Methods This retrospective study included 78 pediatric patients (52 male, 26 female), with ages ranging from 2 to 18 years, with medulloblastomas, from three different sites (28 from hospital A, 18 from hospital B, and 32 from hospital C), who had data available from three clinical MRI protocols (gadolinium-enhanced T1-weighted, T2-weighted, and fluid-attenuated inversion recovery). The scans were retrospectively collected from the year 2000 until May 2019. Reference standard annotations of the tumor habitat, including enhancing tumor, edema, and cystic core plus nonenhancing tumor subcompartments, were performed by two experienced neuroradiologists. Preprocessing included registration to age-appropriate atlases, skull stripping, bias correction, and intensity matching. The two models were trained as follows: (a) the transfer learning nnU-Net model was pretrained on an adult glioma cohort (n = 484) and fine-tuned on medulloblastoma studies using Models Genesis and (b) the direct deep learning nnU-Net model was trained directly on the medulloblastoma datasets, across fivefold cross-validation. Model robustness was evaluated on the three datasets when using different combinations of training and test sets, with data from two sites at a time used for training and data from the third site used for testing. Results Analysis on the three test sites yielded Dice scores of 0.81, 0.86, and 0.86 and 0.80, 0.86, and 0.85 for tumor habitat; 0.68, 0.84, and 0.77 and 0.67, 0.83, and 0.76 for enhancing tumor; 0.56, 0.71, and 0.69 and 0.56, 0.71, and 0.70 for edema; and 0.32, 0.48, and 0.43 and 0.29, 0.44, and 0.41 for cystic core plus nonenhancing tumor for the transfer learning and direct nnU-Net models, respectively. The models were largely robust to site-specific variations. Conclusion nnU-Net segmentation models hold promise for accurate, robust automated delineation of medulloblastoma tumor subcompartments, potentially leading to more effective radiation therapy planning in pediatric medulloblastoma. Keywords: Pediatrics, MR Imaging, Segmentation, Transfer Learning, Medulloblastoma, nnU-Net, MRI Supplemental material is available for this article. © RSNA, 2024 See also the commentary by Rudie and Correia de Verdier in this issue.

Abstract LB331: Integrating multiple spatial transcriptomic and proteomic technologies to generate a cancer atlas to understand the tumor microenvironment

Abstract Deep single-cell genomics has provided a monumental leap in our understanding and categorical analysis of cancer cells and their unique molecular properties. With spatial platforms that preserve the tissue architecture, an emerging understanding of complex cell interactions and local environment influences is providing new perspectives for targeted therapies. Here we describe a comprehensive approach at profiling the tumor immune microenvironment across multiple solid malignancies to provide an atlas of tumor biology. To generate this atlas, we created a MERFISH panel of 500 genes representing over 40 cell phenotyping pathways and dozens of signaling pathways including chemokines, cytokines, interferon, and ligand-receptor interactions. This panel was used across 15 patients representing multiple solid malignancies. We also utilized the Immuno-Oncology focused Xenium panel to profile 16 different patients across the same indications. With the Xenium data we simultaneously assessed proteomic expression using a 16 plex panel focused on immuno-oncology targets and subsequently did H&amp;E staining, allowing multiple -omic data collections within a single slide. After initial filtering and quality control this projects produced a total dataset with over 15 million cells and billions of transcripts. The 500 gene MERFISH panel revealed complex relationships between diverse cell types within the tumor microenvironment that was dependent on cellular location and surrounding area. While the presence of major immune cell types was observed in all cancer tissues, the ratio and spatial relationship to cancer cells differed between each tumor. Accordingly, we observed that differences in cellular composition of the microenvironment corresponded with associated changes in cellular signaling. Similarly, with Xenium we identified new spatial relationships between cancer and immune cells with their environment while confirming part of these relationships with subsequent immunofluorescence data. The H&amp;E data provided an opportunity to overlay this data with standard pathological annotation. We further investigated how each cellular microneighborhood varied throughout the tumor and varied from patient. This comprehensive map of the tumor immune microenvironment is a deep cellular dive into cellular architecture. As this Cancer Atlas evolves this will drive the discovery of several new potential biological targets and cell types of interest. This will also be critical for the assessment of current cellular and drug therapies in preclinical and clinical models. This effort will continue with more patient samples across a greater number of disease indications and will include more technologies. We will also expand the development and implementation of analytical tools to maximum the knowledge and insights gained from this Atlas. Together, this will be a critical tool in the continuing fight against cancer. Citation Format: Colles Price, Kazuho Nishimura, Som Bandyopadhyay, Darrell Borger, Russell Weiner. Integrating multiple spatial transcriptomic and proteomic technologies to generate a cancer atlas to understand the tumor microenvironment [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 2 (Late-Breaking, Clinical Trial, and Invited Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(7_Suppl):Abstract nr LB331.

Significance Associated with Phenotype Score Aids in Variant Prioritization for Exome Sequencing Analysis

Several in silico annotation-based methods have been developed to prioritize variants in exome sequencing analysis. This study introduces a novel metric, the Significance Associated with Phenotypes (SAP) score, which generates a statistical score by comparing an individual's observed phenotypes against existing gene-phenotype associations. To evaluate the SAP score, a retrospective analysis was performed on 219 exomes. Among them, 82 family-based and 35 singleton exomes had at least one disease-causing variant explaining the patient’s clinical features. SAP scores were calculated, and the rank of the disease-causing variant was compared to a known method, Exomiser. Using the SAP score, the known causative variant was ranked in the top 10 retained variants for 94% (77/82) of the family-based exomes and in the first place for 73% of these cases. For singleton exomes, the SAP score analysis ranked the known pathogenic variants within the top 10 for 80% (28/35) of the cases. The SAP score, which is independent of detected variants, demonstrates comparable performance to Exomiser, which considers both phenotype and variant-level evidence simultaneously. Among 102 cases with negative results or variants of uncertain significance, SAP score analysis revealed two cases with a potential new diagnosis based on rank. The SAP score, a phenotypic quantitative metric, can be utilized in conjunction with standard variant filtration and annotation to enhance variant prioritization in exome analysis.

Motion-compensation approach for quantitative digital subtraction angiography and its effect on in-vivo blood velocity measurement.

Quantitative monitoring of flow-altering interventions has been proposed using algorithms that quantify blood velocity from time-resolved two-dimensional angiograms. These algorithms track the movement of contrast oscillations along a vessel centerline. Vessel motion may occur relative to a statically defined vessel centerline, corrupting the blood velocity measurement. We provide a method for motion-compensated blood velocity quantification. The motion-compensation approach utilizes a vessel segmentation algorithm to perform frame-by-frame vessel registration and creates a dynamic vessel centerline that moves with the vasculature. Performance was evaluated in-vivo through comparison with manually annotated centerlines. The method was also compared to a previous uncompensated method using best- and worst-case static centerlines chosen to minimize and maximize centerline placement accuracy. Blood velocities determined through quantitative DSA (qDSA) analysis for each centerline type were compared through linear regression analysis. Centerline distance errors were relative to gold standard manual annotations. For the uncompensated approach, the best- and worst-case static centerlines had distance errors of and , respectively. Linear regression analysis found a high -squared between qDSA-derived blood velocities using gold standard centerlines and motion-compensated centerlines () with a slope of 1.15 and a small offset of . The use of static centerlines resulted in low coefficients of determination for the best case () and worst-case () scenarios, with slopes close to zero. In-vivo validation of motion-compensated qDSA analysis demonstrated improved velocity quantification accuracy in vessels with motion, addressing an important clinical limitation of the current qDSA algorithm.

Optimal Annotation of Local Activation Time in Ventricular Tachycardia Substrate Mapping

BackgroundAccurate annotation of electrogram local activation time (LAT) is critical to the functional assessment of ventricular tachycardia (VT) substrate. Contemporary methods of annotation include: 1) earliest bipolar electrogram (LATearliest); 2) peak bipolar electrogram (LATpeak); 3) latest bipolar electrogram (LATlatest); and 4) steepest unipolar −dV/dt (LAT−dV/dt). However, no direct comparison of these methods has been performed in a large dataset, and it is unclear which provides the optimal functional analysis of the VT substrate. ObjectivesThis study sought to investigate the optimal method of LAT annotation during VT substrate mapping. MethodsPatients with high-density VT substrate maps and a defined critical site for VT re-entry were included. All electrograms were annotated using 5 different methods: LATearliest, LATpeak, LATlatest, LAT−dV/dt, and the novel steepest unipolar −dV/dt using a dynamic window of interest (LATDWOI). Electrograms were also tagged as either late potentials and/or fractionated signals. Maps, utilizing each annotation method, were then compared in their ability to identify critical sites using deceleration zones. ResultsFifty cases were identified with 1,.813 ± 811 points per map. Using LATlatest, a deceleration zone was present at the critical site in 100% of cases. There was no significant difference with LATearliest (100%) or LATpeak (100%). However, this number decreased to 54% using LAT−dV/dt and 76% for LATDWOI. Using LAT−dV/dt, only 33% of late potentials were correctly annotated, with the larger far field signals often annotated preferentially. ConclusionsAnnotation with LAT−dV/dt and LATDWOI are suboptimal in VT substrate mapping. We propose that LATlatest should be the gold standard annotation method, as this allows identification of critical sites and is most suited to automation.

Automated threat modelling and risk analysis in e-Government using BPMN

Recent progress integrates security requirements into BPMN, enhancing its framework. Extensions aim to seamlessly embed security concepts, yet the inherent ambiguity of security terms may lead to misinterpretations and vulnerabilities. Unfortunately, many business process experts lack the expertise to accurately interpret and integrate vital security concepts. In this study, we present an innovative automated methodology tailored to assist business process experts in identifying security threats and conducting risk assessments, particularly in the context of e-Government processes. Our approach streamlines the process, requiring only a business specialist to annotate BPMN entities with high-level, non-security-related information. Based on these annotations, potential threats to the system can be automatically identified. To develop our methodology, we leverage the standard BPMN annotation mechanism. From the annotated BPMN, the methodology utilises the ENISA Threat Landscape knowledge base for threat identification and employs the OWASP Risk Rating Methodology for risk assessment. To demonstrate the effectiveness of our approach, we applied it to a straightforward case study within the e-Government domain. Through this example, we illustrate how our methodology can be employed to ensure compliance with the General Data Protection Regulation and meet the mandatory Data Protection Impact Assessment requirements.

Deep computational phenotyping of genomic variants impacting the SET domain of KMT2C reveal molecular mechanisms for their dysfunction.

Introduction: Kleefstra Syndrome type 2 (KLEFS-2) is a genetic, neurodevelopmental disorder characterized by intellectual disability, infantile hypotonia, severe expressive language delay, and characteristic facial appearance, with a spectrum of other distinct clinical manifestations. Pathogenic mutations in the epigenetic modifier type 2 lysine methyltransferase KMT2C have been identified to be causative in KLEFS-2 individuals. Methods: This work reports a translational genomic study that applies a multidimensional computational approach for deep variant phenotyping, combining conventional genomic analyses, advanced protein bioinformatics, computational biophysics, biochemistry, and biostatistics-based modeling. We use standard variant annotation, paralog annotation analyses, molecular mechanics, and molecular dynamics simulations to evaluate damaging scores and provide potential mechanisms underlying KMT2C variant dysfunction. Results: We integrated data derived from the structure and dynamics of KMT2C to classify variants into SV (Structural Variant), DV (Dynamic Variant), SDV (Structural and Dynamic Variant), and VUS (Variant of Uncertain Significance). When compared with controls, these variants show values reflecting alterations in molecular fitness in both structure and dynamics. Discussion: We demonstrate that our 3D models for KMT2C variants suggest distinct mechanisms that lead to their imbalance and are not predictable from sequence alone. Thus, the missense variants studied here cause destabilizing effects on KMT2C function by different biophysical and biochemical mechanisms which we adeptly describe. This new knowledge extends our understanding of how variations in the KMT2C gene cause the dysfunction of its methyltransferase enzyme product, thereby bearing significant biomedical relevance for carriers of KLEFS2-associated genomic mutations.

Topological data analysis assisted automated sleep stage scoring using airflow signals

Objective:Breathing pattern variability (BPV), as a universal physiological feature, encodes rich health information. We aim to show that, a high-quality automatic sleep stage scoring based on a proper quantification of BPV extracting from the single airflow signal can be achieved. Methods:Topological data analysis (TDA) is applied to characterize BPV from the intrinsically nonstationary airflow signal. The extracted features from TDA are utilized to train an automatic sleep stage scoring model using the XGBoost learner. Additionally, the noise and artifacts that are typically present in the air flow signal are leveraged to improve the performance of the trained system. To evaluate the effectiveness of the proposed approach, a state-of-the-art method is implemented for comparison purposes. Results:A leave-one-subject-out cross-validation was conducted on a dataset comprising 30 whole-night polysomnogram signals with standard annotations. The results show that the proposed features outperform those considered in the state-of-the-art work in terms of overall accuracy (78.8% ± 8.7% vs. 75.0% ± 9.6%) and Cohen’s kappa (0.56 ± 0.15 vs. 0.50 ± 0.15) for automatically scoring wake, rapid eye movement (REM), and non-REM (NREM) stages. An external validation conducted on a dataset comprising 80 whole-night polysomnogram signals with standard annotations shows a result of overall accuracy 74.1%±11.6% and Cohen’s kappa 0.42±0.15, which again outperforms the state-of-the-art work. Furthermore, the analysis of feature importance reveals that the TDA features provide complementary information to the traditional features commonly used in the literature, and the respiratory quality index is identified as an essential component. Conclusion:The proposed TDA-assisted automatic annotation system can accurately distinguish wake, REM and NREM from the airflow signal. Significance:The utilization of a single air flow channel and the universality of BPV suggest the potential of TDA-assisted signal processing in addressing various biomedical signals and homecare issues beyond sleep stage annotation.

Confidence Contours: Uncertainty-Aware Annotation for Medical Semantic Segmentation

Medical image segmentation modeling is a high-stakes task where understanding of uncertainty is crucial for addressing visual ambiguity. Prior work has developed segmentation models utilizing probabilistic or generative mechanisms to infer uncertainty from labels where annotators draw a singular boundary. However, as these annotations cannot represent an individual annotator's uncertainty, models trained on them produce uncertainty maps that are difficult to interpret. We propose a novel segmentation representation, Confidence Contours, which uses high- and low-confidence ``contours’’ to capture uncertainty directly, and develop a novel annotation system for collecting contours. We conduct an evaluation on the Lung Image Dataset Consortium (LIDC) and a synthetic dataset. From an annotation study with 30 participants, results show that Confidence Contours provide high representative capacity without considerably higher annotator effort. We also find that general-purpose segmentation models can learn Confidence Contours at the same performance level as standard singular annotations. Finally, from interviews with 5 medical experts, we find that Confidence Contour maps are more interpretable than Bayesian maps due to representation of structural uncertainty.

A Transformer Natural Language Processing Algorithm for Cancer Associated Thrombosis Phenotype

Introduction: Cancer patients have high risk of venous thromboembolism (VTE). The ability to accurately capture VTE outcome in large electronic health record (EHR) is critical for both surveillance and risk stratification. We previously published a rule-based natural language processing (NLP) algorithm that relied on radiology reports alone with a precision (positive predictive value) of 89% and recall (sensitivity) of 69-78% (sTable 5 PMID36626707). The low recall metric was expected as VTE diagnosis that occur at outside hospital would not be captured by radiology reports. In the current study, we aimed to assess the performance of a machine learning NLP algorithm using both radiology and medical notes to detect cancer associated VTE longitudinally in a large cancer cohort. Methods: VTEwasdefined as symptomatic or incidental pulmonary embolism, lower or upper extremity deep vein thrombosis. Unusual thrombosis such as splanchnic vein thrombosis (mostly tumor thrombi) was classified as negative. Figure 1 details the study design: i) patient selection, ii) pre-annotation filtering, iii) gold standard annotation, iv) transformer model development. From a cohort of 9,769 patients with active cancer receiving first-line systemic therapy, we first pre-filtered all clinic progress notes, discharge summaries, and radiology reports to exclude sections with little clinical relevance (e.g. review of systems, physical exam), while maintaining sections with high clinical value (e.g. history of present illness, assessment/plan, hospital problems/course). Then, we kept notes with VTE keyword stems (e.g. VTE, PE, DVT, thromb, embol, clot, filling defect, phlebitis). We selected 808 patients with at least 1 keyword to determine the gold standard VTE date and location through independent chart review process. This cohort was randomly split into a training and testing cohort at an 7:1 ratio. In the training cohort (n=708), we further annotated all keyword-containing phrases within 30 days (d) from VTE date and from index date until 180d if no VTE (n=~2,000). The ~800 positive and ~1,200 negative keyword-containing phrases were used to train ClinicalBERT (bidirectional encoder representation transformer) large language model through split-sample cross validation to finetune the loss function and to determine the prediction threshold. Finally, we applied the entire trained model package to every available notes/reports (n=~2300) in the test cohort (n=100) to assess if the first positively predicted date would correlate with the gold standard VTE date. Results: The finetuned ClinicalBERT modelachieved 98% precision and 98% recall on the patient level in the training cohort (n=708). When applied on the 100-patient test cohort, the model accurately predicted first VTE events in 50/54 patients (93% precision) and captured 50/54 of patients with VTE (93% recall sensitivity) (Table 1). Most of the tumor thrombi and superficial venous thrombi were correctly classified as negative. The false positives were related to unusual sentences with both positive and negative findings such as “Thrombosis surrounding the left brachial catheter in basilic vein,” “CT chest with PE was done,” “Lovenox was initiated for UE DVT, but this was port thrombosis in the basilic vein.” The false negatives were related to isolated VTE diagnosis close to death or loss to follow-up. Notably, the precision and recall tradeoff can be further finetuned by changing the minimum number of positively predicted notes from the NLP. Conclusion: In conclusion, we successfully developed and tested a transformer NLP model to detect cancer associated VTE longitudinally using a combination of clinical notes and radiology reports that achieved a precision of 93% and recall of 93%. This represented an improvement over our original algorithm with radiology report alone. Further validation of the model in external cohorts and finetuning through federated or transferred learning are ongoing to ensure model generalizability and usability.

EXSCLAIM!: Harnessing materials science literature for self-labeled microscopy datasets

This work introduces the EXSCLAIM! toolkit for the automatic extraction, separation, and caption-based natural language annotation of images from scientific literature. EXSCLAIM! is used to show how rule-based natural language processing and image recognition can be leveraged to construct an electron microscopy dataset containing thousands of keyword-annotated nanostructure images. Moreover, it is demonstrated how a combination of statistical topic modeling and semantic word similarity comparisons can be used to increase the number and variety of keyword annotations on top of the standard annotations from EXSCLAIM! With large-scale imaging datasets constructed from scientific literature, users are well positioned to train neural networks for classification and recognition tasks specific to microscopy-tasks often otherwise inhibited by a lack of sufficient annotated training data.

Advanced variant classification framework reduces the false positive rate of predicted loss-of-function variants in population sequencing data

Predicted loss of function (pLoF) variants are often highly deleterious and play an important role in disease biology, but many pLoF variants may not result in loss of function (LoF). Here we present a framework that advances interpretation of pLoF variants in research and clinical settings by considering three categories of LoF evasion: (1) predicted rescue by secondary sequence properties, (2) uncertain biological relevance, and (3) potential technical artifacts. We also provide recommendations on adjustments to ACMG/AMP guidelines' PVS1 criterion. Applying this framework to all high-confidence pLoF variants in 22 genes associated with autosomal-recessive disease from the Genome Aggregation Database (gnomAD v.2.1.1) revealed predicted LoF evasion or potential artifacts in 27.3% (304/1,113) of variants. The major reasons were location in the last exon, in a homopolymer repeat, in a low proportion expressed across transcripts (pext) scored region, or the presence of cryptic in-frame splice rescues. Variants predicted to evade LoF or to be potential artifacts were enriched for ClinVar benign variants. PVS1 was downgraded in 99.4% (162/163) of pLoF variants predicted as likely not LoF/not LoF, with 17.2% (28/163) downgraded as a result of our framework, adding to previous guidelines. Variant pathogenicity was affected (mostly from likely pathogenic to VUS) in 20 (71.4%) of these 28 variants. This framework guides assessment of pLoF variants beyond standard annotation pipelines and substantially reduces false positive rates, which is key to ensure accurate LoF variant prediction in both a research and clinical setting.

MOIT: A Novel task for mining opinions towards implicit targets

The extraction of opinions and their corresponding targets has gained significant interest recently, as it offers valuable insights into Opinion Mining (OM) at a granular level. Opinion and target terms to be extracted by existing OM tasks need to be explicitly present in reviews. Targets that are not present but implied in contextual semantics, are neglected by existing OM tasks, even though an investigation reported that about 60% of reviews contain implicit targets. To enable implicit target extraction, a novel task named Mining Opinions towards Implicit Targets (MOIT) under the fine-grained OM, is proposed to extract both opinions and their corresponding implicit targets, enabling a more comprehensive analysis of reviews. To set up the basis for follow-up research on MOIT, two large-scale datasets were constructed as resources in two languages, where the Chinese dataset was built from scratch via a standard human annotation process, and the English dataset was built semi-automatically through machine translation and manual checking. Furthermore, three baseline models adapting three representative paradigms of information extraction, namely sequence labeling, question answering, and text generation, were proposed to solve MOIT. Extensive experiments demonstrated the effectiveness of the models. The proposed MOIT task extends the field of OM research, and the datasets and models establish a foundation for future studies in this area.

Dual attention fusion UNet for COVID-19 lesion segmentation from CT images.

Chest CT scan is an effective way to detect and diagnose COVID-19 infection. However, features of COVID-19 infection in chest CT images are very complex and heterogeneous, which make segmentation of COVID-19 lesions from CT images quite challenging. To overcome this challenge, this study proposes and tests an end-to-end deep learning method called dual attention fusion UNet (DAF-UNet). The proposed DAF-UNet improves the typical UNet into an advanced architecture. The dense-connected convolution is adopted to replace the convolution operation. The mixture of average-pooling and max-pooling acts as the down-sampling in the encoder. Bridge-connected layers, including convolution, batch normalization, and leaky rectified linear unit (leaky ReLU) activation, serve as the skip connections between the encoder and decoder to bridge the semantic gap differences. A multiscale pyramid pooling module acts as the bottleneck to fit the features of COVID-19 lesion with complexity. Furthermore, dual attention feature (DAF) fusion containing channel and position attentions followed the improved UNet to learn the long-dependency contextual features of COVID-19 and further enhance the capacity of the proposed DAF-UNet. The proposed model is first pre-trained on the pseudo label dataset (generated by Inf-Net) containing many samples, then fine-tuned on the standard annotation dataset (provided by the Italian Society of Medical and Interventional Radiology) with high-quality but limited samples to improve performance of COVID-19 lesion segmentation on chest CT images. The Dice coefficient and Sensitivity are 0.778 and 0.798 respectively. The proposed DAF-UNet has higher scores than the popular models (Att-UNet, Dense-UNet, Inf-Net, COPLE-Net) tested using the same dataset as our model. The study demonstrates that the proposed DAF-UNet achieves superior performance for precisely segmenting COVID-19 lesions from chest CT scans compared with the state-of-the-art approaches. Thus, the DAF-UNet has promising potential for assisting COVID-19 disease screening and detection.