Ground Truth Labels Research Articles

An Unsupervised Anomaly Detection in Electricity Consumption Using Reinforcement Learning and Time Series Forest Based Framework

Abstract Anomaly detection (AD) plays a crucial role in time series applications, primarily because time series data is employed across real-world scenarios. Detecting anomalies poses significant challenges since anomalies take diverse forms making them hard to pinpoint accurately. Previous research has explored different AD models, making specific assumptions with varying sensitivity toward particular anomaly types. To address this issue, we propose a novel model selection for unsupervised AD using a combination of time series forest (TSF) and reinforcement learning (RL) approaches that dynamically chooses an AD technique. Our approach allows for effective AD without explicitly depending on ground truth labels that are often scarce and expensive to obtain. Results from the real-time series dataset demonstrate that the proposed model selection approach outperforms all other AD models in terms of the F1 score metric. For the synthetic dataset, our proposed model surpasses all other AD models except for KNN, with an impressive F1 score of 0.989. The proposed model selection framework also exceeded the performance of GPT-4 when prompted to act as an anomaly detector on the synthetic dataset. Exploring different reward functions revealed that the original reward function in our proposed AD model selection approach yielded the best overall scores. We evaluated the performance of the six AD models on an additional three datasets, having global, local, and clustered anomalies respectively, showing that each AD model exhibited distinct performance depending on the type of anomalies. This emphasizes the significance of our proposed AD model selection framework, maintaining high performance across all datasets, and showcasing superior performance across different anomaly types.

Automated Assessment of Sarcopenia with Hounsfield Unit Average Calculation in Computed Tomography Scans Using Deep Learning Techniques

Abstract Introduction Skeletal muscle is increasingly plastic with an ability to gain or lose tissue. Depletion of muscle mass and quality occurs due to various factors such as aging, disease, and disuse. Sarcopenia can be loosely defined as a significant loss of muscle mass and function. Sarcopenia is now recognized as an independent risk factor for various patient-related negative outcomes after various surgeries. Various computed tomography (CT) based imaging indices for assessment of sarcopenia exist in practice. The psoas muscle Hounsfield unit average calculation (HUAC) has been proven to be an effective one as it is independent of patient anthropometric data, and it can be calculated in the images provided. Aim The aim of this study is to develop automated tools for estimation of the HUAC using deep learning algorithms. Materials and Methods A total of 41 abdominal CTs were used. Ground truth was established and validated by two radiologists with more than 5 and 10 years of experience each. Models were trained to identify the psoas muscle among the slices and calculate the HUAC. Results At inference, an average intersection over union (IoU) value of 90% was obtained between the deep learning model outputs and the original annotated test images for the CT slices. The Dice coefficient was 0.90 between the ground truth labels and the output from the model. Conclusion We have demonstrated the accuracy of our deep learning–based algorithm for quantifying the psoas muscle HUAC, which is a marker for sarcopenia. There is a potential for a fully automated measure to calculate the HUAC for any patient undergoing CT scan.

Boosting grape bunch detection in RGB-D images using zero-shot annotation with Segment Anything and GroundingDINO

Latest advances in artificial intelligence, particularly in object recognition and segmentation, provide unprecedented opportunities for precision agriculture. This work investigates the use of state-of-the-art AI models, namely Meta’s Segment Anything (SAM) and GroundingDino, for the task of grape cluster detection in vineyards. Three different methods aimed at enhancing the instance segmentation process are proposed: (i) SAM-Refine (SAM-R), which refines a previously proposed depth-based clustering approach, referred to as DepthSeg, using SAM; (ii) SAM-Segmentation (SAM-S), which integrates SAM with a pre-trained semantic segmentation model to improve cluster separation; and (iii) AutoSAM-Dino (ASD), which eliminates the need for manual labeling and transfer learning through the combined use of GroundingDino and SAM. Analysis is conducted on both the object counting and pixel-level segmentation accuracy against a manually labeled ground truth. Metrics such as mean Average Precision (mAP), Intersection over Union (IoU), and precision and recall are calculated to assess the system performance. Compared to the original DepthSeg algorithm, SAM-R slightly advances object counting (mAP: +0.5%) and excels in pixel-level segmentation (IoU: +17.0%). SAM-S, despite a mAP decrease, improves segmentation accuracy (IoU: +13.9%, Precision: +9.2%, Recall: +11.7%). Similarly, ASD, although with a lower mAP, shows significant accuracy enhancement (IoU: +7.8%, Precision: +4.2%, Recall: +4.9%). Additionally, from a labor effort point of view, instance segmentation techniques require much less time for training than manual labeling.

Quantification of Empty Lacunae in Tissue Sections of Osteonecrosis of the Femoral Head Using YOLOv8 Artificial Intelligence Model

ABSTRACTHistomorphometry is an important technique in the evaluation of non‐traumatic osteonecrosis of the femoral head (ONFH). Quantification of empty lacunae and pyknotic cells on histological images is the most reliable measure of ONFH pathology, yet it is time and manpower consuming. This study focused on the application of artificial intelligence (AI) technology to tissue image evaluation. The aim of this study is to establish an automated cell counting platform using YOLOv8 as an object detection model on ONFH tissue images and to evaluate and validate its accuracy. From 30 ONFH model rabbits, 270 tissue images were prepared; based on evaluations by three researchers, ground truth labels were created to classify each cell in the image into two classes (osteocytes and empty lacunae) or three classes (osteocytes, pyknotic cells, and empty lacunae). Two and three classes were then annotated on each image. Transfer learning based on annotated data (80% for training and 20% for validation) was performed using YOLOv8n and YOLOv8x with different parameters. To evaluate the detection accuracy of the training model, the mean average precision (mAP (50)) and precision‐recall curve were identified. In addition, the reliability of cell counting by YOLOv8 relative to manual cell counting was evaluated by linear regression analysis using five histological images unused in previous experiments. The mAP (50) for the detection of empty lacunae was 0.868 for the YOLOv8n and 0.883 for the YOLOv8x. The mAP (50) for the three classes was 0.735 for the YOLOv8n model and 0.750 for the YOLOv8x model. The quantification of empty lacunae by automated cell counting obtained in the learning was highly correlated with the manual counting data. The development of an AI‐applied automated cell counting platform will significantly reduce the time and effort of manual cell counting in histological analysis.

Automatic Detection and Classification of Aurora in THEMIS All‐Sky Images

AbstractWe report a novel machine‐learning algorithm for automatically detecting and classifying aurora in all–sky images (ASI) that is largely trained without requiring ground–truth labels. By including a small number of labeled images, we are able to automatically label all of the approximately 700 million images in the Time History of Events and Macroscale Interactions during Substorms (THEMIS) ASI data set from 2008 to 2022. We use a two–stage approach. In the first stage, we adapt the Simple framework for Contrastive Learning of Representations (SimCLR) algorithm to learn latent representations of THEMIS all–sky images. We then finetune a classifier network on the latent representations our model learns of the manually labeled Oslo aurora THEMIS (OATH) data set. We demonstrate that this two–stage approach achieves excellent classification results on data for which there is no current ML classification benchmark. The outcome of this work will facilitate efficient information retrieval for researchers interested in specific categories of aurora and will enable large scale statistical studies and machine learning analyses of THEMIS all–sky images that have not previously been possible. To demonstrate possible ways to utilize this database, we performed a statistical analysis of the occurrence rates of auroral labels with respect to solar wind parameters, interplanetary magnetic field vector, and geomagnetic indices. We further investigate the occurrence rates of auroral phenomena in the annotated data set and their geoeffectiveness by utilizing the co–located THEMIS ground magnetometer data set.

Self-Supervised Adversarial Training of Monocular Depth Estimation Against Physical-World Attacks.

Monocular Depth Estimation (MDE) plays a vital role in applications such as autonomous driving. However, various attacks target MDE models, with physical attacks posing significant threats to system security. Traditional adversarial training methods, which require ground-truth labels, are not directly applicable to MDE models that lack ground-truth depth. Some self-supervised model hardening techniques (e.g., contrastive learning) overlook the domain knowledge of MDE, resulting in suboptimal performance. In this work, we introduce a novel self-supervised adversarial training approach for MDE models, leveraging view synthesis without the need for ground-truth depth. We enhance adversarial robustness against real-world attacks by incorporating L0-norm-bounded perturbation during training. We evaluate our method against supervised learning-based and contrastive learning-based approaches specifically designed for MDE. Our experiments with two representative MDE networks demonstrate improved robustness against various adversarial attacks, with minimal impact on benign performance.

Validation of Deep Learning-based Automatic Retinal Layer Segmentation Algorithms for AMD with Two SD-OCT Devices

PurposeSegmentations of retinal layers in spectral-domain optical coherence tomography (SD-OCT) images serve as a crucial tool for identifying and analyzing the progression of various retinal diseases, encompassing a broad spectrum of abnormalities associated with age-related macular degeneration (AMD). The training of deep-learning algorithms necessitates well-defined ground-truth labels, validated by experts, to delineate boundaries accurately. However, this resource-intensive process has constrained the widespread application of such algorithms across diverse OCT devices. This work validates deep learning image segmentation models across multiple OCT devices by testing robustness in generating clinically relevant metrics. DesignProspective, comparative study. ParticipantsAdults over 50 years of age with no AMD to advanced AMD, as defined in the Age-Related Eye Disease Study (AREDS)), in at least one eye, were enrolled. 402 SD-OCT scans were used in this study. MethodsWe evaluate two separate state-of-the-art segmentation algorithms through a training process using images obtained from one OCT device (Heidelberg-Spectralis) and subsequent testing using images acquired from two OCT devices (Heidelberg-Spectralis and Zeiss-Cirrus). This assessment is performed on a dataset that encompasses a range of retinal pathologies, spanning from disease-free conditions to severe forms of AMD, with a focus on evaluating the device independence of the algorithms. Main Outcome MeasuresPerformance metrics (mean-squared-error, mean-absolute-error, dice-coefficients) for the segmentations of the internal limiting membrane (ILM), retinal-pigment-epithelium (RPE), and RPE to Bruch’s-membrane (BM) region, along with en face thickness maps, volumetric estimations (in mm3). Violin plots and Bland-Altman plots comparing predictions against ground-truth are also presented. ResultsThe UNet and DeepLabv3, trained on Spectralis B-scans, demonstrate clinically useful outcomes when applied to Cirrus test B-scans. Review of the Cirrus test-data by two independent annotators revealed that the aggregated-mean-absolute-error in pixels for ILM was 1.82±0.24 (equivalent to 7.0±0.9 μm) and for RPE was 2.46±0.66 (9.5±2.6 μm). Additionally, the dice-similarity-coefficient for the RPE-drusen complex (RPE-DC) region, comparing predictions to ground truth, reached 0.87±0.01. ConclusionIn the pursuit of task-specific goals such as retinal layer segmentation, a segmentation network has the capacity to acquire domain-independent features from a large training dataset. This enables the utilization of the network to execute tasks in domains where ground truth is hard to generate.

Classification of Pneumonia, Tuberculosis and Covid-19 from Chest X-Ray Images Using Convolution Neural Network Model

Accurate and timely diagnosis of respiratory ailments like pneumonia, tuberculosis (TB), and COVID-19 is pivotal for effective patient care and public health interventions. Deep learning algorithms have emerged as potent tools in medical image classification, offering promise for automated diagnosis and screening. This study presents a deep learning-based approach for categorizing chest X-ray images into three classes: pneumonia, tuberculosis, and COVID-19. Utilizing convolutional neural networks (CNNs) as the primary architecture, owing to their ability to automatically extract relevant features from raw image data. The proposed model is trained on a sizable dataset of chest X-ray images annotated with ground truth labels for pneumonia, TB, and COVID-19. Extensive experiments are conducted to evaluate the model's performance in terms of classification accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve (AUC-ROC). Furthermore, we compare the performance of our deep learning model with traditional machine learning approaches and assess its generalization ability on an independent test set. Our findings demonstrate that the proposed deep learning model achieves high accuracy in classifying chest X-ray images of pneumonia, TB, and COVID-19, outperforming traditional methods and showing potential for clinical deployment as a screening tool, especially in resource-limited settings.

Closing the gap in domain adaptation for semantic segmentation: a time-aware method

Semantic segmentation models need a large number of images to be effectively trained but manual annotation of such images has a high cost. Active domain adaptation addresses this problem by pretraining the model with a synthetically generated dataset and then fine-tuning it with a few selected label annotations (the “budget”) on real images to account for the domain shift. Previous works annotate a percentage of either individual pixels or whole target images. We argue that the first is infeasible in practice, and the second spends part of the budget on classes that the pretrained model may have already learned well. We propose a method based on the annotation of regions computed by Segment Anything, a recently introduced foundation model for class-agnostic image segmentation. The key idea is to assign a ground truth label to each of a tiny subset of regions, those for which the model is more uncertain. In order to increase the number of annotated regions we propagate the ground truth labels to most similar regions according to a hierarchical clustering algorithm that uses the features learned by the pretrained model. Our method outperforms the state-of-the-art on the GTA5 to Cityscapes benchmark by using fewer annotations, almost closing the gap between the synthetically pre-trained model and that obtained with full supervision of the real images. Furthermore, we present competitive results for budgets less than 1% of samples and also for a larger and more challenging target dataset, Mapillary Vistas.

ScVAG: Unified single-cell clustering via variational-autoencoder integration with Graph Attention Autoencoder

Single-cell RNA sequencing (scRNA-seq) enables high-resolution transcriptional profiling of cell heterogeneity. However, analyzing this noisy, high-dimensional matrix remains challenging. We present scVAG, an integrated deep learning framework combining Variational-Autoencoder (VAE) and Graph Attention Autoencoder (GATE) for enhanced single-cell clustering. Building upon scGAC, our approach replaces its restrictive linear principal component analysis (PCA) with nonlinear dimensionality reduction better suited for scRNA-seq data. Specifically, we integrate VAE and GATE to enable more flexible latent space encoding. Extensive experiments on 20 datasets demonstrate scVAG's superior performance over previous state-of-the-art methods including scGAC, SCEA, SC3, Seurat, scGNN, scASGC, DESC, NIC, scLDS2, DRJCC, sLMIC, and jSRC. On average, scVAG improves clustering accuracy by 5 percent in ARI and 4 percent in NMI parameters. Visualizations highlight scVAG's capacity to recover interpretable biological structures. Our VAE-GATE pipeline extracts intricate expression patterns into compact representations that precisely delineate cell subpopulations consistent with ground truth labels. Overall, scVAG establishes a robust architecture for elucidating cell taxonomies from noisy transcriptomic inputs.

Motion-Aware Self-Supervised RGBT Tracking with Multi-Modality Hierarchical Transformers

Supervised RGBT (SRGBT) tracking tasks need both expensive and time-consuming annotations. Therefore, the implementation of Self-Supervised RGBT (SSRGBT) tracking methods has become increasingly important. Straightforward SSRGBT tracking methods use pseudo-labels for tracking, but inaccurate pseudo-labels can lead to object drift, which severely affects tracking performance. This article proposes a self-supervised RGBT object tracking method (S2OTFormer) to bridge the gap between tracking methods supervised under pseudo-labels and ground truth labels. Firstly, to provide more robust appearance features for motion cues, we introduce a multi-modality hierarchical transformer (MHT) module for feature fusion. This module allocates weights to both modalities and strengthens the expressive capability of the MHT module through multiple nonlinear layers to fully utilize the complementary information of the two modalities. Secondly, in order to solve the problems of motion blur caused by camera motion and inaccurate appearance information caused by pseudo-labels, we introduce a motion-aware mechanism (MAM). The MAM extracts the average motion vectors from the previous multi-frame search frame features and constructs the consistency loss with the motion vectors of the current search frame features. The motion vectors of inter-frame objects are obtained by reusing the inter-frame attention map to predict coordinate positions. Finally, to further reduce the effect of inaccurate pseudo-labels, we propose an Attention-Based Multi-Scale Enhancement Module. By introducing cross-attention to achieve more precise and accurate object tracking, this module overcomes the receptive field limitations of traditional CNN tracking heads. We demonstrate the effectiveness of S2OTFormer on four large-scale public datasets through extensive comparisons as well as numerous ablation experiments. The source code is available at https://github.com/LiShenglana/S2OTFormer .

Performance of an AI-powered visualization software platform for precision surgery in breast cancer patients

Surgery remains the primary treatment modality in the management of early-stage invasive breast cancer. Artificial intelligence (AI)-powered visualization platforms offer the compelling potential to aid surgeons in evaluating the tumor’s location and morphology within the breast and accordingly optimize their surgical approach. We sought to validate an AI platform that employs dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) to render three-dimensional (3D) representations of the tumor and 5 additional chest tissues, offering clear visualizations as well as functionalities for quantifying tumor morphology, tumor-to-landmark structure distances, excision volumes, and approximate surgical margins. This retrospective study assessed the visualization platform’s performance on 100 cases with ground-truth labels vetted by 2 breast-specialized radiologists. We assessed features including automatic AI-generated clinical metrics (e.g., tumor dimensions) as well as visualization tools including convex hulls at desired margins around the tumor to help visualize lumpectomy volume. The statistical performance of the platform’s automated features was robust and within the range of inter-radiologist variability. These detailed 3D tumor and surrounding multi-tissue depictions offer both qualitative and quantitative comprehension of cancer topology and may aid in formulating an optimal surgical approach for breast cancer treatment. We further establish the framework for broader data integration into the platform to enhance precision cancer care.

Gaze Zone Classification for Driving Studies Using YOLOv8 Image Classification.

Gaze zone detection involves estimating where drivers look in terms of broad categories (e.g., left mirror, speedometer, rear mirror). We here specifically focus on the automatic annotation of gaze zones in the context of road safety research, where the system can be tuned to specific drivers and driving conditions, so that an easy to use but accurate system may be obtained. We show with an existing dataset of eye region crops (nine gaze zones) and two newly collected datasets (12 and 10 gaze zones) that image classification with YOLOv8, which has a simple command line interface, achieves near-perfect accuracy without any pre-processing of the images, as long as a model is trained on the driver and conditions for which annotation is required (such as whether the drivers wear glasses or sunglasses). We also present two apps to collect the training images and to train and apply the YOLOv8 models. Future research will need to explore how well the method extends to real driving conditions, which may be more variable and more difficult to annotate for ground truth labels.

Abstract 4135928: Externally Validated Deep Learning Model for Patent Ductus Arteriosus Detection by Echocardiography in Preterm Infants

Introduction: Patent ductus arteriosus (PDA) is a common form of congenital heart disease in premature infants and a source of significant morbidity. Echocardiography is the mainstay modality for diagnosis and monitoring but is resource-intensive and requires expertise for interpretation. Aims: To develop and externally validate a deep learning model to identify PDA in preterm infants by echocardiography. Methods: We trained, validated, and tested a convolutional neural network to detect PDA in raw echocardiogram clips from infants (gestational age <37 weeks) with and without PDAs and no significant additional congenital heart disease at the Medical University of South Carolina (MUSC). For training and validation, we employed a 12-fold cross-validation procedure. Experienced cardiologists defined the ground truth labels. We assessed classification performance on an internal model naïve dataset using the area under the receiver operator curve (AUROC), sensitivity, specificity, positive predictive value, and negative predictive value. For external multicenter testing, we used studies from Children’s Hospital of Orange County (CHOC) and Boston Children’s Hospital (BCH), both of which utilize a different ultrasound system vendor. Results: Training and validation were completed using 1,145 color/color-compare clips (661 clips with PDA and 484 no PDA) from 66 patients. The internal test dataset consisted of 142 clips from 16 patients. The external CHOC and BCH cohorts consisted of 146 clips from 50 patients and 294 clips from 68 patients, respectively. Model performance was similar for internal and external testing (Table 1). Conclusions: Our deep learning model detected the presence of PDA with acceptable performance. This performance generalized well across multiple institutions and ultrasound vendors. This work shows promise for the future development of an automated tool for PDA diagnosis enabling the democratization of pediatric cardiology resources.

Development of a Dual-Plane MRI-Based Deep Learning Model to Assess the 1-Year Postoperative Outcomes in Lumbar Disc Herniation After Tubular Microdiscectomy.

Tubular microdiscectomy (TMD) is a treatment for lumbar disc herniation (LDH). Although the combination of MRI and deep learning (DL) has shown promise, its application in evaluating postoperative outcomes in TMD has not been fully explored. To evaluate whether integrating preoperative dual-plane MRI-based DL features with clinical features can assess 1-year outcomes in TMD for LDH. Retrospective. The study involved 548 patients who underwent TMD between January 2016 and January 2021. Training set (N = 305, mean age 51.85 ± 13.84 years, 56.4% male). Internal validation set (N = 131, mean age 51.85 ± 13.84 years, 54.2% male). External validation set (N = 112, mean age 51.54 ± 14.43 years, 50.9% male). 3 T MRI with sagittal and transverse T2-weighted sequences (Fast Spin Echo). Ground truth labels were based on improvement rate in 1-year Japanese Orthopaedic Association (JOA) scores. Information on 42 preoperative clinical features was collected. The largest protrusions were identified from T2 MRI by three clinicians and were used to train deep learning models (ResNet50, ResNet101, and ResNet152) to extract DL features. After feature selection, three models were built, namely, clinical, DL, and combined models. Chi-square or Fisher's exact tests was used for group comparisons. Quantitative differences were analyzed using the t-test or Mann-Whitney U test. P-values <0.05 were considered significant. Models were validated on internal and external datasets using metrics such as the area under the curve (AUC). The AUCs of the clinical models achieved 0.806 (internal) and 0.779 (external). ResNet152 performed best in three DL models, with AUCs of 0.858 (internal) and 0.834 (external). The combined model achieved AUCs of 0.889 (internal) and 0.857 (external). A model combining preoperative dual-plane MRI DL features and clinical features can assess 1-year outcomes of TMD for LDH. 4 TECHNICAL EFFICACY: Stage 2.

Annotation-free multi-organ anomaly detection in abdominal CT using free-text radiology reports: A multi-centre retrospective study

Artificial intelligence (AI) systems designed to detect abnormalities in abdominal computed tomography (CT) could reduce radiologists' workload and improve diagnostic processes. However, development of such models has been hampered by the shortage of large expert-annotated datasets. Here, we used information from free-text radiology reports, rather than manual annotations, to develop a deep-learning-based pipeline for comprehensive detection of abdominal CT abnormalities. In this multicentre retrospective study, we developed a deep-learning-based pipeline to detect abnormalities in the liver, gallbladder, pancreas, spleen, and kidneys. Abdominal CT exams and related free-text reports obtained during routine clinical practice collected from three institutions were used for training and internal testing, while data collected from six institutions were used for external testing. A multi-organ segmentation model and an information extraction schema were used to extract specific organ images and disease information, CT images and radiology reports, respectively, which were used to train a multiple-instance learning model for anomaly detection. Its performance was evaluated using the area under the receiver operating characteristic curve (AUC), accuracy, sensitivity, specificity, and F1 score against radiologists' ground-truth labels. We trained the model for each organ on images selected from 66,684 exams (39,255 patients) and tested it on 300 (295 patients) and 600 (596 patients) exams for internal and external validation, respectively. In the external test cohort, the overall AUC for detecting organ abnormalities was 0.886. Whereas models trained on human-annotated labels performed better with the same number of exams, those trained on larger datasets with labels auto-extracted via the information extraction schema significantly outperformed human-annotated label-derived models. Using disease information from routine clinical free-text radiology reports allows development of accurate anomaly detection models without requiring manual annotations. This approach is applicable to various anatomical sites and could streamline diagnostic processes. Japan Science and Technology Agency.

Automated dentition segmentation: 3D UNet-based approach with MIScnn framework

Advancements in technology have led to the adoption of digital workflows in dentistry, which require the segmentation of regions of interest from cone-beam computed tomography (CBCT) scans. These segmentations assist in diagnosis, treatment planning, and research. However, manual segmentation is an expensive and labor-intensive process. Therefore, automated methods, such as convolutional neural networks (CNNs), provide a more efficient way to generate segmentations from CBCT scans. A three-dimensional UNet-based CNN model, utilizing the Medical Image Segmentation CNN framework, was used for training and generating predictions from CBCT scans. A dataset of 351 CBCT scans, with ground-truth labels created through manual segmentation using AI-assisted segmentation software, was prepared. Data preprocessing, augmentation, and model training were performed, and the performance of the proposed CNN model was analyzed. The CNN model achieved high accuracy in segmenting maxillary and mandibular teeth from CBCT scans, with average Dice Similarity Coefficient values of 91.83% and 91.35% for maxillary and mandibular teeth, respectively. Performance metrics, including Intersection over Union, precision, and recall, further confirmed the model's effectiveness. The study demonstrates the efficacy of the three-dimensional UNet-based CNN model within the Medical Image Segmentation CNN framework for automated segmentation of maxillary and mandibular dentition from CBCT scans. Automated segmentation using CNNs has the potential to deliver accurate and efficient results, offering a significant advantage over traditional segmentation methods.

Computational staining of CD3/CD20 positive lymphocytes in human tissues with experimental confirmation in a genetically engineered mouse model.

Immune dysregulation plays a major role in cancer progression. The quantification of lymphocytic spatial inflammation may enable spatial system biology, improve understanding of therapeutic resistance, and contribute to prognostic imaging biomarkers. In this paper, we propose a knowledge-guided deep learning framework to measure the lymphocytic spatial architecture on human H&E tissue, where the fidelity of training labels is maximized through single-cell resolution image registration of H&E to IHC. We demonstrate that such an approach enables pixel-perfect ground-truth labeling of lymphocytes on H&E as measured by IHC. We then experimentally validate our technique in a genetically engineered, immune-compromised Rag2 mouse model, where Rag2 knockout mice lacking mature lymphocytes are used as a negative experimental control. Such experimental validation moves beyond the classical statistical testing of deep learning models and demonstrates feasibility of more rigorous validation strategies that integrate computational science and basic science. Using our developed approach, we automatically annotated more than 111,000 human nuclei (45,611 CD3/CD20 positive lymphocytes) on H&E images to develop our model, which achieved an AUC of 0.78 and 0.71 on internal hold-out testing data and external testing on an independent dataset, respectively. As a measure of the global spatial architecture of the lymphocytic microenvironment, the average structural similarity between predicted lymphocytic density maps and ground truth lymphocytic density maps was 0.86 ± 0.06 on testing data. On experimental mouse model validation, we measured a lymphocytic density of 96.5 ± %1% in a Rag2+/- control mouse, compared to an average of 16.2 ± %5% in Rag2-/- immune knockout mice (p<0.0001, ANOVA-test). These results demonstrate that CD3/CD20 positive lymphocytes can be accurately detected and characterized on H&E by deep learning and generalized across species. Collectively, these data suggest that our understanding of complex biological systems may benefit from computationally-derived spatial analysis, as well as integration of computational science and basic science.

Necessity and impact of specialization of large foundation model for medical segmentation tasks.

Large foundation models, such as the Segment Anything Model (SAM), have shown remarkable performance in image segmentation tasks. However, the optimal approach to achieve true utility of these models for domain-specific applications, such as medical image segmentation, remains an open question. Recent studies have released a medical version of the foundation model MedSAM by training on vast medical data, who promised SOTA medical segmentation. Independent community inspection and dissection is needed. Foundation models are developed for general purposes. On the other hand, stable delivery of reliable performance is key to clinical utility. This study aims at elucidating the potential advantage and limitations of landing the foundation models in clinical use by assessing the performance of off-the-shelf medical foundation model MedSAM for the segmentation of anatomical structures in pelvic MR images. We also explore the simple remedies by evaluating the dependency on prompting scheme. Finally, we demonstrate the need and performance gain of further specialized fine-tuning. MedSAM and its lightweight version LiteMedSAM were evaluated out-of-the-box on a public MR dataset consisting of 589 pelvic images split 80:20 for training and testing. An nnU-Net model was trained from scratch to serve as a benchmark and to provide bounding box prompts for MedSAM. MedSAM was evaluated using different quality bounding boxes, those derived from ground truth labels, those derived from nnU-Net, and those derived from the former two but with 5-pixel isometric expansion. Lastly, LiteMedSAM was refined on the training set and reevaluated on this task. Out-of-the-box MedSAM and LiteMedSAM both performed poorly across the structure set, especially for disjoint or non-convex structures. Varying prompt with different bounding box inputs had minimal effect. For example, the mean Dice score and mean Hausdorff distances (in mm) for obturator internus using MedSAM and LiteMedSAM were {0.251±0.110, 0.101±0.079} and {34.142±5.196, 33.688±5.306}, respectively. Fine-tuning of LiteMedSAM led to significant performance gain, improving Dice score and Hausdorff distance for the obturator internus to 0.864±0.123 and 5.022±10.684, on par with nnU-Net with no significant difference in evaluation of most structures. All segmentation structures benefited significantly from specialized refinement, at varying improvement margin. While our study alludes to the potential of deep learning models like MedSAM and LiteMedSAM for medical segmentation, it highlights the need for specialized refinement and adjudication. Off-the-shelf use of such large foundation models is highly likely to be suboptimal, and specialized fine-tuning is often necessary to achieve clinical desired accuracy and stability.

Optimizing Football Formation Analysis via LSTM-Based Event Detection

The process of manually annotating sports footage is a demanding one. In American football alone, coaches spend thousands of hours reviewing and analyzing videos each season. We aim to automate this process by developing a system that generates comprehensive statistical reports from full-length football game videos. Having previously demonstrated the proof of concept for our system, here, we present optimizations to our preprocessing techniques along with an inventive method for multi-person event detection in sports videos. Employing a long short-term memory (LSTM)-based architecture to detect the snap in American football, we achieve an outstanding LSI (Levenshtein similarity index) of 0.9445, suggesting a normalized difference of less than 0.06 between predictions and ground truth labels. We also illustrate the utility of snap detection as a means of identifying the offensive players’ assuming of formation. Our results exhibit not only the success of our unique approach and underlying optimizations but also the potential for continued robustness as we pursue the development of our remaining system components.