U-Net Model Research Articles

HyPhAICC v1.0: a hybrid physics–AI approach for probability fields advection shown through an application to cloud cover nowcasting

Abstract. This work proposes a hybrid approach that combines physics and artificial intelligence (AI) for cloud cover nowcasting. It addresses the limitations of traditional deep-learning methods in producing realistic and physically consistent results that can generalise to unseen data. The proposed approach, named HyPhAICC, enforces a physical behaviour. In the first model, denoted as HyPhAICC-1, a multi-level advection dynamics is considered a hard constraint for a trained U-Net model. Our experiments show that the hybrid formulation outperforms not only traditional deep-learning methods but also the EUMETSAT Extrapolated Imagery model (EXIM) in terms of both qualitative and quantitative results. In particular, we illustrate that the hybrid model preserves more details and achieves higher scores based on similarity metrics in comparison to U-Net. Remarkably, these improvements are achieved while using only one-third of the data required by the other models. Another model, denoted as HyPhAICC-2, adds a source term to the advection equation, it impaired the visual rendering but displayed the best performance in terms of accuracy. These results suggest that the proposed hybrid physics–AI architecture provides a promising solution to overcome the limitations of classical AI methods and contributes to open up new possibilities for combining physical knowledge with deep-learning models.

A model integration approach for stratigraphic boundary delineation based on local data augmentation

Identification of stratigraphic boundaries is a fundamental task in the seismic interpretation of oil and gas reservoir locations. When employing deep learning techniques to interpret stratigraphic boundaries, insufficient training data and sample imbalances are common challenges affecting model training. In regions with intricate geological structural changes, conventional deep-learning segmentation algorithms, such as U-Net often struggle to accurately capture the features of complex local structures. To address these limitations, we propose a model integration approach that incorporates global and local uneven-type stratigraphic data augmentation to enhance the accuracy of stratigraphic boundary identification in uneven-type regions. To address the problems of class imbalance and insufficient complex variation samples, we adopted a strategy of separately training global and local data and integrating predictions, thereby handling the disparity between uneven-type and flat-type stratigraphic data during model training. By testing the Netherlands F3 dataset with sparsely labeled profiles, it was demonstrated that the proposed method can effectively improve the delineation accuracy of stratigraphic boundaries compared to the benchmark U-Net model.

AxonFinder: Automated segmentation of tumor innervating neuronal fibers.

Neurosignaling is increasingly recognized as a critical factor in cancer progression, where neuronal innervation of primary tumors contributes to the disease's advancement. This study focuses on segmenting individual axons within the prostate tumor microenvironment, which have been challenging to detect and analyze due to their irregular morphologies. We present a novel deep learning-based approach for the automated segmentation of axons, AxonFinder, leveraging a U-Net model with a ResNet-101 encoder, based on a multiplexed imaging approach. Utilizing a dataset of whole-slide images from low-, intermediate-, and high-risk prostate cancer patients, we manually annotated axons to train our model, achieving significant accuracy in detecting axonal structures that were previously hard to segment. Our analysis includes a comprehensive assessment of axon density and morphological features across different CAPRA-S prostate cancer risk categories, providing insights into the correlation between tumor innervation and cancer progression. Our paper suggests the potential utility of neuronal markers in the prognostic assessment of prostate cancer in aiding the pathologist's assessment of tumor sections and advancing our understanding of neurosignaling in the tumor microenvironment.

Development, deployment and scaling of operating room-ready artificial intelligence for real-time surgical decision support

Deep learning for computer vision can be leveraged for interpreting surgical scenes and providing surgeons with real-time guidance to avoid complications. However, neither generalizability nor scalability of computer-vision-based surgical guidance systems have been demonstrated, especially to geographic locations that lack hardware and infrastructure necessary for real-time inference. We propose a new equipment-agnostic framework for real-time use in operating suites. Using laparoscopic cholecystectomy and semantic segmentation models for predicting safe/dangerous (“Go”/”No-Go”) zones of dissection as an example use case, this study aimed to develop and test the performance of a novel data pipeline linked to a web-platform that enables real-time deployment from any edge device. To test this infrastructure and demonstrate its scalability and generalizability, lightweight U-Net and SegFormer models were trained on annotated frames from a large and diverse multicenter dataset from 136 institutions, and then tested on a separate prospectively collected dataset. A web-platform was created to enable real-time inference on any surgical video stream, and performance was tested on and optimized for a range of network speeds. The U-Net and SegFormer models respectively achieved mean Dice scores of 57% and 60%, precision 45% and 53%, and recall 82% and 75% for predicting the Go zone, and mean Dice scores of 76% and 76%, precision 68% and 68%, and recall 92% and 92% for predicting the No-Go zone. After optimization of the client-server interaction over the network, we deliver a prediction stream of at least 60 fps and with a maximum round-trip delay of 70 ms for speeds above 8 Mbps. Clinical deployment of machine learning models for surgical guidance is feasible and cost-effective using a generalizable, scalable and equipment-agnostic framework that lacks dependency on hardware with high computing performance or ultra-fast internet connection speed.

Automated segmentation and classification of supraspinatus fatty infiltration in shoulder magnetic resonance image using a convolutional neural network.

Goutallier's fatty infiltration of the supraspinatus muscle is a critical condition in degenerative shoulder disorders. Deep learning research primarily uses manual segmentation and labeling to detect this condition. Employing unsupervised training with a hybrid framework of segmentation and classification could offer an efficient solution. To develop and assess a two-step deep learning model for detecting the region of interest and categorizing the magnetic resonance image (MRI) supraspinatus muscle fatty infiltration according to Goutallier's scale. A retrospective study was performed from January 1, 2019 to September 20, 2020, using 900 MRI T2-weighted images with supraspinatus muscle fatty infiltration diagnoses. A model with two sequential neural networks was implemented and trained. The first sub-model automatically detects the region of interest using a U-Net model. The second sub-model performs a binary classification using the VGG-19 architecture. The model's performance was computed as the average of five-fold cross-validation processes. Loss, accuracy, Dice coefficient (CI. 95%), AU-ROC, sensitivity, and specificity (CI. 95%) were reported. Six hundred and six shoulders MRIs were analyzed. The Goutallier distribution was presented as follows: 0 (66.50%); 1 (18.81%); 2 (8.42%); 3 (3.96%); 4 (2.31%). Segmentation results demonstrate high levels of accuracy (0.9977 ± 0.0002) and Dice score (0.9441 ± 0.0031), while the classification model also results in high levels of accuracy (0.9731 ± 0.0230); sensitivity (0.9000 ± 0.0980); specificity (0.9788 ± 0.0257); and AUROC (0.9903 ± 0.0092). The two-step training method proposed using a deep learning model demonstrated strong performance in segmentation and classification tasks.

Optimizing Glaucoma Diagnosis with Deep Learning-Based Segmentation and Classification of Retinal Images

Glaucoma, a leading cause of permanent blindness worldwide, necessitates early detection to prevent vision loss, a task that is challenging and time-consuming when performed manually. This study proposes an automatic glaucoma detection method on enhanced retinal images using deep learning. The system analyzes retinal images, generating masks for the optic disc and optic cup, and providing a classification for glaucoma diagnosis. We employ a U-Net architecture with a pretrained residual neural network (ResNet34) for segmentation and an EfficientNetB0 for classification. The proposed framework is tested on publicly available datasets, including ORIGA, REFUGE, RIM-ONE DL, and HRF. Our work evaluated the U-Net model with five pretrained backbones (ResNet34, ResNet50, VGG19, DenseNet121, and EfficientNetB0) and examined preprocessing effects. We optimized model training with limited data using transfer learning and data augmentation techniques. The segmentation model achieves a mean intersection over union (mIoU) value of 0.98. The classification model shows remarkable performance with 99.9% training and 100% testing accuracy on ORIGA, 99.9% training and 99% testing accuracy on RIM-ONE DL, and 98% training and 100% testing accuracy on HRF. The proposed model outperforms related works and demonstrates potential for accurate glaucoma classification and detection tasks.

Beyond PhacoTrainer: Deep Learning for Enhanced Trabecular Meshwork Detection in MIGS Videos.

The purpose of this study was to develop deep learning models for surgical video analysis, capable of identifying minimally invasive glaucoma surgery (MIGS) and locating the trabecular meshwork (TM). For classification of surgical steps, we had 313 video files (265 for cataract surgery and 48 for MIGS procedures), and for TM segmentation, we had 1743 frames (1110 for TM and 633 for no TM). We used transfer learning to update a classification model pretrained to recognize standard cataract surgical steps, enabling it to also identify MIGS procedures. For TM localization, we developed three different models: U-Net, Y-Net, and Cascaded. Segmentation accuracy for TM was measured by calculating the average pixel error between the predicted and ground truth TM locations. Using transfer learning, we developed a model which achieved 87% accuracy for MIGS frame classification, with area under the receiver operating characteristic curve (AUROC) of 0.99. This model maintained a 79% accuracy for identifying 14 standard cataract surgery steps. The overall micro-averaged AUROC was 0.98. The U-Net model excelled in TM segmentation with an Intersection over union (IoU) score of 0.9988 and an average pixel error of 1.47. Building on prior work developing computer vision models for cataract surgical video, we developed models that recognize MIGS procedures and precisely localize the TM with superior performance. Our work demonstrates the potential of transfer learning for extending our computer vision models to new surgeries without the need for extensive additional data collection. Computer vision models in surgical videos can underpin the development of systems offering automated feedback for trainees, improving surgical training and patient care.

Multi-label deep learning for comprehensive optic nerve head segmentation through data of fundus images

Early diagnosis and continuous monitoring of patients with eye diseases are critical in computer-aided detection (CAD) techniques. Semantic segmentation, a key component in computer vision, enables pixel-level classification and provides detailed information about objects within images. In this study, we present three U-Net models designed for multi-class semantic segmentation, leveraging the U-Net architecture with transfer learning. To generate ground truth for the HRF dataset, we combine two U-Net models, namely MSU-Net and BU-Net, to predict probability maps for the optic disc and cup regions. Binary masks are then derived from these probability maps to extract the optic disc and cup regions from retinal images. The dataset used in this study includes pre-existing blood vessels and manually annotated peripapillary atrophy zones (alpha and beta) provided by expert ophthalmologists. This comprehensive dataset, integrating existing blood vessels and expert-marked peripapillary atrophy zones, fulfills the study's objectives. The effectiveness of the proposed approach is validated by training nine pre-trained models on the HRF dataset comprising 45 retinal images, successfully segmenting the optic disc, cup, blood vessels, and peripapillary atrophy zones (alpha and beta). The results demonstrate 87.7 % pixel accuracy, 87 % Intersection over Union (IoU), 86.9 % F1 Score, 85 % mean IoU (mIoU), and 15 % model loss, significantly contributing to the early diagnosis and monitoring of glaucoma and optic nerve disorders.

Recognition of the lumbar spine using MRI plane-based FiLM conditioning and patient dependent batching on semantic segmentation

Abstract Lower back pain (LBP) is a common symptom that can be linked to the degeneration of intervertebral discs or injuries to vertebrae. Accurate segmentation of lumbar spinal structures is vital for image-based surgery planning but can be resource intensive. Recent studies have shown that U-Net models can be used reliably to segment vertebrae and intervertebral discs from medical imaging data. This study investigated the potential of combining multiple planar MRI images to segment these structures automatically. The study used MRI data from 83 patients with back pain of the lower spine, with ground truth segmentations performed by a spine surgery specialist using Mimics software. The segmentation models were trained using a specialized patient batching method and a FiLMed U-Net model, which applies a Feature-wise Linear Modulation layer to improve mask generation. The models were trained using a 10-fold cross validation study, and the dice loss was used as the objective function. The study found that both patient batching and FiLM layers improve results by around 5% compared to a baseline U-Net. The average IoU scores of the patient batching method were 0.896 for vertebrae and 0.876 intervertebral disc segmentation. The FiLMed U-Net model achieved average IoU scores of 0.844 and 0.814 and against a baseline of 0.837 and 0.774, respectively. A web-based tool was developed to enable medical personnel to assess segmentation quality visually. Overall, the study indicates that U-Nets show better segmentation potential when using imaging data with complete 3D information of one patient. Future studies should focus on increasing the dataset size and exploring other 3D model architectures to improve segmentation performance.

A model-based deep learning framework for damage classification and detection in polycarbonate infused with AEROSIL under dynamic loading conditions

Composite 3D printing is a significant engineering application owing to its robustness, ability to achieve complex geometries, and ease of use. Polycarbonate, particularly when infused with AEROSIL, is an interesting thermoplastic and potential candidate for 3D printing with enhanced properties. The primary objective of this research is to develop a new model-based deep-learning framework to classify and detect damage in this material under dynamic loading conditions. To achieve this, a FASTCAM high-speed camera was placed in front of the SHPB test setup to capture dynamic damage. The test results were then used as label inputs for training the advanced deep learning algorithms, focusing on dense image recognition techniques for detailed damage analysis. The study involved a series of fully convolutional networks (FCNs), evaluating semantic segmentation with U-Net and instance segmentation with state-of-the-art frameworks such as YOLOv8 and Mask R–CNN. A comparative analysis revealed that deep learning models outperform traditional methods, providing efficient and accurate damage classification and detection. The U-Net model demonstrated the ability to recognize cubes and bars but was limited in detecting minor damage regardless of size. YOLO-V8, which specializes in case segmentation, achieved remarkable performance in detecting significant damage but struggled to accurately identify minor damage. By leveraging deep learning techniques, this study enables an efficient and accurate damage assessment, which is crucial for ensuring the reliability and safety of composite structures in various industries.

Three contrasts in 3 min: Rapid, high-resolution, and bone-selective UTE MRI for craniofacial imaging with automated deep-learning skull segmentation.

Ultrashort echo time (UTE) MRI can be a radiation-free alternative to CT for craniofacial imaging of pediatric patients. However, unlike CT, bone-specific MR imaging is limited by long scan times, relatively low spatial resolution, and a time-consuming bone segmentation workflow. A rapid, high-resolution UTE technique for brain and skull imaging in conjunction with an automatic segmentation pipeline was developed. A dual-RF, dual-echo UTE sequence was optimized for rapid scan time (3 min) and smaller voxel size (0.65 mm3). A weighted least-squares conjugate gradient method for computing the bone-selective image improves bone specificity while retaining bone sensitivity. Additionally, a deep-learning U-Net model was trained to automatically segment the skull from the bone-selective images. Ten healthy adult volunteers (six male, age 31.5 ± 10 years) and three pediatric patients (two male, ages 12 to 15 years) were scanned at 3 T. Clinical CT for the three patients were obtained for validation. Similarities in 3D skull reconstructions relative to clinical standard CT were evaluated based on the Dice similarity coefficient and Hausdorff distance. Craniometric measurements were used to assess geometric accuracy of the 3D skull renderings. The weighted least-squares method produces images with enhanced bone specificity, suppression of soft tissue, and separation from air at the sinuses when validated against CT in pediatric patients. Dice similarity coefficient overlap was 0.86 ± 0.05, and the 95th percentile Hausdorff distance was 1.77 ± 0.49 mm between the full-skull binary masks of the optimized UTE and CT in the testing dataset. An optimized MRI acquisition, reconstruction, and segmentation workflow for craniofacial imaging was developed.

Performance Evaluation of the U-Net Model for Medical Image Segmentation Using Dice Coefficient, IOU, and Loss Metrics

Performance Evaluation of the U-Net Model for Medical Image Segmentation Using Dice Coefficient, IOU, and Loss Metrics

Enhancing brain tumor segmentation in MRI images: A hybrid approach using UNet, attention mechanisms, and transformers

Accurate brain tumor segmentation in MRI images is crucial for effective treatment planning and monitoring. Traditional methods often encounter challenges due to the complexity and variability of tumor shapes and textures. Consequently, there is a growing need for automated solutions to assist healthcare professionals in segmentation tasks, improving efficiency and reducing workload. This study introduces an innovative method for accurately segmenting brain tumors in MRI images by employing a refined 3D UNet model integrated with a Transformer. The goal is to leverage self-attention mechanisms to enhance segmentation capabilities. The proposed model combines Contextual Transformer (CoT) and Double Attention (DA) architectures. CoT is extended to a 3D format and integrated with the baseline model to exploit intricate contextual details in MRI images. DA blocks in skip connections aggregate and distribute long-range features, emphasizing inter-dependencies within an expanded spatial scope. Experimental results demonstrate superior segmentation performance compared to current state-of-the-art methods. With its ability to accurately segment and delineate tumors in 3D, our segmentation model promises to be a powerful tool for medical image processing and performance optimization, saving time for healthcare professionals and healthcare systems.

Deep-learning-based method for the segmentation of ureter and renal pelvis on non-enhanced CT scans

This study aimed to develop a deep-learning (DL) based method for three-dimensional (3D) segmentation of the upper urinary tract (UUT), including ureter and renal pelvis, on non-enhanced computed tomography (NECT) scans. A total of 150 NECT scans with normal appearance of the left UUT were chosen for this study. The dataset was divided into training (n = 130) and validation sets (n = 20). The test set contained 29 randomly chosen cases with computed tomography urography (CTU) and NECT scans, all with normal appearance of the left UUT. An experienced radiologist marked out the left renal pelvis and ureter on each scan. Two types of frameworks (entire and sectional) with three types of DL models (basic UNet, UNet3 + and ViT-UNet) were developed, and evaluated. The sectional framework with basic UNet model achieved the highest mean precision (85.5%) and mean recall (71.9%) on the test set compared to all other tested methods. Compared with CTU scans, this method had higher axial UUT recall than CTU (82.5% vs 69.1%, P < 0.01). This method achieved similar or better visualization of UUT than CTU in many cases, however, in some cases, it exhibited a non-ignorable false-positive rate. The proposed DL method demonstrates promising potential in automated 3D UUT segmentation on NECT scans. The proposed DL models could remarkably improve the efficiency of UUT reconstruction, and have the potential to save many patients from invasive examinations such as CTU. DL models could also serve as a valuable complement to CTU.

Intelligent void identification of particle packing system of caved ore and rock

Analyzing the void structure of the packing system of caved ore and rock and establishing the causal relationship between void structure and macroscopic properties are critical research areas in the field of gravity flow. However, the current method of analyzing void structure primarily involves manual threshold segmentation, leading to low analysis efficiency and the need for improved accuracy. This study introduces a novel approach using a dataset of two-dimensional Computed Tomography (CT) slices of an irregular limestone particle packing system. The primary goal is to enhance existing algorithms by focusing on data and network aspects to address the challenges in identifying this type of image dataset. As a result, a model named Data Attention gate with Recurrent Residual convolutional neural network based on U-net (DA2RU-net) is developed and evaluated for its convergence, accuracy, and robustness. The findings indicate the following: (1) To identify voids in the packing system of caved ore and rock using CT slices, it is advisable to employ 1000 images with dimensions between 300 and 400 pixels. It is essential to ensure that both large and small voids are represented in the image dataset. (2) The DA2RU-net model demonstrates superior performance with an average Dice value of 0.9859 for identifying large voids and 0.9654 for identifying small voids, surpassing other iterations of the U-net model and traditional algorithms. (3) The DA2RU-net model shows robustness to variations in brightness levels.

Spatial attention U-Net model with Harris hawks optimization for retinal blood vessel and optic disc segmentation in fundus images.

The state of the human eye's blood vessels is a crucial aspect in the diagnosis of ophthalmological illnesses. For many computer-aided diagnostic systems, precise retinal vessel segmentation is an essential job. However, it remains a difficult task due to the intricate vascular system of the eye. Although many different vascular segmentation techniques have already been presented, additional study is still required to address the problem of inadequate segmentation of thin and tiny vessels. In this work, we introduce the Spatial Attention U-Net (SAU-Net) modelwith harris hawks' optimization (HHO), a lightweight network that can be applied as a data augmentation technique to improve the efficiency of the existingannotated samples without the need of thousands of training instances for Retinal Blood Vessel and Optic Disc Segmentation. The SAU-Net-HHO implementation uses a spatially inferred attention map multiplied by the input feature map for adaptive feature enhancement. U-Net convolutional blocks have been replaced with structured dropout blocks in the proposed network to prevent overfitting. Data from both vascular extraction (DRIVE) and structured analysis of the retina (STARE) are used to evaluate SAU-Net-HHO performance. The results show that the proposed SAU-Net-HHO performs well on both datasets. Analysing the obtained results, an average of 98.5% accuracy and Specificity 96.7% was achieved for DRIVE dataset and 97.8% accuracy and specificity 94.5% for STARE dataset. The proposed method yields numerical results with average values that are on par with those of state-of-the-art methods. Visual inspection has revealed that the suggested method can segment thin and tiny vessels with greater accuracy than previous methods. It also demonstrates its potential for real-life clinical application.

Automatic Wound Image Segmentation with U-Net Model for Smartphone Application

Pengenalan citra luka memiliki potensi yang sangat penting dalam analisis luka, termasuk klasifikasi jenis luka, identifikasi infeksi, estimasi penyembuhan, dan penentuan perawatan yang tepat. Salah satu perkembangan terbaru dalam bidang ini adalah segmentasi otomatis pada citra, yang memanfaatkan kemajuan dalam deep learning untuk melakukan ekstraksi citra luka dengan menghilangkan piksel yang tidak relevan dengan luka. Dalam penelitian ini, kami melakukan evaluasi terhadap kinerja arsitektur U-Net dasar dan membandingkannya dengan tiga model pre-trained yang terkenal, termasuk MobileNet, MobileNetV2, EfficientNetB0, dan NasNet mobile sebagai backbone untuk meningkatkan kualitas segmentasi citra luka

Preparing the Electrical Signal Data of the Heart by Performing Segmentation Based on the Neural Network U-Net

Research on the automated extraction of essential data from an electrocardiography (ECG) recording has been a significant topic for a long time. The main focus of digital processing processes is to measure fiducial points that determine the beginning and end of the P, QRS, and T waves based on their waveform properties. The presence of unavoidable noise during ECG data collection and inherent physiological differences among individuals make it challenging to accurately identify these reference points, resulting in suboptimal performance. This is done through several primary stages that rely on the idea of preliminary processing of the ECG electrical signal through a set of steps (preparing raw data and converting them into files that are read and then processed by removing empty data and unifying the width of the signal at a length of 250 in order to remove noise accurately, and then performing the process of identifying the QRS in the first place and P-T implicitly, and then the task stage is determining the required peak and making a cut based on it. The U-Net pre-trained model is used for deep learning. It takes an ECG signal with a customisable sampling rate as input and generates a list of the beginning and ending points of P and T waves, as well as QRS complexes, as output. The distinguishing features of our segmentation method are its high speed, minimal parameter requirements, and strong generalization capabilities, which are used to create data that can be used in diagnosing diseases or biometric systems.

Pretrained subtraction and segmentation model for coronary angiograms

This study introduces a novel self-supervised learning method for single-frame subtraction and vessel segmentation in coronary angiography, addressing the scarcity of annotated medical samples in AI applications. We pretrain a U-Net model on a large dataset of unannotated coronary angiograms using an image-to-image translation framework, then fine-tune it on a limited set of manually annotated samples. The pretrained model excels at comprehensive single-frame subtraction, outperforming existing DSA methods. Fine-tuning with just 40 samples yields a Dice coefficient of 0.828 for vessel segmentation. On the public XCAD dataset, our model sets a new state-of-the-art benchmark with a Dice coefficient of 0.755, surpassing both unsupervised and supervised learning approaches. This method achieves robust single-frame subtraction and demonstrates that combining pretraining with minimal fine-tuning enables accurate coronary vessel segmentation with limited manual annotations. We successfully apply this approach to assist physicians in visualizing potential vascular stenosis sites during coronary angiography. Code, dataset, and a live demo will be available available at: https://github.com/newfyu/DeepSA.

Drought Awareness over Continental United States

Understanding the relationship between droughts and drought awareness is vital towards decision making and policy for water management and conservation strategies, and socioeconomic outcomes. We used computer vision (UNet models) to analyze nonlinear, heterogeneous, lagged correlations between Standardized Precipitation Evapotranspiration Index (SPEI) and Google Trends Search Interest within the Continental United States (CONUS). The most important drivers of the relationship between drought occurrence and drought awareness are the variability and ranges of drought trends and severity, as well as extreme drought conditions. This relationship was the strongest for Western states, followed by Northeastern, Southeastern, and Central regions. Search interest tends to lag droughts by a period of̃1-3 months. We also found evidence that reductionist linear approaches, such as a Principal Component Analysis, might not be as effective as UNet models in capturing the nuanced relationship between droughts and drought awareness at various dimensions and scales.