U-Net Architecture Research Articles

Dual-encoder architecture for metal artifact reduction for kV-cone-beam CT images in head and neck cancer radiotherapy.

During a radiotherapy(RT) course, geometrical variations of target volumes, organs at risk, weight changes (loss/gain), tumor regression and/or progression can significantly affect the treatment outcome. Adaptive RT has become the effective methods along with technical advancements in imaging modalities including cone-beam computed tomography (CBCT). Planning CT (pCT) can be modified via deformable image registration (DIR), which is applied to the pair of pCT and CBCT. However, the artifact existed in both pCT and CBCT is a vulnerable factor in DIR. The dose calculation on CBCT is also suggested. Missing information due to the artifacts hinders the accurate dose calculation on CBCT. In this study, we aim to develop a deep learning-based metal artifact reduction(MAR) model to reduce the metal artifacts in CBCT for head and neck cancer RT. To train the proposed MAR model, we synthesized the kV-CBCT images including metallic implants, with and without metal artifacts (simulated image data pairs) through sinogram image handling process. We propose the deep learning architecture which focuses on both artifact removal and reconstruction of anatomic structure using a dual-encoder architecture. We designed four single-encoder models and three dual-encoder models based on UNet (for an artifact removal) and FusionNet (for a tissue restoration). Each single-encoder model contains either UNet or FusionNet, while the dual-encoder models have both UNet and FusionNet architectures. In the dual-encoder models, we implemented different feature fusion methods, including simple addition, spatial attention, and spatial/channel wise attention. Among the models, a dual-encoder model with spatial/channel wise attention showed the highest scores in terms of peak signal-to-noise ratio, mean squared error, structural similarity index, and Pearson correlation coefficient. CBCT images from 34 head and neck cancer patients were used to test the developed models. The dual-encoder model with spatial/channel wise attention showed the best results in terms of artifact index. By using the proposed model to CBCT, one can achieve more accurate synthetic pCT for head and neck patients as well as better tissue recognition and structure delineation for CBCT image itself.

Abstract 4138272: CMR Quantification of Myocardial Oxygenation Extraction Fraction Maps Using a Model-based Self-supervised Learning Network: Initial Experience in Patients with Cardiomyopathies

Background: Measurements of myocardial oxygenation extraction fraction (mOEF) maps with cardiovascular magnetic resonance (CMR) has been recently developed to assess myocardial oxygen metabolism in healthy. Recent studies focus on developing supervised machine learning methods to improve the quantification of mOEF. However, these supervised learning methods may be limited by the lack of ground-truth mOEF maps because MRI cannot directly measure mOEF with the same precision as PET. Hypothesis: Model-based self-supervised learning neural networks maybe feasible to generate mOEF maps accurately. Aims: To develop a physical model-based self-supervised learning neural network to accurately estimate mOEF for the patients with a variety of cardiomyopathies. Methods: An ECG triggered, two-dimensional asymmetric spin-echo (ASE) prepared sequence was employed to scan 71 patients with dilated, hypertrophic, or ischemic cardiomyopathies. At least 3 slices were scanned for each patient and each ASE scan was performed free-breathing with a spatial resolution of 1.7 × 1.7 × 8 mm 3 and scan time of 18 RR-interval. Acquisition of late gadolinium enhancement (LGE) was performed in each patient after the administration of 0.15 mmol/kg gadolinium agent for the visualization of myocardial infarction regions. A fully convolutional neural network with U-Net architecture was optimized with a physical model-based signal loss function (see Figure 1) on the ASE training dataset from 67 patients to generate mOEF maps. To evaluate if the method can accurately quantify abnormal mOEF, four patients with myocardial infarction were selected as the test set. Results: A previous publication reported normal mOEF levels at approximately 0.6-0.7 in healthy subjects. Figure 2 shows our proposed method generated mOEF maps from two representative patients and their LGE images for comparisons. In comparison with the remote region (mOEF = 0.70±0.10), the mOEF of myocardial regions with infarction was significantly lower (mOEF = 0.36±0.12, p < 0.001). The infarction areas predicted by the network overlapped well with the LGE. Conclusion: Quantification of mOEF using the proposed model-based self-supervised U-Net is feasible to detect abnormalities in regional myocardial oxygenation. Current study is ongoing to evaluate the proposed technique in a large cardiac patient dataset.

Building Extraction from Unmanned Aerial Vehicle (UAV) Data in a Landslide-Affected Scattered Mountainous Area Based on Res-Unet

Building extraction in landslide-affected scattered mountainous areas is essential for sustainable development, as it improves disaster risk management, fosters sustainable land use, safeguards the environment, and bolsters socio-economic advancement; however, this process entails considerable challenges. This study proposes a Res-Unet-based model to extract landslide-affected buildings from unmanned aerial vehicle (UAV) data in scattered mountain regions, leveraging the feature extraction capabilities of ResNet and the precise localization abilities of U-Net. A landslide-affected, scattered mountainous region within the Three Gorges Reservoir area was selected as a case study to validate the model’s performance. Experimental results indicate that Res-Unet displays high accuracy and robustness in building recognition, attaining accuracy (ACC), intersection-over-union (IOU), and F1-score values of 0.9849, 0.9785, and 0.9892, respectively. This enhancement can be attributed to the combined model, which amalgamates the skip connections, the symmetric architecture of U-Net, and the residual blocks of ResNet. This integration preserves low-level detail during recovery at higher levels, facilitating the extraction of multi-scale features while also mitigating the vanishing gradient problem prevalent in deep network training through the residual block structure, thus enabling the extraction of more complex features. The proposed Res-Unet approach shows significant potential for the accurate recognition and extraction of buildings in complex terrains through the efficient processing of remote sensing images.

Adaptive Depth Estimation via SoftHebbLayer in a Hybrid ResNet-UNet Architecture

Abstract. Depth estimation is a key task in computer vision, critical for applications such as autonomous navigation and augmented reality. This paper introduces a novel hybrid neural network that combines ResNet and UNet architectures with a SoftHebbLayer, inspired by Hebbian learning principles, to improve depth estimation from RGB images. The ResNet backbone extracts robust hierarchical features, while the UNet decoder reconstructs fine-grained depth maps. The SoftHebbLayer dynamically adjusts feature connections based on co-activation, enhancing the models adaptability to diverse environments. This approach addresses common challenges in depth estimation, including poor generalization and computational inefficiency. We evaluated the model on the DIODE dataset, achieving strong results in Mean Squared Error (MSE), which is 0.0800 and Root Mean Squared Error (RMSE), which is 0.2805, demonstrating improved accuracy in both indoor and outdoor scenes. While the model excels in precision, further refinement is needed to reduce computational overhead and improve performance in challenging environments. This research paves the way for more efficient, adaptable depth estimation models, with potential applications in mobile robotics and real-time edge computing systems.

Integrating Super-Resolution with Deep Learning for Enhanced Periodontal Bone Loss Segmentation in Panoramic Radiographs

Periodontal disease is a widespread global health concern that necessitates an accurate diagnosis for effective treatment. Traditional diagnostic methods based on panoramic radiographs are often limited by subjective evaluation and low-resolution imaging, leading to suboptimal precision. This study presents an approach that integrates Super-Resolution Generative Adversarial Networks (SRGANs) with deep learning-based segmentation models to enhance the segmentation of periodontal bone loss (PBL) areas on panoramic radiographs. By transforming low-resolution images into high-resolution versions, the proposed method reveals critical anatomical details that are essential for precise diagnostics. The effectiveness of this approach was validated using datasets from the Chungbuk National University Hospital and the Kaggle data portal, demonstrating significant improvements in both image resolution and segmentation accuracy. The SRGAN model, evaluated using the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) metrics, achieved a PSNR of 30.10 dB and an SSIM of 0.878, indicating high fidelity in image reconstruction. When applied to semantic segmentation using a U-Net architecture, the enhanced images resulted in a dice similarity coefficient (DSC) of 0.91 and an intersection over union (IoU) of 84.9%, compared with 0.72 DSC and 65.4% IoU for native low-resolution images. These results underscore the potential of SRGAN-enhanced imaging to improve PBL area segmentation and suggest broader applications in medical imaging, where enhanced image clarity is crucial for diagnostic accuracy. This study also highlights the importance of further research to expand the dataset diversity and incorporate clinical validation to fully realize the benefits of super-resolution techniques in medical diagnostics.

Machine Learning Models Based on [18F]FDG PET Radiomics for Bone Marrow Assessment in Non-Hodgkin Lymphoma

Non-Hodgkin lymphoma is a heterogeneous group of cancers that triggers bone marrow infiltration in 20–40% of cases. Bone marrow biopsy in combination with a visual assessment of [18F]FDG PET/CT images is used to assess the marrow status. Despite the potential of both techniques, they still have limitations due to the subjectivity of visual assessment. The present study aims to develop models based on bone marrow uptake in [18F]FDG PET/CT images at the time of diagnosis to differentiate bone marrow status. For this purpose, a model trained for skeleton segmentation and based on the U-Net architecture is retrained for bone marrow segmentation from CT images. The mask obtained from this segmentation together with the [18F]FDG PET image is used to extract radiomics features with which 11 machine learning models for marrow status differentiation are trained. The segmentation model yields very satisfactory results with Jaccard and Dice index values of 0.933 and 0.964, respectively. As for the classification models, a maximum F1_score_weighted and F1_score_macro of 0.962 and 0.747, respectively, are achieved. This highlights the potential of these features for bone marrow assessment, laying the foundation for a new clinical decision support system.

A Novel Defect Segmentation Model via Integrating Enhanced U-Net with Convolutional Block Attention Mechanisms

Abstract. Defect segmentation in industrial quality control is pivotal for maintaining high product surface standards, but it remains a challenge due to complex defect patterns and varying backgrounds. Traditional segmentation methods often struggle with sensitivity and specificity, leading to either missed defects or false positives. In order to address the above issues this paper presents an advanced defect segmentation model that integrates a Convolutional Block Attention Module (CBAM) with an enhanced U-Net architecture. Our method leverages the spatial and channel-wise attention mechanisms to focus more accurately on subtle defects and remove background interference, enhancing the discriminative power of the feature maps generated by the U-Net. Experimental results on SD-saliency-900 industrial datasets demonstrate significant improvements in segmentation accuracy and robustness compared to existing state-of-the-art methods, confirming the effectiveness of integrating attention mechanisms in deep learning models for complex defect segmentation tasks. Specifically, our model achieved a mean Intersection over Union (mIoU) of 0.87, which is an increase of 5% over the traditional U-Net model and a 4% improvement compared to the DeepLabv3+ model. These results indicate a substantial enhancement in defect detection performance, validating the superiority of our approach.

Mitigating coarse spatial reconstruction to generate missing bands for the HLS dataset

Abstract. The Harmonized LandSat-Sentinel (HLS) dataset has significantly advanced Earth Observation by integrating data from Landsat and Sentinel satellites. However, challenges persist in achieving spectral band parity between LandSat and Sentinel-derived HLS products. This paper presents an extended investigation aimed at enhancing spatial reconstruction accuracy to enable spectral band parity within HLS products. Building upon our previous work, which utilized generative neural networks to address partial feature mismatches between S30 and L30 products, we introduce a refined approach that fully integrates a Self-Supervised Learning (SSL)-pretrained encoder into a U-Net architecture. This method aims to access multi-scale features and improve spatial reconstruction accuracy, addressing the limitations in spatial resolution observed in our earlier study. Our methodology incorporates a comprehensive ablation study to assess various SSL-pretrained backbone architectures. Preliminary results demonstrate significant improvements in spatial reconstruction accuracy compared to our previous work. The adapted U-Net architecture, leveraging SSL-pretrained encoders, shows enhanced capability in capturing intricate spatial features within the HLS dataset. Our experiments demonstrate a substantial improvement in spatial resolution and feature reconstruction for L30 products, particularly in bands not natively present in Landsat data, paving the way for more accurate multi-sensor analyses.

Cloud shadow and uneven illumination detection and correction using the U-net architecture in near-surface images of complex forest canopies

Cloud shadow and uneven illumination detection and correction using the U-net architecture in near-surface images of complex forest canopies

A Strong Noise Reduction Network for Seismic Records

Noise reduction is a critical step in seismic data processing. A novel strong noise reduction network is proposed in this study. The network enhances the U-Net architecture with an improved inception module and coordinate attention (CA) mechanism, suppressing noise and enhancing signal clarity. These enhancements improve the network’s capability to distinguish between signal and noise in the time–frequency domain. We trained and tested our model on the STEAD dataset, which eliminated noise across various frequency bands, improved the signal-to-noise ratio (SNR) of seismic records, and reduced the waveform distortion significantly. Comparative analyses against U-Net, DeepDenoiser, and DnRDB models, using signals with SNRs ranging from −14 dB to 0 dB, demonstrated our model’s superior performance. At the same time, we demonstrated that the Inception Conv Block has a significant impact on the denoising ability of the network. Furthermore, validation using the “Di Ting” dataset and real noisy signals confirmed the model’s generalizability. These results show that the proposed model significantly outperforms the comparative methods in terms of the SNR, correlation coefficient (r), and root mean square error (RMSE), delivering higher-quality seismograms. The enhanced phase-picking accuracy underscores the potential of our approach to advance in geophysics applications.

Multi-level digital-twin models of pulmonary mechanics: correlation analysis of 3D CT lung volume and 2D Chest motion

Creating multi-level digital-twin models for mechanical ventilation requires a detailed estimation of regional lung volume. An accurate generic map between 2D chest surface motion and 3D regional lung volume could provide improved regionalisation and clinically acceptable estimates localising lung damage. This work investigates the relationship between CT lung volumes and the forced vital capacity (FVC) a surrogate of tidal volume proven linked to 2D chest motion. In particular, a convolutional neural network (CNN) with U-Net architecture is employed to build a lung segmentation model using a benchmark CT scan dataset. An automated thresholding method is proposed for image morphology analysis to improve model performance. Finally, the trained model is applied to an independent CT dataset with FVC measurements for correlation analysis of CT lung volume projection to lung recruitment capacity. Model training results show a clear improvement of lung segmentation performance with the proposed automated thresholding method compared to a typically suggested fixed value selection, achieving accuracy greater than 95% for both training and independent validation sets. The correlation analysis for 160 patients shows a good correlation ofRsquared value of 0.73 between the proposed 2D volume projection and the FVC value, which indicates a larger and denser projection of lung volume relative to a greater FVC value and lung recruitable capacity. The overall results thus validate the potential of using non-contact, non-invasive 2D measures to enable regionalising lung mechanics models to equivalent 3D models with a generic map based on the good correlation. The clinical impact of improved lung mechanics digital twins due to regionalising the lung mechanics and volume to specific lung regions could be very high in managing mechanical ventilation and diagnosing or locating lung injury or dysfunction based on regular monitoring instead of intermittent and invasive lung imaging modalities.

Correction to: Automatic liver segmentation using U-Net deep learning architecture for additive manufacturing

Correction to: Automatic liver segmentation using U-Net deep learning architecture for additive manufacturing

Improving deep learning U-Net++ by discrete wavelet and attention gate mechanisms for effective pathological lung segmentation in chest X-ray imaging.

Since its introduction in 2015, the U-Net architecture used in Deep Learning has played a crucial role in medical imaging. Recognized for its ability to accurately discriminate small structures, the U-Net has received more than 2600 citations in academic literature, which motivated continuous enhancements to its architecture. In hospitals, chest radiography is the primary diagnostic method for pulmonary disorders, however, accurate lung segmentation in chest X-ray images remains a challenging task, primarily due to the significant variations in lung shapes and the presence of intense opacities caused by various diseases. This article introduces a new approach for the segmentation of lung X-ray images. Traditional max-pooling operations, commonly employed in conventional U-Net++ models, were replaced with the discrete wavelet transform (DWT), offering a more accurate down-sampling technique that potentially captures detailed features of lung structures. Additionally, we used attention gate (AG) mechanisms that enable the model to focus on specific regions in the input image, which improves the accuracy of the segmentation process. When compared with current techniques like Atrous Convolutions, Improved FCN, Improved SegNet, U-Net, and U-Net++, our method (U-Net++-DWT) showed remarkable efficacy, particularly on the Japanese Society of Radiological Technology dataset, achieving an accuracy of 99.1%, specificity of 98.9%, sensitivity of 97.8%, Dice Coefficient of 97.2%, and Jaccard Index of 96.3%. Its performance on the Montgomery County dataset further demonstrated its consistent effectiveness. Moreover, when applied to additional datasets of Chest X-ray Masks and Labels and COVID-19, our method maintained high performance levels, achieving up to 99.3% accuracy, thereby underscoring its adaptability and potential for broad applications in medical imaging diagnostics.

Enhancing Visual Perception in Immersive VR and AR Environments: AI-Driven Color and Clarity Adjustments Under Dynamic Lighting Conditions

The visual fidelity of virtual reality (VR) and augmented reality (AR) environments is essential for user immersion and comfort. Dynamic lighting often leads to chromatic distortions and reduced clarity, causing discomfort and disrupting user experience. This paper introduces an AI-driven chromatic adjustment system based on a modified U-Net architecture, optimized for real-time applications in VR/AR. This system adapts to dynamic lighting conditions, addressing the shortcomings of traditional methods like histogram equalization and gamma correction, which struggle with rapid lighting changes and real-time user interactions. We compared our approach with state-of-the-art color constancy algorithms, including Barron’s Convolutional Color Constancy and STAR, demonstrating superior performance. Experimental results from 60 participants show significant improvements, with up to 41% better color accuracy and 39% enhanced clarity under dynamic lighting conditions. The study also included eye-tracking data, which confirmed increased user engagement with AI-enhanced images. Our system provides a practical solution for developers aiming to improve image quality, reduce visual discomfort, and enhance overall user satisfaction in immersive environments. Future work will focus on extending the model’s capability to handle more complex lighting scenarios.

Segmentation and classification of white blood SMEAR images using modified CNN architecture

The classification and recognition of leukocytes or WBCs in blood smear images presents a key role in the corresponding diagnosis of specific diseases, such as leukemia, tumor, hematological disorders, etc. The computerized framework for automated segmentation & classification of WBCs nucleus contributes an important role for the recognition of WBCs related disorders. Therefore, this work emphasizes WBCs nucleus segmentation using modified U-Net architecture and the segmented WBCs nucleus are further classified into their subcategory i.e., basophil, eosinophil, neutrophil, monocyte and lymphocyte. The classification and nucleus characterization task has been performed using VGGNet and MobileNet V2 architecture. Initially, collected instances are passed to the preprocessing phase for image rescaling and normalization. The rescaled and normalized instances are passed to the U-Net model for nucleus segmentation. Extracted nucleus are forwarded to the classification phase for their class identifications. Furthermore, the functioning of the intended design will be compared with other modern methods. By the end of this study a successful model classifying various nucleus morphologies such as Basophil, Eosinophil, Lymphocyte, Monocyte and Neutrophil was obtained where overall test accuracy achieved was 97.0% for VGGNet classifier and 94.0% for MobileNet V2 classifier.

Echocardiographic mitral valve segmentation model

Segmentation of mitral valve is not only important for clinical diagnosis, but also has far-reaching impact on prevention and prognosis of the disease by experts and doctors.In this paper, the multi-channel cross fusion transformer based U-Net network model (MCCT-UNet) is proposed according to the classical U-Net architecture. First, the jump connection part of MCCT-UNet is designed by using a multi-channel cross-fusion based attention mechanism module (MCCT) instead of the original jump connection, and this module fuses the feature maps from different scales in different stages of the encoder. Second, the optimization of the feature fusion method is proposed in the decoding stage by designing the cross-compression excitation sub-module (C-SENet) to replace the simple feature splicing, and the C-SENet is used to bridge the inconsistency of the semantic hierarchy by effectively combining the deeper information in the encoding stage with the shallower information. This two modules can establish a close connection between the encoder and decoder by exploring multi-scale global contextual information to solve the semantic divide problem, thus it significantly improves the segmentation performance of the network. The experimental results show that the improvement is effective, and the MCCT-UNet model outperforms the other 9 network models. Specifically, the MCCT-UNet achieved a Dice coefficient of 0.8734, an IoU of 0.7854, and an accuracy of 0.9977, demonstrating significant improvements over the compared models.

Seismic random noise suppression via mining multi-scale local and global information

Suppressing random noise is critical for revealing real subsurface structures. Convolutional neural networks (CNNs), the leading seismic data denoising methods, excel at extracting local features but struggle to capture global representations. Unet can extract and reuse multi-scale features, aiding in the precise detection of details and semantic information; however, being based on convolutional operations, it struggles to capture global information. To capture global representations, researchers normally employ Transformers in high-level visual tasks, owing to their self-attention mechanisms. This paper introduces a method for mining multi-scale local and global information based on hybrid-gated Unet (HGUnet), which integrates Transformer, CNN, and Unet architectures to enhance the feature representation capability for seismic random noise suppression tasks. HGUnet comprises hybrid-gated blocks (HGB) embedded within a U-shaped architecture, employing a concurrent structure of Octave convolution and lightweight multi-head self-attention mechanism to efficiently extract multi-scale local and global features simultaneously. Moreover, at the conclusion of the HGB, to precisely leverage information and reduce computing costs, a gated feedforward network is designed to retain valuable information and prune redundancies for feature fusion. Synthetic and field experimental results demonstrate that HGUnet improves denoising quality over traditional and CNN methods without adding significant computing costs.

FINE VESSEL SEGMENTATION WITH REFINEMENT GATE IN DEEP LEARNING ARCHITECTURES

Automated vessel segmentation is essential in diagnosing eye-related disorders and monitoring progressive retinal diseases. State-of-the-art methods have achieved excellent results in this field, but very few have considered the post-processing of feature maps. As a result, there is often a lack of small and fine vessels or discontinuities in segmented vessels. To address this issue, this study introduces a novel post-processing technique called the refinement gate, which works with a deep learning model during training. The refinement gate enhances contextual information to extract important features from feature maps better. The proposed technique is applied with U-net architecture and placed after every convolution block in the encoder path. Visual and statistical comparisons demonstrate the robustness of the proposed method using three publicly available datasets, namely: the DRIVE DB, the STARE DB, and CHASE_DB1 datasets, showing significant improvements to segment weak and tiny vessels. The reported results confirm the potential of the model to be used as a segmentation tool in the medical field. This study is the first to propose such a gating mechanism without additional trainable parameters or standalone networks as in other literature.

Chimeric U-Net – Modifying the standard U-Net towards explainability

Healthcare guided by semantic segmentation has the potential to improve our quality of life through early and accurate disease detection. Convolutional Neural Networks, especially the U-Net-based architectures, are currently the state-of-the-art learning-based segmentation methods and have given unprecedented performances. However, their decision-making processes are still an active field of research. In order to reliably utilize such methods in healthcare, explainability of how the segmentation was performed is mandated. To date, explainability is studied and applied heavily in classification tasks. In this work, we propose the Chimeric U-Net, a U-Net architecture with an invertible decoder unit, that inherently brings explainability into semantic segmentation tasks. We find that having the restriction of an invertible decoder does not hinder the performance of the segmentation task. However, the invertible decoder helps to disentangle the class information in the latent space embedding and to construct meaningful saliency maps. Furthermore, we found that with a simple k-Nearest-Neighbours classifier, we could predict the Intersection over Union scores of unseen data, demonstrating that the latent space, constructed by the Chimeric U-Net, encodes an interpretable representation of the segmentation quality. Explainability is an emerging field, and in this work, we propose an alternative approach, that is, rather than building tools for explaining a generic architecture, we propose constraints on the architecture which induce explainability. With this approach, we could peer into the architecture to reveal its class correlations and local contextual dependencies, taking an insightful step towards trustworthy and reliable AI. Code to build and utilize the Chimeric U-Net is made available under:https://github.com/kenrickschulze/Chimeric-UNet---Half-invertible-UNet-in-Pytorch

RECONSTRUCTION OF THE SURFACE PROFILE OF OPTICAL-MECHANICAL STRUCTURE WITH WHITE LIGHT INTERFERENCE BY DEEP LEARNING U-NET MODEL

This research proposes a method to accurately determine the location of the interference signal peak in white light interferometry. This improves the accuracy and resolution of the measurement, enabling precise measurement and reconstruction of three-dimensional microstructures on optical surfaces. By using a deep learning algorithm with a U-net neural network architecture to generate a predicted signal that closely matches the reference signal, combined with Fast Fourier Transform to filter noise and fit the proposed function, the exact location of the envelope signal peak is determined, providing information about the height of the point. This method can achieve an accuracy of around 0.9 nanometers at a noise level of 50 dB, which is over 40% more accurate than the traditional Fourier transform method at various noise levels. The persistent issues of the traditional Fourier transform method, such as determining the location of the interference signal peak at a signal position and the accuracy being heavily influenced by noise and the fitting signal, are effectively addressed.