Variants Of U-Net Research Articles

Deep Learning-Based Flap Detection System Using Thermographic Images in Plastic Surgery

In reconstructive surgery, flaps are the cornerstone for repairing tissue defects, but postoperative monitoring of their viability remains a challenge. Among the imagistic techniques for monitoring flaps, the thermal camera has demonstrated its value as an efficient indirect method that is easy to use and easy to integrate into clinical practice. This provides a narrow color spectrum image that is amenable to the development of an artificial neural network in the context of current technological progress. In the present study, we introduce a novel attention-enhanced recurrent residual U-Net (AER2U-Net) model that is able to accurately segment flaps on thermographic images. This model was trained on a uniquely generated database of thermographic images obtained by monitoring 40 patients who required flap surgery. We compared the proposed AER2U-Net with several state-of-the-art neural networks used for multi-modal segmentation of medical images, all of which are based on the U-Net architecture (U-Net, R2U-Net, AttU-Net). Experimental results demonstrate that our model (AER2U-Net) achieves significantly better performance on our unique dataset compared to these existing U-Net variants, showing an accuracy of 0.87. This deep learning-based algorithm offers a non-invasive and precise method to monitor flap vitality and detect postoperative complications early, with further refinement needed to enhance its clinical applicability and effectiveness.

Automated High-Precision Recognition of Solar Filaments Based on an Improved U2-Net

Solar filaments are a significant solar activity phenomenon, typically observed in full-disk solar observations in the H-alpha band. They are closely associated with the magnetic fields of solar active regions, solar flare eruptions, and coronal mass ejections. With the increasing volume of observational data, the automated high-precision recognition of solar filaments using deep learning is crucial. In this study, we processed full-disk H-alpha solar images captured by the Chinese H-alpha Solar Explorer in 2023 to generate labels for solar filaments. The preprocessing steps included limb-darkening removal, grayscale transformation, K-means clustering, particle erosion, multiple closing operations, and hole filling. The dataset containing solar filament labels is constructed for deep learning. We developed the Attention U2-Net neural network for deep learning on the solar dataset by introducing an attention mechanism into U2-Net. In the results, Attention U2-Net achieved an average Accuracy of 0.9987, an average Precision of 0.8221, an average Recall of 0.8469, an average IoU of 0.7139, and an average F1-score of 0.8323 on the solar filament test set, showing significant improvements compared to other U-net variants.

USCT-UNet: Rethinking the Semantic Gap in U-Net Network from U-shaped Skip Connections with Multichannel Fusion Transformer.

Medical image segmentation is a crucial component of computer-aided clinical diagnosis, with state-of-the-art models often being variants of U-Net. Despite their success, these models' skip connections introduce an unnecessary semantic gap between the encoder and decoder, which hinders their ability to achieve the high precision required for clinical applications. Awareness of this semantic gap and its detrimental influences have increased over time. However, a quantitative understanding of how this semantic gap compromises accuracy and reliability remains lacking, emphasizing the need for effective mitigation strategies. In response, we present the first quantitative evaluation of the semantic gap between corresponding layers of U-Net and identify two key characteristics: 1) The direct skip connection (DSC) exhibits a semantic gap that negatively impacts models' performance; 2) The magnitude of the semantic gap varies across different layers. Based on these findings, we re-examine this issue through the lens of skip connections. We introduce a Multichannel Fusion Transformer (MCFT) and propose a novel USCT-UNet architecture, which incorporates U-shaped skip connections (USC) to replace DSC, allocates varying numbers of MCFT blocks based on the semantic gap magnitude at different layers, and employs a spatial channel cross-attention (SCCA) module to facilitate the fusion of features between the decoder and USC. We evaluate USCT-UNet on four challenging datasets, and the results demonstrate that it effectively eliminates the semantic gap. Compared to using DSC, our USC and SCCA strategies achieve maximum improvements of 4.79% in the Dice coefficient, 5.70% in mean intersection over union (MIoU), and 3.26 in Hausdorff distance.

Shape-intensity-guided U-net for medical image segmentation

Medical image segmentation has achieved impressive results thanks to U-Net or its alternatives. Yet, most existing methods perform segmentation by classifying individual pixels, tending to ignore the shape-intensity prior information. This may yield implausible segmentation results. Besides, the segmentation performance often drops greatly on unseen datasets. One possible reason is that the model is biased towards texture information, which varies more than shape information across different datasets. In this paper, we introduce a novel Shape-Intensity-Guided U-Net (SIG-UNet) for improving the generalization ability of variants of U-Net in segmenting medical images. Specifically, we adopt the U-Net architecture to reconstruct class-wisely averaged images that only contain the shape-intensity information. We then add an extra similar decoder branch with the reconstruction decoder for segmentation, and apply skip fusion between them. Since the class-wisely averaged image has no any texture information, the reconstruction decoder focuses more on shape and intensity features than the encoder on the original image. Therefore, the final segmentation decoder has less texture bias. Extensive experiments on three segmentation tasks of medical images with different modalities demonstrate that the proposed SIG-UNet achieves promising intra-dataset results while significantly improving the cross-dataset segmentation performance. The source code will be publicly available after acceptance.

Compact biologically inspired camera with computational compound eye

Abstract The growing interests have been witnessed in the evolution and improvement of artificial compound eyes (CE) inspired by arthropods. However, the existing CE cameras are suffering from a defocusing problem due to the incompatibility with commercial CMOS cameras. Inspired by the CEs of South American Shrimps, we report a compact biologically inspired camera that enables wide-field-of-view (FOV), high-resolution imaging and sensitive 3D moving trajectory reconstruction. To overcome the defocusing problem, a deep learning architecture with distance regulation is proposed to achieve wide-range-clear imaging, without any hardware or complex front-end design, which greatly reduces system complexity and size. The architecture is composed of a variant of Unet and Pyramid-multi-scale attention, with designed short, middle and long distance regulation. Compared to the current competitive well-known models, our method is at least 2 dB ahead. Here we describe the high-resolution computational-CE camera with 271 ommatidia, with a weight of 5.4 g an area of 3 × 3 cm2 and 5-mm thickness, which achieves compatibility and integration of CE with commercial CMOS. The experimental result illustrates this computational-CE camera has competitive advantages in enhanced resolution and sensitive 3D live moving trajectory reconstruction. The compact camera has promising applications in nano-optics fields such as medical endoscopy, panoramic imaging and vision robotics.

VisioRenalNet: Spatial Vision Transformer UNet for enhanced T2-Weighted Kidney MRI Segmentation

Chronic Kidney Disease (CKD) afflicts approximately 20% of the global population, underscoring the pressing need for precise detection and diagnosis. Accurate kidney segmentation in medical imaging is pivotal for capturing intricate details, facilitating effective evaluation and treatment planning. This work consists implementation of U-Net Variants along with enhanced variant for kidney segmentation in Magnetic Resonance Images (MRI). We systematically assess the performance of various U-Net variants, including U-Net, VGGUNet, DenseUNet, ResUNet, U-Net++, and AttentionUNet, utilizing a publicly available dataset of T2-weighted abdomen MRI scans from 100 subjects. Subsequently, we propose an enhanced U-Net framework that seamlessly integrates transformer blocks, spatial Attention Mechanism, and swish activation to enhance performance and robustness. Enhanced UNet provides improved feature learning, enhanced contextual information, selective focus and improved performance metrices for Kidney Segmentation. This significant advancement in medical image segmentation can contribute to enhanced diagnosis, more effective treatment planning, and improved disease management for individuals suffering from kidney diseases. Notably, our enhanced U-Net variant, after 50 epochs, achieved an average Dice coefficient of 89.1%, a Jaccard score of 80.4%, and an impressive accuracy of 96.2%, surpassing all baseline U-Net variants in all evaluation metrics, thereby affirming its superiority in kidney segmentation.

PL-Net: progressive learning network for medical image segmentation.

In recent years, deep convolutional neural network-based segmentation methods have achieved state-of-the-art performance for many medical analysis tasks. However, most of these approaches rely on optimizing the U-Net structure or adding new functional modules, which overlooks the complementation and fusion of coarse-grained and fine-grained semantic information. To address these issues, we propose a 2D medical image segmentation framework called Progressive Learning Network (PL-Net), which comprises Internal Progressive Learning (IPL) and External Progressive Learning (EPL). PL-Net offers the following advantages: 1) IPL divides feature extraction into two steps, allowing for the mixing of different size receptive fields and capturing semantic information from coarse to fine granularity without introducing additional parameters; 2) EPL divides the training process into two stages to optimize parameters and facilitate the fusion of coarse-grained information in the first stage and fine-grained information in the second stage. We conducted comprehensive evaluations of our proposed method on five medical image segmentation datasets, and the experimental results demonstrate that PL-Net achieves competitive segmentation performance. It is worth noting that PL-Net does not introduce any additional learnable parameters compared to other U-Net variants.

Medical Image Segmentation Review: The Success of U-Net.

Automatic medical image segmentation is a crucial topic in the medical domain and successively a critical counterpart in the computer-aided diagnosis paradigm. U-Net is the most widespread image segmentation architecture due to its flexibility, optimized modular design, and success in all medical image modalities. Over the years, the U-Net model has received tremendous attention from academic and industrial researchers who have extended it to address the scale and complexity created by medical tasks. These extensions are commonly related to enhancing the U-Net's backbone, bottleneck, or skip connections, or including representation learning, or combining it with a Transformer architecture, or even addressing probabilistic prediction of the segmentation map. Having a compendium of different previously proposed U-Net variants makes it easier for machine learning researchers to identify relevant research questions and understand the challenges of the biological tasks that challenge the model. In this work, we discuss the practical aspects of the U-Net model and organize each variant model into a taxonomy. Moreover, to measure the performance of these strategies in a clinical application, we propose fair evaluations of some unique and famous designs on well-known datasets. Furthermore, we provide a comprehensive implementation library with trained models. In addition, for ease of future studies, we created an online list of U-Net papers with their possible official implementation. All information is gathered in a GitHub repository https://github.com/NITR098/Awesome-U-Net.

Structural analysis of U-Net and its variants in the field of medical image segmentation

Medical image segmentation can provide valuable information for doctors, it has important research value in the medical field. Meanwhile, U-Net, as the fundamental networks for such tasks, brings a substantial improvement in the segmentation performance of traditional medical images. With the increasingly widespread use of U-Net, researchers have designed various U-Net variants according to different task requirements. However, most of the current summaries of U-Net variants are divided according to the direction of network applications, and the structural relationship between the variant networks and U-Net is not elaborated. Therefore, this paper classifies U-Net variants according to their network framework by elaborating the principles of U-Net structure. According to the U-Net network structure, it is divided into three main categories: backbone improvement, module addition and cross-network fusion. Further, the characteristics, advantages and disadvantages of different categories of variants are introduced, and the directions of the variants for U-Net optimization are analyzed. Finally, the article summarizes the current development direction of U-Net variants and provides an outlook on the future directions that can continue to be optimized.

DualA-Net: A generalizable and adaptive network with dual-branch encoder for medical image segmentation

Medical image segmentation is a critical task in early disease detection and diagnosis. In recent years, numerous variants of U-Net and Transformer-based models have demonstrated success in medical image segmentation. But these models still have certain limitations, particularly regarding the extraction of semantic information and high computational complexity. In order to tackle these challenges, we introduce a cutting-edge model called DualA-Net, which incorporates dual-branch encoder, multi-scale skip connections and Adaptive Receptive Field Selection Decoder(ARFSD). These innovations we proposed enable the model to intelligently adapt to and focus on relevant areas, ensuring its adaptability and thus improving the accuracy and efficiency of the segmentation process. To assess the performance of DualA-Net, its generalization capability was evaluated on five datasets of different segmentation tasks. The experimental results showed that the DualA-Net model performed the best on these datasets. Moreover, it minimized the parameter count and computational complexity. These findings provide evidence supporting the versatility and effectiveness of DualA-Net in medical image segmentation. Codes are available at https://github.com/Ziii1/DualA-Net.

Exploring advanced architectural variations of nnUNet

The nnUNet is a state-of-the-art deep learning based segmentation framework which automatically and systematically configures the entire network training pipeline. We extend the network architecture component of the nnUNet framework by newly integrating mechanisms from advanced U-Net variations including residual, dense, and inception blocks as well as three forms of the attention mechanism. We propose the following extensions to nnUNet, namely Residual-nnUNet, Dense-nnUNet, Inception-nnUNet, Spatial-Single-Attention-nnUNet, Spatial- Multi-Attention-nnUNet, and Channel-Spatial-Attention-nnUNet. Furthermore, within Channel-Spatial- Attention-nnUNet we integrate our newly proposed variation of the channel-attention mechanism. We demonstrate that use of the nnUNet allows for consistent and transparent comparison of U-Net architectural modifications, while maintaining network architecture as the sole independent variable across experiments with respect to a dataset. The proposed variants are evaluated on eight medical imaging datasets consisting of 20 anatomical regions which is the largest collection of datasets on which attention U-Net variants have been compared in a single work. Our results suggest that attention variants are effective at improving performance when applied to tumour segmentation tasks consisting of two or more target anatomical regions, and that segmentation performance is influenced by use of the deep supervision architectural feature.

Improved distinct bone segmentation in upper-body CT through multi-resolution networks

PurposeAutomated distinct bone segmentation from CT scans is widely used in planning and navigation workflows. U-Net variants are known to provide excellent results in supervised semantic segmentation. However, in distinct bone segmentation from upper-body CTs a large field of view and a computationally taxing 3D architecture are required. This leads to low-resolution results lacking detail or localisation errors due to missing spatial context when using high-resolution inputs.MethodsWe propose to solve this problem by using end-to-end trainable segmentation networks that combine several 3D U-Nets working at different resolutions. Our approach, which extends and generalizes HookNet and MRN, captures spatial information at a lower resolution and skips the encoded information to the target network, which operates on smaller high-resolution inputs. We evaluated our proposed architecture against single-resolution networks and performed an ablation study on information concatenation and the number of context networks.ResultsOur proposed best network achieves a median DSC of 0.86 taken over all 125 segmented bone classes and reduces the confusion among similar-looking bones in different locations. These results outperform our previously published 3D U-Net baseline results on the task and distinct bone segmentation results reported by other groups.ConclusionThe presented multi-resolution 3D U-Nets address current shortcomings in bone segmentation from upper-body CT scans by allowing for capturing a larger field of view while avoiding the cubic growth of the input pixels and intermediate computations that quickly outgrow the computational capacities in 3D. The approach thus improves the accuracy and efficiency of distinct bone segmentation from upper-body CT.

English

U-Net is adequate for various biomedical image segmentation tasks. Using this in the Age-related Macular Degeneration (AMD) images would be better for segmenting only the affected region. The variant of U-Net uses residual connections. A residual link is a type of connection that allows the network to learn the residual mapping between the input and output rather than the complete mapping. This technique can help the network learn more efficiently and avoid the vanishing gradients problem. This research aims to develop a method for segmenting the retinal pigment epithelium, the Bruch’s membrane, and the Inner Limiting Membranes (ILM) in OCT scans of individuals in good health and with intermediate AMD.

Historical document image analysis using controlled data for pre-training

Using neural networks for semantic labeling has become a dominant technique for layout analysis of historical document images. However, to train or fine-tune appropriate models, large labeled datasets are needed. This paper addresses the case when only limited labeled data are available and promotes a novel approach using so-called controlled data to pre-train the networks. Two different strategies are proposed: The first addresses the real labeling task by using artificial data; the second uses real data to pre-train the networks with a pretext task. To assess these strategies, a large set of experiments has been carried out on a text line detection and classification task using different variants of U-Net. The observations, obtained from two different datasets, show that globally the approach reduces the training time while offering similar or better performance. Furthermore, the effect is bigger on lightweight network architectures.

MemU-Net: A new volumetric dose prediction model using deep learning techniques in radiation treatment planning

Objective:Automatic radiation treatment planning systems are gaining significance nowadays. This paper presents a novel deep learning architecture for automatically generating the radiotherapy dose information given to the cancer patients. Methods:A new deep learning architecture MemU-Net is proposed. MemU-Net incorporates the principles of two existing deep learning architectures: Memnet, which was proposed for image restoration, and U-Net, which was proposed for biomedical image segmentation. The proposed model is first trained and tested with the public dataset (OpenKBP). The predicted dose distributions are evaluated based on dosimetric values and Dose volume histogram measures. The results are compared with the predictions made by a standard 3D U-Net, Residual U-Net, Dense U-Net and 3D GAN. The statistical significance of the proposed model is tested by computing the paired t-test P-values. Finally the proposed model is trained and tested on a private dataset collected from MVR Cancer Center and Research Institute. The model performance was evaluated by plotting DVHs and analyzing dosimetric values. Results:The dose distributions of MemU-Net shows superiority in performance compared to other models. MemU-Net had a lower mean absolute error of the predicted and ground truth doses than other models for all targets and OARs except spinal cord. The dosimetric measures of the MemU-Net outperformed the state of the art models except in the case of D0.1cc of OARs. Conclusion:A new variant of U-Net, MemU-Net is proposed, that performs better than the state of the art models, in the dose prediction domain in radiotherapy.

PDF-UNet: A semi-supervised method for segmentation of breast tumor images using a U-shaped pyramid-dilated network

Rapid and precise segmentation of breast tumors is a severe challenge for the global research community to diagnose breast cancer in younger females. An ultrasound system is a non-invasive and efficient way of breast screening. The area, shape, and texture of different breast tumors play a vital role for clinicians in making accurate diagnostic decisions. Furthermore, the limited availability of breast tumor annotated datasets is another challenge for properly training deep neural networks. This research proposes a semi-supervised learning-based method, which incorporates a Data expansion network (DEN), Probability map generator network (PMG), and U-shaped pyramid-dilated fusion network (PDF-UNet) for accurate breast tumor segmentation. The first DEN network is trained on breast unannotated tumor images and generates synthetic images for the data expansion task. The second PMG network generates corresponding probability map images against synthetic unannotated images. Finally, we proposed a segmentation network (PDF-UNet), a modified variant of UNet, to segment the breast tumor images. The results demonstrate that compared with classical UNet, our proposed PDF-UNet achieves an increment of DSC (2.42%) on the Mendeley dataset and an increment of DSC (1.52%) observed on the SIIT dataset. The results reflect that the proposed method is effective when annotated breast ultrasound data is insufficient to train the network. Furthermore, the proposed method can be helpful in relieving the annotation burden of radiologists. The implementation source code is available at GitHub: https://github.com/ahmedeqbal/PDF-UNet.

Mapping the Distribution and Dynamics of Coniferous Forests in Large Areas from 1985 to 2020 Combining Deep Learning and Google Earth Engine

Mapping the distribution of coniferous forests is of great importance to the sustainable management of forests and government decision-making. The development of remote sensing, cloud computing and deep learning has provided the support of data, computing power and algorithms for obtaining large-scale forest parameters. However, few studies have used deep learning algorithms combined with Google Earth Engine (GEE) to extract coniferous forests in large areas and the performance remains unknown. In this study, we thus propose a cloud-enabled deep-learning approach using long-time series Landsat remote sensing images to map the distribution and obtain information on the dynamics of coniferous forests over 35 years (1985–2020) in the northwest of Liaoning, China, through the combination of GEE and U2-Net. Firstly, to assess the reliability of the proposed method, the U2-Net model was compared with three Unet variants (i.e., Resnet50-Unet, Mobile-Unet and U-Net) in coniferous forest extraction. Secondly, we evaluated U2-Net’s temporal transferability of remote sensing images from Landsat-5 TM, Landsat-7 ETM+ and Landsat-8 OLI. Finally, we compared the results obtained by the proposed approach with three publicly available datasets, namely GlobeLand30-2010, GLC_FCS30-2010 and FROM_GLC30-2010. The results show that (1) the cloud-enabled deep-learning approach proposed in this paper that combines GEE and U2-Net achieves a high performance in coniferous forest extraction with an F1 score, overall accuracy (OA), precision, recall and kappa of 95.4%, 94.2%, 96.6%, 95.5% and 94.0%, respectively, outperforming the other three Unet variants; (2) the proposed model trained by the sample blocks collected from a specific time can be applied to predict the coniferous forests in different years with satisfactory precision; (3) Compared with three global land-cover products, the distribution of coniferous forests extracted by U2-Net was most similar to that of actual coniferous forests; (4) The area of coniferous forests in Northwestern Liaoning showed an upward trend in the past 35 years. The area of coniferous forests has grown from 945.64 km2 in 1985 to 6084.55 km2 in 2020 with a growth rate of 543.43%. This study indicates that the proposed approach combining GEE and U2-Net can extract coniferous forests quickly and accurately, which helps obtain dynamic information and assists scientists in developing sustainable strategies for forest management.

Macerals particle characteristics analysis of tar-rich coal in northern Shaanxi based on image segmentation models via the U-Net variants and image feature extraction

To investigate the particle characteristics of tar-rich coal macerals before separation, this study focuses on the tar-rich coal in northern Shaanxi, China, as the research object. The image segmentation models of U-Net variants (Mobile-Unet, VGG-Unet, Res-Unet, and TransUNet) are combined with OpenCV feature extraction to systematically study the particle morphology, particle size, liberation characteristics, and density separation process of coal macerals. The results indicate that the morphology of vitrinite tends to an olive-like shape as the particle size decreases, whereas the morphology of inertinite is complex and changeable. Further, the particle sizes of raw coal, vitrinite, and inertinite range from 0.09 mm to 0.075 mm, and their morphology uniformity is the best in this range. The statistical difference between the grid calculation point method (GCPM) and the pixel area statistics method (PASM) is mainly due to the particle size uniformity. When the particle size is less than 0.03 mm, the error is less than 3.0 %. By narrowing the range of the liberation densities and separating them preferentially, samples within the range of the unliberated densities are returned to the regrinding process to separate and enrich the vitrinite and inertinite groups in coal, which effectively improves their purity and recovery. In summary, Res-UNet, TransUNet, and the PASM not only have considerable potential for the qualitative analysis of maceral compositions and their liberation types but also provide powerful computer-assisted measurement to achieve reasonable separation and utilization of coal macerals.

SCL-Net: Structured Collaborative Learning for PET/CT Based Tumor Segmentation.

Collaborative learning methods for medical image segmentation are often variants of UNet, where the constructions of classifiers depend on each other and their outputs are supervised independently. However, they cannot explicitly ensure that optimizing auxiliary classifier heads leads to improved segmentation of target classifier. To resolve this problem, we propose a structured collaborative learning (SCL) method, which consists of a context-aware structured classifier population generation (CA-SCPG) module, where the feature propagation of the target classifier path is directly enhanced by the outputs of auxiliary classifiers via a light-weighted high-level context-aware dense connection (HLCA-DC) mechanism, and a knowledge-aware structured classifier population supervision (KA-SCPS) module, where the auxiliary classifiers are properly supervised under the guidance of target classifier's segmentations. Specifically, SCL is proposed based on a recurrent-dense-siamese decoder (RDS-Decoder), which consists of multiple siamese-decoder paths. CA-SCPG enhances the feature propagation of the decoder paths by HLCA-DC, which densely reuses previous decoder paths' output prediction maps to belong to the target classes as inputs to the layers of latter decoder paths. KA-SCPS supervises the classifier heads simultaneously with KA-SCPS loss, which consists of a generalized weighted cross-entropy loss for deep class-imbalanced learning and a novel knowledge-aware Dice loss (KA-DL). KA-DL is a weighted Dice loss broadcasting knowledges learnt by the target classifier to other classifier heads, harmonizing the learning process of the classifier population. Experiments are performed based on PET/CT volumes with malignant melanoma, lymphoma, or lung cancer. Experimental results demonstrate the superiority of our SCL, when compared to the state-of-the-art methods and baselines.

Well-Log Information-Assisted High-Resolution Waveform Inversion Based on Deep Learning

The high-resolution waveform inversion for seismic velocities is gaining increasing interest as we start to deal with complex structures. Although full waveform inversion (FWI) has been used for several years, obtaining high-resolution velocity models still presents many obstacles, such as the high computational cost and the limited bandwidth of the data. Thus, we propose a deep learning (DL)-based algorithm to build high-resolution velocity models using low-resolution velocity models, migration images, and well-log velocities as inputs. The well information, specifically, helps enhance the resolution with ground-truth information, especially around the well. These three inputs are fed to an improved neural network, a variant of U-Net, as three channels to predict the corresponding true velocity models, which serve as labels in the training. The incorporation of well velocities from several locations is crucial for improving the resolution of the output model. Numerical experiments on complex models demonstrate the robust performance of this network and the crucial role that well information plays, especially in generalizing the approach to models that differ from the trained ones and achieving superior performance compared with FWI.