U-Net Structure Research Articles

Graph Attention U-Net for Retinal Layer Surface Detection and Choroid Neovascularization Segmentation in OCT Images.

Choroidal neovascularization (CNV) is a typical symptom of age-related macular degeneration (AMD) and is one of the leading causes for blindness. Accurate segmentation of CNV and detection of retinal layers are critical for eye disease diagnosis and monitoring. In this paper, we propose a novel graph attention U-Net (GA-UNet) for retinal layer surface detection and CNV segmentation in optical coherence tomography (OCT) images. Due to retinal layer deformation caused by CNV, it is challenging for existing models to segment CNV and detect retinal layer surfaces with the correct topological order. We propose two novel modules to address the challenge. The first module is a graph attention encoder (GAE) in a U-Net model that automatically integrates topological and pathological knowledge of retinal layers into the U-Net structure to achieve effective feature embedding. The second module is a graph decorrelation module (GDM) that takes reconstructed features by the decoder of the U-Net as inputs, it then decorrelates and removes information unrelated to retinal layer for improved retinal layer surface detection. In addition, we propose a new loss function to maintain the correct topological order of retinal layers and the continuity of their boundaries. The proposed model learns graph attention maps automatically during training and performs retinal layer surface detection and CNV segmentation simultaneously with the attention maps during inference. We evaluated the proposed model on our private AMD dataset and another public dataset. Experiment results show that the proposed model outperformed the competing methods for retinal layer surface detection and CNV segmentation and achieved new state of the arts on the datasets.

CUDU-Net: Collaborative up-sampling decoder U-Net for leaf vein segmentation

Leaf vein is a common visual pattern in nature which provides potential clues for species identification, health evaluation, and variety selection of plants. However, as a critical step in leaf vein pattern analysis, segmenting vein from leaf image remains unaddressed due to its hierarchical curvilinear structure and busy background. In this study, we for the first time design a deep model which is tailored to address the segmentation of overall leaf vein structure. The proposed deep model, termed Collaborative Up-sampling Decoder U-Net (CUDU-Net), is an improved U-Net structure consisting of a fine-tuned ResNet extractor and a collaborative up-sampling decoder. The ResNet extractor utilizes residual module to explore high-dimensional features that are representative and abstract in the hidden layers of the network. The core of CUDU-Net is the collaborative up-sampling decoder which utilizes the complementarity of the bilinear-interpolation and deconvolution, to enhance the decoding capability of the model. The bilinear-interpolation can recovery key veins while the deconvolution actively learns to supplement more fine-grained features of the tertiary veins. In addition, we embed the strip pooling in the skip-connection to distill the vein-related semantics for performance boosting. Two leaf vein segmentation datasets, termed SoyVein500 and CottVein20, are built for model validation and generalization ability test. The extensive experimental results show that our proposed CUDU-Net outperforms the state-of-the-art methods in both segmentation accuracy and generalization ability.

Brain tumor image segmentation based on improved FPN

PurposeAutomatic segmentation of brain tumors by deep learning algorithm is one of the research hotspots in the field of medical image segmentation. An improved FPN network for brain tumor segmentation is proposed to improve the segmentation effect of brain tumor.Materials and methodsAiming at the problem that the traditional full convolutional neural network (FCN) has weak processing ability, which leads to the loss of details in tumor segmentation, this paper proposes a brain tumor image segmentation method based on the improved feature pyramid networks (FPN) convolutional neural network. In order to improve the segmentation effect of brain tumors, we improved the model, introduced the FPN structure into the U-Net structure, captured the context multi-scale information by using the different scale information in the U-Net model and the multi receptive field high-level features in the FPN convolutional neural network, and improved the adaptability of the model to different scale features.ResultsPerformance evaluation indicators show that the proposed improved FPN model has 99.1% accuracy, 92% DICE rating and 86% Jaccard index. The performance of the proposed method outperforms other segmentation models in each metric. In addition, the schematic diagram of the segmentation results shows that the segmentation results of our algorithm are closer to the ground truth, showing more brain tumour details, while the segmentation results of other algorithms are smoother.ConclusionsThe experimental results show that this method can effectively segment brain tumor regions and has certain generalization, and the segmentation effect is better than other networks. It has positive significance for clinical diagnosis of brain tumors.

CFANet: Context Feature Fusion and Attention Mechanism Based Network for Small Target Segmentation in Medical Images.

Medical image segmentation plays a crucial role in clinical diagnosis, treatment planning, and disease monitoring. The automatic segmentation method based on deep learning has developed rapidly, with segmentation results comparable to clinical experts for large objects, but the segmentation accuracy for small objects is still unsatisfactory. Current segmentation methods based on deep learning find it difficult to extract multiple scale features of medical images, leading to an insufficient detection capability for smaller objects. In this paper, we propose a context feature fusion and attention mechanism based network for small target segmentation in medical images called CFANet. CFANet is based on U-Net structure, including the encoder and the decoder, and incorporates two key modules, context feature fusion (CFF) and effective channel spatial attention (ECSA), in order to improve segmentation performance. The CFF module utilizes contextual information from different scales to enhance the representation of small targets. By fusing multi-scale features, the network captures local and global contextual cues, which are critical for accurate segmentation. The ECSA module further enhances the network's ability to capture long-range dependencies by incorporating attention mechanisms at the spatial and channel levels, which allows the network to focus on information-rich regions while suppressing irrelevant or noisy features. Extensive experiments are conducted on four challenging medical image datasets, namely ADAM, LUNA16, Thoracic OAR, and WORD. Experimental results show that CFANet outperforms state-of-the-art methods in terms of segmentation accuracy and robustness. The proposed method achieves excellent performance in segmenting small targets in medical images, demonstrating its potential in various clinical applications.

Structural analysis of U-Net and its variants in the field of medical image segmentation

Medical image segmentation can provide valuable information for doctors, it has important research value in the medical field. Meanwhile, U-Net, as the fundamental networks for such tasks, brings a substantial improvement in the segmentation performance of traditional medical images. With the increasingly widespread use of U-Net, researchers have designed various U-Net variants according to different task requirements. However, most of the current summaries of U-Net variants are divided according to the direction of network applications, and the structural relationship between the variant networks and U-Net is not elaborated. Therefore, this paper classifies U-Net variants according to their network framework by elaborating the principles of U-Net structure. According to the U-Net network structure, it is divided into three main categories: backbone improvement, module addition and cross-network fusion. Further, the characteristics, advantages and disadvantages of different categories of variants are introduced, and the directions of the variants for U-Net optimization are analyzed. Finally, the article summarizes the current development direction of U-Net variants and provides an outlook on the future directions that can continue to be optimized.

Scene Uyghur Recognition Based on Visual Prediction Enhancement.

Aiming at the problems of Uyghur oblique deformation, character adhesion and character similarity in scene images, this paper proposes a scene Uyghur recognition model with enhanced visual prediction. First, the content-aware correction network TPS++ is used to perform feature-level correction for skewed text. Then, ABINet is used as the basic recognition network, and the U-Net structure in the vision model is improved to aggregate horizontal features, suppress multiple activation phenomena, better describe the spatial characteristics of character positions, and alleviate the problem of character adhesion. Finally, a visual masking semantic awareness (VMSA) module is added to guide the vision model to consider the language information in the visual space by masking the corresponding visual features on the attention map to obtain more accurate visual prediction. This module can not only alleviate the correction load of the language model, but also distinguish similar characters using the language information. The effectiveness of the improved method is verified by ablation experiments, and the model is compared with common scene text recognition methods and scene Uyghur recognition methods on the self-built scene Uyghur dataset.

SAR U-Net: Spatial attention residual U-Net structure for water body segmentation from remote sensing satellite images

SAR U-Net: Spatial attention residual U-Net structure for water body segmentation from remote sensing satellite images

Deep Learning-based Land-cover Change Detection in Remote-sensing Imagery

With the significant advancement in deep-learning methods and their feature representation, deep-learning methods are more prevalent in solving change-detection tasks. The prime purpose of change detection is to detect the changes on the surface of the earth. In this work, an end-to-end encoder-decoder architecture is used to detect the changes in the land cover. The proposed method uses residual U-Net to find land-cover image changes. The UNet structure is used as the backbone of the network. The effectiveness of the proposed method has been experimented through LEVIR-CD datasets. The results showed that the proposed method outperforms the state-of-the-art techniques and gives reliable results. These techniques can be used to examine changes in the earth's crest due to natural events, such as landslides, earthquakes, erosion and geo-hazards or human activity, like mining and development. KEYWORDS: Change detection, Remote sensing, Residual UNet, Deep learning, Land cover, Climate.

Two-Stage Channel Estimation for mmWave Massive MIMO Systems Based on ResNet-UNet

For millimeter wave massive multiple-input multiple-output systems, the transceiver usually adopts a hybrid precoding structure to reduce complexity and cost, which poses great challenges to the acquisition of channel state information, especially in the case of low signal-to-noise ratio regime. In this article, residual network (ResNet) is employed to address this problem. Firstly, we design a two-stage channel estimation structure to improve the accuracy of channel estimation. Then, we take ResNet as the basic network and integrate UNet structure to build ResNet-UNet model to solve the model degradation problem. Moreover, we use noise2noise algorithm to train the neural network in order to implement the channel estimation in the case that a clean pilot cannot be obtained. Numerical results show that compared with the traditional channel estimation algorithms and deep convolutional neural network algorithm, the proposed approach has higher accuracy and robustness, and achieves performance close to the denoising algorithm using clean targets that is very difficult to be implemented in practical situations.

Multi-level adaptive perception guidance based infrared and visible image fusion

The current constraint rules of end-to-end infrared and visible image fusion (IVIF) networks based on deep learning solely focus on the pixel level, disregarding the consideration of deep features and global information. To address this limitation, this paper proposes an multi-level adaptive perception-guided image fusion approach, named MAPFusion. The main idea is to design a specific network structure and use features to guide the training of the network. Specifically, on the one hand, a new loss function strategy is designed, which combines pixel-level, structure-level, and feature-level strategies to comprehensively enhance the information of the fused image. In particular, the multi-level adaptive perceptual loss is introduced as a feature-level strategy, which preserves both low-level positional data and high-level semantic data by constraining the original image and fused image features. Moreover, adaptive weights are constructed using the information measure between feature maps, which measure the degree of information preservation of different source images. On the other hand, a new fusion network is proposed, which uses the improved U-Net structure to extract the multi-scale features of the image, and adds the Atrous Spatial Pyramid Pooling (ASPP) to the specific network layer, which can increase the receptive field and effectively take advantage of the global information of the image. Comparative experiments demonstrate that the proposed method effectively utilizes the information existing in infrared and visible images, resulting in superior visual quality and objective evaluation compared to SOTA algorithms. Ablation and supplementary experiments further validate the rationality of the proposed method and its potential for advanced vision tasks.

Graph isomorphism U-Net

Graph embedding learning is a fundamental task when dealing with diverse datasets. While encoder–decoder architectures, such as U-Nets, have shown great success in image pixel-wise prediction tasks, applying similar methods to graph data poses challenges due to the lack of natural pooling and up-sampling operations for graphs. Recent methods leverage learnable parameters to extract structural information from neural networks and extend pooling and unpooling to graphs using node features and graph structural information. This paper proposes a novel model called GIUNet (Graph Isomorphism U-Net) for the graph classification task. The proposed Graph U-Net structure is based on graph isomorphism convolution while using a comprehensive pqPooling layer. The pqPooling layer in our approach effectively combines node features and graph structure information during the graph down-sampling stage. To incorporate graph structure information, we utilize both the spectral representation and node centrality measurements. Node centrality measurements capture various structural aspects of nodes in the graph, while the spectral representation helps us focus on the informative low-frequency components of the graph structure. Through ablation studies, we have demonstrated that leveraging the GIUNet model leads to significant improvements compared to state-of-the-art methods across multiple benchmark datasets.

A global-frequency-domain network for medical image segmentation

The UNet series networks have been a leader in the field of medical image segmentation since their introduction. However, encoder and decoder structures of the traditional UNet series network are complex, with a large number of parameters and floating-point operations. This requires a large amount of data as support for model training, but most medical datasets only contain limited numbers of samples. To address this issue, we propose a global frequency domain UNet (GFUNet), a novel architecture for fast medical image segmentation. Inspired by recent modified Multi-Layer Perceptron(MLP)-like models, we combine Fourier Transform with UNet structure to achieve more efficient and effective encoding and decoding processes. Meanwhile, A dual-domain encoding module is designed to improve the performance of the encoder and decoder by fully used frequency domain feature. Furthermore, due to the excellent property of the Fourier Transform and its optimization, our network greatly reduces the number of parameters compared to other UNets. We evaluate GFUNet on several medical segmentation tasks, achieving improved segmentation performance compared to state-of-the-art network architectures for medical image segmentation. Compared to the original UNet, the results show that we reduce the number of parameters by 46 times, reduce computational complexity by 114 times, and improved the considerable dice score.

Enhanced LDR Detail Rendering for HDR Fusion by TransU- Fusion Network

High Dynamic Range (HDR) images are widely used in automotive, aerospace, AI, and other fields but are limited by the maximum dynamic range of a single data acquisition using CMOS image sensors. High dynamic range images are usually synthesized through multiple exposure techniques and image processing techniques. One of the most challenging task in multiframe Low Dynamic Range (LDR) images fusion for HDR is to eliminate ghosting artifacts caused by motion.In traditional algorithms, optical flow is generally used to align dynamic scenes before image fusion, which can achieve good results in cases of small-scale motion scenes but causes obvious ghosting artifacts when motion magnitude is large. Recently, attention mechanisms have been introduced during the alignment stage to enhance the network’s ability to remove ghosts. However, significant ghosting artifacts still occur in some scenarios with large-scale motion or oversaturated areas. We proposea novel Distilled Feature TransformerBlock (DFTB) structure to distill and re-extract information from deep image features obtained after U-Net downsampling, achieving ghost removal at the semantic level for HDR fusion. We introduce a Feature Distillation Transformer Block (FDTB), based on the Swin-Transformer and RFDB structure. FDTB uses multiple distillation connections to learn more discriminative feature representations. For the multiexposure moving scene image fusion HDR ghost removal task, in the previous method, the use of deep learning to remove the ghost effect in the composite image has been perfect, and it is almost difficult to observe the ghost residue of moving objects in the composite HDR image. The method in this paper focuses more on how to save the details of LDR image more completely after removing the ghost to synthesize high-quality HDR image. After using the proposed FDTB, the edge texture details of the synthesized HDR image are saved more perfectly, which shows that FDTB has a better effect in saving the details of image fusion. Futhermore, we propose a new depth framework based on DFTB for fusing and removing ghosts from deep image features, called TransU-Fusion. First of all, we use the encoder in U-Net to extract image features of different exposures and map them to different dimensional feature spaces. By utilizing the symmetry of the U-Net structure, we can ultimately output these feature images as original size HDR images. Then, we further fuse high-dimensional space features using Dilated Residual Dense Block (DRDB) to expand the receptive field, which is beneficial for repairing over-saturated regions. We use the transformer in DFTB to perform low-pass filtering on low-dimensional space features and interact with global information to remove ghosts. Finally, the processed features are merged and output as an HDR image without ghosting artifacts through the decoder. After testing on datasets and comparing with benchmark and state-of-the-art models, the results demonstrate our model’s excellent information fusion ability and stronger ghost removal capability.

IPDNet: A dual convolutional network combined with image prior for single image dehazing

Dehazing based on deep learning neural networks (CNNs) has achieved remarkable results. However, the most existing dehazing CNNs perform well only on synthetic images and struggle with realistic hazy images. Moreover, training complex dehazing models is challenging due to the limited availability of realistic image datasets, leading to suboptimal performance. For this purpose, we develop a novel Image Prior Dehazing Network called IPDNet to tackle the challenge of dehazing realistic images with limited training data. The IPDNet comprises two sub-networks and a learnable fusion block. The global and local features are obtained by atmospheric scattering and direct mapping via two sub-networks with a sparse mechanism. And the learnable fusion block is settled to acquire the optimal fusion solution to improve the dehazing quality. The IPDNet offers several benefits: (1) a dual network with a learnable fusion block can effectively generate high-quality dehazed images at low computational cost; (2) an image preprocessing block based on dehazing prior can acquires the salient features of realistic hazy image, enhancing the dehazing performance on realistic hazy images; (3) the sub-network is designed by incorporating a feature attention (FA) block into the U-net structure, allowing flexible feature extraction on limited datasets. Extensive experiments on SOTS, NH-HAZE, and DENSE-HAZE datasets show that IPDNet outperforms other state-of-the-art methods on synthetic and realistic datasets. Specifically, our model achieved an improvement of approximately 0.84 dB in PSNR and 11.6% in SSIM, demonstrating its effectiveness in realistic scenarios, which contributes to improving traffic safety in adverse weather conditions.

IESRGAN: Enhanced U-Net Structured Generative Adversarial Network for Remote Sensing Image Super-Resolution Reconstruction

With the continuous development of modern remote sensing satellite technology, high-resolution (HR) remote sensing image data have gradually become widely used. However, due to the vastness of areas that need to be monitored and the difficulty in obtaining HR images, most monitoring projects still rely on low-resolution (LR) data for the regions being monitored. The emergence of remote sensing image super-resolution (SR) reconstruction technology effectively compensates for the lack of original HR images. This paper proposes an Improved Enhanced Super-Resolution Generative Adversarial Network (IESRGAN) based on an enhanced U-Net structure for a 4× scale detail reconstruction of LR images using NaSC-TG2 remote sensing images. In this method, in-depth research has been performed and consequent improvements have been made to the generator and discriminator within the GAN network. Specifically, before introducing Residual-in-Residual Dense Blocks (RRDB), in the proposed method, input images are subjected to reflective padding to enhance edge information. Meanwhile, a U-Net structure is adopted for the discriminator, incorporating spectral normalization to focus on semantic and structural changes between real and fake images, thereby improving generated image quality and GAN performance. To evaluate the effectiveness and generalization ability of our proposed model, experiments were conducted on multiple real-world remote sensing image datasets. Experimental results demonstrate that IESRGAN exhibits strong generalization capabilities while delivering outstanding performance in terms of PSNR, SSIM, and LPIPS image evaluation metrics.

A simulation study of 1D U-Net-based inter-crystal scatter event recovery of PET detectors

To achieve high spatial resolution of reconstructed images in positron emission tomography (PET), the size of the scintillation crystal element is set small in current PET systems, which greatly increases the inter-crystal scattering (ICS) frequency. The ICS is a type of Compton scattering of the gamma photons from one crystal element to its neighborhood element, which obscures the determination of the first interaction position. In this study, we propose a 1D U-Net convolutional neural network to predict the first interaction position, which provides a universal way to efficiently solve the ICS recovery problem. The network is trained using the dataset collected from the GATE Monte Carlo simulation. The 1D U-Net structure is applied due to its capability of synthesizing both low-level and high-level information, which shows superiority in solving the ICS recovery problem. After being well trained, the 1D U-Net can generate a prediction accuracy of 78.1%. Compared to the coincidence events only composed from two photoelectric gamma photons, the sensitivity is improved by 149%. The contrast-to-noise ratio of the reconstructed contrast phantom increases from 6.973 to 10.795 for the 16 mm hot sphere. Compared to the take-energy-centroid method, the spatial resolution of the reconstructed resolution phantom can obtain the best improvement of 33.46%. Compared with the previous deep learning method based on the fully connected network, the proposed 1D U-Net can work more stably with considerably fewer network parameters. The 1D U-Net network model shows good universality when predicting different phantoms, and the computation speed is fast.

Two-Branch Underwater Image Enhancement and Original Resolution Information Optimization Strategy in Ocean Observation

In complex marine environments, underwater images often suffer from color distortion, blur, and poor visibility. Existing underwater image enhancement methods predominantly rely on the U-net structure, which assigns the same weight to different resolution information. However, this approach lacks the ability to extract sufficient detailed information, resulting in problems such as blurred details and color distortion. We propose a two-branch underwater image enhancement method with an optimized original resolution information strategy to address this limitation. Our method comprises a feature enhancement subnetwork (FEnet) and an original resolution subnetwork (ORSnet). FEnet extracts multi-resolution information and utilizes an adaptive feature selection module to enhance global features in different dimensions. The enhanced features are then fed into ORSnet as complementary features, which extract local enhancement features at the original image scale to achieve semantically consistent and visually superior enhancement effects. Experimental results on the UIEB dataset demonstrate that our method achieves the best performance compared to the state-of-the-art methods. Furthermore, through comprehensive application testing, we have validated the superiority of our proposed method in feature extraction and enhancement compared to other end-to-end underwater image enhancement methods.

Kidney Segmentation from Dynamic Contrast-Enhanced Magnetic Resonance Imaging Integrating Deep Convolutional Neural Networks and Level Set Methods.

The dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) technique has taken on a significant and increasing role in diagnostic procedures and treatments for patients who suffer from chronic kidney disease. Careful segmentation of kidneys from DCE-MRI scans is an essential early step towards the evaluation of kidney function. Recently, deep convolutional neural networks have increased in popularity in medical image segmentation. To this end, in this paper, we propose a new and fully automated two-phase approach that integrates convolutional neural networks and level set methods to delimit kidneys in DCE-MRI scans. We first develop two convolutional neural networks that rely on the U-Net structure (UNT) to predict a kidney probability map for DCE-MRI scans. Then, to leverage the segmentation performance, the pixel-wise kidney probability map predicted from the deep model is exploited with the shape prior information in a level set method to guide the contour evolution towards the target kidney. Real DCE-MRI datasets of 45 subjects are used for training, validating, and testing the proposed approach. The valuation results demonstrate the high performance of the two-phase approach, achieving a Dice similarity coefficient of 0.95 ± 0.02 and intersection over union of 0.91 ± 0.03, and 1.54 ± 1.6 considering a 95% Hausdorff distance. Our intensive experiments confirm the potential and effectiveness of that approach over both UNT models and numerous recent level set-based methods.

Neural-network-based wavefront solution algorithm for a wide field survey telescope.

The wide field survey telescope (WFST) is a 2.5m optical survey telescope currently under construction in China. The telescope employs a primary-focus optical design to achieve a wide field of view of 3deg, and its focal plane is equipped with four pairs of curvature sensors to perform wavefront sensing and active optics. Currently, there are several wavefront solution algorithms available for curvature sensors, including the iterative fast Fourier transform method, orthogonal series expansion method, Green's function method, and sensitivity matrix method. However, each of these methods has limitations in practical use. This study proposes a solution method based on a convolutional neural network model with a U-Net structure for the curvature wavefront sensing of the WFST. Numerical simulations show that the model, when properly trained, has a high accuracy and performs a curvature wavefront solution effectively. Upon a comparison with the sensitivity matrix method, this new method demonstrates its superiority. Finally, the study is summarized, and the drawbacks of the proposed method are discussed, which leads to direction for future optimizations.

Enhancing the security of image steganography via multiple adversarial networks and channel attention modules

Existing steganographic methods based on adversarial images can only design adversarial images for a single steganalyzer and cannot resist detection from the latest steganalyzers using convolutional neural networks such as SRNet and Zhu-Net. To address this issue, this paper proposes a novel method for enhancing the security of image steganography using multiple adversarial networks and channel attention modules. The proposed method employs generative adversarial networks based on the U-Net structure to generate high-quality adversarial images and uses the self-learning properties of the adversarial networks to iteratively optimize the parameters of multiple adversarial steganographic networks. This process generates high-quality adversarial images capable of misleading multiple steganalyzers. Additionally, the proposed scheme adaptively adjusts the distribution of adversarial noise in the original image using multiple lightweight channel attention modules in the generator, thus enhancing the anti-steganalysis ability of adversarial images. Furthermore, the proposed method utilizes multiple discrimination losses and MSE loss, dynamically combined to improve the quality of adversarial images and facilitate the network's rapid and stable convergence. Extensive experimental results demonstrate that the proposed algorithm can generate adversarial images with an average PSNR of 40.3 dB, and the success rate of misleading the advanced steganalyzers is over 93%. The security and generalization of the algorithm we propose exceed those of the compared steganographic methods.