U-Net Structure Research Articles

2D–3D cascade network for glioma segmentation in multisequence MRI images using multiscale information

Background and objectiveGlioma segmentation is an important procedure for the treatment plan and follow-up evaluation of patients with glioma. UNet-based networks are widely used in medical image segmentation tasks and have achieved state-of-the-art performance. However, context information along the third dimension is ignored in 2D convolutions, whereas difference between z-axis and in-plane resolutions is large in 3D convolutions. Moreover, an original UNet structure cannot capture fine details because of the reduced resolution of feature maps near bottleneck layers. MethodsTo address these issues, a novel 2D–3D cascade network with multiscale information module is proposed for the multiclass segmentation of gliomas in multisequence MRI images. First, a 2D network is applied to fully exploit potential intra-slice features. A variational autoencoder module is incorporated into 2D DenseUNet to regularize a shared encoder, extract useful information, and represent glioma heterogeneity. Second, we integrated 3D DenseUNet with the 2D network in cascade mode to extract useful inter-slice features and alleviate the influence of large difference between z-axis and in-plane resolutions. Moreover, a multiscale information module is used in the 2D and 3D networks to further capture the fine details of gliomas. Finally, the whole 2D–3D cascade network is trained in an end-to-end manner, where the intra-slice and inter-slice features are fused and optimized jointly to take full advantage of 3D image information. ResultsOur method is evaluated on publicly available and clinical datasets and achieves competitive performance in these two datasets. ConclusionsThese results indicate that the proposed method may be a useful tool for glioma segmentation.

CUMTGAN: An instance-level controllable U-Net GAN for facial makeup transfer

Instance-level makeup transfer refers to transferring a specific makeup style to a particular face without cosmetic, while keeping its facial structure. To solve the challenges such as face structure loss and uncontrollable shade of makeup transfer in the existing makeup transfer methods, in this paper, we propose an instance-level face makeup transfer algorithm with generative adversarial network (CUMTGAN, Controllable U-Net Makeup Transfer GAN). The network uses the results of face semantic segmentation to put on or remove makeup on specific areas such as eyes, lips, while constraining the parts such as image background. It uses the characteristics of the skip connections of the U-Net structure to achieve hierarchical feature fusion. A Superimposed Module is also proposed to provide a controllable shade of makeup for the generated images, making the control of the makeup more arbitrary. Especially, we propose a new quantitative evaluation method to evaluate facial structure and makeup style respectively. To our best knowledge, it is the first time to evaluate generated samples from two aspects. The experimental results on the MT dataset show that the proposed method can generate natural and visual makeup images and makeup removal images, which is superior to the baseline methods in facial structure and makeup style.

Multi-input adaptive neural network for automatic detection of cervical vertebral landmarks on X-rays

Cervical vertebral landmark detection is a significant pre-task for vertebral relative motion parameter measurement, which is helpful for doctors to diagnose cervical spine diseases. Accurate cervical vertebral landmark detection could provide reliable motion parameter measurement results. However, different cervical spines in X-rays with various poses and angles have imposed quite challenges. It is observed that there are similar appearances of vertebral bones in different cervical spine X-rays. For this, to fully use these similar features, a multi-input adaptive U-Net (MultiIA-UNet) focusing on the similar local features between different cervical spine X-rays is put forward to do cervical vertebral landmark detection accurately and effectively. MultiIA-UNet used an improved U-Net structure as backbone network combining with the novel adaptive convolution module to better extract changing global features. At training, MultiIA-UNet applied a multi-input strategy to extract features from random pairs of training data at the same time, and then learned their similar local features through a subspace alignment module. We collected a dataset including 688 cervical spine X-rays to evaluate MultiIA-UNet. The results exhibited that our method demonstrated the state-of-the-art performance (the minimum average point to point error of 12.988 pixels). In addition, we further evaluated the effect of these landmark detection results on cervical motion angle parameter measurement. It showed that our method was capable to obtain more accurate cervical spine motion angle parameters (the minimum symmetric mean absolute percentage is 26.969%). MultiIA-UNet could be an efficient and accurate landmark detection method for doctors to do cervical vertebral motion analysis.

Research on Pavement Crack Detection Algorithm based on Deep Residual Unet Neural Network

Pavement crack is one of the main factors threatening highway safety. Rapid and accurate identification of pavement crack is very important to protect highway. Traditional image processing methods have been difficult to meet the requirements of practical engineering. So, this paper proposes an improved pavement crack detection method based on deep convolution neural network model, to overcome the inaccurate segmentation caused by complex background and fuzzy crack edge in asphalt pavement image, which is an encoder-decoder network structure, and integrates the idea of semantic segmentation into image detection. The encoder of the proposed method adopts the ResNet’s residual structure to strengthen image feature extraction, the decoder uses transpose convolution from the original UNet structure for up sampling, and integrates the encoder features with the up sampled features after cutting, Strengthen the learning of network features. Experimental results show that the F1 score, Precision and Recall are significantly improved on the asphalt crack dataset. Compared with the traditional algorithm, the proposed algorithm can segment the cracks better and improve the accuracy of pavement crack detection.

Robust high-resolution direction-of-arrival estimation method using DenseBlock-based U-net.

Direction-of-arrival (DOA) estimation is widely used in underwater detection and localization. To address the high-resolution DOA estimation problem, a DenseBlock-based U-net structure is proposed in this paper. U-net is a U-shaped fully convolutional neural network, which yields a two-dimensional image. DenseBlock is a more efficient structure than typical convolutional layers. The proposed network replaces the concatenated convolutional layers in the original U-net with DenseBlocks. Through training, the network can remove the interference of sidelobes and noise in a conventional beam forming bearing-time record (BTR) and get a clean BTR; hence, this method has narrow beam width and few sidelobes. In addition, the network can be trained by simulation data and applied in actual data when the simulated and actual data are similar in BTR features, so the method has high generalization. For a multi-target problem, the network does not need to be trained on all cases with different target quantities and therefore can reduce the training set size. As a data-driven method, it does not rely on prior assumptions of the array model and possesses better robustness to array imperfections than typical model-based DOA algorithms. Simulations and experiments verify the advantages of the proposed method.

MFFE: Multi-scale Feature Fusion Enhanced Net for image dehazing

It is very challenging to perform dehazing operations on single haze images taken due to information degradation, for which we propose a novel single-image dehazing network based on U-Net. Most existing image dehazing algorithms only focus on whether they can remove the haze, but ignore the quality of the final dehazed image, which can lead to problems such as loss of image detail information and texture blurring, affecting the final recovery. We use the enhancement and error feedback mechanisms inside the super-resolution algorithm to gradually recover the haze-free images by introducing the Strengthen-Operate-subtract strategy in the decoder and demonstrate their effectiveness for the dehazing problem. Meanwhile, to solve the problem of missing information and utilization in the traditional U-Net structure, we design a multiscale feature fusion module that can effectively compensate for the missing contextual information and make full use of the disjoint features. In addition, an improved local binary pattern and SE attention mechanism are used to help the network obtain clearer details and texture images, and the accuracy of the enhanced dehazing network is improved using residual learning. We analyze the effectiveness of the proposed algorithm and show through extensive evaluation that our proposed model can be effectively used for single image dehazing and also has good performance compared to state-of-the-art methods on the test dataset.

Denoising of scanning electron microscope images for biological ultrastructure enhancement.

Scanning electron microscopy (SEM) is of great significance for analyzing the ultrastructure. However, due to the requirements of data throughput and electron dose of biological samples in the imaging process, the SEM image of biological samples is often occupied by noise which severely affects the observation of ultrastructure. Therefore, it is necessary to analyze and establish a noise model of SEM and propose an effective denoising algorithm that can preserve the ultrastructure. We first investigated the noise source of SEM images and introduced a signal-related SEM noise model. Then, we validated the effectiveness of the noise model through experiments, which are designed with standard samples to reflect the relation between real signal intensity and noise. Based on the SEM noise model and traditional variance stabilization denoising strategy, we proposed a novel, two-stage denoising method. In the first stage variance stabilization, our VS-Net realizes the separation of signal-dependent noise and signal in the SEM image. In the second stage denoising, our D-Net employs the structure of U-Net and combines the attention mechanism to achieve efficient noise removal. Compared with other existing denoising methods for SEM images, our proposed method is more competitive in objective evaluation and visual effects. Source code is available on GitHub (https://github.com/VictorCSheng/VSID-Net).

Speech Enhancement Using U-Net with Compressed Sensing

With the development of deep learning, speech enhancement based on deep neural networks had made a great breakthrough. The methods based on U-Net structure achieved good denoising performance. However, part of them rely on ordinary convolution operation, which may ignore the contextual information and detailed features of input speech. To solve this issue, many studies have improved model performance by adding additional network modules, such as attention mechanism, long and short-term memory (LSTM), etc. In this work, therefore, a time-domain U-Net speech enhancement model which combines lightweight Shuffle Attention mechanism and compressed sensing loss (CS loss) is proposed. The time-domain dilated residual blocks are constructed and used for down-sampling and up-sampling in this model. The Shuffle Attention is added to the final output of the encoder for focusing on features of speech and suppressing irrelevant audio information. A new loss is defined by using the measurements of clean speech and enhanced speech based on compressed sensing, it can further remove noise in noisy speech. In the experimental part, the influence of different loss functions on model performance is proved through ablation experiments, and the effectiveness of CS loss is verified. Compared with the reference models, the proposed model can obtain higher speech quality and intelligibility scores with fewer parameters. When dealing with the noise outside the dataset, the proposed model still achieves good denoising performance, which proves that the proposed model can not only achieve a good enhancement effect, but also has good generalization ability.

Application Research of Deep Learning Technology in Natural Landscape Animation Design

Due to the limitation of technology and cost, the animation design of natural landscape in the past was often dealt with relative simplicity. With the increasing level of audience appreciation and the continuous development of animation technology, new requirements are put forward for the design of natural landscape animation. In order to make the animation design effect of natural landscape more real, the synthetic aperture radar image is firstly analyzed to obtain the location of mountains, farmland, rivers, villages, roads, and buildings. Considering the superiority of U-Net network in image semantic segmentation, this paper constructs a semantic segmentation model based on U-Net structure. In this model, dense connection module is introduced in downsampling, and spatial void pyramid structure is introduced in upsampling to retain more image features, to achieve accurate segmentation of satellite images. Experimental results show that the proposed algorithm has higher segmentation accuracy than other algorithms. After accurate classification of natural scene images, it can provide a guarantee for designing more real natural landscape animation design effects.

U-Net-Based Medical Image Segmentation.

Deep learning has been extensively applied to segmentation in medical imaging. U-Net proposed in 2015 shows the advantages of accurate segmentation of small targets and its scalable network architecture. With the increasing requirements for the performance of segmentation in medical imaging in recent years, U-Net has been cited academically more than 2500 times. Many scholars have been constantly developing the U-Net architecture. This paper summarizes the medical image segmentation technologies based on the U-Net structure variants concerning their structure, innovation, efficiency, etc.; reviews and categorizes the related methodology; and introduces the loss functions, evaluation parameters, and modules commonly applied to segmentation in medical imaging, which will provide a good reference for the future research.

DRs-UNet: A Deep Semantic Segmentation Network for the Recognition of Active Landslides from InSAR Imagery in the Three Rivers Region of the Qinghai–Tibet Plateau

At present, Synthetic Aperture Radar Interferometry (InSAR) has been an important technique for active landslides recognition in the geological survey field. However, the traditional interpretation method through human–computer interaction highly relies on expert experience, which is time-consuming and subjective. To solve the problem, this study designed an end-to-end semantic segmentation network, called deep residual shrinkage U-Net (DRs-UNet), to automatically extract potential active landslides in InSAR imagery. The proposed model was inspired by the structure of U-Net and adopted a residual shrinkage building unit (RSBU) as the feature extraction block in its encoder part. The method of this study has three main advantages: (1) The RSBU in the encoder part incorporated with soft thresholding can reduce the influence of noise from InSAR images. (2) The residual connection of the RSBU makes the training of the network easier and accelerates the convergency process. (3) The feature fusion of the corresponding layers between the encoder and decoder effectively improves the classification accuracy. Two widely used networks, U-Net and SegNet, were trained under the same experiment environment to compare with the proposed method. The experiment results in the test set show that our method achieved the best performance; specifically, the F1 score is 1.48% and 4.1% higher than U-Net and SegNet, which indicates a better balance between precision and recall. Additionally, our method has the best IoU score of over 90%. Furthermore, we applied our network to a test area located in Zhongxinrong County along Jinsha River where landslides are highly evolved. The quantitative evaluation results prove that our method is effective for the automatic recognition of potential active landslide hazards from InSAR imagery.

Unmanned aerial vehicle image biological soil crust recognition based on UNet++

ABSTRACT Biological soil crust is an important feature of desert ecosystem composition and surface landscape. Determining the role of biological soil crust in the energy flow and logistics cycle in the desert ecosystem is one of the frontier areas of ecological restoration in arid and semi-arid areas. Obtaining biological soil crust information from drone images is an efficient, fast, and low-cost method. However, due to the scattered and uneven growth of biological soil crusts and the complexity of the field environment, it is difficult to accurately extract biological soil crusts. In view of this, this study used the improved UNet++ model to extract biological soil crusts based on UAV image data. Firstly, the optimal Epoch, Backbone, and Loss function are selected and trained based on the network structure of UNet++ model. Then, the improved UNet++ model proposed in this paper, which takes ResNeXt as the Backbone and Soft Cross-Entropy Loss+Dice Loss as the Loss Function, is obtained. Finally, the test results of UNet++, U-Net, LinkNet, FPN, and PSPNet are compared with the improved UNet++ model in this paper. The results showed that the improved UNet++ model had the best segmentation effect, and the precision, recall, F1-Score, and IoU were 0.9788, 0.9501, 0.9495, and 0.9309, respectively. UAV image biological soil crust recognition based on the improved UNet++ model in this paper can obtain high-precision extraction results, provide good data support for studying the development of biological soil crusts in arid areas, and provide a new method for precise segmentation of different features in arid areas. In particular, it is of great significance to evaluate the governance effect of ecological restoration projects.

Intelligent Identification Method of Crop Species Using Improved U-Net Network in UAV Remote Sensing Image

Aiming at the problems of incomplete classification features extraction of remote sensing images and low accuracy of crop classification in existing crop classification and recognition methods, a crop classification and recognition method using improved U-Net in Unmanned Aerial Vehicle (UAV) remote sensing images is proposed. First, the experimental data set is preprocessed. The data set is expanded by flipping transformation, translation transformation, and random cutting, which expands the number of data sets, and then the original image is cut to meet the requirements of the experiment for image size; Moreover, the network structure of U-Net is deepened, and the atrous spatial pyramid pooling (ASPP) structure is introduced after the encoder to better understand the semantic information and improve the ability of model mining data features; Finally, in order to prevent the overfitting of the deepened model, the dropout layer is introduced to weaken the joint adaptability between various neurons. Experiments show that the comprehensive OA and Kappa indexes of the proposed crop classification and recognition method based on improved U-Net are 92.14% and 0.896, respectively, which are better than the comparison methods. Therefore, the proposed method has good ability of crop classification and recognition.

DBGAN: A dual-branch generative adversarial network for undersampled MRI reconstruction

Compressed sensing magnetic resonance imaging (CS-MRI) greatly accelerates the acquisition process and yield considerable reconstructed images. Deep learning was introduced into CS-MRI to further speed up the reconstruction process and improve the image quality. Recently, generative adversarial network (GAN) using two-stage cascaded U-Net structure as generator has been proven to be effective in MRI reconstruction. However, previous cascaded structure was limited to few feature information propagation channels thus may lead to information missing. In this paper, we proposed a GAN-based model, DBGAN, for MRI reconstruction from undersampled k-space data. The model uses cross-stage skip connection (CSSC) between two end-to-end cascaded U-Net in our generator to widen the channels of feature propagation. To avoid discrepancy between training and inference, we replaced classical batch normalization (BN) with instance normalization (IN) . A stage loss is involved in the loss function to boost the training performance. In addition, a bilinear interpolation decoder branch is introduced in the generator to supplement the missing information of the deconvolution decoder. Tested under five variant patterns with four undersampling rates on different modality of MRI data, the quantitative results show that DBGAN model achieves mean improvements of 3.65 dB in peak signal-to-noise ratio (PSNR) and 0.016 in normalized mean square error (NMSE) compared with state-of-the-art GAN-based methods on T1-Weighted brain dataset from MICCAI 2013 grand challenge. The qualitative visual results show that our method can reconstruct considerable images on brain and knee MRI data from different modality. Furthermore, DBGAN is light and fast – the model parameters are fewer than half of state-of-the-art GAN-based methods and each 256 × 256 image is reconstructed in 60 milliseconds, which is suitable for real-time processing.

Kidney Tumor Segmentation Based on FR2PAttU-Net Model.

The incidence rate of kidney tumors increases year by year, especially for some incidental small tumors. It is challenging for doctors to segment kidney tumors from kidney CT images. Therefore, this paper proposes a deep learning model based on FR2PAttU-Net to help doctors process many CT images quickly and efficiently and save medical resources. FR2PAttU-Net is not a new CNN structure but focuses on improving the segmentation effect of kidney tumors, even when the kidney tumors are not clear. Firstly, we use the R2Att network in the “U” structure of the original U-Net, add parallel convolution, and construct FR2PAttU-Net model, to increase the width of the model, improve the adaptability of the model to the features of different scales of the image, and avoid the failure of network deepening to learn valuable features. Then, we use the fuzzy set enhancement algorithm to enhance the input image and construct the FR2PAttU-Net model to make the image obtain more prominent features to adapt to the model. Finally, we used the KiTS19 data set and took the size of the kidney tumor as the category judgment standard to enhance the small sample data set to balance the sample data set. We tested the segmentation effect of the model at different convolution and depths, and we got scored a 0.948 kidney Dice and a 0.911 tumor Dice results in a 0.930 composite score, showing a good segmentation effect.

A Novel Elastomeric UNet for Medical Image Segmentation.

Medical image segmentation is of important support for clinical medical applications. As most of the current medical image segmentation models are limited in the U-shaped structure, to some extent the deep convolutional neural network (CNN) structure design is hard to be accomplished. The design in this study mimics the way the wave is elastomeric propagating, extending the structure from both the horizontal and spatial dimensions for realizing the Elastomeric UNet (EUNet) structure. The EUNet can be divided into two types: horizontal EUNet and spatial EUNet, based on the propagation direction. The advantages of this design are threefold. First, the training structure can be deepened effectively. Second, the independence brought by each branch (a U-shaped design) makes the flexible design redundancy available. Finally, a horizontal and vertical series-parallel structure helps on feature accumulation and recursion. Researchers can adjust the design according to the requirements to achieve better segmentation performance for the independent structural design. The proposed networks were evaluated on two datasets: a self-built dataset (multi-photon microscopy, MPM) and publicly benchmark retinal datasets (DRIVE). The results of experiments demonstrated that the performance of EUNet outperformed the UNet and its variants.

Fast and accurate dose predictions for novel radiotherapy treatments in heterogeneous phantoms using conditional 3D‐UNet generative adversarial networks

Novel radiotherapy techniques like synchrotron X-ray microbeam radiation therapy (MRT) require fast dose distribution predictions that are accurate at the sub-mm level, especially close to tissue/bone/air interfaces. Monte Carlo (MC) physics simulations are recognized to be one of the most accurate tools to predict the dose delivered in a target tissue but can be very time consuming and therefore prohibitive for treatment planning. Faster dose prediction algorithms are usually developed for clinically deployed treatments only. In this work, we explore a new approach for fast and accurate dose estimations suitable for novel treatments using digital phantoms used in preclinical development and modern machine learning techniques. We develop a generative adversarial network (GAN) model, which is able to emulate the equivalent Geant4 MC simulation with adequate accuracy and use it to predict the radiation dose delivered by a broad synchrotron beam to various phantoms. The energy depositions used for the training of the GAN are obtained using full Geant4 MC simulations of a synchrotron radiation broad beam passing through the phantoms. The energy deposition is scored and predicted in voxel matrices of size 140 × 18 × 18 with a voxel edge length of 1 mm. The GAN model consists of two competing 3D convolutional neural networks, which are conditioned on the photon beam and phantom properties. The generator network has a U-Net structure and is designed to predict the energy depositions of the photon beam inside three phantoms of variable geometry with increasing complexity. The critic network is a relatively simple convolutional network, which is trained to distinguish energy depositions predicted by the generator from the ones obtained with the full MC simulation. The energy deposition predictions inside all phantom geometries under investigation show deviations of less than 3% of the maximum deposited energy from the simulation for roughly 99% of the voxels in the field of the beam. Inside the most realistic phantom, a simple pediatric head, the model predictions deviate by less than 1% of the maximal energy deposition from the simulations in more than 96% of the in-field voxels. For all three phantoms, the model generalizes the energy deposition predictions well to phantom geometries, which have not been used for training the model but are interpolations of the training data in multiple dimensions. The computing time for a single prediction is reduced from several hundred hours using Geant4 simulation to less than a second using the GAN model. The proposed GAN model predicts dose distributions inside unknown phantoms with only small deviations from the full MC simulation with computations times of less than a second. It demonstrates good interpolation ability to unseen but similar phantom geometries and is flexible enough to be trained on data with different radiation scenarios without the need for optimization of the model parameter. This proof-of-concept encourages to apply and further develop the model for the use in MRT treatment planning, which requires fast and accurate predictions with sub-mm resolutions.

Deep Learning Approach for Cardiac MRI Images

Deep Learning (DL) is the most widely used image-analysis process, especially in medical image processing. Though DL has entered image processing to solve Machine Learning (ML) problems, identifying the most suitable model based on evaluation of the epochs is still an open question for scholars in the field. There are so many types of function approximators like Decision Tree, Gaussian Processes and Deep Learning, used in multi-layered Neural Networks (NNs), which should be evaluated to determine their effectiveness. Therefore, this study aimed to assess an approach based on DL techniques for modern medical imaging methods according to Magnetic Resonance Imaging (MRI) segmentation. To do so, an experiment with a random sampling approach was conducted. One hundred patient cases were used in this study for training, validation, and testing. The method used in this study was based on full automatic processing of segmentation and disease classification based on MRI images. U-Net structure was used for the segmentation process, with the use of cardiac Right Ventricular Cavity (RVC), Left Ventricular Cavity (LVC), Left Ventricular Myocardium (LVM), and information extracted from the segmentation step. With train and using random forest classifier, and Multilayer Perceptron (MLP), the task of predicting the pathologic target class was conducted. Segmentation extracted information was in the form of comprehensive features handcrafted to reflect demonstrative clinical strategies. Our study suggests 92% test accuracy for cardiac MRI image segmentation and classification. As for the MLP ensemble, and for the random forest, test accuracy was equal to 91% and 90%, respectively. This study has implications for scholars in the field of medical image processing.

Elimination of stripe artifacts in light sheet fluorescence microscopy using an attention-based residual neural network.

Stripe artifacts can deteriorate the quality of light sheet fluorescence microscopy (LSFM) images. Owing to the inhomogeneous, high-absorption, or scattering objects located in the excitation light path, stripe artifacts are generated in LSFM images in various directions and types, such as horizontal, anisotropic, or multidirectional anisotropic. These artifacts severely degrade the quality of LSFM images. To address this issue, we proposed a new deep-learning-based approach for the elimination of stripe artifacts. This method utilizes an encoder-decoder structure of UNet integrated with residual blocks and attention modules between successive convolutional layers. Our attention module was implemented in the residual blocks to learn useful features and suppress the residual features. The proposed network was trained and validated by generating three different degradation datasets with different types of stripe artifacts in LSFM images. Our method can effectively remove different stripes in generated and actual LSFM images distorted by stripe artifacts. Besides, quantitative analysis and extensive comparison results demonstrated that our method performs the best compared with classical image-based processing algorithms and other powerful deep-learning-based destriping methods for all three generated datasets. Thus, our method has tremendous application prospects to LSFM, and its use can be easily extended to images reconstructed by other modalities affected by the presence of stripe artifacts.

Binarization of Printed Wiring Board Images with Scratches Using Convolutional Neural Network with U-Net Structure

This paper presents the method by which binarize Printed Wiring Board(PWB) images. Multilayer printed wired boards have many wiring layers inside the board, and the images capturing these wiring layers contains a large number of scratches and stains, because the wiring layer images are captured while the surface is thinly scraped. We aim to recover the wiring pattern as a binarized image from such images. We have devised an input layer based on the U-Net structure, which consists of 1) 3-channel RGB input (conventional U-Net), 2) 5-channel input by adding L and S channels of HLS color space to RGB, 3) 4-channel input by adding a binary image by Otsu's method of L to RGB, 4) a loop that repeats the structure of method 3), and compared them. As a result, we found that method 3) with Otsu's binarization added to the input was superior in all the indices of accuracy, F-measure, mIoU, and the number of breaks and short circuits.