Deep Encoder-decoder Network Research Articles

Web-SpikeSegNet: Deep Learning Framework for Recognition and Counting of Spikes From Visual Images of Wheat Plants

Computer vision with deep learning is emerging as a significant approach for non-invasive and non-destructive plant phenotyping. Spikes are the reproductive organs of wheat plants. Detection and counting of spikes considered the grain-bearing organ have great importance in the phenomics study of large sets of germplasms. In the present study, we developed an online platform, “Web-SpikeSegNet,” based on a deep-learning framework for spike detection and counting from the wheat plant’s visual images. The architecture of the Web-SpikeSegNet consists of 2 layers. First Layer, Client-Side Interface Layer, deals with end user’s requests and corresponding responses management. In contrast, the second layer, Server Side Application Layer, consists of a spike detection and counting module. The backbone of the spike detection module comprises of deep encoder-decoder network with hourglass network for spike segmentation. The Spike counting module implements the “Analyze Particle” function of imageJ to count the number of spikes. For evaluating the performance of Web-SpikeSegNet, we acquired the wheat plant’s visual images, and the satisfactory segmentation performances were obtained as Type I error 0.00159, Type II error 0.0586, Accuracy 99.65%, Precision 99.59% and F1 score 99.65%. As spike detection and counting in wheat phenotyping are closely related to the yield, Web-SpikeSegNet is a significant step forward in the field of wheat phenotyping and will be very useful to the researchers and students working in the domain.

Non-Rigid Respiratory Motion Estimation of Whole-Heart Coronary MR Images Using Unsupervised Deep Learning.

Non-rigid motion-corrected reconstruction has been proposed to account for the complex motion of the heart in free-breathing 3D coronary magnetic resonance angiography (CMRA). This reconstruction framework requires efficient and accurate estimation of non-rigid motion fields from undersampled images at different respiratory positions (or bins). However, state-of-the-art registration methods can be time-consuming. This article presents a novel unsupervised deep learning-based strategy for fast estimation of inter-bin 3D non-rigid respiratory motion fields for motion-corrected free-breathing CMRA. The proposed 3D respiratory motion estimation network (RespME-net) is trained as a deep encoder-decoder network, taking pairs of 3D image patches extracted from CMRA volumes as input and outputting the motion field between image patches. Using image warping by the estimated motion field, a loss function that imposes image similarity and motion smoothness is adopted to enable training without ground truth motion field. RespME-net is trained patch-wise to circumvent the challenges of training a 3D network volume-wise which requires large amounts of GPU memory and 3D datasets. We perform 5-fold cross-validation with 45 CMRA datasets and demonstrate that RespME-net can predict 3D non-rigid motion fields with subpixel accuracy (0.44 ± 0.38 mm) within ~10 seconds, being ~20 times faster than a GPU-implemented state-of-the-art non-rigid registration method. Moreover, we perform non-rigid motion-compensated CMRA reconstruction for 9 additional patients. The proposed RespME-net has achieved similar motion-corrected CMRA image quality to the conventional registration method regarding coronary artery length and sharpness.

Capsules for biomedical image segmentation.

Our work expands the use of capsule networks to the task of object segmentation for the first time in the literature. This is made possible via the introduction of locally-constrained routing and transformation matrix sharing, which reduces the parameter/memory burden and allows for the segmentation of objects at large resolutions. To compensate for the loss of global information in constraining the routing, we propose the concept of "deconvolutional" capsules to create a deep encoder-decoder style network, called SegCaps. We extend the masked reconstruction regularization to the task of segmentation and perform thorough ablation experiments on each component of our method. The proposed convolutional-deconvolutional capsule network, SegCaps, shows state-of-the-art results while using a fraction of the parameters of popular segmentation networks. To validate our proposed method, we perform experiments segmenting pathological lungs from clinical and pre-clinical thoracic computed tomography (CT) scans and segmenting muscle and adipose (fat) tissue from magnetic resonance imaging (MRI) scans of human subjects' thighs. Notably, our experiments in lung segmentation represent the largest-scale study in pathological lung segmentation in the literature, where we conduct experiments across five extremely challenging datasets, containing both clinical and pre-clinical subjects, and nearly 2000 computed-tomography scans. Our newly developed segmentation platform outperforms other methods across all datasets while utilizing less than 5% of the parameters in the popular U-Net for biomedical image segmentation. Further, we demonstrate capsules' ability to generalize to unseen handling of rotations/reflections on natural images.

Semantic Labeling in Remote Sensing Corpora Using Feature Fusion-Based Enhanced Global Convolutional Network with High-Resolution Representations and Depthwise Atrous Convolution

One of the fundamental tasks in remote sensing is the semantic segmentation on the aerial and satellite images. It plays a vital role in applications, such as agriculture planning, map updates, route optimization, and navigation. The state-of-the-art model is the Enhanced Global Convolutional Network (GCN152-TL-A) from our previous work. It composes two main components: (i) the backbone network to extract features and ( i i ) the segmentation network to annotate labels. However, the accuracy can be further improved, since the deep learning network is not designed for recovering low-level features (e.g., river, low vegetation). In this paper, we aim to improve the semantic segmentation network in three aspects, designed explicitly for the remotely sensed domain. First, we propose to employ a modern backbone network called “High-Resolution Representation (HR)” to extract features with higher quality. It repeatedly fuses the representations generated by the high-to-low subnetworks with the restoration of the low-resolution representations to the same depth and level. Second, “Feature Fusion (FF)” is added to our network to capture low-level features (e.g., lines, dots, or gradient orientation). It fuses between the features from the backbone and the segmentation models, which helps to prevent the loss of these low-level features. Finally, “Depthwise Atrous Convolution (DA)” is introduced to refine the extracted features by using four multi-resolution layers in collaboration with a dilated convolution strategy. The experiment was conducted on three data sets: two private corpora from Landsat-8 satellite and one public benchmark from the “ISPRS Vaihingen” challenge. There are two baseline models: the Deep Encoder-Decoder Network (DCED) and our previous model. The results show that the proposed model significantly outperforms all baselines. It is the winner in all data sets and exceeds more than 90% of F 1 : 0.9114, 0.9362, and 0.9111 in two Landsat-8 and ISPRS Vaihingen data sets, respectively. Furthermore, it achieves an accuracy beyond 90% on almost all classes.

Look More Into Occlusion: Realistic Face Frontalization and Recognition With BoostGAN.

Many factors can affect face recognition, such as occlusion, pose, aging, and illumination. First and foremost are occlusion and large-pose problems, which may even lead to more than 10% accuracy degradation. Recently, generative adversarial net (GAN) and its variants have been proved to be effective in processing pose and occlusion. For the former, pose-invariant feature representation and face frontalization based on GAN models have been studied to solve the pose variation problem. For the latter, frontal face completion on occlusions based on GAN models have also been presented, which is much concerned with facial structure and realistic pixel details rather than identity preservation. However, synthesizing and recognizing the occluded but profile faces is still an understudied problem. Therefore, in this article, to address this problem, we contribute an efficient but effective solution on how to synthesize and recognize faces with large-pose variations and simultaneously corrupted regions (e.g., nose and eyes). Specifically, we propose a boosting GAN (BoostGAN) for occluded but profile face frontalization, deocclusion, and recognition, which has two aspects: 1) with the assumption that face occlusion is incomplete and partial, multiple images with patch occlusion are fed into our model for knowledge boosting, i.e., identity and texture information and 2) a new aggregation structure integrated with a deep encoder-decoder network for coarse face synthesis and a boosting network for fine face generation is carefully designed. Exhaustive experiments on benchmark data sets with regular and irregular occlusions demonstrate that the proposed model not only shows clear photorealistic images but also presents powerful recognition performance over state-of-the-art GAN models for occlusive but profile face recognition in both the controlled and uncontrolled environments. To the best of our knowledge, this article proposes to solve face synthesis and recognition under poses and occlusions for the first time.

Cascaded deep learning-based efficient approach for license plate detection and recognition

Automatic license plate (ALP) detection and recognition is an important task for both traffic surveillance and parking management systems, as well as being crucial to maintaining the flow of modern civic life. Various ALP detection and recognition methods have been proposed to date. These methods generally use various image processing and machine learning techniques. In this paper, a cascaded deep learning approach is proposed in order to construct an efficient ALP detection and recognition system for the vehicles of northern Iraq. The license plates in northern Iraq contain three regions, namely a plate number, a city region, and a country region. The proposed method initially employs several preprocessing techniques such as Gaussian filtering and adaptive image contrast enhancement to make the input images more suited to further processing. Then, a deep semantic segmentation network is used in order to determine the three license plate regions of the input image. Segmentation is then carried out via deep encoder-decoder network architecture. The determined license plate regions are fed into two separate convolutional neural network (CNN) models for both Arabic number recognition and the city determination. For Arabic number recognition, an end-to-end CNN model was constructed and trained, whilst for the city recognition, a pretrained CNN model was further fine-tuned. A new license plate dataset was also constructed and used in the experimental works of the study. The performance of the proposed method was evaluated both in terms of detection and recognition. For detection, recall, precision and F-measure scores were used, and for recognition, classification accuracy was used. The obtained results showed the proposed method to be efficient in both license plate detection and recognition. The calculated recall, precision and F-measure scores were 92.10%, 94.43%, and 91.01%, respectively. Moreover, the classification accuracies for Arabic numbers and city labels were shown to be 99.37% and 92.26%, respectively.

Representation learning in a deep network for license plate recognition

The goal of license plate recognition (LPR) is to read the license plate characters. Due to image degradation, there are many difficulties in the way of achieving this goal. In this paper, the proposed method recognizes the license plate characters without employing the traditional segmentation and binarization techniques. This method uses a deep learning algorithm and tries to achieve better learning experience by engaging a multi-task learning algorithm based on sharing features. The features of license plate characters are extracted by a deep encoder-decoder network, and transferred to 8 parallel classifiers for recognition. To evaluate the current work, a database of 11,000 license plate images, collected from a currently working surveillance system installed on a dual carriageway, is employed. The proposed method achieved the correct character recognition rate of 96% for 4000 test images that is acceptable in comparison to the competing methods.

Building extraction in pixel level from aerial imagery with a deep encoder-decoder network

Building extraction in pixel level from aerial imagery with a deep encoder-decoder network

Neural machine translation of Hindi and English

Translation has been one of the oldest problems in natural language processing. Despite its age, it is still one where there is a tremendous scope for improvement and creativity; the quantity and quality of research in it is testament to that fact. The subfield of primarily using deep neural networks for translation has recently started to gain traction. Many techniques have been developed using deep encoder-decoder networks for bilingual translation using both parallel as well as non-parallel corpora. There is a lot of potential in applying concepts such as bilingual embeddings to create generic translation architecture, which doesn’t need huge parallel corpora to train. These ideas are particularly pertinent in the case of Indic languages, where it is generally difficult to obtain such corpus. In this paper, we try to adapt some of newest techniques in autoencoder networks and bilingual embeddings to the task of translating between English and Hindi. The models considerably outperform state of the art translating systems for these languages.

Contour-Aware Polyp Segmentation in Colonoscopy Images Using Detailed Upsampling Encoder-Decoder Networks

Colorectal cancer has become one of the most common cause of cancer mortality worldwide, with a five-year survival rate of over 50%. Additionally, the potential of some common polyp types to progress to colorectal cancer is considered high. Colonoscopy is the most common method for finding and removing polyps. However, during colonoscopy, a significant number of polyps is missed as a result of human error mistakes. Thus, this study was primarily motivated by the need to obtain an early and accurate diagnosis of polyps detected in colonoscopy images. In this paper, we propose a new polyp segmentation method based on an architecture of multi-model deep encoder-decoder networks called MED-Net. Not only does this architecture obtain multi-level contextual information by extracting discriminative features at different effective fields-of-view and multiple image scales, it also can substantially do upsample more correctly to produce better prediction. It is also able to capture more accurate polyp boundaries by using multi-scale effective decoders. Moreover, we also present a complementary strategy for improving the method's segmentation performance based on a combination of a boundary-aware data augmentation method and an effective weighted loss function. The purpose of this strategy is to allow our deep learning network to sequentially focus on poorly defined polyp boundaries, which are caused by the non-specular transition zone between the polyp and non-polyp regions. To provide a general view of the proposed method, our network was trained and evaluated on four well-known dataset CVC-ColonDB, CVC-ClinicDB, ASU-Mayo Clinic Colonoscopy Video Database, and ETIS-LaribPolypDB. Our results show that our MED-Net significantly outperforms state-of-the-art methods.

3D deep encoder-decoder network for fluorescence molecular tomography.

Fluorescence molecular tomography (FMT) is a promising and noninvasive in vivo functional imaging modality. However, the quality of FMT reconstruction is limited by the simplified linear model of photon propagation. Here, an end-to-end three-dimensional deep encoder-decoder (3D-En-Decoder) network is proposed to improve the quality of FMT reconstruction. It directly establishes the nonlinear mapping relationship between the inside fluorescent source distribution and the boundary fluorescent signal distribution. Thus the reconstruction inaccuracy caused by the simplified linear model can be fundamentally avoided by the proposed network. Both numerical simulations and phantom experiments were carried out, and the results demonstrated that the 3D-En-Decoder network can greatly improve image quality and significantly reduce reconstruction time compared with conventional methods.

DeepPET: A deep encoder-decoder network for directly solving the PET image reconstruction inverse problem.

The purpose of this research was to implement a deep learning network to overcome two of the major bottlenecks in improved image reconstruction for clinical positron emission tomography (PET). These are the lack of an automated means for the optimization of advanced image reconstruction algorithms, and the computational expense associated with these state-of-the art methods. We thus present a novel end-to-end PET image reconstruction technique, called DeepPET, based on a deep convolutional encoder-decoder network, which takes PET sinogram data as input and directly and quickly outputs high quality, quantitative PET images. Using simulated data derived from a whole-body digital phantom, we randomly sampled the configurable parameters to generate realistic images, which were each augmented to a total of more than 291,000 reference images. Realistic PET acquisitions of these images were simulated, resulting in noisy sinogram data, used for training, validation, and testing the DeepPET network. We demonstrated that DeepPET generates higher quality images compared to conventional techniques, in terms of relative root mean squared error (11%/53% lower than ordered subset expectation maximization (OSEM)/filtered back-projection (FBP), structural similarity index (1%/11% higher than OSEM/FBP), and peak signal-to-noise ratio (1.1/3.8dB higher than OSEM/FBP). In addition, we show that DeepPET reconstructs images 108 and 3 times faster than OSEM and FBP, respectively. Finally, DeepPET was successfully applied to real clinical data. This study shows that an end-to-end encoder-decoder network can produce high quality PET images at a fraction of the time compared to conventional methods.

Coarse-to-fine road scene segmentation via hierarchical graphical models

The road scene segmentation is an important problem which is helpful for a higher level of the scene understanding. This article presents a novel approach for image semantic segmentation of road scenes via a hierarchical graph-based inference. A deep encoder–decoder network is first applied for a fast pixel-wise classification. Then, hierarchical graph-based inference is performed to get an accurate segmentation result. In the inference process, all the object classes are grouped into fewer categories which contains at least one class. The category labels are assigned to image superpixels using Markov random field model. For each category, a pixel-level labeling based on fully connected conditional random fields is performed to divide image into different classes. After the inference for all categories, the results are integrated together to get the final segmentation. In additional to low-level affinity functions, the feature maps from network are integrated in pairwise potentials of the graphical models. This hierarchical inference scheme can alleviate the confusion of classes belonging to different categories. It performs well for small objects without adding more computational burden. Both qualitative and quantitative assessments are adopted to evaluate the proposed method. The results on benchmark data sets prove the effectiveness of the proposed hierarchical scheme, and the performance is competitive with the state-of-the-art methods.

MSDF-Net: Multi-Scale Deep Fusion Network for Stroke Lesion Segmentation

Lesion segmentation is of great research interest due to its capability in facilitating accurate stroke diagnosis and surgical planning. Existing deep neural networks, such as U-net, have demonstrated encouraging progress in biomedical image segmentation. Nevertheless, there are still many challenges related to the segmentation of stroke lesions, including dealing with diverse lesion locations, variations in lesion scales, and fuzzy lesion boundaries. In order to address these challenges, this paper proposes a deep neural network architecture denoted as the Multi-Scale Deep Fusion Network (MSDF-Net) with Atrous Spatial Pyramid Pooling (ASPP) for the feature extraction at different scales, and the inclusion of capsules to deal with complicated relative entities. The proposed method is essentially an end-to-end deep encoder-decoder neural network. The cross connection between the encoder and the decoder guarantees the high resolution of the feature mapping. Experimental results on the open-source Anatomical Tracings of Lesions After Stroke (ATLAS) dataset shows that the proposed model achieved a higher evaluating score compared to 5 existing models.

Robust Boundary Segmentation in Medical Images Using a Consecutive Deep Encoder-Decoder Network

Image segmentation is typically used to locate objects and boundaries. It is essential in many clinical applications, such as the pathological diagnosis of hepatic diseases, surgical planning, and postoperative assessment. The segmentation task is hampered by fuzzy boundaries, complex backgrounds, and appearances of objects of interest, which vary considerably. The success of the procedure is still highly dependent on the operator’s skills and the level of hand–eye coordination. Thus, this paper was strongly motivated by the necessity to obtain an early and accurate diagnosis of a detected object in medical images. In this paper, we propose a new polyp segmentation method based on the architecture of a multiple deep encoder–decoder networks combination called CDED-net. The architecture can not only hold multi-level contextual information by extracting discriminative features at different effective fields-of-view and multiple image scales but also learn rich information features from missing pixels in the training phase. Moreover, the network is also able to capture object boundaries by using multiscale effective decoders. We also propose a novel strategy for improving the method’s segmentation performance based on a combination of a boundary-emphasization data augmentation method and a new effective dice loss function. The goal of this strategy is to make our deep learning network available with poorly defined object boundaries, which are caused by the non-specular transition zone between the background and foreground regions. To provide a general view of the proposed method, our network was trained and evaluated on three well-known polyp datasets, CVC-ColonDB, CVC-ClinicDB, and ETIS-Larib PolypDB. Furthermore, we also used the Pedro Hispano Hospital (PH2), ISBI 2016 skin lesion segmentation dataset, and CT healthy abdominal organ segmentation dataset to depict our network’s ability. Our results reveal that the CDED-net significantly surpasses the state-of-the-art methods.

Monocular Vision-Based Obstacle Detection and Avoidance for a Multicopter

This article presents a monocular vision-based algorithm for detecting obstacles and identifying obstacle-aware regions, developed to be used for collision avoidance by a multicopter. The first step of our algorithm is to predict a disparity image from a single-view image via implementing a deep encoder-decoder network. All pixels in this disparity prediction are then categorized as one of three classes, obstacle, road, or obstacle-free, by combining V-disparity analysis and a fuzzy inference system. For pixels belonging to obstacle objects, obstacle-aware regions are generated within the field of visual perception. To accommodate the safety margins of a multicopter, intermediate waypoints are then added to obtain a new flyable path that passes through an unknown environment safely. Experimental results verified the effectiveness of the detection of obstacles and the identification of obstacle-aware regions. The accuracy of disparity prediction and monocular depth estimation were quantitatively compared to support the feasibility of monocular vision in obstacle avoidance. Furthermore, the entire algorithm was successfully tested on a robotic platform, autonomously flying a hexacopter in an outdoor space with obstacles. In conclusion, the proposed monocular algorithm performs well for obstacle detection and depth estimation and is potentially an alternative to a binocular solution.

Prominent edge detection with deep metric expression and multi-scale features

Edge detection is one of today’s hottest computer vision issues with widely applications. It is beneficial for improving the capability of many vision systems, such as semantic segmentation, salient object detection and object recognition. Deep convolution neural networks (CNNs) recently have been employed to extract robust features, and have achieved a definite improvement. However, there is still a long run to study this hotspot with the main reason that CNNs-based approaches may cause the edges thicker. To address this problem, a novel semantic edge detection algorithm using multi-scale features is proposed. Our model is deep symmetrical metric learning network, which includes 3 key parts. Firstly, the deep detail layer, as a preprocessing layer and a guide module, is employed to remove some low-frequency information and still maintain the edge. Secondly, the deep encoder-decoder networks extract multi-scale features of original image, integrated for complementing information among each level feature. Finally, metric learning is introduced to generate a metric space used to predict edge result. It is easy to distinguish different categories, such as edge space and object space. Simulations and comparisons on benchmark datasets demonstrate the proposed algorithm is superior to the others through visual and quantitative evaluation, and specifically, the score of ODS reachs 0.788.

Fusion of encoder-decoder deep networks improves delineation of multiple nuclear phenotypes

BackgroundNuclear segmentation is an important step for profiling aberrant regions of histology sections. If nuclear segmentation can be resolved, then new biomarkers of nuclear phenotypes and their organization can be predicted for the application of precision medicine. However, segmentation is a complex problem as a result of variations in nuclear geometry (e.g., size, shape), nuclear type (e.g., epithelial, fibroblast), nuclear phenotypes (e.g., vesicular, aneuploidy), and overlapping nuclei. The problem is further complicated as a result of variations in sample preparation (e.g., fixation, staining). Our hypothesis is that (i) deep learning techniques can learn complex phenotypic signatures that rise in tumor sections, and (ii) fusion of different representations (e.g., regions, boundaries) contributes to improved nuclear segmentation.ResultsWe have demonstrated that training of deep encoder-decoder convolutional networks overcomes complexities associated with multiple nuclear phenotypes, where we evaluate alternative architecture of deep learning for an improved performance against the simplicity of the design. In addition, improved nuclear segmentation is achieved by color decomposition and combining region- and boundary-based features through a fusion network. The trained models have been evaluated against approximately 19,000 manually annotated nuclei, and object-level Precision, Recall, F1-score and Standard Error are reported with the best F1-score being 0.91. Raw training images, annotated images, processed images, and source codes are released as a part of the Additional file 1.ConclusionsThere are two intrinsic barriers in nuclear segmentation to H&E stained images, which correspond to the diversity of nuclear phenotypes and perceptual boundaries between adjacent cells. We demonstrate that (i) the encoder-decoder architecture can learn complex phenotypes that include the vesicular type; (ii) delineation of overlapping nuclei is enhanced by fusion of region- and edge-based networks; (iii) fusion of ENets produces an improved result over the fusion of UNets; and (iv) fusion of networks is better than multitask learning. We suggest that our protocol enables processing a large cohort of whole slide images for applications in precision medicine.

Multi-scale deep encoder-decoder network for salient object detection

Abstract Deep convolutional neural networks (CNNs) have recently made revolutionary improvements in salient object detection. However, most existing CNN-based models fail to precisely separate the whole salient object(s) from a cluttered background due to the downsampling effects or the patch-level operation. In this paper, we propose a multi-scale deep encoder-decoder network which learns discriminative saliency cues and computes confidence scores in an end-to-end fashion. The encoder network extracts meaningful and informative features in a global view, and the decoder network recovers lost detailed object structure in a local perspective. By taking multiple resized images as the inputs, the proposed model incorporates multi-scale features from a shared network and predicts a fine-grained saliency map at the pixel level. To easily and efficiently train the whole network, the light-weighted decoder breaks through the limit of conventional symmetric structure. In addition, a two-stage training strategy is designed to encourage the robustness and accuracy of the network. Without any post-processing steps, our method is capable of significantly reducing the computation complexity while densely segmenting foreground objects from an image. Extensive experiments on six challenging datasets demonstrate that the proposed model outperforms other state-of-the-art approaches in terms of various evaluation metrics.

Quicksilver: Fast predictive image registration – A deep learning approach

This paper introduces Quicksilver, a fast deformable image registration method. Quicksilver registration for image-pairs works by patch-wise prediction of a deformation model based directly on image appearance. A deep encoder-decoder network is used as the prediction model. While the prediction strategy is general, we focus on predictions for the Large Deformation Diffeomorphic Metric Mapping (LDDMM) model. Specifically, we predict the momentum-parameterization of LDDMM, which facilitates a patch-wise prediction strategy while maintaining the theoretical properties of LDDMM, such as guaranteed diffeomorphic mappings for sufficiently strong regularization. We also provide a probabilistic version of our prediction network which can be sampled during the testing time to calculate uncertainties in the predicted deformations. Finally, we introduce a new correction network which greatly increases the prediction accuracy of an already existing prediction network. We show experimental results for uni-modal atlas-to-image as well as uni-/multi-modal image-to-image registrations. These experiments demonstrate that our method accurately predicts registrations obtained by numerical optimization, is very fast, achieves state-of-the-art registration results on four standard validation datasets, and can jointly learn an image similarity measure. Quicksilver is freely available as an open-source software.