Neural Network For Semantic Segmentation Research Articles

Clustering of spatial cues by semantic segmentation for anechoic binaural source separation

The recent introduction of neural networks to speech separation has dramatically boosted the separation performance. This paper presents a novel psychoacoustic approach for speech source separation in anechoic conditions, using semantic segmentation of the interaural spectrograms of the audio mixtures. We have trained two separate U-Nets (a specialized neural network for semantic segmentation) on the interaural level difference (ILD) spectrogram, and the interaural phase difference (IPD) spectrogram of a single source. After training, these U-Nets are used to predict the class of each time frequency (TF) unit of the interaural spectrogram of the audio mixture. The ILD and IPD soft masks obtained from these U-Nets are combined by a novel scheme which utilizes the strength of the interaural cues in different frequency bands. The results show improved separation over two state of the art machine learning source separation systems utilizing the same interaural cues. There is average improvement of 7.32 dB in signal to distortion ratio (SDR) and 0.3 points improvement in short term objective intelligibility (STOI) over degenerate un-mixing estimation technique (DUET) algorithm and 2.51 dB improvement in SDR with comparable intelligibility over model-based expectation–maximization source separation and localization (MESSL) algorithm.

Failure signature classification in solar photovoltaic plants using RGB images and convolutional neural networks

Physical fault detection in panels that are part of photovoltaic (PV) plants typically involves the analysis of thermal and electroluminescent images, which makes it either difficult or impossible to identify the source of the fault in the plant. This paper proposes a method of automatic physical fault classification for PV plants using convolutional neural networks for semantic segmentation and classification from RGB images. This study shows experimental results for 2 output classes identified as a fault and no fault, and 4 output classes as no fault, cracks, shadows, and dust that cannot be easily detected. The proposed method presents an average accuracy of 75% for 2 output classes and 70% for 4 classes, showing a positive approach to the proposed classification method for PV systems.

SEMI-SUPERVISED SEMANTIC SEGMENTATION NETWORK VIA LEARNING CONSISTENCY FOR REMOTE SENSING LAND-COVER CLASSIFICATION

Abstract. Current popular deep neural networks for semantic segmentation are almost supervised and highly rely on a large amount of labeled data. However, obtaining a large amount of pixel-level labeled data is time-consuming and laborious. In remote sensing area, this problem is more urgent. To alleviate this problem, we propose a novel semantic segmentation neural network (S4Net) based on semi-supervised learning by using unlabeled data. Our model can learn from unlabeled data by consistency regularization, which enforces the consistency of output under different random transforms and perturbations, such as random affine transform. Thus, the network is trained by the weighted sum of a supervised loss from labeled data and a consistency regularization loss from unlabeled data. The experiments we conducted on DeepGlobe land cover classification challenge dataset verified that our network can make use of unlabeled data to obtain precise results of semantic segmentation and achieve competitive performance when compared to other methods.

Enabling Robust Horizon Picking From Small Training Sets

Seismic interpretation is a complex procedure that depends on many and interdependent data analyses. One of the essential steps in this process is picking horizons in seismic images, which is time-consuming and prone to errors when performed manually. In this context, having a reliable horizon picking tool is fundamental for accurate seismic interpretation. Although several methods for horizon picking have been proposed in the literature and many tools made available in the industry, most require numerous iterations and manual corrections for delivering satisfactory results. In this article, we present a three-step approach that improves the robustness of horizon picking using state-of-art semantic segmentation neural networks followed by geometry-based processing of horizon points to identify and remove outliers. For the well-known Netherlands F3 Block data set, this approach allows accurate results from a few training annotated inlines and delivers geometry-consistent horizons through all the cubes. We further present results for an additional set comprising four cubes with different seismic structures to provide some robustness evidence of our approach. The results reported in this study indicate that the proposed methodology can provide a good trade-off between accuracy and time spent on manually picking horizons.

Panoptic Segmentation of Individual Pigs for Posture Recognition.

Behavioural research of pigs can be greatly simplified if automatic recognition systems are used. Systems based on computer vision in particular have the advantage that they allow an evaluation without affecting the normal behaviour of the animals. In recent years, methods based on deep learning have been introduced and have shown excellent results. Object and keypoint detector have frequently been used to detect individual animals. Despite promising results, bounding boxes and sparse keypoints do not trace the contours of the animals, resulting in a lot of information being lost. Therefore, this paper follows the relatively new approach of panoptic segmentation and aims at the pixel accurate segmentation of individual pigs. A framework consisting of a neural network for semantic segmentation as well as different network heads and postprocessing methods will be discussed. The method was tested on a data set of 1000 hand-labeled images created specifically for this experiment and achieves detection rates of around 95% (F1 score) despite disturbances such as occlusions and dirty lenses.

Evaluation of Deep Neural Networks for Semantic Segmentation of Prostate in T2W MRI.

In this paper, we present an evaluation of four encoder–decoder CNNs in the segmentation of the prostate gland in T2W magnetic resonance imaging (MRI) image. The four selected CNNs are FCN, SegNet, U-Net, and DeepLabV3+, which was originally proposed for the segmentation of road scene, biomedical, and natural images. Segmentation of prostate in T2W MRI images is an important step in the automatic diagnosis of prostate cancer to enable better lesion detection and staging of prostate cancer. Therefore, many research efforts have been conducted to improve the segmentation of the prostate gland in MRI images. The main challenges of prostate gland segmentation are blurry prostate boundary and variability in prostate anatomical structure. In this work, we investigated the performance of encoder–decoder CNNs for segmentation of prostate gland in T2W MRI. Image pre-processing techniques including image resizing, center-cropping and intensity normalization are applied to address the issues of inter-patient and inter-scanner variability as well as the issue of dominating background pixels over prostate pixels. In addition, to enrich the network with more data, to increase data variation, and to improve its accuracy, patch extraction and data augmentation are applied prior to training the networks. Furthermore, class weight balancing is used to avoid having biased networks since the number of background pixels is much higher than the prostate pixels. The class imbalance problem is solved by utilizing weighted cross-entropy loss function during the training of the CNN model. The performance of the CNNs is evaluated in terms of the Dice similarity coefficient (DSC) and our experimental results show that patch-wise DeepLabV3+ gives the best performance with DSC equal to . This value is the highest DSC score compared to the FCN, SegNet, and U-Net that also competed the recently published state-of-the-art method of prostate segmentation.

Pedestrian Recognition Based On Improved Semantic Segmentation Neural Network

At present, the existing Re-ID methods can’t align the target and the target in the image that should to be detected, which has been affecting the recognition accuracy. Although using semantic segmentation method can solve this problem, it brings new problems. The region which is considered as background by semantic segmentation neural network and discarded is likely to contain feature representations that are helpful for recognition. In this paper, an improved semantic segmentation method is proposed to re-divide the foreground and background regions. In order to avoid the influence of semantic segmentation errors on recognition accuracy, a new method is used to combine background and foreground regions. Experiments show that the accuracy of Re-ID is 84.52%, which is 3.33% higher than before.

The effects of different levels of realism on the training of CNNs with only synthetic images for the semantic segmentation of robotic instruments in a head phantom.

The manual generation of training data for the semantic segmentation of medical images using deep neural networks is a time-consuming and error-prone task. In this paper, we investigate the effect of different levels of realism on the training of deep neural networks for semantic segmentation of robotic instruments. An interactive virtual-reality environment was developed to generate synthetic images for robot-aided endoscopic surgery. In contrast with earlier works, we use physically based rendering for increased realism. Using a virtual reality simulator that replicates our robotic setup, three synthetic image databases with an increasing level of realism were generated: flat, basic, and realistic (using the physically-based rendering). Each of those databases was used to train 20 instances of a UNet-based semantic-segmentation deep-learning model. The networks trained with only synthetic images were evaluated on the segmentation of 160 endoscopic images of a phantom. The networks were compared using the Dwass-Steel-Critchlow-Fligner nonparametric test. Our results show that the levels of realism increased the mean intersection-over-union (mIoU) of the networks on endoscopic images of a phantom ([Formula: see text]). The median mIoU values were 0.235 for the flat dataset, 0.458 for the basic, and 0.729 for the realistic. All the networks trained with synthetic images outperformed naive classifiers. Moreover, in an ablation study, we show that the mIoU of physically based rendering is superior to texture mapping ([Formula: see text]) of the instrument (0.606), the background (0.685), and the background and instruments combined (0.672). Using physical-based rendering to generate synthetic images is an effective approach to improve the training of neural networks for the semantic segmentation of surgical instruments in endoscopic images. Our results show that this strategy can be an essential step in the broad applicability of deep neural networks in semantic segmentation tasks and help bridge the domain gap in machine learning.

Generalized Sparse Convolutional Neural Networks for Semantic Segmentation of Point Clouds Derived from Tri-Stereo Satellite Imagery

We studied the applicability of point clouds derived from tri-stereo satellite imagery for semantic segmentation for generalized sparse convolutional neural networks by the example of an Austrian study area. We examined, in particular, if the distorted geometric information, in addition to color, influences the performance of segmenting clutter, roads, buildings, trees, and vehicles. In this regard, we trained a fully convolutional neural network that uses generalized sparse convolution one time solely on 3D geometric information (i.e., 3D point cloud derived by dense image matching), and twice on 3D geometric as well as color information. In the first experiment, we did not use class weights, whereas in the second we did. We compared the results with a fully convolutional neural network that was trained on a 2D orthophoto, and a decision tree that was once trained on hand-crafted 3D geometric features, and once trained on hand-crafted 3D geometric as well as color features. The decision tree using hand-crafted features has been successfully applied to aerial laser scanning data in the literature. Hence, we compared our main interest of study, a representation learning technique, with another representation learning technique, and a non-representation learning technique. Our study area is located in Waldviertel, a region in Lower Austria. The territory is a hilly region covered mainly by forests, agriculture, and grasslands. Our classes of interest are heavily unbalanced. However, we did not use any data augmentation techniques to counter overfitting. For our study area, we reported that geometric and color information only improves the performance of the Generalized Sparse Convolutional Neural Network (GSCNN) on the dominant class, which leads to a higher overall performance in our case. We also found that training the network with median class weighting partially reverts the effects of adding color. The network also started to learn the classes with lower occurrences. The fully convolutional neural network that was trained on the 2D orthophoto generally outperforms the other two with a kappa score of over 90% and an average per class accuracy of 61%. However, the decision tree trained on colors and hand-crafted geometric features has a 2% higher accuracy for roads.

Boosting Real-Time Driving Scene Parsing With Shared Semantics

Real-time scene parsing is a fundamental feature for autonomous driving vehicles with multiple cameras. In this letter we demonstrate that sharing semantics between cameras with different perspectives and overlapped views can boost the parsing performance when compared with traditional methods, which individually process the frames from each camera. Our framework is based on a deep neural network for semantic segmentation but with two kinds of additional modules for sharing and fusing semantics. On the one hand, a semantics sharing module is designed to establish the pixel-wise mapping between the input images. Features as well as semantics are shared by the map to reduce duplicated workload which leads to more efficient computation. On the other hand, feature fusion modules are designed to combine different modal of semantic features, which leverage the information from both inputs for better accuracy. To evaluate the effectiveness of the proposed framework, we have applied our network to a dual-camera vision system for driving scene parsing. Experimental results show that our network outperforms the baseline method on the parsing accuracy with comparable computations.

An Efficient Solution for Semantic Segmentation of Three Ground‐based Cloud Datasets

AbstractThe machine learning approach has shown its state‐of‐the‐art ability to handle segmentation and detection tasks. It is increasingly employed to extract patterns and spatiotemporal features from the ever‐increasing stream of Earth system data. However, there is still a significant challenge, which is the generalization capability of the model on cloud images in different types and weather conditions. After studying several popular methods, we propose a semantic segmentation neural network for cloud segmentation. It extracts features learned by source and target domains in an end‐to‐end behavior, which can address the problem of significant lack of labels in the observed cloud image data. It is further evaluated on the Singapore Whole Sky Image Segmentation (SWIMSEG) dataset by using Mean Intersection‐over‐Union, recall, F‐score, and accuracy matrices. The scores of these matrices are 86%, 97%, 92%, and 96%, which prove that it has excellent efficiency and robustness. Most importantly, a new benchmark based on the SWIMSEG dataset for the task of cloud segmentation is introduced. The others, BENCHMARK, Cirrus Cumulus Stratus Nimbus are evaluated through the model trained from the SWIMSEG dataset by way of visualization.

A Multi-Scale Water Extraction Convolutional Neural Network (MWEN) Method for GaoFen-1 Remote Sensing Images

Automatic water body extraction method is important for monitoring floods, droughts, and water resources. In this study, a new semantic segmentation convolutional neural network named the multi-scale water extraction convolutional neural network (MWEN) is proposed to automatically extract water bodies from GaoFen-1 (GF-1) remote sensing images. Three convolutional neural networks for semantic segmentation (fully convolutional network (FCN), Unet, and Deeplab V3+) are employed to compare with the water bodies extraction performance of MWEN. Visual comparison and five evaluation metrics are used to evaluate the performance of these convolutional neural networks (CNNs). The results show the following. (1) The results of water body extraction in multiple scenes using the MWEN are better than those of the other comparison methods based on the indicators. (2) The MWEN method has the capability to accurately extract various types of water bodies, such as urban water bodies, open ponds, and plateau lakes. (3) By fusing features extracted at different scales, the MWEN has the capability to extract water bodies with different sizes and suppress noise, such as building shadows and highways. Therefore, MWEN is a robust water extraction algorithm for GaoFen-1 satellite images and has the potential to conduct water body mapping with multisource high-resolution satellite remote sensing data.

An End-to-End and Localized Post-Processing Method for Correcting High-Resolution Remote Sensing Classification Result Images

Since the result images obtained by deep semantic segmentation neural networks are usually not perfect, especially at object borders, the conditional random field (CRF) method is frequently utilized in the result post-processing stage to obtain the corrected classification result image. The CRF method has achieved many successes in the field of computer vision, but when it is applied to remote sensing images, overcorrection phenomena may occur. This paper proposes an end-to-end and localized post-processing method (ELP) to correct the result images of high-resolution remote sensing image classification methods. ELP has two advantages. (1) End-to-end evaluation: ELP can identify which locations of the result image are highly suspected of having errors without requiring samples. This characteristic allows ELP to be adapted to an end-to-end classification process. (2) Localization: Based on the suspect areas, ELP limits the CRF analysis and update area to a small range and controls the iteration termination condition. This characteristic avoids the overcorrections caused by the global processing of the CRF. In the experiments, ELP is used to correct the classification results obtained by various deep semantic segmentation neural networks. Compared with traditional methods, the proposed method more effectively corrects the classification result and improves classification accuracy.

Diabetic Retinopathy Detection Using Prognosis of Microaneurysm and Early Diagnosis System for Non-Proliferative Diabetic Retinopathy Based on Deep Learning Algorithms

Predicting the presence of Microaneurysms in the fundus images and the identification of diabetic retinopathy in early-stage has always been a major challenge for decades. Diabetic Retinopathy (DR) is affected by prolonged high blood glucose level which leads to microvascular complications and irreversible vision loss. Microaneurysms formation and macular edema in the retinal is the initial sign of DR and diagnosis at the right time can reduce the risk of non proliferated diabetic retinopathy. The rapid improvement of deep learning makes it gradually become an efficient technique to provide an interesting solution for medical image analysis problems. The proposed system analysis the presence of microaneurysm in fundus image using convolutional neural network algorithms that embeds deep learning as a core component accelerated with GPU(Graphics Processing Unit) which will perform medical image detection and segmentation with high-performance and low-latency inference. The semantic segmentation algorithm is utilized to classify the fundus picture as normal or infected. Semantic segmentation divides the image pixels based on their common semantic to identify the feature of microaneurysm. This provides an automated system that will assist ophthalmologists to grade the fundus images as early NPDR, moderate NPDR, and severe NPDR. The Prognosis of Microaneurysm and early diagnosis system for non - proliferative diabetic retinopathy system has been proposed that is capable to train effectively a deep convolution neural network for semantic segmentation of fundus images which can increase the efficiency and accuracy of NPDR (non proliferated diabetic retinopathy) prediction.

Mask-R-FCN: A Deep Fusion Network for Semantic Segmentation

Remote sensing image classification plays a significant role in urban applications, precision agriculture, water resource management. The task of classification in the field of remote sensing is to map raw images to semantic maps. Typically, fully convolutional network (FCN) is one of the most effective deep neural networks for semantic segmentation. However, small objects in remote sensing images can be easily overlooked and misclassified as the majority label, which is often the background of the image. Although many works have attempted to deal with this problem, making a trade-off between background semantics and edge details is still a problem. This is mainly because they are based on a single neural network model. To deal with this problem, a convolutional deep network with regions (R-CNN), which is highly effective for object detection is leveraged as a complementary component in our work. A learning-based and decision-level strategy is applied to fuse both semantic maps from a semantic model and an object detection model. The proposed network is referred to as Mask-R-FCN. Experimental results on real remote sensing images from the Zurich dataset, Gaofen Image Dataset (GID), and DataFountain2017 show that the proposed network can obtain higher accuracy than single deep neural networks and other machine learning algorithms. The proposed network achieved better average accuracies, which are approximately 2% higher than those of any other single deep neural networks on the Zurich, GID, and DataFoundation2017 datasets.

A New Multi-Channel Deep Convolutional Neural Network for Semantic Segmentation of Remote Sensing Image

The semantic segmentation of remote sensing (RS) image is a hot research field. With the development of deep learning, the semantic segmentation based on a full convolution neural network greatly improves the segmentation accuracy. The amount of information on the RS image is very large, but the sample size is extremely uneven. Therefore, even the common network can segment RS images to a certain extent, but the segmentation accuracy can still be greatly improved. The common neural network deepens the network to improve the classification accuracy, but it has a lot of loss to the target spatial features and scale features, and the existing common feature fusion methods can only solve some problems. A segmentation network is built to solve the above problems very well. The network employs the InceptionV-4 network as the backbone and improves it. We modify the network structure and introduce the changed Atrous Spatial Pyramid Pooling module to extract the multi-scale features of the target from different training stages. Without losing the depth of the network, using Inception blocks to strengthen the width of the network can obtain more abstract features. At the same time, the backbone network is used for semantic fusion of the context, it can retain more spatial features, then an effective decoder network is designed. Finally, evaluate our model on the ISPRS 2D Semantic Labeling Contest Potsdam and Inria Aerial Image Labeling Dataset. The results show that the network has very superior performance, reaching 89.62% IOU score and 94.49% F1 score on the Potsdam dataset, and the IOU score on the Inria dataset has been greatly improved.

Learning Unbiased Zero-Shot Semantic Segmentation Networks Via Transductive Transfer

Semantic segmentation aims to obtain a detailed understanding of images. Deep learning has achieved great advances in semantic segmentation over the past years. In practice, however, the classes do not always correspond to the ones in the training stage. Since it is impractical to collect sufficient labeled data for all classes, zero-shot semantic segmentation has received increasing attentions recently. Although semantic segmentation neural networks can transfer knowledge from seen classes to unseen classes by incorporating the class-level semantic information, it shows a strong bias towards seen classes. In this letter, we propose an easy-to-implement transductive approach to alleviate the prediction bias in zero-shot semantic segmentation. We assume that both source images with full pixel-level labels and unlabeled target images are available for training. The source images are used to build the relationship between visual images and class-level semantic embeddings. On the other hand, the target images are used to alleviate the bias towards seen classes. Comprehensive experiments over the PASCAL dataset clearly demonstrate the effectiveness of our approach.

A Visual-Inertial Localization Method for Unmanned Aerial Vehicle in Underground Tunnel Dynamic Environments

Unmanned Aerial Vehicles (UAVs) can significantly improve the autonomy of the mining industry, and self-localization is the key to autonomous flights of underground UAVs. A localization method of visual-inertial sensor data fusion is proposed in this paper. The method aims to improve the localization accuracy and robustness of underground UAVs in dynamic environments. First, an algorithm for dynamic point detection and rejection is presented, which combines a semantic segmentation neural network, an optical flow method, and an epipolar constraint method. Second, a visual-inertial sensor fusion algorithm is used to enhance performance in areas lacking static visual features. It can also provide absolute scales to the localization results, as opposed to monocular systems. Finally, a hand-held multi-sensor data collection system is developed with accurate calibration, for imitating flights of underground UAVs and easing data collection in real underground tunnels. We evaluate our proposed localization method and compare it with state-of-art method VINS-Mono on both the public EuRoC dataset and our own collected data in underground tunnels. Experimental results show that the proposed visual-inertial localization method can improve the accuracy by more than 67% over VINS-Mono in high dynamic environments, and it can be applied to underground dynamic scenes with high robustness and accuracy.

Uncertainty and interpretability in convolutional neural networks for semantic segmentation of colorectal polyps.

Colorectal polyps are known to be potential precursors to colorectal cancer, which is one of the leading causes of cancer-related deaths on a global scale. Early detection and prevention of colorectal cancer is primarily enabled through manual screenings, where the intestines of a patient is visually examined. Such a procedure can be challenging and exhausting for the person performing the screening. This has resulted in numerous studies on designing automatic systems aimed at supporting physicians during the examination. Recently, such automatic systems have seen a significant improvement as a result of an increasing amount of publicly available colorectal imagery and advances in deep learning research for object image recognition. Specifically, decision support systems based on Convolutional Neural Networks (CNNs) have demonstrated state-of-the-art performance on both detection and segmentation of colorectal polyps. However, CNN-based models need to not only be precise in order to be helpful in a medical context. In addition, interpretability and uncertainty in predictions must be well understood. In this paper, we develop and evaluate recent advances in uncertainty estimation and model interpretability in the context of semantic segmentation of polyps from colonoscopy images. Furthermore, we propose a novel method for estimating the uncertainty associated with important features in the input and demonstrate how interpretability and uncertainty can be modeled in DSSs for semantic segmentation of colorectal polyps. Results indicate that deep models are utilizing the shape and edge information of polyps to make their prediction. Moreover, inaccurate predictions show a higher degree of uncertainty compared to precise predictions.

FABnet: feature attention-based network for simultaneous segmentation of microvessels and nerves in routine histology images of oral cancer

Perineural invasion (PNI), lymphovascular invasion (LVI) and tumor angiogenesis have strong correlation with cancer recurrence, metastasis and poor patient survival. The accurate segmentations of nerves and microvessels can be considered as the preliminary step in objective identification of PNI, LVI and tumor angiogenic analysis in histology images. We proposed a deep network for simultaneous segmentation of microvessel and nerves in routinely used H&E-stained histology images. The network is designed as an encoder–decoder architecture with embedded feature attention blocks and an uncertainty prediction. The proposed network uses Xception residual blocks, followed by atrous spatial pyramid pooling for feature extraction at multiple scales. Feature attention blocks are used in the skip connections from encoder to decoder as well as in the decoder up-sampling, which enables the network in focusing on more salient features while making prediction for segmentation. The method is evaluated using 7780 images of size 512 × 512 pixels, extracted from 20 WSIs of oral squamous cell carcinoma tissue at 20× magnification. The ensemble of network outputs at test time is used to obtain a better segmentation result and simultaneous generation of network prediction uncertainty map. The proposed network achieves state-of-the-art results compared to currently used deep neural networks for semantic segmentation (FCN-8, U-Net, Segnet and DeepLabV3+). The proposed network also gives robust segmentation performance when applied to the full digital whole slide image.