Neural Network For Semantic Segmentation Research Articles

RSI-Net: Two-Stream Deep Neural Network for Remote Sensing Images-Based Semantic Segmentation

For semantic segmentation of remote sensing images (RSI), trade-off between representation power and location accuracy is quite important. How to get the trade-off effectively is an open question, where current approaches of utilizing very deep models result in complex models with large memory consumption. In contrast to previous work that utilizes dilated convolutions or deep models, we propose a novel two-stream deep neural network for semantic segmentation of RSI (RSI-Net) to obtain improved performance through modeling and propagating spatial contextual structure effectively and a decoding scheme with image-level and graph-level combination. The first component explicitly models correlations between adjacent land covers and conduct flexible convolution on arbitrarily irregular image regions by using graph convolutional network, while densely connected atrous convolution network (DenseAtrousCNet) with multi-scale atrous convolution can expand the receptive fields and obtain image global information. Extensive experiments are implemented on the Vaihingen, Potsdam and Gaofen RSI datasets, where the comparison results demonstrate the superior performance of RSI-Net in terms of overall accuracy (91.83%, 93.31% and 93.67% on three datasets, respectively), F1 score (90.3%, 91.49% and 89.35% on three datasets, respectively) and kappa coefficient (89.46%, 90.46% and 90.37% on three datasets, respectively) when compared with six state-of-the-art RSI semantic segmentation methods.

Transformer and CNN Hybrid Deep Neural Network for Semantic Segmentation of Very-High-Resolution Remote Sensing Imagery

This article presents a transformer and convolutional neural network (CNN) hybrid deep neural network for semantic segmentation of very high resolution (VHR) remote sensing imagery. The model follows an encoder–decoder structure. The encoder module uses a new universal backbone Swin transformer to extract features to achieve better long-range spatial dependencies modeling. The decoder module draws on some effective blocks and successful strategies of CNN-based models in remote sensing image segmentation. In the middle of the framework, an atrous spatial pyramid pooling block based on depthwise separable convolution (SASPP) is applied to obtain a multiscale context. A U-shaped decoder is used to gradually restore the size of the feature maps. Three skip connections are built between the encoder and decoder feature maps of the same size to maintain the transmission of local details and enhance the communication of multiscale features. A squeeze-and-excitation (SE) channel attention block is added before segmentation for feature augmentation. An auxiliary boundary detection branch is combined to provide edge constraints for semantic segmentation. Extensive ablation experiments were conducted on the International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen and Potsdam benchmarks to test the effectiveness of multiple components of the network. At the same time, the proposed method is compared with the current state-of-the-art methods on the two benchmarks. The proposed hybrid network achieved the second highest overall accuracy (OA) on both the Potsdam and Vaihingen benchmarks (code and models are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/zq7734509/mmsegmentation-</uri> multilayer).

Robust Segmentation Models Using an Uncertainty Slice Sampling-Based Annotation Workflow

Semantic segmentation neural networks require pixel-level annotations in large quantities to achieve a good performance. In the medical domain, such annotations are expensive because they are time-consuming and require expert knowledge. Active learning optimizes the annotation effort by devising strategies to select cases for labeling that are the most informative to the model. In this work, we propose an uncertainty slice sampling (USS) strategy for the semantic segmentation of 3D medical volumes that selects 2D image slices for annotation and we compare it with various other strategies. We demonstrate the efficiency of USS on a CT liver segmentation task using multisite data. After five iterations, the training data resulting from USS consisted of 2410 slices (4&#x0025; of all slices in the data pool) compared to 8121(13&#x0025;), 8641(14&#x0025;), and 3730(6&#x0025;) slices for uncertainty volume (UVS), random volume (RVS), and random slice (RSS) sampling, respectively. Despite being trained on the smallest amount of data, the model based on the USS strategy evaluated on 234 test volumes significantly outperformed models trained according to the UVS, RVS, and RSS strategies and achieved a mean Dice index of 0.964, a relative volume error of 4.2&#x0025;, a mean surface distance of 1.35mm, and a Hausdorff distance of 23.4mm. This was only slightly inferior to 0.967, 3.8&#x0025;, 1.18mm, and 22.9mm achieved by a model trained on all available data. Our robustness analysis using the 5^th percentile of Dice and the 95^th percentile of the remaining metrics demonstrated that USS not only resulted in the most robust model compared to other strategies, but also outperformed the model trained on all data according to the 5^th percentile of Dice (0.946 vs. 0.945) and the 95^th percentile of mean surface distance (1.92mm vs. 2.03mm).

Interweave features of Deep Convolutional Neural Networks for semantic segmentation

Although semantic segmentation based on Deep Convolutional Neural Networks (DCNN) has made great signs of progress, the issue that features generated by deep models are of low-resolution and will negatively affect the final semantic segmentation performances is not fully addressed yet. In this paper, we propose to adaptively combine high-level and low-level features of DCNN to improve the quality of the features used for semantic segmentation. To this end, we design a feature interweaving neural network module to fuse features from different layers of DCNN to effectively take advantage of their complementary properties. And, in order to enhance complementarity and diminish contradiction of the features for better feature fusion, we propose a feature modulation neural network module to modulate the features before interweaving. Furthermore, global information of images is summarized and used to augment the features for providing guidance for feature interweaving. The proposed method is extensively evaluated and compared to state-of-the-art methods based on two benchmark semantic segmentation datasets Cityscapes and PASCAL VOC 2012 in the experiments. Obtained results demonstrate the effectiveness of the proposed method.

SGA-Net: Self-Constructing Graph Attention Neural Network for Semantic Segmentation of Remote Sensing Images

Semantic segmentation of remote sensing images is always a critical and challenging task. Graph neural networks, which can capture global contextual representations, can exploit long-range pixel dependency, thereby improving semantic segmentation performance. In this paper, a novel self-constructing graph attention neural network is proposed for such a purpose. Firstly, ResNet50 was employed as backbone of a feature extraction network to acquire feature maps of remote sensing images. Secondly, pixel-wise dependency graphs were constructed from the feature maps of images, and a graph attention network is designed to extract the correlations of pixels of the remote sensing images. Thirdly, the channel linear attention mechanism obtained the channel dependency of images, further improving the prediction of semantic segmentation. Lastly, we conducted comprehensive experiments and found that the proposed model consistently outperformed state-of-the-art methods on two widely used remote sensing image datasets.

Deep convolutional neural networks for semantic segmentation of cracks

Deep convolutional neural networks for semantic segmentation of cracks

A data-driven semantic segmentation model for direct cardiac functional analysis based on undersampled radial MR cine series.

Imageacquisition and subsequent manual analysis of cardiac cine MRIis time-consuming. The purpose of this study was to train and evaluate a 3D artificial neural network for semanticsegmentation ofradially undersampledcardiacMRI to accelerate both scan time and postprocessing. A database ofCartesianshort-axisMR images of the heart(148,500 images, 484 examinations)was assembled fromanopenly accessibledatabaseandradial undersampling wassimulated.A3D U-Net architecture waspretrainedfor segmentation of undersampled spatiotemporal cineMRI.Transfer learning wasthen performed using samples from asecond database, comprising108non-Cartesian radial cine series of the midventricularmyocardium to optimize the performancefor authentic data.The performance was evaluated for different levelsof undersampling by theDice similarity coefficient(DSC)with respecttoreferencelabels, as well as by deriving ventricular volumes and myocardial masses. Without transfer learning,thepretrained modelperformedmoderately ontrueradial data [maximum number of projections tested, P=196;DSC =0.87(left ventricle),DSC=0.76(myocardium),and DSC =0.64(right ventricle)]. After transfer learning with authentic data, the predictions achieved human levelevenforhighundersamplingrates (P=33, DSC=0.95, 0.87, and 0.93)withoutsignificant difference compared withsegmentations derived fromfully sampled data. A 3D U-Net architecture can be used for semantic segmentation of radially undersampled cine acquisitions, achieving a performance comparable with human experts in fully sampled data. This approach can jointly acceleratetime-consumingcineimage acquisition andcumbersome manual image analysis.

Comparison of Semantic Convolution Neural Networks on the Example of Crack Segmentation in Asphalt Images

The article is devoted to a comparative analysis of the effectiveness of convolutional neural networks for semantic segmentation of road surface damage marking. Currently, photo and video surveillance methods are used to control the condition of the road surface. Assessing and analyzing new manual data can take too long. Thus, a completely different procedure is required to inspect and assess the state of controlled objects using technical vision. The authors compared 3 neural networks (Unet, Linknet, PSPNet) used in semantic segmentation using the example of the Crack500 dataset. The proposed architectures have been implemented in the Keras and TensorFlow frameworks. The compared models of neural convolutional networks effectively solve the assigned tasks even with a limited amount of training data. High accuracy is observed. The considered models can be used in various segmentation tasks. The results obtained can be used in the process of modeling, monitoring, and predicting the wear of the road surface.

Simplified object-based deep neural network for very high resolution remote sensing image classification

For the object-based classification of high resolution remote sensing images, many people expect that introducing deep learning methods can improve then classification accuracy. Unfortunately, the input shape for deep neural networks (DNNs) is usually rectangular, whereas the shapes of the segments output by segmentation methods are usually according to the corresponding ground objects; this inconsistency can lead to confusion among different types of heterogeneous content when a DNN processes a segment. Currently, most object-based methods utilizing convolutional neural networks (CNNs) adopt additional models to overcome the detrimental influence of such heterogeneous content; however, these heterogeneity suppression mechanisms introduce additional complexity into the whole classification process, and these methods are usually unstable and difficult to use in real applications. To address the above problems, this paper proposes a simplified object-based deep neural network (SO-DNN) for very high resolution remote sensing image classification. In SO-DNN, a new segment category label inference method is introduced, in which a deep semantic segmentation neural network (DSSNN) is used as the classification model instead of a traditional CNN. Since the DSSNN can obtain a category label for each pixel in the input image patch, different types of content are not mixed together; therefore, SO-DNN does not require an additional heterogeneity suppression mechanism. Moreover, SO-DNN includes a sample information optimization method that allows the DSSNN model to be trained using only pixel-based training samples. Because only a single model is used and only a pixel-based training set is needed, the whole classification process of SO-DNN is relatively simple and direct. In experiments, we use very high-resolution aerial images from Vaihingen and Potsdam from the ISPRS WG II/4 dataset as test data and compare SO-DNN with 6 traditional methods: O-MLP, O+CNN, OHSF-CNN, 2-CNN, JDL and U-Net. Compared with the best-performing method among these traditional methods, the classification accuracy of SO-DNN is improved by up to 7.71% and 10.78% for single images from Vaihingen and Potsdam, respectively, and the average classification accuracy is improved by 2.46% and 2.91% for the Vaihingen and Potsdam images, respectively. SO-DNN relies on fewer models and easier-to-obtain samples than traditional methods, and its stable performance makes SO-DNN more valuable for practical applications.

3D point cloud semantic segmentation toward large-scale unstructured agricultural scene classification

In recent years, with the development of computer vision, deep learning, and artificial intelligence technologies, the popularity of depth sensors and lidar has promoted the rapid development of three-dimensional (3D) point cloud semantic segmentation. The semantic segmentation of 3D point clouds for large-scale unstructured agricultural scenes is important for agricultural robots to perceive their surrounding environment, and for autonomous navigation and positioning and autonomous scene understanding. In this study, the problem of 3D point cloud semantic segmentation for large-scale unstructured agricultural scenes was studied. By improving the neural network structure of RandLA-Net, a deeper 3D point cloud semantic segmentation neural network model for large-scale unstructured agricultural scenes was built, and good experimental results were obtained. The local feature aggregation module in RandLA-Net was integrated and improved to achieve 3D point cloud semantic segmentation for large-scale unstructured agricultural scenes. To test the influence of the 3D point cloud sampling algorithm on the overall accuracy (OA) and mean intersection-over-union (mIoU) of semantic segmentation, the random sampling algorithm and farthest point sampling algorithm were used to build two models with the same neural network structure. The test results show that the sampling algorithm has little effect on the OA and mIoU of 3D point cloud semantic segmentation, and the final result depends mainly on the extraction of 3D point cloud features. In addition, two different Semantic3D datasets were used to test the effect of the datasets on the generalization ability of the model, and the results showed that the datasets had an important effect on the neural network model.

Convolutional neural network for semantic segmentation of fetal echocardiography based on four-chamber view

The acute shortage of trained and experienced sonographers causes the detection of congenital heart defects (CHDs) extremely difficult. In order to minimize this difficulty, an accurate fetal heart segmentation to the early location of such structural heart abnormalities prior to delivery is essential. However, the segmentation process is not an easy task due to the small size of the fetal heart structure. Moreover, the manual task for identifying the standard cardiac planes, primarily based on a four-chamber view, requires a well-trained clinician and experience. In this paper, a CNN method using U-Net architecture was proposed to automate fetal cardiac standard planes segmentation from ultrasound images. A total of 519 fetal cardiac images was obtained from three videos. All data is divided into training and testing data. The testing data consist of 106 slices of the four-chamber segmentation tasks, i.e. atrial septal defect (ASD), ventricular septal defect (VSD), and normal. The segmentation of the post-processing method is needed to enhanced the segmentation result. In this paper, a combination technique with U-Net and Otsu thresholding gives the best performances with 99.48%-pixel accuracy, 96.73% mean accuracy, 94.92% mean intersection over union, and 0.21% segmentation error. In the future, the implementation of Deep Learning in the study of CHDs holds significant potential for identifying novel CHDs in heterogeneous fetal hearts.

Hybridizing Cross-Level Contextual and Attentive Representations for Remote Sensing Imagery Semantic Segmentation

Semantic segmentation of remote sensing imagery is a fundamental task in intelligent interpretation. Since deep convolutional neural networks (DCNNs) performed considerable insight in learning implicit representations from data, numerous works in recent years have transferred the DCNN-based model to remote sensing data analysis. However, the wide-range observation areas, complex and diverse objects and illumination and imaging angle influence the pixels easily confused, leading to undesirable results. Therefore, a remote sensing imagery semantic segmentation neural network, named HCANet, is proposed to generate representative and discriminative representations for dense predictions. HCANet hybridizes cross-level contextual and attentive representations to emphasize the distinguishability of learned features. First of all, a cross-level contextual representation module (CCRM) is devised to exploit and harness the superpixel contextual information. Moreover, a hybrid representation enhancement module (HREM) is designed to fuse cross-level contextual and self-attentive representations flexibly. Furthermore, the decoder incorporates DUpsampling operation to boost the efficiency losslessly. The extensive experiments are implemented on the Vaihingen and Potsdam benchmarks. In addition, the results indicate that HCANet achieves excellent performance on overall accuracy and mean intersection over union. In addition, the ablation study further verifies the superiority of CCRM.

SketchGNN: Semantic Sketch Segmentation with Graph Neural Networks

We introduce SketchGNN , a convolutional graph neural network for semantic segmentation and labeling of freehand vector sketches. We treat an input stroke-based sketch as a graph with nodes representing the sampled points along input strokes and edges encoding the stroke structure information. To predict the per-node labels, our SketchGNN uses graph convolution and a static-dynamic branching network architecture to extract the features at three levels, i.e., point-level, stroke-level, and sketch-level. SketchGNN significantly improves the accuracy of the state-of-the-art methods for semantic sketch segmentation (by 11.2% in the pixel-based metric and 18.2% in the component-based metric over a large-scale challenging SPG dataset) and has magnitudes fewer parameters than both image-based and sequence-based methods.

A COMPARATIVE STUDY OF POINT CLOUDS SEMANTIC SEGMENTATION USING THREE DIFFERENT NEURAL NETWORKS ON THE RAILWAY STATION DATASET

Abstract. Point cloud data have rich semantic representations and can benefit various applications towards a digital twin. However, they are unordered and anisotropically distributed, thus being unsuitable for a typical Convolutional Neural Networks (CNN) to handle. With the advance of deep learning, several neural networks claim to have solved the point cloud semantic segmentation problem. This paper evaluates three different neural networks for semantic segmentation of point clouds, namely PointNet++, PointCNN and DGCNN. A public indoor scene of the Amersfoort railway station is used as the study area. Unlike the typical indoor scenes and even more from the ubiquitous outdoor ones in currently available datasets, the station consists of objects such as the entrance gates, ticket machines, couches, and garbage cans. For the experiment, we use subsets from the data, remove the noise, evaluate the performance of the selected neural networks. The results indicate an overall accuracy of more than 90% for all the networks but vary in terms of mean class accuracy and mean Intersection over Union (IoU). The misclassification mainly occurs in the classes of couch and garbage can. Several factors that may contribute to the errors are analyzed, such as the quality of the data and the proportion of the number of points per class. The adaptability of the networks is also heavily dependent on the training location: the overall characteristics of the train station make a trained network for one location less suitable for another.

Convolutional neural network for semantic segmentation of fetal echocardiography based on four-chamber view

The acute shortage of trained and experienced sonographers causes the detection of congenital heart defects (CHDs) extremely difficult. In order to minimize this difficulty, an accurate fetal heart segmentation to the early location of such structural heart abnormalities prior to delivery is essential. However, the segmentation process is not an easy task due to the small size of the fetal heart structure. Moreover, the manual task for identifying the standard cardiac planes, primarily based on a four-chamber view, requires a well-trained clinician and experience. In this paper, a CNN method using U-Net architecture was proposed to automate fetal cardiac standard planes segmentation from ultrasound images. A total of 519 fetal cardiac images was obtained from three videos. All data is divided into training and testing data. The testing data consist of 106 slices of the four-chamber segmentation tasks, i.e. atrial septal defect (ASD), ventricular septal defect (VSD), and Normal. The segmentation of the post-processing method is needed to enhanced the segmentation result. In this paper, a combination technique with U-Net and Otsu thresholding gives the best performances with 99.48%-pixel accuracy, 96.73% mean accuracy, 94.92% mean intersection over union, and 0.21% segmentation error. In the future, the implementation of Deep Learning in the study of CHDs holds significant potential for identifying novel CHDs in heterogeneous fetal hearts.

A new deep distortion convolutional neural network for semantic segmentation of panoramic images

Semantic segmentation of panoramic images plays a crucial role in many applications, such as scene understanding, autonomous navigation, and community security. However, the traditional deep learning algorithms achieve lower accuracy for panoramic images due to the serious image distortion. This paper proposes a new deep convolutional neural network to segment panoramic images of outdoor scenes semantically. The proposed network includes two branches: the semantic segmentation sub-branch (SS-branch) and the feature enhancement sub-branch (FE-branch). The SS-branch contains two parts: Encoder and decoder. In the encoder, a new distortion convolution layer (DCL) is defined to reduce the distortion of panoramic images. In the FE-branch, a canny is used to detect the edge and a lightweight network is designed to enhance the features of image boundaries. Finally, a weighted loss function is used. The experimental results show that the proposed semantic segmentation method has better performance for different outdoor scenes.

NEURAL NETWORK METHODS FOR PLANAR IMAGE ANALYSIS IN AUTOMATED SCREENING SYSTEMS

Nowadays, means of preventive management in various spheres of human life are actively developing. The task of automated screening is to detect hidden problems at an early stage without human intervention, while the cost of responding to them is low. Visual inspection is often used to perform a screening task. Deep artificial neural networks are especially popular in image processing. One of the main problems when working with them is the need for a large amount of well-labeled data for training. In automated screening systems, available neural network approaches have limitations on the reliability of predictions due to the lack of accurately marked training data, as obtaining quality markup from professionals is very expensive, and sometimes not possibleinprinciple. Therefore, there is a contradiction between increasing the requirements for the precision of predictions of neuralnetwork models without increasing the time spent on the one hand, and the need to reduce the cost of obtaining the markup of educational data. In this paper, we propose the parametric model of the segmentation dataset, which can be used to generate training datafor model selection and benchmarking; and the multi-task learning method for training and inference of deep neural networks for semantic segmentation. Based on the proposed method, we develop a semi-supervised approach for segmentation of salient regions for classification task. The main advantage of the proposed method is that it uses semantically-similar general tasks, thathave better labeling than original one, what allows users to reduce the cost of the labeling process. We propose to use classification task as a more general to the problem of semantic segmentation. As semantic segmentation aims to classify each pixel in the input image, classification aims to assign a class to all of the pixels in the input image. We evaluate our methods using the proposed dataset model, observing the Dice score improvement by seventeen percent. Additionally, we evaluate the robustness ofthe proposed method to different amount of the noise in labels and observe consistent improvement over baseline version.

Manipulation Planning for Object Re-Orientation Based on Semantic Segmentation Keypoint Detection.

In this paper, a manipulation planning method for object re-orientation based on semantic segmentation keypoint detection is proposed for robot manipulator which is able to detect and re-orientate the randomly placed objects to a specified position and pose. There are two main parts: (1) 3D keypoint detection system; and (2) manipulation planning system for object re-orientation. In the 3D keypoint detection system, an RGB-D camera is used to obtain the information of the environment and can generate 3D keypoints of the target object as inputs to represent its corresponding position and pose. This process simplifies the 3D model representation so that the manipulation planning for object re-orientation can be executed in a category-level manner by adding various training data of the object in the training phase. In addition, 3D suction points in both the object’s current and expected poses are also generated as the inputs of the next operation stage. During the next stage, Mask Region-Convolutional Neural Network (Mask R-CNN) algorithm is used for preliminary object detection and object image. The highest confidence index image is selected as the input of the semantic segmentation system in order to classify each pixel in the picture for the corresponding pack unit of the object. In addition, after using a convolutional neural network for semantic segmentation, the Conditional Random Fields (CRFs) method is used to perform several iterations to obtain a more accurate result of object recognition. When the target object is segmented into the pack units of image process, the center position of each pack unit can be obtained. Then, a normal vector of each pack unit’s center points is generated by the depth image information and pose of the object, which can be obtained by connecting the center points of each pack unit. In the manipulation planning system for object re-orientation, the pose of the object and the normal vector of each pack unit are first converted into the working coordinate system of the robot manipulator. Then, according to the current and expected pose of the object, the spherical linear interpolation (Slerp) algorithm is used to generate a series of movements in the workspace for object re-orientation on the robot manipulator. In addition, the pose of the object is adjusted on the z-axis of the object’s geodetic coordinate system based on the image features on the surface of the object, so that the pose of the placed object can approach the desired pose. Finally, a robot manipulator and a vacuum suction cup made by the laboratory are used to verify that the proposed system can indeed complete the planned task of object re-orientation.

Non-destructive internal disorder detection of Conference pears by semantic segmentation of X-ray CT scans using deep learning

Long term storage is required to deliver high quality pear fruit year-round. Under suboptimal storage conditions, internal disorders, such as internal browning and cavity formation, can develop and are often invisible from the outside. We present a non-destructive inspection method to quantify internal disorders in X-ray CT scans of pear fruit using a deep neural network for semantic segmentation. Herein, a U-net based model was trained to automatically indicate healthy tissue, core and regions affected by internal disorders, i.e., cavity formation and internal browning. The quantitative data resulting from the segmentations was used to measure the severity of internal disorders. Excellent classification accuracies of 99.4 and 92.2% were obtained for the classification of “consumable” vs “non-consumable” fruit on the one hand and “healthy” vs “defect but consumable” vs “non-consumable” fruit on the other hand. The identification of “defect but consumable” fruit showed to be the most difficult.

Development of Coral Investigation System Based on Semantic Segmentation of Single-Channel Images.

Among aquatic biota, corals provide shelter with sufficient nutrition to a wide variety of underwater life. However, a severe decline in the coral resources can be noted in the last decades due to global environmental changes causing marine pollution. Hence, it is of paramount importance to develop and deploy swift coral monitoring system to alleviate the destruction of corals. Performing semantic segmentation on underwater images is one of the most efficient methods for automatic investigation of corals. Firstly, to design a coral investigation system, RGB and spectral images of various types of corals in natural and artificial aquatic sites are collected. Based on single-channel images, a convolutional neural network (CNN) model, named DeeperLabC, is employed for the semantic segmentation of corals, which is a concise and modified deeperlab model with encoder-decoder architecture. Using ResNet34 as a skeleton network, the proposed model extracts coral features in the images and performs semantic segmentation. DeeperLabC achieved state-of-the-art coral segmentation with an overall mean intersection over union (IoU) value of 93.90%, and maximum F1-score of 97.10% which surpassed other existing benchmark neural networks for semantic segmentation. The class activation map (CAM) module also proved the excellent performance of the DeeperLabC model in binary classification among coral and non-coral bodies.