Problem Of Semantic Segmentation Research Articles

Large-scale rice mapping under different years based on time-series Sentinel-1 images using deep semantic segmentation model

Identifying spatial distribution of crop planting in large-scale is one of the most significant applications of remote sensing imagery. As an active remote sensing system, synthetic aperture radar (SAR) provides high-resolution polarimetric information of land covers. Nowadays, it is possible to carry out continuous multi-temporal analysis of crops in large-scales since an increased number of spaceborne SAR systems has been launched. This paper formulates rice mapping as a semantic segmentation problem and proposes to use deep learning techniques to exploit the phenological similarity of rice production to identify the rice distribution in large-scales. The study area (i.e., about 58504 km2) located in Arkansas River Basin is selected to develop an adapted U-Net for large-scale rice mapping. The Sentinel-1 data in previous years (i.e., data collected in 2017 and 2018) are used to train and fine-tune the network, and current season data (i.e., data collected in 2019) is selected to test the robustness of the network. Experimental results show that the proposed method achieves the state-of-the-art performance as it benefits from the spatial characteristics and phenological similarity of rice. The experiments of rice extraction in different planting pattern regions and extracted features visual projection are conducted to explain the features mined by the adapted U-Net. Furthermore, the advantages of temporal generalization in large-scale are validated by the comparison between space migration and time migration, which indicates that the difference of rice in different years is smaller than that of rice in different spaces. Finally, the issues for operational implementation are discussed.

EDC-Net: Edge Detection Capsule Network for 3D Point Clouds

Edge features in point clouds are prominent due to the capability of describing an abstract shape of a set of points. Point clouds obtained by 3D scanner devices are often immense in terms of size. Edges are essential features in large scale point clouds since they are capable of describing the shapes in down-sampled point clouds while maintaining the principal information. In this paper, we tackle challenges of edge detection tasks in 3D point clouds. To this end, we propose a novel technique to detect edges of point clouds based on a capsule network architecture. In this approach, we define the edge detection task of point clouds as a semantic segmentation problem. We built a classifier through the capsules to predict edge and non-edge points in 3D point clouds. We applied a weakly-supervised learning approach in order to improve the performance of our proposed method and built in the capability of testing the technique in wider range of shapes. We provide several quantitative and qualitative experimental results to demonstrate the robustness of our proposed EDC-Net for edge detection in 3D point clouds. We performed a statistical analysis over the ABC and ShapeNet datasets. Our numerical results demonstrate the robust and efficient performance of EDC-Net.

Seismic Stratum Segmentation Using an Encoder–Decoder Convolutional Neural Network

As an essential step in reservoir characterization, seismic stratigraphic interpretation is often dependent on the development of powerful computer-based interpretation tools that can simulate the intelligence of experienced interpreters. With the success of machine/deep learning applications in many aspects of geoscience in recent decades, geophysicists have become more dedicated to exploring seismic big data in a smarter and more sophisticated way to better image subsurface reservoirs/structures. In this paper, a specific U-shaped fully convolutional network (U-Net) is established for automatic seismic stratigraphic interpretation. Specifically, this task is formulated as a semantic segmentation problem by identifying strata at the pixel level and classifying each pixel in the image into a specific stratum category. An experiment using the Netherlands F3 seismic dataset suggests that, compared with previously established deep learning models requiring a large number of training sets, the proposed U-Net method can achieve high evaluation indicators and better stratum segmentations in the case of a limited training set. During the test, the proposed U-Net model outperforms the Bayesian neural network (BNN) model for seismic stratum segmentation with regard to the training time, prediction speed, and segmentation accuracy. These results indicate the great potential of using U-Net-based deep learning for intelligent seismic stratigraphic interpretation.

Learning Semantic Segmentation of Large-Scale Point Clouds With Random Sampling.

We study the problem of efficient semantic segmentation of large-scale 3D point clouds. By relying on expensive sampling techniques or computationally heavy pre/post-processing steps, most existing approaches are only able to be trained and operate over small-scale point clouds. In this paper, we introduce RandLA-Net, an efficient and lightweight neural architecture to directly infer per-point semantics for large-scale point clouds. The key to our approach is to use random point sampling instead of more complex point selection approaches. Although remarkably computation and memory efficient, random sampling can discard key features by chance. To overcome this, we introduce a novel local feature aggregation module to progressively increase the receptive field for each 3D point, thereby effectively preserving geometric details. Comparative experiments show that our RandLA-Net can process 1 million points in a single pass up to 200× faster than existing approaches. Moreover, extensive experiments on five large-scale point cloud datasets, including Semantic3D, SemanticKITTI, Toronto3D, NPM3D and S3DIS, demonstrate the state-of-the-art semantic segmentation performance of our RandLA-Net.

ContexedNet: Context–Aware Ear Detection in Unconstrained Settings

Ear detection represents one of the key components of contemporary ear recognition systems. While significant progress has been made in the area of ear detection over recent years, most of the improvements are direct results of advances in the field of visual object detection. Only a limited number of techniques presented in the literature are domain–specific and designed explicitly with ear detection in mind. In this paper, we aim to address this gap and present a novel detection approach that does not rely only on general ear (object) appearance, but also exploits contextual information, i.e., face–part locations, to ensure accurate and robust ear detection with images captured in a wide variety of imaging conditions. The proposed approach is based on a Contex t–aware ${E}$ ar ${D}$ etection Net work (ContexedNet) and poses ear detection as a semantic image segmentation problem. ContexedNet consists of two processing paths: i) a context–provider that extracts probability maps corresponding to the locations of facial parts from the input image, and ii) a dedicated ear segmentation model that integrates the computed probability maps into a context–aware segmentation-based ear detection procedure. ContexedNet is evaluated in rigorous experiments on the AWE and UBEAR datasets and shown to ensure competitive performance when evaluated against state–of–the–art ear detection models from the literature. Additionally, because the proposed contextualization is model agnostic, it can also be utilized with other ear detection techniques to improve performance.

Light-Weight Semantic Segmentation Network for UAV Remote Sensing Images

Semantic segmentation for unmanned aerial vehicle (UAV) remote sensing images has become one of the research focuses in the field of remote sensing at present, which could accurately analyze the ground objects and their relationships. However, conventional semantic segmentation methods based on deep learning require large-scale models that are not suitable for resource-constrained UAV remote sensing tasks. Therefore, it is important to construct a light-weight semantic segmentation method for UAV remote sensing images. With this motivation, we propose a light-weight neural network model with fewer parameters to solve the problem of semantic segmentation of UAV remote sensing images. The network adopts an encoder–decoder architecture. In the encoder, we build a light-weight convolutional neural network model with fewer channels of each layer to reduce the number of model parameters. Then, feature maps of different scales from the encoder are concatenated together after resizing to carry out the multiscale fusion. Moreover, we employ two attention modules to capture the global semantic information from the context and the correlation among channels in UAV remote sensing images. In the decoder part, the model obtains predictions of each pixel through the softmax function. We conducted experiments on the ISPRS Vaihingen dataset, UAVid dataset, and UDD6 dataset to verify the effectiveness of the light-weight network. Our method obtains quality semantic segmentation results evaluated on UAV remote sensing datasets with only 9 M parameters the model owns, which is competitive among popular methods with the same level of parameters.

Triplet-Graph Reasoning Network for Few-Shot Metal Generic Surface Defect Segmentation

Metal surface defect segmentation can play an important role in dealing with the issue of quality control during the production and manufacturing stages. There are still two major challenges in industrial applications. One is the case that the number of metal surface defect samples is severely insufficient, and the other is that the most existing algorithms can only be used for specific surface defects and it is difficult to generalize to other metal surfaces. In this work, a theory of few-shot metal generic surface defect segmentation is introduced to solve these challenges. Simultaneously, the Triplet-Graph Reasoning Network (TGRNet) and a novel dataset Surface Defects- 4 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> are proposed to achieve this theory. In our TGRNet, the surface defect triplet (including triplet encoder and trip loss) is proposed and is used to segment background and defect area, respectively. Through triplet, the few-shot metal surface defect segmentation problem is transformed into few-shot semantic segmentation problem of defect area and background area. For few-shot semantic segmentation, we propose a method of multi-graph reasoning to explore the similarity relationship between different images. And to improve segmentation performance in the industrial scene, an adaptive auxiliary prediction module is proposed. For Surface Defects- 4 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> , it includes multiple categories of metal surface defect images to verify the generalization performance of our TGRNet and adds the nonmetal categories (leather and tile) as extensions. Through extensive comparative experiments and ablation experiments, it is proved that our architecture can achieve state-of-the-art results.

Phase unwrapping in ICF target interferometric measurement via deep learning.

This paper proposes an unwrapping algorithm based on deep learning for inertial confinement fusion (ICF) target interferograms. With a deep convolutional neural network (CNN), the task of phase unwrapping is transferred into a problem of semantic segmentation. A method for producing the data set for the ICF target measurement system is demonstrated. The noisy wrapped phase is preprocessed using a guided filter. Postprocessing is introduced to refine the final result, ensuring the proposed method can still accurately unwrap the phase even when the segmentation result of the CNN is not perfect. Simulations and actual interferograms show that our method has better accuracy and antinoise ability than some classical unwrapping approaches. In addition, the generalization capability of our method is verified by successfully applying it to an aspheric nonnull test system. By adjusting the data set, the proposed method may be transferred to other systems.

Map-Guided Curriculum Domain Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation.

We address the problem of semantic nighttime image segmentation and improve the state-of-the-art, by adapting daytime models to nighttime without using nighttime annotations. Moreover, we design a new evaluation framework to address the substantial uncertainty of semantics in nighttime images. Our central contributions are: 1) a curriculum framework to gradually adapt semantic segmentation models from day to night through progressively darker times of day, exploiting cross-time-of-day correspondences between daytime images from a reference map and dark images to guide the label inference in the dark domains; 2) a novel uncertainty-aware annotation and evaluation framework and metric for semantic segmentation, including image regions beyond human recognition capability in the evaluation in a principled fashion; 3) the Dark Zurich dataset, comprising 2416 unlabeled nighttime and 2920 unlabeled twilight images with correspondences to their daytime counterparts plus a set of 201 nighttime images with fine pixel-level annotations created with our protocol, which serves as a first benchmark for our novel evaluation. Experiments show that our map-guided curriculum adaptation significantly outperforms state-of-the-art methods on nighttime sets both for standard metrics and our uncertainty-aware metric. Furthermore, our uncertainty-aware evaluation reveals that selective invalidation of predictions can improve results on data with ambiguous content such as our benchmark and profit safety-oriented applications involving invalid inputs.

Ice Monitoring in Swiss Lakes from Optical Satellites and Webcams Using Machine Learning

Continuous observation of climate indicators, such as trends in lake freezing, is important to understand the dynamics of the local and global climate system. Consequently, lake ice has been included among the Essential Climate Variables (ECVs) of the Global Climate Observing System (GCOS), and there is a need to set up operational monitoring capabilities. Multi-temporal satellite images and publicly available webcam streams are among the viable data sources capable of monitoring lake ice. In this work we investigate machine learning-based image analysis as a tool to determine the spatio-temporal extent of ice on Swiss Alpine lakes as well as the ice-on and ice-off dates, from both multispectral optical satellite images (VIIRS and MODIS) and RGB webcam images. We model lake ice monitoring as a pixel-wise semantic segmentation problem, i.e., each pixel on the lake surface is classified to obtain a spatially explicit map of ice cover. We show experimentally that the proposed system produces consistently good results when tested on data from multiple winters and lakes. Our satellite-based method obtains mean Intersection-over-Union (mIoU) scores &gt; 93%, for both sensors. It also generalises well across lakes and winters with mIoU scores &gt; 78% and &gt;80% respectively. On average, our webcam approach achieves mIoU values of ≈87% and generalisation scores of ≈71% and ≈69% across different cameras and winters respectively. Additionally, we generate and make available a new benchmark dataset of webcam images (Photi-LakeIce) which includes data from two winters and three cameras.

Choosing Hyperparameter Values of the Convolution Neural Network When Solving the Problem of Semantic Segmentation of Images Obtained by Remote Sensing of the Earth’s Surface

Among the tasks solved by artificial neural networks are the tasks of analyzing objects on the images of the underlying Earth’s surface, obtained by the on-board equipment of unmanned aerial vehicle (UAV). For the solution of such problems, the convolutional neural networks (CNN), operating semantic segmentation of the received image, are widely used. In this case, the designer of such networks has to solve the difficult task of selecting hyperparameter values for them. These values’ choice is one of the most critical tasks that have to be solved when forming a CNN. Existing attempts to solve this problem are usually based on one of two approaches. The first one involves a set of experiments with different values of hyperparameters of the CNN with learning each of the network variants. These experiments are performed until a CNN with acceptable characteristics is obtained. This approach is simple to implement but does not guarantee a CNN with high performance. The second approach treats the selection of hyperparameter values in the network as an optimization problem. If this problem is successfully solved, it is possible to obtain a CNN with sufficiently high characteristics. However, this task has a significant complexity and also requires a large consumption of computing resources. Images in the form of multidimensional arrays are used as source data to analyze objects on the underlying surface. It means that CNN will contain a significant number of parameters. Accordingly, it will take considerable time to find a suitable CNN by searching for possible hyperparameter values. This paper proposes an alternative approach to the problem of selecting the hyperparameter values of CNN based on the analysis of the processes running in the network. The effectiveness of this approach is demonstrated by solving the problem of semantic segmentation of the underlying surface obtained by remote sensing of the Earth’s surface.

Semantic part segmentation of single-view point cloud

Single-view point cloud is the most commonly type of raw 3D data. Previous studies have focused on the object classification problem, and a few studies have been concerned with segmenting and labeling the semantic parts of single-view point cloud. In this paper, we propose a new method to solve the point cloud semantic part segmentation problem via annotation transference. First, we established a database of 3D synthetic CAD models. Taking a single-view point cloud as input, we retrieved the matching models from the database. Using the point-level correspondences, we transferred the annotations onto the input. We performed experiments on two public benchmarks and one raw scanned dataset. Compared to five other state-of-the-art methods, our method achieves a comparable accuracy with a low cost.

Semantic segmentation with deep learning: detection of cracks at the cut edge of glass

In this paper, artificial intelligence (AI) will be applied for the first time in the context of glass processing. The goal is to use an algorithm based on artificial intelligence to detect the fractured edge of a cut glass in order to generate a so-called mask image by AI. In the context of AI, this is a classical problem of semantic segmentation, in which objects (here the cut-edge of the cut glass) are automatically surrounded by the power of AI or detected and drawn. An original image of a cut glass edge is implemented into a deep neural net and processed in such a way that a mask image, i.e. an image of the cut edge, is automatically generated. Currently, this is only possible by manual tracing the cut-edge due to the fact that the crack contour of glass can sometimes only be recognized roughly. After manually marking the crack using an image processing program, the contour is then automatically evaluated further. AI and deep learning may provide the potential to automate the step of manual detection of the cut-edge of cut glass to great extent. In addition to the enormous time savings, the objectivity and reproducibility of detection is an important aspect, which will be addressed in this paper.

DEEP SEMANTIC SEGMENTATION FOR THE OFF-ROAD AUTONOMOUS DRIVING

Abstract. This paper is devoted to the problem of image semantic segmentation for machine vision system of off-road autonomous robotic vehicle. Most modern convolutional neural networks require large computing resources that go beyond the capabilities of many robotic platforms. Therefore, the main drawback of such models is extremely high complexity of the convolutional neural network used, whereas tasks in real applications must be performed on devices with limited resources in real-time. This paper focuses on the practical application of modern lightweight architectures as applied to the task of semantic segmentation on mobile robotic systems. The article discusses backbones based on ResNet18, ResNet34, MobileNetV2, ShuffleNetV2, EfficientNet-B0 and decoders based on U-Net and DeepLabV3 as well as additional components that can increase the accuracy of segmentation and reduce the inference time. In this paper we propose a model using ResNet34 and DeepLabV3 decoding with Squeeze &amp;amp; Excitation blocks that was optimal in terms of inference time and accuracy. We also demonstrate our off-road dataset and simulated dataset for semantic segmentation. Furthermore, we present that using pre-trained weights on simulated dataset achieves to increase 2.7% mIoU on our off-road dataset compared pre-trained weights on the Cityscapes. Moreover, we achieve 75.6% mIoU on the Cityscapes validation set and 85.2% mIoU on our off-road validation set with a speed of 37 FPS for a 1,024×1,024 input on one NVIDIA GeForce RTX 2080 card using NVIDIA TensorRT.

PCANet: Pyramid convolutional attention network for semantic segmentation

Pyramid Convolutional Attention Network is proposed to efficiently capture long-range dependency and fuse features from different levels for benefitting semantic segmentation problems. In this paper, we focus on how to extract more representative features for segmentation object recognition and design a decoder to recover details in a more efficient way. Inspired by atrous sampling and attention mechanism, we propose Pyramid Atrous Attention module to capture long-range dependency for learning richer contextual features. We also find that features of different levels have diverse representation so we design Convolutional Attention Refinement module to provide global context for low-level features and local details for high-level features. By combining with these two efficient module, we construct our Pyramid Convolutional Attention Network (PCANet), which achieves state-of-the-art results on Pascal VOC 2012 and Cityscapes benchmark.

LAKE ICE DETECTION FROM SENTINEL-1 SAR WITH DEEP LEARNING

Abstract. Lake ice, as part of the Essential Climate Variable (ECV) lakes, is an important indicator to monitor climate change and global warming. The spatio-temporal extent of lake ice cover, along with the timings of key phenological events such as freeze-up and break-up, provide important cues about the local and global climate. We present a lake ice monitoring system based on the automatic analysis of Sentinel-1 Synthetic Aperture Radar (SAR) data with a deep neural network. In previous studies that used optical satellite imagery for lake ice monitoring, frequent cloud cover was a main limiting factor, which we overcome thanks to the ability of microwave sensors to penetrate clouds and observe the lakes regardless of the weather and illumination conditions. We cast ice detection as a two class (frozen, non-frozen) semantic segmentation problem and solve it using a state-of-the-art deep convolutional network (CNN).We report results on two winters (2016–17 and 2017–18) and three alpine lakes in Switzerland. The proposed model reaches mean Intersection-over-Union (mIoU) scores &gt;90% on average, and &gt;84% even for the most difficult lake. Additionally, we perform cross-validation tests and show that our algorithm generalises well across unseen lakes and winters.

Cardiac MRI Segmentation With a Dilated CNN Incorporating Domain-Specific Constraints

Semantic segmentation of cardiac MR images is a challenging task due to its importance in medical assessment of heart diseases. Having a detailed localization of specific regions of interest such as Right and Left Ventricular Cavities and Myocardium, doctors can infer important information about the presence of cardiovascular diseases, which are today a major cause of death globally. This paper addresses the problem of semantic segmentation in cardiac MR images using a dilated Convolutional Neural Network. Opting for dilated convolutions allowed us to work in full resolution throughout the network's layers, preserving localization accuracy, while maintaining a relatively small number of trainable parameters. To assist the network's training process we designed a custom loss function. Furthermore, we developed new augmentation techniques and also adapted existing ones, to cope for the lack of sufficient training images. Consequently, the training set increases not only by amount, but by substance as well, and the network trains quickly and efficiently without overfitting. Our pre- and post-processing steps are also crucial to the whole process. We apply our methodology for the Right and Left Ventricles (RV, LV) and also the Myocardium (MYO) according to the Automated Cardiac Diagnosis Challenge (ACDC) with promising results. Submitting our algorithm's predictions to the Post-2017-MICCAI-challenge testing phase, we achieved similar scores (average Dice coefficient 0.916) on the test data set compared to the state of the art featured in the ACDC leaderboard, but with significantly fewer parameters than the leading method. Our approach outperforms other methods featuring dilated convolutions in this challenge up until now.

Efficient Dense Spatial Pyramid Network for Lane Detection

Lane detection separating the roads and positioning each lane is a significant task in Autonomous-driving field. Most of existing methods take lane detection as a semantic segmentation problem, and the methods based on encoder-decoder networks achieve robust performances than traditional methods with the disadvantages of large amounts of parameters and high computational cost. To improve efficiency and better adapt to adverse environments, we propose an EDSPnet by compiling the efficient dense module of depthwise dilated separable convolution (EDD module) and dense spatial pyramid (DSP) module. The two proposed modules enable our network to make full use of the contextual information from different feature maps when compared to a chain of layers. Different from symmetrical segmentation networks, we concentrate on the encoder block and a simple decoder is designed to match the input resolution. The decoder is complemented by two methods of upsampling: bilinear interpolation and deconvolution, which guarantee the accuracy and efficiency simultaneously. Our method is evaluated on the TuSimple dataset, and the experimental results show that our network has less parameters and is still efficient even compared with the state-of-the-art semantic segmentation networks.

Automated ECG Waveform Annotation Based on Stacked Long Short-Term Memory

The classification of electrocardiogram (ECG) waveform segmentation techniques can be difficult due to physiological variation of heart rate and different characteristics of the different ECG waves in terms of shape, frequency, amplitude, and duration. The P-wave, PR-segment, QRS-complex, ST-segment, and T-wave are extracted as the feature for classification algorithm to diagnose specified cardiac disorders. This requires the implementation of algorithms that identify specific points within the ECG wave. Some previous computational algorithms for automatic classification of ECG segmentation are proposed to overcome limitations of manual inspection of the ECG. This study presents new insight into the ECG semantic segmentation problem is surmounted by a deep learning approach for automatic ECG wave-form. Long short-term memory (LSTM) is proposed for this task. This experimental study has been performed for six different waveforms of ECG signal that represents cardiac disorders obtained from the Physionet: QT database. Overall, LSTM performance achieved accuracy, sensitivity, specificity, precision, F1-score, is 93.36%, 86.85%, 95.78%, 81.79%, and 83.09%, respectively.

Automated ECG Waveform Annotation Based on Stacked Long Short-Term Memory

The classification of electrocardiogram (ECG) waveform segmentation techniques can be difficult due to physiological variation of heart rate and different characteristics of the different ECG waves in terms of shape, frequency, amplitude, and duration. The P-wave, PR-segment, QRS-complex, ST-segment, and T-wave are extracted as the feature for classification algorithm to diagnose specified cardiac disorders. This requires the implementation of algorithms that identify specific points within the ECG wave. Some previous computational algorithms for automatic classification of ECG segmentation are proposed to overcome limitations of manual inspection of the ECG. This study presents new insight into the ECG semantic segmentation problem is surmounted by a deep learning approach for automatic ECG wave-form. Long short-term memory (LSTM) is proposed for this task. This experimental study has been performed for six different waveforms of ECG signal that represents cardiac disorders obtained from the Physionet: QT database. Overall, LSTM performance achieved accuracy, sensitivity, specificity, precision, F1-score, is 93.36%, 86.85%, 95.78%, 81.79%, and 83.09%, respectively.