Semantic Labeling Research Articles

DGCN-ED: dynamic graph convolutional networks with encoder–decoder structure and its application for airborne LiDAR point classification

ABSTRACT Achieving high-precision automatic classification in real-world applications for airborne laser scanning point clouds is a challenging task duo to their unstructured nature, uneven density distribution, high redundancy, incompleteness and scene complexity. Graph Convolutional Neural Networks can process scattered point clouds directly without regularization, which avoids the loss of depth information，and has recently become a topic of increased interest. Therefore, this study proposed an extension of the Graph-Unet network architecture named DGCN-ED for airborne LiDAR point classification, which uses a Graph Convolutional Neural Network as a representation to describe complex object relationships and an encoder-decoder architecture to capture the multi-scaled point features and describe objects in the high-level feature space. The two-layer dynamic update Graph Convolutional Neural Network is designed to expand the effective range of nodes and enhance the representation ability of learned pointwise features. The effectiveness of the proposed method is evaluated by an experiment on the ISPRS Vaihingen 3D semantic labelling benchmark dataset. Moreover, experiments on the IEEE 2019 Data Fusion Contest Dataset were conducted to demonstrate the generalization abilities of the proposed method. The results show that our method achieved on average 2.8% higher overall accuracy than existing methods, with an overall accuracy of 98.0% and an average F1 score of 0.797.

DPSNet: Multitask Learning Using Geometry Reasoning for Scene Depth and Semantics.

Multitask joint learning technology continues gaining more attention as a paradigm shift and has shown promising performance in many applications. Depth estimation and semantic understanding from monocular images emerge as a challenging problem in computer vision. While the other joint learning frameworks establish the relationship between the semantics and depth from stereo pairs, the lack of learning camera motion renders the frameworks that fail to model the geometric structure of the image scene. We make a further step in this article by proposing a multitask learning method, namely DPSNet, which can jointly perform depth and camera pose estimation and semantic scene segmentation. Our core idea for depth and camera pose prediction is that we present the rigid semantic consistency loss to overcome the limitation of moving pixels from image reconstruction technology and further infer the segmentation of moving instances based on them. In addition, our proposed model performs semantic segmentation by reasoning the geometric correspondences between the pixel semantic outputs and the semantic labels at multiscale resolutions. Experiments on open-source datasets and a video dataset captured on a micro-smart car show the effectiveness of each component of DPSNet, and DPSNet achieves state-of-the-art results in all three tasks compared with the best popular methods. All our models and code are available at https://github.com/jn-z/DPSNet: Multitask Learning Using Geometry Reasoning for Scene Depth and semantics.

A Large-scale Virtual Dataset and Egocentric Localization for Disaster Responses.

With the increasing social demands of disaster response, methods of visual observation for rescue and safety have become increasingly important. However, because of the shortage of datasets for disaster scenarios, there has been little progress in computer vision and robotics in this field. With this in mind, we present the first large-scale synthetic dataset of egocentric viewpoints for disaster scenarios. We simulate pre- and post-disaster cases with drastic changes in appearance, such as buildings on fire and earthquakes. The dataset consists of more than 300K high-resolution stereo image pairs, all annotated with ground-truth data for the semantic label, depth in metric scale, optical flow with sub-pixel precision, and surface normal as well as their corresponding camera poses. To create realistic disaster scenes, we manually augment the effects with 3D models using physically-based graphics tools. We train various state-of-the-art methods to perform computer vision tasks using our dataset, evaluate how well these methods recognize the disaster situations, and produce reliable results of virtual scenes as well as real-world images. We also present a convolutional neural network-based egocentric localization method that is robust to drastic appearance changes, such as the texture changes in a fire, and layout changes from a collapse. To address these key challenges, we propose a new model that learns a shape-based representation by training on stylized images, and incorporate the dominant planes of query images as approximate scene coordinates. We evaluate the proposed method using various scenes including a simulated disaster dataset to demonstrate the effectiveness of our method when confronted with significant changes in scene layout. Experimental results show that our method provides reliable camera pose predictions despite vastly changed conditions.

Cross-Viewpoint Semantic Mapping: Integrating Human and Robot Perspectives for Improved 3D Semantic Reconstruction.

Allocentric semantic 3D maps are highly useful for a variety of human-machine interaction related tasks since egocentric viewpoints can be derived by the machine for the human partner. Class labels and map interpretations, however, may differ or could be missing for the participants due to the different perspectives. Particularly, when considering the viewpoint of a small robot, which significantly differs from the viewpoint of a human. In order to overcome this issue, and to establish common ground, we extend an existing real-time 3D semantic reconstruction pipeline with semantic matching across human and robot viewpoints. We use deep recognition networks, which usually perform well from higher (i.e., human) viewpoints but are inferior from lower viewpoints, such as that of a small robot. We propose several approaches for acquiring semantic labels for images taken from unusual perspectives. We start with a partial 3D semantic reconstruction from the human perspective that we transfer and adapt to the small robot's perspective using superpixel segmentation and the geometry of the surroundings. The quality of the reconstruction is evaluated in the Habitat simulator and a real environment using a robot car with an RGBD camera. We show that the proposed approach provides high-quality semantic segmentation from the robot's perspective, with accuracy comparable to the original one. In addition, we exploit the gained information and improve the recognition performance of the deep network for the lower viewpoints and show that the small robot alone is capable of generating high-quality semantic maps for the human partner. The computations are close to real-time, so the approach enables interactive applications.

Joint semantic–geometric learning for polygonal building segmentation from high-resolution remote sensing images

As a fundamental task for geographical information updating, 3D city modeling, and other critical applications, the automatic extraction of building footprints from high-resolution remote sensing images has been substantially explored and received increasing attention over recent years. Among different types of building extraction methods, the polygonal segmentation methods produce vector building polygons that are in a more realistic format compared with those obtained from pixel-wise semantic labeling and contour-based methods. However, existing polygonal building segmentation methods usually require a perfect segmentation map and a complex post-processing procedure to guarantee the polygonization quality, or produce inaccurate vertex prediction results that suffer from wrong vertex sequence, self-intersections, fixed vertex quantity, etc. In our previous work, we have proposed a method for polygonal building segmentation from remote sensing images that addresses the above limitations of existing methods. In this paper, we propose PolyCity, which further extends and improves our previous work in terms of the application scenario, methodology design, and experimental results. Our proposed PolyCity contains the following three components: (1) a pixel-wise multi-task network for learning the semantic and geometric information via three tasks, i.e., building segmentation, boundary prediction, and edge orientation prediction; (2) a simple but effective vertex selection module (VSM), which effectively bridges the gap between pixel-wise and graph-based models via transforming the segmentation map into valid polygon vertices; (3) a graph-based vertex refinement network (VRN) for automatically adjusting the coordinates of VSM-generated valid polygon vertices, producing the final building polygons with more precise vertices. Results on three large-scale building extraction datasets demonstrate that our proposed PolyCity generates vector building footprints with more accurate vertices, edges, shapes, etc., achieving significant vertex score improvements while maintaining high segmentation and boundary scores compared with the current state-of-the-art. The code of PolyCity will be released at https://github.com/liweijia/polycity.

Pseudo-supervised image clustering based on meta-features

Stable semantics is a prerequisite for achieving excellent image clustering. However, most current methods suffer from inaccurate class semantic estimation, which limits the clustering performance. For the sake of addressing the issue, we propose a pseudo-supervised clustering framework based on meta-features. First, the framework mines meta-semantic features (i.e., meta-features) of image categories based on instance-level features, which not only preserves instance-level information but also ensures the semantic robustness of meta-features. Ulteriorly, we propagate pseudo-labels to its global neighbor samples with meta-features as the center, which effectively avoids the accumulation of errors caused by the misclassification of samples at the cluster boundary. Finally, we exploit the cross-entropy loss with label smoothing to optimize the pseudo-label optimization network. This optimization method not only achieves a direct mapping from features to stable semantic labels, but also effectively avoids suboptimal solutions caused by multi-level optimization. Extensive experiments demonstrate that our method significantly outperforms twenty-one competing clustering methods on six challenging datasets.

An automated image-based workflow for detecting megabenthic fauna in optical images with examples from the Clarion–Clipperton Zone

Recent advances in optical underwater imaging technologies enable the acquisition of huge numbers of high-resolution seafloor images during scientific expeditions. While these images contain valuable information for non-invasive monitoring of megabenthic fauna, flora and the marine ecosystem, traditional labor-intensive manual approaches for analyzing them are neither feasible nor scalable. Therefore, machine learning has been proposed as a solution, but training the respective models still requires substantial manual annotation. Here, we present an automated image-based workflow for Megabenthic Fauna Detection with Faster R-CNN (FaunD-Fast). The workflow significantly reduces the required annotation effort by automating the detection of anomalous superpixels, which are regions in underwater images that have unusual properties relative to the background seafloor. The bounding box coordinates of the detected anomalous superpixels are proposed as a set of weak annotations, which are then assigned semantic morphotype labels and used to train a Faster R-CNN object detection model. We applied this workflow to example underwater images recorded during cruise SO268 to the German and Belgian contract areas for Manganese-nodule exploration, within the Clarion–Clipperton Zone (CCZ). A performance assessment of our FaunD-Fast model showed a mean average precision of 78.1% at an intersection-over-union threshold of 0.5, which is on a par with competing models that use costly-to-acquire annotations. In more detail, the analysis of the megafauna detection results revealed that ophiuroids and xenophyophores were among the most abundant morphotypes, accounting for 62% of all the detections within the surveyed area. Investigating the regional differences between the two contract areas further revealed that both megafaunal abundance and diversity was higher in the shallower German area, which might be explainable by the higher food availability in form of sinking organic material that decreases from east-to-west across the CCZ. Since these findings are consistent with studies based on conventional image-based methods, we conclude that our automated workflow significantly reduces the required human effort, while still providing accurate estimates of megafaunal abundance and their spatial distribution. The workflow is thus useful for a quick but objective generation of baseline information to enable monitoring of remote benthic ecosystems.

Machine learning-based segmentation of aerial LiDAR point cloud data on building roof

ABSTRACT Three-dimensional (3D) reconstruction of a building can be facilitated by correctly segmenting different feature points (e.g. in the form of boundary, fold edge, and planar points) over the building roof, and then, establishing relationships among the constructed feature lines and planar patches using the segmented points. Present machine learning-based segmentation approaches of Light Detection and Ranging (LiDAR) point cloud data are confined only to different object classes or semantic labelling. In the context of fine-grained feature point classification over the extracted building roof, machine learning approaches have not yet been explored. In this paper, after generating the ground truth data for the extracted building roofs from three different datasets, we apply machine learning methods to segment the roof point cloud based on seven different effective geometric features. The goal is not to semantically enhance the point cloud, but rather to facilitate the application of 3D building reconstruction algorithms, making them easier to use. The calculated F1-scores for each class confirm the competitive performances over the state-of-the-art techniques, which are more than 95% almost in each area of the used datasets.

Semantic Real-Time Mapping with UAVs

Whilst mapping with UAVs has become an established tool for geodata acquisition in many domains, certain time-critical applications, such as crisis and disaster response, demand fast geodata processing pipelines rather than photogrammetric post-processing approaches. Based on our 3D-capable real-time mapping pipeline, this contribution presents not only an array of optimisations of the original implementation but also an extension towards understanding the image content with respect to land cover and object detection using machine learning. This paper (1) describes the pipeline in its entirety, (2) compares the performance of the semantic labelling and object detection models quantitatively and (3) showcases real-world experiments with qualitative evaluations.

Historical document image analysis using controlled data for pre-training

Using neural networks for semantic labeling has become a dominant technique for layout analysis of historical document images. However, to train or fine-tune appropriate models, large labeled datasets are needed. This paper addresses the case when only limited labeled data are available and promotes a novel approach using so-called controlled data to pre-train the networks. Two different strategies are proposed: The first addresses the real labeling task by using artificial data; the second uses real data to pre-train the networks with a pretext task. To assess these strategies, a large set of experiments has been carried out on a text line detection and classification task using different variants of U-Net. The observations, obtained from two different datasets, show that globally the approach reduces the training time while offering similar or better performance. Furthermore, the effect is bigger on lightweight network architectures.

Acoustic Scene Classification using Deep Fisher network

Acoustic Scene Classification (ASC) is the task of assigning a semantic label to an audio recording, based on the surrounding environment. In this work, a Fisher network is introduced for ASC. The proposed method mimics the working mechanism of a feed-forward Convolutional Neural Network (CNN) where, output of a layer is fed as an input to the succeeding layer. The Fisher network consists of a feature extraction step followed by a Fisher layer. The Fisher layer has three sub-layers, namely, Fisher Vector (FV) encoder, temporal pyramid and normalization layers along with feature reduction layer. Gammatone Time Cepstral Coefficients (GTCCs) and Mel-spectrograms are the features encoded as Fisher vector representation in FV encoder sub-layer. Temporal information of the Fisher vectors is retained using temporal pyramid sub-layer. After temporal pyramids are extracted from Fisher vectors, they are available as a feature vector. Irrelevant dimensions of the temporal pyramids are reduced further using Principal Component Analysis (PCA) in normalization and PCA sub-layers. The proposed model is evaluated on five DCASE datasets, TUT Urban Acoustic Scenes 2018 and Mobile, DCASE 2019 Acoustic Scene Classification Task 1(a) and Task 1(b), TAU Urban Acoustic Scenes 2020 datasets. The overall classification accuracy is 93%, 91%, 92%, 91% and 89% for TUT 2018, TUT Mobile 2018, DCASE Task 1(a) 2019, DCASE Task 1(b) 2019, and TAU Urban Acoustic Scenes 2020 datasets, respectively. The proposed model performed much better than the state-of-the-art ASC systems.

MDC-Net: a multi-directional constrained and prior assisted neural network for wood and leaf separation from terrestrial laser scanning

ABSTRACT Wood-leaf separation from terrestrial laser scanning (TLS) is a crucial prerequisite for quantifying many biophysical properties and understanding ecological functions. In this study, we propose a novel multi-directional collaborative convolutional neural network (MDC-Net) that takes the original 3D coordinates and useful features from prior knowledge (prior features) as input, and outputs the semantic labels of TLS point clouds. The MDC-Net contains two key units: (1) a multi-directional neighborhood construction (MDNC) unit to obtain more representative neighbors and enable directionally aware feature encoding in the subsequent local feature extraction, to mitigate occlusion effects; (2) a collaborative feature encoding (CFE) unit is introduced to incorporate useful features from prior knowledge into the network through a collaborative cross coding to enhance the discrimination for thin structures (e.g. small branches and leaf). The MDC-Net is evaluated on five plots from forests in Guangxi, China, with different branch architectures and leaf distributions. Experimental results showed that the MDC-Net achieved an OA of 0.973 and a mIoU of 0.821 and outperformed other related methods. We believe the MDC-Net would facilitate the usage of TLS in ecology studies for quantifying tree size and morphology and thus promote the development of relevant ecological applications.

Flow imaging as an alternative to non-intrusive measurements and surrogate models through vision transformers and convolutional neural networks

A numerical framework is proposed whereby flow imaging data are leveraged to extract relevant information from flowfield visualizations. To this end, a vision transformer (ViT) model is developed to predict quantities of interest from images of unsteady flows. Here, the unsteady pressure distribution, the aerodynamic coefficients, and the skin friction coefficient are computed for an airfoil under dynamic stall as an example. The network is capable of identifying relevant flow features present in the images and associate them to the airfoil response. Results demonstrate that the model is effective in interpolating and extrapolating between flow regimes and for different airfoil motions, meaning that ViT-based models may offer a promising alternative for sensors in experimental campaigns and for building robust surrogate models of complex unsteady flows. In addition, we uniquely treat the image semantic segmentation as an image-to-image translation task that infers semantic labels of structures from the input images in a supervised way. Given an input image of the velocity field, a resulting convolutional neural network generates synthetic images of any corresponding fluid property of interest. In particular, we convert the velocity field data into pressure in order to subsequently estimate the pressure distribution over the airfoil in a robust manner. This approach proves to be effective in mapping between flowfield properties.

Using deep learning to generate design spaces for architecture

We present an early prototype of a design system that uses Deep Learning methodology—Conditional Variational Autoencoders (CVAE)—to arrive at custom design spaces that can be interactively explored using semantic labels. Our work is closely tied to principles of parametric design. We use parametric models to create the dataset needed to train the neural network, thus tackling the problem of lacking 3D datasets needed for deep learning. We propose that the CVAE functions as a parametric tool in itself: The solution space is larger and more diverse than the combined solution spaces of all parametric models used for training. We showcase multiple methods on how this solution space can be navigated and explored, supporting explorations such as object morphing, object addition, and rudimentary 3D style transfer. As a test case, we implemented some examples of the geometric taxonomy of “Operative Design” by Di Mari and Yoo.

Computational modeling of a generative grammar for clitic order in River Plate Spanish

Abstract The order of clitics (CLs) in Romance languages has been studied in depth by many scholars since Perlmutter’s first approach. This author points out that person and case features impose a filter that results in the following pattern in Spanish: Se- II- I- III dat - III acc . This pattern has been revisited because it is not restrictive enough: it gives rise to impossible sequences. In this context, the goal of this work is to define a pattern of morphosyntactic features for River Plate Spanish clitics, and an ordering pattern, with the purpose of developing a computational model for an automatic analysis of this phenomenon. This kind of analysis is justified because the algorithm output shows the degree of success of the descriptive proposal. To this end, we use NooJ, a linguistic development environment that has diverse tools. The computational modeling involves two stages: (i) the creation of an electronic dictionary and (ii) the creation of a computational grammar. The algorithm developed can generate all correct sentences conformed by Nominative Pronoun + CL + CL + Verb, Non-Finite Verb-CL-CL and Imperative Verb-CL-CL, and it can recognize these kinds of expressions in written text assigning the correct semantic labels. With these results, we conclude that our descriptive proposal is adequate for the analysis of sequences of clitics in River Plate Spanish.

Human Parsing with Joint Learning for Dynamic mmWave Radar Point Cloud

Human sensing and understanding is a key requirement for many intelligent systems, such as smart monitoring, human-computer interaction, and activity analysis, etc. In this paper, we present mmParse, the first human parsing design for dynamic point cloud from commercial millimeter-wave radar devices. mmParse proposes an end-to-end neural network design that addresses the inherent challenges in parsing mmWave point cloud (e.g., sparsity and specular reflection). First, we design a novel multi-task learning approach, in which an auxiliary task can guide the network to understand human structural features. Secondly, we introduce a multi-task feature fusion method that incorporates both intra-task and inter-task attention to aggregate spatio-temporal features of the subject from a global view. Through extensive experiments in both indoor and outdoor environments, we demonstrate that our proposed system is able to achieve ~ 92% accuracy and ~ 84% IoU accuracy. We also show that the predicted semantic labels can increase the performance of two downstream tasks (pose estimation and action recognition) by ~ 18% and ~ 6% respectively.

Trajectory Clustering-Based Anomaly Detection in Indoor Human Movement

Human movement anomalies in indoor spaces commonly involve urgent situations, such as security threats, accidents, and fires. This paper proposes a two-phase framework for detecting indoor human trajectory anomalies based on density-based spatial clustering of applications with noise (DBSCAN). The first phase of the framework groups datasets into clusters. In the second phase, the abnormality of a new trajectory is checked. A new metric called the longest common sub-sequence using indoor walking distance and semantic label (LCSS_IS) is proposed to calculate the similarity between trajectories, extending from the longest common sub-sequence (LCSS). Moreover, a DBSCAN cluster validity index (DCVI) is proposed to improve the trajectory clustering performance. The DCVI is used to choose the epsilon parameter for DBSCAN. The proposed method is evaluated using two real trajectory datasets: MIT Badge and sCREEN. The experimental results show that the proposed method effectively detects human trajectory anomalies in indoor spaces. With the MIT Badge dataset, the proposed method achieves 89.03% in terms of F1-score for hypothesized anomalies and above 93% for all synthesized anomalies. In the sCREEN dataset, the proposed method also achieves impressive results in F1-score on synthesized anomalies: 89.92% for rare location visit anomalies (τ = 0.5) and 93.63% for other anomalies.

Path Segmentation from Point Cloud Data for Autonomous Navigation

Autonomous vehicles require in-depth knowledge of their surroundings, making path segmentation and object detection crucial for determining the feasible region for path planning. Uniform characteristics of a road portion can be denoted by segmentations. Currently, road segmentation techniques mostly depend on the quality of camera images under different lighting conditions. However, Light Detection and Ranging (LiDAR) sensors can provide extremely precise 3D geometry information about the surroundings, leading to increased accuracy with increased memory consumption and computational overhead. This paper introduces a novel methodology which combines LiDAR and camera data for road detection, bridging the gap between 3D LiDAR Point Clouds (PCs). The assignment of semantic labels to 3D points is essential in various fields, including remote sensing, autonomous vehicles, and computer vision. This research discusses how to select the most relevant geometric features for path planning and improve autonomous navigation. An automatic framework for Semantic Segmentation (SS) is introduced, consisting of four processes: selecting neighborhoods, extracting classification features, and selecting features. The aim is to make the various components usable for end users without specialized knowledge by considering simplicity, effectiveness, and reproducibility. Through an extensive evaluation of different neighborhoods, geometric features, feature selection methods, classifiers, and benchmark datasets, the outcomes show that selecting the appropriate neighborhoods significantly develops 3D path segmentation. Additionally, selecting the right feature subsets can reduce computation time, memory usage, and enhance the quality of the results.

Prior Semantic Information Guided Change Detection Method for Bi-temporal High-Resolution Remote Sensing Images

High-resolution remote sensing image change detection technology compares and analyzes bi-temporal or multitemporal high-resolution remote sensing images to determine the change areas. It plays an important role in land cover/use monitoring, natural disaster monitoring, illegal building investigation, military target strike effect analysis, and land and resource investigation. The change detection of high-resolution remote sensing images has developed rapidly from data accumulation to algorithm models because of the rapid development of technologies such as deep learning and earth observation in recent years. However, the current deep learning-based change detection methods are strongly dependent on large sample data, and the training model has insufficient cross-domain generalization ability. As a result, a prior semantic information-guided change detection framework (PSI-CD), which alleviates the change detection model’s dependence on datasets by making full use of prior semantic information, is proposed in this paper. The proposed method mainly includes two parts: one is a prior semantic information generation network that uses the semantic segmentation dataset to extract robust and reliable prior semantic information; the other is the prior semantic information guided change detection network that makes full use of prior semantic information to reduce the sample size of the change detection. To verify the effectiveness of the proposed method, we produced pixel-level semantic labels for the bi-temporal images of the public change detection dataset (LEVIR-CD). Then, we performed extensive experiments on the WHU and LEVIR-CD datasets, including comparisons with existing methods, experiments with different amounts of data, and ablation study, to show the effectiveness of the proposed method. Compared with other existing methods, our method has the highest IoU for all training samples and different amounts of training samples on WHU and LEVIR-CD, reaching a maximum of 83.25% and 83.80%, respectively.

Human activity classification using deep learning based on 3D motion feature

Human activity classification is needed to support various fields. The health sector, for example, requires the ability to monitor the activities of patients, the elderly, or people with special needs to provide services with fast response as needed. In the traditional classification model, the steps taken to start from the input of data and then proceed with feature extraction, representation, classifier and end with semantic labels. The classification stage uses Convolutional Neural Network (CNN) deep learning to data input, CNN, and semantic labels. This paper proposes a novel method of classifying nine activities based on the movement features of changes in joint distance using Euclidean on the order of frames in each activity segment as input to the CNN model. This study’s motion feature extraction technique was tested using various window sizes to obtain the best classification accuracy. The experimental results show that the selection of window size 16 on the motion feature setting will produce an optimal model accuracy of 94.08% in classifying human activities.