Scene Segmentation Research Articles

How different levels of semantic segmentation affect human perception of driving scenes

As automated vehicles continue to advance, teleoperation has emerged as a critical support system for navigating complex and unpredictable environments that exceed the vehicles' current autonomous capabilities. A main issue in the implementation of teleoperation is latency caused by the high bandwidth required to transmit the video feed from the vehicle to the remote teleoperation station. A possible approach for addressing the latency problem is the transfer of lower-resolution or compressed videos between the vehicle and the teleoperation station. When applying semantic segmentation on the video feed, many pixels are mapped to a limited set of possible colors according to the types of objects that they represent. This concept has been commonly used in autonomous driving algorithms and has the potential to enable the transferring of smaller-sized videos thus reducing bandwidth. In this study, we examine how presenting semantically segmented driving scenes to humans affects their perception of the scene, and specifically, how it affects their hazard perception and situation awareness. We conducted two user studies comparing the effects of using different levels and types of semantic segmentation. Our results indicate that viewing partly segmented scenes, such that only a selected set of object types are colored, commonly achieves the same effect, and sometimes even outperforms a realistic view. Our study and its insights may pave the way for future research, development, and design of teleoperation systems of automated vehicles.

Plant leaf image segmentation in natural scenes: a multi-layer graph queries propagation approach

Plant leaf image segmentation in natural scenes: a multi-layer graph queries propagation approach

Calibration‐Jitter: Augmentation of hyperspectral data for improved surgical scene segmentation

Calibration‐Jitter: Augmentation of hyperspectral data for improved surgical scene segmentation

Research Status and Prospect of the Key Technologies for Environment Perception of Intelligent Excavators

With the urgent need of the industry and the continuous development of artificial intelligence, research into intelligent excavators has achieved certain progress. However, intelligent excavators often face strong vibrations, dense dust, and complex objectives. These have brought severe challenges to environmental perception, and are important research difficulties that must be overcome in realizing the practical engineering applications of intelligent excavators. Many researchers have studied these problems in reducing vibration and dust noise for light detection and ranging (LiDAR) scanners, multi-sensor information fusion, and the segmentation and recognition of 3D scenes. This paper reviews the research status of these key technologies and discusses their development trends.

Channel2DTransformer: A Multi-level Features Self-attention Fusion Module for Semantic Segmentation

Semantic segmentation is a crucial technology for intelligent vehicles, enabling scene understanding in complex driving environments. However, complex real-world scenarios often contain diverse multi-scale objects, which bring challenges to the accurate semantic segmentation. To address this challenge, we propose a multi-level features self-attention fusion module called Channel2DTransformer. The module utilizes self-attention mechanisms to dynamically fuse multi-level features by computing self-attention weights between their channels, resulting in a consistent and comprehensive representation of scene features. We perform the module on the Cityscapes and NYUDepthV2 datasets, which contain a large number of multi-scale objects. The experimental results validate the positive contributions of the module in enhancing the semantic segmentation accuracy of multi-scale objects and improving the performance of semantic segmentation in complex scenes.

Spatial Attention-Based Kernel Point Convolution Network for Semantic Segmentation of Transmission Corridor Scenarios in Airborne Laser Scanning Point Clouds

Accurate semantic segmentation in transmission corridor scenes is crucial for the maintenance and inspection of power infrastructure, facilitating the timely detection of potential hazards. In this study, we propose SA-KPConv, an advanced segmentation model specifically designed for transmission corridor scenarios. Traditional approaches, including Random Forest and point-based deep learning models such as PointNet++, demonstrate limitations in segmenting critical infrastructure components, particularly power lines and towers, primarily due to their inadequate capacity to capture complex spatial relationships and local geometric details. Our model effectively addresses these challenges by integrating a spatial attention module with kernel point convolution, enhancing both global context and local feature extraction. Experiments demonstrate that SA-KPConv outperforms state-of-the-art methods, achieving a mean Intersection over Union (mIoU) of 89.62%, particularly excelling in challenging terrains such as mountainous areas. Ablation studies further validate the significance of our model’s components in enhancing overall performance and effectively addressing class imbalance. This study presents a robust solution for semantic segmentation, with considerable potential for monitoring and maintaining power infrastructure.

DRMNet: more efficient bilateral networks for real-time semantic segmentation of road scenes

DRMNet: more efficient bilateral networks for real-time semantic segmentation of road scenes

An Algorithmic Study of Transformer-Based Road Scene Segmentation in Autonomous Driving

Applications such as autonomous driving require high-precision semantic image segmentation technology to identify and understand the content of each pixel in the images. Compared with traditional deep convolutional neural networks, the Transformer model is based on pure attention mechanisms, without convolutional layers or recurrent neural network layers. In this paper, we propose a new network structure called SwinLab, which is an improvement upon the Swin Transformer. Experimental results demonstrate that the improved SwinLab model achieves a segmentation accuracy comparable to that of deep convolutional neural network models in applications such as autonomous driving, with an MIoU of 77.61. Additionally, comparative experiments on the CityScapes dataset further validate the effectiveness and generalization of this structure. In conclusion, by refining the Swin Transformer, this paper simplifies the model structure, improves the training and inference speed, and maintains high accuracy, providing a more reliable semantic image segmentation solution for applications such as autonomous driving.

SAFENet: Semantic-Aware Feature Enhancement Network for unsupervised cross-domain road scene segmentation

Unsupervised cross-domain road scene segmentation has attracted substantial interest because of its capability to perform segmentation on new and unlabeled domains, thereby reducing the dependence on expensive manual annotations. This is achieved by leveraging networks trained on labeled source domains to classify images on unlabeled target domains. Conventional techniques usually use adversarial networks to align inputs from the source and the target in either of their domains. However, these approaches often fall short in effectively integrating information from both domains due to Alignment in each space usually leads to bias problems during feature learning. To overcome these limitations and enhance cross-domain interaction while mitigating overfitting to the source domain, we introduce a novel framework called Semantic-Aware Feature Enhancement Network (SAFENet) for Unsupervised Cross-domain Road Scene Segmentation. SAFENet incorporates the Semantic-Aware Enhancement (SAE) module to amplify the importance of class information in segmentation tasks and uses the semantic space as a new domain to guide the alignment of the source and target domains. Additionally, we integrate Adaptive Instance Normalization with Momentum (AdaIN-M) techniques, which convert the source domain image style to the target domain image style, thereby reducing the adverse effects of source domain overfitting on target domain segmentation performance. Moreover, SAFENet employs a Knowledge Transfer (KT) module to optimize network architecture, enhancing computational efficiency during testing while maintaining the robust inference capabilities developed during training. To further improve the segmentation performance, we further employ Curriculum Learning, a self-training mechanism that uses pseudo-labels derived from the target domain to iteratively refine the network. Comprehensive experiments on three well-known datasets, “Synthia→Cityscapes” and “GTA5→Cityscapes”, demonstrate the superior performance of our method. In-depth examinations and ablation studies verify the efficacy of each module within the proposed method.

Improved automated parallel implementation of GMM background subtraction on a multicore digital signal processor

<p><span>Scene segmentation is an essential step in a wide range of video processing applications, for instance, object recognition and tracking. The Gaussian mixture model (GMM) for background subtraction (BS) has gained widespread usage in scene segmentation, despite its known computational intensity. To tackle this challenge, we propose a practical solution to accelerate processing through a parallel implementation on an embedded multicore platform. In this paper, we present an improved automated parallel implementation of the GMM algorithm using the Orphan directive provided by open multiprocessing (OpenMP). Experimental assessments conducted on the eight cores of the C6678 digital signal processor (DSP) demonstrate significant advancements in parallel efficiency, particularly when handling high-resolution frames, including high-definition (HD) and full-HD resolutions. The achieved parallel efficiency surpasses the results obtained with classical OpenMP scheduling modes, encompassing dynamic, static, and guided approaches. Specifically, the parallel efficiency reaches approximately 82% for full-HD resolution frames and, 99.3% for low-resolution frames, respectively.</span></p>

Filtering-Assisted Airborne Point Cloud Semantic Segmentation for Transmission Lines.

Point cloud semantic segmentation is crucial for identifying and analyzing transmission lines. Due to the number of point clouds being huge, complex scenes, and unbalanced sample proportion, the mainstream machine learning methods of point cloud segmentation cannot provide high efficiency and accuracy when extending to transmission line scenes. This paper proposes a filter-assisted airborne point cloud semantic segmentation for transmission lines. First, a large number of ground point clouds is identified by introducing the well-developed cloth simulation filter to alleviate the impact of the imbalance of the target object proportion on the classifier's performance. The multi-dimensional features are then defined, and the classification model is trained to achieve the multi-element semantic segmentation of the transmission line scene. The experimental results and analysis indicate that the proposed filter-assisted algorithm can significantly improve the semantic segmentation performance of the transmission line point cloud, enhancing both the point cloud segmentation efficiency and accuracy by more than 25.46% and 3.15%, respectively. The filter-assisted point cloud semantic segmentation method reduces the volume of sample data, the number of sample classes, and the sample imbalance index in power line scenarios to a certain extent, thereby improving the classification accuracy of classifiers and reducing time consumption. This research holds significant theoretical reference value and engineering application potential for scene reconstruction and intelligent understanding of airborne laser point cloud transmission lines.

Deep Plug-and-Play Non-Iterative Cluster for 3D Global Feature Extraction

Efficient and accurate point cloud feature extraction is crucial for critical tasks such as 3D recognition and semantic segmentation. However, existing global feature extraction methods for 3D data often require designing different models for different input types (point clouds, voxels, and maps). This article proposes an efficient plug-and-play non-iterative clustering method (NICM) to establish a unified point cloud global feature extraction paradigm suitable for any input type to solve the above problems. The core idea of the NICM is to construct the connection between a single point and other global points based only on the cosine similarity between center points to achieve global feature extraction, which has linear complexity characteristics and can be combined with any existing feature extraction model. Additionally, to better integrate the features extracted by NICM and the original model, this article designs an adaptive feature fusion module is designed based on the gate unit, which retains similar features and effectively fuses dissimilar features based on their importance to downstream tasks. We have applied our method to downstream tasks such as point cloud recognition, part segmentation and scene segmentation. Sufficient experiments have proven that our method can provide comprehensive and robust features for the original model, and effectively improve the performance of downstream tasks.

Dexterous Manipulation Based on Object Recognition and Accurate Pose Estimation Using RGB-D Data.

This study presents an integrated system for object recognition, six-degrees-of-freedom pose estimation, and dexterous manipulation using a JACO robotic arm with an Intel RealSense D435 camera. This system is designed to automate the manipulation of industrial valves by capturing point clouds (PCs) from multiple perspectives to improve the accuracy of pose estimation. The object recognition module includes scene segmentation, geometric primitives recognition, model recognition, and a color-based clustering and integration approach enhanced by a dynamic cluster merging algorithm. Pose estimation is achieved using the random sample consensus algorithm, which predicts position and orientation. The system was tested within a 60° field of view, which extended in all directions in front of the object. The experimental results show that the system performs reliably within acceptable error thresholds for both position and orientation when the objects are within a ±15° range of the camera's direct view. However, errors increased with more extreme object orientations and distances, particularly when estimating the orientation of ball valves. A zone-based dexterous manipulation strategy was developed to overcome these challenges, where the system adjusts the camera position for optimal conditions. This approach mitigates larger errors in difficult scenarios, enhancing overall system reliability. The key contributions of this research include a novel method for improving object recognition and pose estimation, a technique for increasing the accuracy of pose estimation, and the development of a robot motion model for dexterous manipulation in industrial settings.

Domain adaptation for semantic segmentation of road scenes via two-stage alignment of traffic elements

Unsupervised domain adaptation has been used to reduce the domain shift, which would improve the performance of semantic segmentation on unlabeled real-world data. However, existing methodologies fall short in effectively addressing the domain shift issue prevalent in traffic scenarios, leading to less than satisfactory segmentation results. In this paper, we propose a novel domain adaptation method for semantic segmentation via unsupervised alignment of traffic elements. Firstly, we introduce a two-stage self-training framework that leverages a blended set of training samples to enhance the training process. In the first stage, we leverage generated mixup training samples as inputs within our two-stage self-training framework and have developed corresponding loss functions for both the source and target domains to direct the training process. Then, the alignment modules for dynamic and static traffic elements are designed to achieve accurate matching between the source and the target domain images. The cosine similarity maximization is applied to the alignment of dynamic traffic elements, while the prototype learning is utilized for the static traffic elements. Additionally, we present a new technique for reducing noise in pseudo labels by constructing thresholds that adjust to each class. Meanwhile, we formulate the associated target domain loss function for vacant pseudo label pixels. The experimental results demonstrate that the proposed method is superior to the existing methods on five different domain adaptation tasks, which is more applicable to semantic segmentation of road scenes.

Towards full autonomous driving: challenges and frontiers

With the rapid advancement of information technology and intelligent systems, autonomous driving has garnered significant attention and research in recent years. Key technologies, such as Simultaneous Localization and Mapping (SLAM), Perception and Localization, and Scene Segmentation, have proven to be essential in this field. These technologies not only evolve independently, each with its own research focus and application paths, but also complement and rely on one another in various complex autonomous driving scenarios. This paper provides a comprehensive review of the development and current state of these technologies, along with a forecast of their future trends.

A point cloud segmentation algorithm based on multi-feature training and weighted random forest

Abstract Point cloud segmentation is the process of dividing point cloud data into a series of coherent subsets according to its attributes. It has been widely used in target recognition, digital protection of cultural relics, medical research and other fields. To improve the classification accuracy of point cloud and achieve accurate segmentation of objects or scenes, a point cloud segmentation algorithm based on multi–features training and weighted random forest (RF) is proposed. Firstly, the feature vector composed of 3D coordinate value, RGB value, echo intensity, point cloud density, normal direction and average curvature is used to train the SVM classifier, and the ‘one–to–one’ strategy is adopted to achieve the initial multivariate rough segmentation of point cloud. Then, the maximum information coefficient and sample correlation coefficient (SCC) are used to evaluate the correlation of the decision tree, and the decision tree is weighted accordingly to build a weak correlation weighted RF, so as to achieve further accurate segmentation of the point cloud. The experiment verifies the effectiveness of the proposed algorithm by segmenting the outdoor scene point cloud data model. The results show that the segmentation algorithm based on multi–features training and weighted RF can achieve accurate point cloud segmentation, and is an effective point cloud segmentation method.

Mesh Convolution With Continuous Filters for 3-D Surface Parsing.

Geometric feature learning for 3-D surfaces is critical for many applications in computer graphics and 3-D vision. However, deep learning currently lags in hierarchical modeling of 3-D surfaces due to the lack of required operations and/or their efficient implementations. In this article, we propose a series of modular operations for effective geometric feature learning from 3-D triangle meshes. These operations include novel mesh convolutions, efficient mesh decimation, and associated mesh (un)poolings. Our mesh convolutions exploit spherical harmonics as orthonormal bases to create continuous convolutional filters. The mesh decimation module is graphics processing unit (GPU)-accelerated and able to process batched meshes on-the-fly, while the (un)pooling operations compute features for upsampled/downsampled meshes. We provide an open-source implementation of these operations, collectively termed Picasso. Picasso supports heterogeneous mesh batching and processing. Leveraging its modular operations, we further contribute a novel hierarchical neural network for perceptual parsing of 3-D surfaces, named PicassoNet++. It achieves highly competitive performance for shape analysis and scene segmentation on prominent 3-D benchmarks. The code, data, and trained models are available at https://github.com/EnyaHermite/Picasso.

A Patch Diversity Transformer for Domain Generalized Semantic Segmentation.

Domain generalization (DG) is one of the critical issues for deep learning in unknown domains. How to effectively represent domain-invariant context (DIC) is a difficult problem that DG needs to solve. Transformers have shown the potential to learn generalized features, since the powerful ability to learn global context. In this article, a novel method named patch diversity Transformer (PDTrans) is proposed to improve the DG for scene segmentation by learning global multidomain semantic relations. Specifically, patch photometric perturbation (PPP) is proposed to improve the representation of multidomain in the global context information, which helps the Transformer learn the relationship between multiple domains. Besides, patch statistics perturbation (PSP) is proposed to model the feature statistics of patches under different domain shifts, which enables the model to encode domain-invariant semantic features and improve generalization. PPP and PSP can help to diversify the source domain at the patch level and feature level. PDTrans learns context across diverse patches and takes advantage of self-attention to improve DG. Extensive experiments demonstrate the tremendous performance advantages of the PDTrans over state-of-the-art DG methods.

CAFA: Cross-Modal Attentive Feature Alignment for Cross-Domain Urban Scene Segmentation

CAFA: Cross-Modal Attentive Feature Alignment for Cross-Domain Urban Scene Segmentation

MSF2Net: multi-stage feature fusion network for real-time semantic segmentation in road scenes

MSF2Net: multi-stage feature fusion network for real-time semantic segmentation in road scenes