Objects In Point Clouds Research Articles

Classification of Terrestrial Laser Scanner Point Clouds: A Comparison of Methods for Landslide Monitoring from Mathematical Surface Approximation

Terrestrial laser scanners (TLS) are contact-free measuring sensors that record dense point clouds of objects or scenes by acquiring coordinates and an intensity value for each point. The point clouds are scattered and noisy. Performing a mathematical surface approximation instead of working directly on the point cloud is an efficient way to reduce the data storage and structure the point clouds by transforming “data” to “information”. Applications include rigorous statistical testing for deformation analysis within the context of landslide monitoring. In order to reach an optimal approximation, classification and segmentation algorithms can identify and remove inhomogeneous structures, such as trees or bushes, to obtain a smooth and accurate mathematical surface of the ground. In this contribution, we compare methods to perform the classification of TLS point clouds with the aim of guiding the reader through the existing algorithms. Besides the traditional point cloud filtering methods, we will analyze machine learning classification algorithms based on the manual extraction of point cloud features, and a deep learning approach with automatic extraction of features called PointNet++. We have intentionally chosen strategies easy to implement and understand so that our results are reproducible for similar point clouds. We show that each method has advantages and drawbacks, depending on user criteria, such as the computational time, the classification accuracy needed, whether manual extraction is performed or not, and if prior information is required. We highlight that filtering methods are advantageous for the application at hand and perform a mathematical surface approximation as an illustration. Accordingly, we have chosen locally refined B-splines, which were shown to provide an optimal and computationally manageable approximation of TLS point clouds.

End-to-End Learning to Grasp via Sampling From Object Point Clouds

The ability to grasp objects is an essential skill that enables many robotic manipulation tasks. Recent works have studied point cloud-based methods for object grasping by starting from simulated datasets and have shown promising performance in real-world scenarios. Nevertheless, many of them still rely on ad-hoc geometric heuristics to generate grasp candidates, which fail to generalize to objects with significantly different shapes with respect to those observed during training. Several approaches exploit complex multi-stage learning strategies and local neighborhood feature extraction while ignoring semantic global information. Furthermore, they are inefficient in terms of number of training samples and time required for inference. In this letter, we propose an end-to-end learning solution to generate 6-DOF parallel-jaw grasps starting from the 3D partial view of the object. Our Learning to Grasp (L2G) method gathers information from the input point cloud through a new procedure that combines a differentiable sampling strategy to identify the visible contact points, with a feature encoder that leverages local and global cues. Overall, L2G is guided by a multi-task objective that generates a diverse set of grasps by optimizing contact point sampling, grasp regression, and grasp classification. With a thorough experimental analysis, we show the effectiveness of L2G as well as its robustness and generalization abilities.

Sensor Fusion for Aircraft Detection at Airport Ramps Using Conditional Random Fields

Self-driving baggage tractors on airport ramps or aprons enable better airport operation procedures and support the expansion of the aviation market. Airport ramps have unique mobility requirements in terms of layout, population, demand, and patterns. Avoiding aircraft movement on an airport apron is a top priority because of critical security and safety issues. Existing aircraft detection approaches use remote-sensing images or surveillance cameras. However, these are not compatible with sensors for low-height equipment at airport ramps. Similarly, public road-based self-driving studies have not considered detecting the massive size and concave contours of movable objects. Camera sensors cannot accurately measure the distance of concave contours, whereas a lidar sensor cannot easily cluster or classify an object among point cloud data. In this paper, we present the fusion of cameras and lidar sensors for aircraft and object detection at airport ramps. We use parallel detection from lidar and camera sensors and then integrate both detection results to compensate for any issues. Using the proposed energy optimization model by adapting a conditional random field, we can handle over- and under-segmentation of the point cloud objects caused by the sparse point cloud generated by the aircraft. Our algorithm achieves 31.1% improvement on tracking and 5.5% improvement on classification over other fusion algorithms when applied to a dataset acquired from the Cincinnati and Northern Kentucky airport.

Masked Autoencoder for Pre-Training on 3D Point Cloud Object Detection

In autonomous driving, the 3D LiDAR (Light Detection and Ranging) point cloud data of the target are missing due to long distance and occlusion. It makes object detection more difficult. This paper proposes Point Cloud Masked Autoencoder (PCMAE), which can provide pre-training for most voxel-based point cloud object detection algorithms. PCMAE improves the feature representation ability of the 3D backbone for long-distance and occluded objects through self-supervised learning. First, a point cloud masking strategy for autonomous driving scenes named PC-Mask is proposed. It is used to simulate the problem of missing point cloud data information due to occlusion and distance in autonomous driving scenarios. Then, a symmetrical encoder–decoder architecture is designed for pre-training. The encoder is used to extract the high-level features of the point cloud after PC-Mask, and the decoder is used to reconstruct the complete point cloud. Finally, the pre-training method proposed in this paper is applied to SECOND (Sparsely Embedded Convolutional Detection) and Part-A2-Net (Part-aware and Aggregate Neural Network) object detection algorithms. The experimental results show that our method can speed up the model convergence speed and improve the detection accuracy, especially the detection effect of long-distance and occluded objects.

Mixed Feature Prediction on Boundary Learning for Point Cloud Semantic Segmentation

Existing point cloud semantic segmentation approaches do not perform well on details, especially for the boundary regions. However, supervised-learning-based methods depend on costly artificial annotations for performance improvement. In this paper, we bridge this gap by designing a self-supervised pretext task applicable to point clouds. Our main innovation lies in the mixed feature prediction strategy during the pretraining stage, which facilitates point cloud feature learning with boundary-aware foundations. Meanwhile, a dynamic feature aggregation module is proposed to regulate the range of receptive field according to the neighboring pattern of each point. In this way, more spatial details are preserved for discriminative high-level representations. Extensive experiments across several point cloud segmentation datasets verify the superiority of our proposed method, including ShapeNet-part, ScanNet v2, and S3DIS. Furthermore, transfer learning on point cloud classification and object detection tasks demonstrates the generalization ability of our method.

Cascaded multi-3D-view fusion for 3D-oriented object detection

Currently, multi-view fusion methods fuse point- or proposal-level features from different views at the end stage of the backbone. This once-end-fusion method is not conducive to the timely adjustment of spatial misalignment for features from different views. Consequently, the discriminative depth and orientation details of the 3D oriented point cloud object may be filtered. To enhance the feature capture capability of the network, we introduce a cascaded multi-3D-view fusion method (CM3DV) to learn the implicit representation of object orientation. In particular, the proposed CM3DV method incorporates the cylindrical front view projection into a voxelised 3D bird’s-eye-view representation in a cascaded manner, and vice versa. Through the learning of 3D-regulated instance representation, this bi-directional mutual fusion module, called cascaded multi-view feature fusion module, alleviates the spatial misalignment of the two views. Furthermore, to learn the rotation- and shape-invariant features of objects, modulated rotation head (MRH) develops a direction-guided adjustment instead of an axis-aligned structure to extract instance-consistent features. By excluding the irrelevant content using MRH, this instance-consistent feature will benefit the object classification and orientation regression. Extensive experiments on the KITTI dataset show that the proposed method achieves a significant improvement over existing advanced methods, especially for orientation estimation.

Zero-Shot Learning on 3D Point Cloud Objects and Beyond

Zero-shot learning, the task of learning to recognize new classes not seen during training, has received considerable attention in the case of 2D image classification. However, despite the increasing ubiquity of 3D sensors, the corresponding 3D point cloud classification problem has not been meaningfully explored and introduces new challenges. In this paper, we identify some of the challenges and apply 2D Zero-Shot Learning (ZSL) methods in the 3D domain to analyze the performance of existing models. Then, we propose a novel approach to address the issues specific to 3D ZSL. We first present an inductive ZSL process and then extend it to the transductive ZSL and Generalized ZSL (GZSL) settings for 3D point cloud classification. To this end, a novel loss function is developed that simultaneously aligns seen semantics with point cloud features and takes advantage of unlabeled test data to address some known issues (e.g., the problems of domain adaptation, hubness, and data bias). While designed for the particularities of 3D point cloud classification, the method is shown to also be applicable to the more common use-case of 2D image classification. An extensive set of experiments is carried out, establishing state-of-the-art for ZSL and GZSL on synthetic (ModelNet40, ModelNet10, McGill) and real (ScanObjectNN) 3D point cloud datasets.

Multiple object tracking in farmland based on fusion point cloud data

Object detection and tracking in a farmland environment is greatly important to unmanned agricultural machinery navigation technology. Based on the existing methods of fusion point cloud acquisition and object detection, the multiple object tracking method was further researched. Constant velocity model was selected as the motion model, and state estimation was achieved using Kalman filter. Data association between the detectors and trackers was realized through an improved GNN method based on the cost of category conversion. A tracking state machine and two threshold mechanisms were proposed to avoid “combined explosion” disaster. Fusion point cloud data were used for motion judgment to achieve the real motion attributes and speed of the objects based on ground reference. As the results, the average static drift of the tracking method was less than 6.5 cm, the discrimination accuracy of static objects exceeded 75%. Besides, the multiple object tracking precision was less than 35 cm, and the multiple object tracking accuracy exceeded 85%. Therefore, the proposed multiple object tracking method could satisfy the requirements in the farmland environment.

다중 시점 RGB 영상의 심층 합성곱 신경망 기반 3 차원 형상 추정 방법

In this paper, we describe point cloud generation method from multiple RGB images based on convolutional neural network. The proposed method is motivated from human’s 3d understanding from orthographic sketches, which infer spatial relationship from front, top, side view of the 3d object. The network of the proposed method employed generative model, to make it predict point clouds of atypical objects. Auto encoder network is utilized to encode three RGB images into latent vectors, and generate point clouds. Loss functions are defined which measure reconstruction performance and uniformity of the point clouds to make the network generate point clouds similar to the original and distribute uniformly along entire region. From this structure, we expected the network to learn spatial relation of the original 3d model from multiple RGB images. The result shows that the proposed method can predict overall shape of the objects, and it is hard to express detailed geometry. As a further work, network structure improvement to generate detailed shape of the object, predict occluded region will be performed.

LiSurveying: A high-resolution TLS-LiDAR benchmark

Outdoor point-cloud object localization is an essential processing step for urban scene analysis and modeling in numerous applications, especially in land surveying and site analysis. Given the increasingly use of Terrestrial Laser Scanning (TLS) and LiDAR 3D data acquisition, numerous annotated point-cloud datasets are available and can be used to evaluate computer vision and machine learning based algorithms. Nevertheless, current point-cloud datasets mainly focus on object detection and classification in autonomous driving or urban planning type of applications, and share redundant object classes, e.g., trees, vehicles and pedestrians, which limit their usefulness for land surveying and site analysis. This paper introduces a novel 3D benchmark dataset LiSurveying, which is a large-scale point-cloud dataset with over a billion points and uncommon urban object categories in complex outdoor environments. Our dataset incorporates more urban object classes than existing datasets. Its instances have diverse point densities, shapes and dimensions, which also impose a challenge for point-cloud detection and classification algorithms. We conducted baselines experiments for point-cloud classification using machine learning classifiers and deep learning methods on different subsets of the LiSurveying dataset, and we are able to demonstrate that the various types of object classes, number of instances per class, distribution of object points, and variety of complex scenes, make this LiSurveying benchmark dataset suitable for evaluating 3D point-cloud classification, semantic segmentation, and object detection algorithms.

Collision Detection and Avoidance for Underwater Vehicles Using Omnidirectional Vision.

Exploration of marine habitats is one of the key pillars of underwater science, which often involves collecting images at close range. As acquiring imagery close to the seabed involves multiple hazards, the safety of underwater vehicles, such as remotely operated vehicles (ROVs) and autonomous underwater vehicles (AUVs), is often compromised. Common applications for obstacle avoidance in underwater environments are often conducted with acoustic sensors, which cannot be used reliably at very short distances, thus requiring a high level of attention from the operator to avoid damaging the robot. Therefore, developing capabilities such as advanced assisted mapping, spatial awareness and safety, and user immersion in confined environments is an important research area for human-operated underwater robotics. In this paper, we present a novel approach that provides an ROV with capabilities for navigation in complex environments. By leveraging the ability of omnidirectional multi-camera systems to provide a comprehensive view of the environment, we create a 360° real-time point cloud of nearby objects or structures within a visual SLAM framework. We also develop a strategy to assess the risk of obstacles in the vicinity. We show that the system can use the risk information to generate warnings that the robot can use to perform evasive maneuvers when approaching dangerous obstacles in real-world scenarios. This system is a first step towards a comprehensive pilot assistance system that will enable inexperienced pilots to operate vehicles in complex and cluttered environments.

2D&3DHNet for 3D Object Classification in LiDAR Point Cloud

Accurate semantic analysis of LiDAR point clouds enables the interaction between intelligent vehicles and the real environment. This paper proposes a hybrid 2D and 3D Hough Net by combining 3D global Hough features and 2D local Hough features with a classification deep learning network. Firstly, the 3D object point clouds are mapped into the 3D Hough space to extract the global Hough features. The generated global Hough features are input into the 3D convolutional neural network for training global features. Furthermore, a multi-scale critical point sampling method is designed to extract critical points in the 2D views projected from the point clouds to reduce the computation of redundant points. To extract local features, a grid-based dynamic nearest neighbors algorithm is designed by searching the neighbors of the critical points. Finally, the two networks are connected to the full connection layer, which is input into fully connected layers for object classification.

OctAttention: Octree-Based Large-Scale Contexts Model for Point Cloud Compression

In point cloud compression, sufficient contexts are significant for modeling the point cloud distribution. However, the contexts gathered by the previous voxel-based methods decrease when handling sparse point clouds. To address this problem, we propose a multiple-contexts deep learning framework called OctAttention employing the octree structure, a memory-efficient representation for point clouds. Our approach encodes octree symbol sequences in a lossless way by gathering the information of sibling and ancestor nodes. Expressly, we first represent point clouds with octree to reduce spatial redundancy, which is robust for point clouds with different resolutions. We then design a conditional entropy model with a large receptive field that models the sibling and ancestor contexts to exploit the strong dependency among the neighboring nodes and employ an attention mechanism to emphasize the correlated nodes in the context. Furthermore, we introduce a mask operation during training and testing to make a trade-off between encoding time and performance. Compared to the previous state-of-the-art works, our approach obtains a 10%-35% BD-Rate gain on the LiDAR benchmark (e.g. SemanticKITTI) and object point cloud dataset (e.g. MPEG 8i, MVUB), and saves 95% coding time compared to the voxel-based baseline. The code is available at https://github.com/zb12138/OctAttention.

Adaptation of Grad-CAM Method to Neural Network Architecture for LiDAR Pointcloud Object Detection

Explainable Artificial Intelligence (XAI) methods demonstrate internal representation of data hidden within neural network trained weights. That information, presented in a form readable to humans, could be remarkably useful during model development and validation. Among others, gradient-based methods such as Grad-CAM are broadly used in an image processing domain. On the other hand, the autonomous vehicle sensor suite consists of auxiliary devices such as radars and LiDARs, for which existing XAI methods do not apply directly. In this article, we present our adaptation approach to utilize Grad-CAM visualization for LiDAR pointcloud specific object detection architectures used in automotive perception systems. We try to solve data and network architecture compatibility problems and answer the question whether Grad-CAM methods could be used with LiDAR sensor data efficiently. We showcase successful results of our method and all the benefits that come with a Grad-CAM XAI application to a LiDAR sensor in an autonomous driving domain.

A TWO-STAGE APPROACH FOR RARE CLASS SEGMENTATION IN LARGE-SCALE URBAN POINT CLOUDS

Abstract. Although deep learning has greatly improved the semantic segmentation accuracy of point clouds, the segmentation of rare classes in large-scale urban scenes has not been targeted in available methods. This paper proposes a two-stage segmentation framework with automated workflows for imbalanced rare classes based on general semantic segmentation. The proposed approach includes two stages: general semantic segmentation and object-based refined semantic segmentation. Firstly, general segmentation networks are utilized to segment general large objects. Secondly, refined semantic segmentation is conducted by an automated workflow: 3D clustering and bounding box (BBox) generation are applied to the point cloud of rare fine-grained objects during the training, followed by object detection to extract fine-grained objects. Afterwards, as the constraints, the extracted BBoxes further refine the segmentation results. Our approach is evaluated on the Hessigheim High-Resolution 3D Point Cloud (H3D) Benchmark and obtains state-of-the-art 89.35% overall accuracy and outstanding 75.70% mean F1-Score. Furthermore, rare classes Vehicle and Chimney achieve breakthroughs from zero to 63.63% and 52.00% in F1-score, respectively.

HCCG: Efficient high compatibility correspondence grouping for 3D object recognition and 6D pose estimation in cluttered scenes

Recognizing and localizing queried objects in point clouds is a critical technique for robotic manipulation and bin picking tasks. Even though it has been steadily studied, it is still a challenging task for scenes with heavy occlusion and clutter. To copy with this intractable problem, this paper presents a fast and robust 3D object recognition framework, especially an efficient high compatibility correspondence grouping (HCCG) technique achieved by the correspondence ranking, clustering and expanding operations. Utilizing the HCCG technique, we first cluster the compatible correspondences into several high compatibility correspondence groups. Then, for each group, a 6DoF pose hypothesis is generated by using a point pair feature constraints (PPFCs)-based outlier removal module and a local reference frame (LRF)-based pose estimation algorithm. Finally, a robust pose verification operator is carried out to reject the false positive pose hypotheses and pick up the correct target object pose. Our approach is not only suitable for the multi-object recognition task, but also inherently yields the capability of detecting multiple instances thanks to the efficient HCCG technique. Extensive experiments on four challenging datasets show that our approach yields efficient and timely solutions and its advantages are further verified by comparing with the latest methods.

A Lightweight Two-End Feature Fusion Network for Object 6D Pose Estimation

Currently, many methods of object pose estimation use images or point clouds alone for pose estimation. This leads to their inability to accurately estimate the object pose in the case of occlusion and poor illumination. Second, these models have large parameters and cannot be deployed on mobile devices. Therefore, we propose a lightweight two-terminal feature fusion network, which can effectively use images and point clouds for accurate object pose estimation. First, Pointno problemNet network is used to extract point cloud features. Then the extracted point cloud features are combined with the images at pixel level and the features are extracted by CNN. Secondly, the extracted image features are combined with the point cloud point by point. Then feature extraction is performed on it using the improved PointNet++ network. Finally, a set of center point features are obtained and pose estimation is performed for each feature. The pose with the highest confidence is selected as the final result. Furthermore, we apply depthwise separable convolutions to reduce the amount of model parameters. Experiments show that the proposed method exhibits better performance on Linemod and Occlusion Linemod datasets. Furthermore, the model parameters are small, and it is robust in occlusion and low-light situations.

Pose Estimation of a Small Connector Attached to the Tip of a Cable Sticking Out of a Circuit Board

In this study, we present the pose estimation of a small connector attached to the tip of a cable sticking out from a circuit board. Since such connectors are generally small and float in various configurations on their workpieces, it is difficult to achieve automation of grasping and inserting the connector into the corresponding socket. We focus on the task of grasping a connector and propose methods for detecting the location of the connector and estimating its 6DoF pose. In this regard, we use a high-precision three-dimensional digitizer to capture the object point cloud and combine several methods, such as deep learning and registration, to perform data processing. We conducted grasp experiments on several connectors using an actual industrial robot and confirmed the effectiveness of our approach in terms of pose estimation.

Research on 3D Point Cloud Object Detection Algorithm for Autonomous Driving

In autonomous driving, lidar has become the main vehicle sensor due to its advantages such as long-range measurement and high accuracy. However, the collected point cloud data is sparse and unevenly distributed, and it lacks characterization capabilities when facing objects with missing or similar shapes, so that the detection accuracy is low while detecting long-distance small targets with similar shapes and a small number of point clouds. In order to improve the detection accuracy of small targets represented by point clouds, this paper adopts a method that fuses point cloud and RGB image to construct a 3D object detection network architecture based on two-stage complementary fusion. In the first stage of the fusion, we use the FPS method to select some points from the raw point cloud data as key points. Then, we voxelize the raw point cloud and use the 3D sparse convolutional neural network to extract multi-scale points cloud features, which would fuse features of different scales with key points. In the second stage of fusion, a 2D object detector is used to obtain the 2D bounding box and category information of the target in the image and take the camera as the origin to extend along the direction of the 2D bounding box to form a frustum; then the point cloud and target category information within the frustum are fused into the key points. This paper uses key points as a bridge to effectively combine the semantic information of the image such as texture and color with the point cloud features. The experimental results show that the network proposed in this paper has excellent performance on small objects.

Semantic-aware 3D-voxel CenterNet for point cloud object detection

It is still challenging to accurately detect the small point cloud objects in the density-varying scenes, such as autonomous driving scenes where the measurements in nearby regions are much more than they in farther-away regions. Most previous methods detect objects with uniformly sampled key points, which may miss small objects. We propose a semantic-aware 3D-voxel CenterNet for 3D object detection (SA-Voxel-CenterNet) to address the above challenge. Specifically, SA-Voxel-CenterNet contains a cluster-enhanced center regression module (CCRM) and a semantic-aware proposal refinement module (SPRM). CCRM designs a voxel-wise semantic class segmentation branch and a center offset learning branch to extract centers of objects based on an improved Connected Component Analysis algorithm. Besides, SPRM generates the interesting region around each candidate with a predefined size. The experiments on the KITTI benchmark show that SA-Voxel-CenterNet delivers high detection accuracy, especially for the small object, e.g., cyclist, achieves state-of-the-art performance.