Objects In Point Clouds Research Articles

FF3D: A Rapid and Accurate 3D Fruit Detector for Robotic Harvesting.

This study presents the Fast Fruit 3D Detector (FF3D), a novel framework that contains a 3D neural network for fruit detection and an anisotropic Gaussian-based next-best view estimator. The proposed one-stage 3D detector, which utilizes an end-to-end 3D detection network, shows superior accuracy and robustness compared to traditional 2D methods. The core of the FF3D is a 3D object detection network based on a 3D convolutional neural network (3D CNN) followed by an anisotropic Gaussian-based next-best view estimation module. The innovative architecture combines point cloud feature extraction and object detection tasks, achieving accurate real-time fruit localization. The model is trained on a large-scale 3D fruit dataset and contains data collected from an apple orchard. Additionally, the proposed next-best view estimator improves accuracy and lowers the collision risk for grasping. Thorough assessments on the test set and in a simulated environment validate the efficacy of our FF3D. The experimental results show an AP of 76.3%, an AR of 92.3%, and an average Euclidean distance error of less than 6.2 mm, highlighting the framework's potential to overcome challenges in orchard environments.

Efficient flexible voxel-based two-stage network for 3D object detection in autonomous driving

3D object detection from the LiDAR point cloud plays an important role in autonomous driving. It is difficult to balance inference speed and detection accuracy when performing 3D point cloud object detection due to the large size of point cloud data and its unstructured storage, which makes it difficult to represent its features. To address the challenge, we propose a two-stage point cloud object detector, AFV-RCNN. In stage-1, the attention flexible voxel feature encoding layer is introduced, which utilizes flexible voxels to enhance feature encoding speed and focuses on foreground points to be detected through voxel attention. In stage-2, the multi-level and grid-based multi-scale RoI (Region of Interest) feature fusion module is designed. It directly extracts complete 3D structures from 3D region proposals and focuses on both local and global features through multi-scale partitioning. In the training stage, GHM-C Loss is applied to address the challenges associated with imbalanced target categories and the imbalance between difficult and easy samples in the classification task. We evaluate the model on the public KITTI Dataset and Waymo Open Dataset. The mAP in KITTI for 3D detection is 73.41% and inference on a single GPU reaches 30.0 fps. Compared with other state-of-the-art methods, AFV-RCNN achieves both the inference speed of a single-stage detector and the detection accuracy of a two-stage detector. It ensures higher detection accuracy while efficiently processing the point cloud.

Multi-Scale Feature Fusion Point Cloud Object Detection Based on Original Point Cloud and Projection

Multi-Scale Feature Fusion Point Cloud Object Detection Based on Original Point Cloud and Projection

Real-virtual Point Cloud Augmentation Method for Test and Evaluation of Autonomous Weapon Systems

Autonomous weapon systems act according to artificial intelligence-based judgement based on recognition through various sensors. Test and evaluation for various scenarios is required depending on the characteristics that artificial intelligence-based judgement is made. As a part of this approach, this paper proposed a LiDAR point cloud augmentation method for mixed-reality based test and evaluation. The augmentation process is achieved by mixing real and virtual LiDAR signals based on the virtual LiDAR synchronized with the pose of the autonomous weapon system. For realistic augmentation of test and evaluation purposes, appropriate intensity values were inserted when generating a point cloud of a virtual object and its validity was verified. In addition, when mixing the generated point cloud of the virtual object with the real point cloud, the proposed method enhances realism by considering the occlusion phenomenon caused by the insertion of the virtual object.

A First Approach to Quantum Logical Shape Classification Framework

Quantum logic is a well-structured theory, which has recently received some attention because of its fundamental relation to quantum computing. However, the complex foundation of quantum logic borrowing concepts from different branches of mathematics as well as its peculiar settings have made it a non-trivial task to devise suitable applications. This article aims to propose for the first time an approach using quantum logic in image processing for shape classification. We show how to make use of the principal component analysis to realize quantum logical propositions. In this way, we are able to assign a concrete meaning to the rather abstract quantum logical concepts, and we are able to compute a probability measure from the principal components. For shape classification, we consider encrypting given point clouds of different objects by making use of specific distance histograms. This enables us to initiate the principal component analysis. Through experiments, we explore the possibility of distinguishing between different geometrical objects and discuss the results in terms of quantum logical interpretation.

Leveraging Deep Learning for Automated Reconstruction of Indoor Unstructured Elements in Scan-to-BIM

Abstract. Achieving automatic 3D reconstruction for indoor scenes is extremely useful in the field of scene understanding. Building information modeling (BIM) models are essential for lowering project costs, assisting in building planning and renovations, as well as improving building management efficiency. However, nearly all current available scan-to-BIM approaches employ manual or semi-automatic methods. These approaches concentrate solely on significant structured objects, neglecting other unstructured elements such as furniture. The limitation arises from challenges in modeling incomplete point clouds of obstructed objects and capturing indoor scene details. Therefore, this research introduces an innovative and effective reconstruction framework based on deep learning semantic segmentation and model-driven techniques to address these limitations. The proposed framework utilizes wall segment recognition, feature extraction, opening detection, and automatic modeling to reconstruct 3D structured models of point clouds with different room layouts in both Manhattan and non-Manhattan architectures. Moreover, it provides 3D BIM models of actual unstructured elements by detecting objects, completing point clouds, establishing bounding boxes, determining type and orientation, and automatically generating 3D BIM models with a parametric algorithm implemented into the Revit software. We evaluated this framework using publicly available and locally generated point cloud datasets with varying furniture combinations and layout complexity. The results demonstrate the proposed framework's efficiency in reconstructing structured indoor elements, exhibiting completeness and geometric accuracy, and achieving precision and recall values greater than 98%. Furthermore, the generated unstructured 3D BIM models keep essential real-scene characteristics such as geometry, spatial locations, numerical aspects, various shapes, and orientations compared to literature methods.

Accurate Segmentation Method for Roadside Lampposts Based on Vehicle-Mounted Lidar Point Clouds

Abstract. The extraction of roadside lampposts constitutes a significant research focus within the domain of vehicular LiDAR point cloud object retrieval. Addressing the complexities inherent in discerning lampposts amidst convoluted vegetation in diverse roadway settings, this study introduces an innovative, progressive technique for the individualized extraction of lampposts utilizing vehicular LiDAR point clouds. The proposed method initiates with a bifurcation of the primary point cloud into terrestrial and aerial subsets via the Cloth Simulation Filter (CSF) algorithm. Subsequent processes involve the extraction of distinct lamppost structures from aerial point clouds through a methodology integrating elevation-normalized spatial partitioning and directional coverage analysis, thereby facilitating precise lamppost localization. The culmination of this process involves the refinement of lamppost point clouds through supervoxel clustering complemented by a non-discretization filter grounded in Principal Component Analysis (PCA). The efficacy of this novel approach is substantiated through empirical studies employing a LiDAR dataset encompassing extensive adhesive scenarios, whereupon comparative analysis with extant methodologies reveals its enhanced proficiency in isolating individual lampposts in complex environments.

Corr-Track: Category-Level 6D Pose Tracking with Soft-Correspondence Matrix Estimation.

Category-level pose tracking methods can continuously track the pose of objects without requiring any prior knowledge of the specific shape of the tracked instance. This makes them advantageous in augmented reality and virtual reality applications. The key challenge is how to train neural networks to accurately predict the poses of objects they have never seen before and exhibit strong generalization performance. We propose a novel category-level 6D pose tracking method Corr-Track, which is capable of accurately tracking objects belonging to the same category from depth video streams. Our approach utilizes direct soft correspondence constraints to train a neural network, which estimates bidirectional soft correspondences between sparsely sampled point clouds of objects in two frames. We first introduce a soft correspondence matrix for pose tracking tasks and establish effective constraints through direct spatial point-to-point correspondence representations in the sparse point cloud correspondence matrix. We propose the "point cloud expansion" strategy to address the "point cloud shrinkage" problem resulting from soft correspondences. This strategy ensures that the corresponding point cloud accurately reproduces the shape of the target point cloud, leading to precise pose tracking results. We evaluated our approach on the NOCS-REAL275 and Wild6D dataset and observed superior performance compared to previous methods. Additionally, we conducted cross-category experiments that further demonstrated its generalization capability.

SSF: Sparse point cloud object detection based on self-adaptive voxel encoding and focal-sparse convolution

Point cloud object detection is gradually playing a key role in autonomous driving tasks. To address the issue of insensitivity to sparse objects in point cloud object detection, we have made improvements to the voxel encoding and 3D backbone network of the PVRCNN++. We have introduced adaptive pooling operations during voxel feature encoding to expand the point cloud information within each voxel, followed by the utilization of multi-layer perceptrons to extract richer point cloud features. On the 3D backbone network, we have employed adaptive sparse convolution operations to make the backbone network’s channel count more flexible, allowing it to accommodate a wider range of input data types. Furthermore, we have integrated Focal Loss to tackle the issue of class imbalance in detection tasks. Experimental results on the public KITTI dataset demonstrate significant improvements over the PVRCNN++, particularly in pedestrian and bicycle detection tasks. Specifically, we have observed 1% increase in detection accuracy for pedestrians and 2.1% improvement for bicycles. Our detection performance also surpasses that of other comparative detection algorithms.

PCKRF: Point Cloud Completion and Keypoint Refinement With Fusion Data for 6D Pose Estimation.

Some robust point cloud registration approaches with controllable pose refinement magnitude, such as ICP and its variants, are commonly used to improve 6D pose estimation accuracy. However, the effectiveness of these methods gradually diminishes with the advancement of deep learning techniques and the enhancement of initial pose accuracy, primarily due to their lack of specific design for pose refinement. In this paper, we propose Point Cloud Completion and Keypoint Refinement with Fusion Data (PCKRF), a new pose refinement pipeline for 6D pose estimation. The pipeline consists of two steps. First, it completes the input point clouds via a novel pose-sensitive point completion network. The network uses both local and global features with pose information during point completion. Then, it registers the completed object point cloud with the corresponding target point cloud by our proposed Color supported Iterative KeyPoint (CIKP) method. The CIKP method introduces color information into registration and registers a point cloud around each keypoint to increase stability. The PCKRF pipeline can be integrated with existing popular 6D pose estimation methods, such as the full flow bidirectional fusion network, to further improve their pose estimation accuracy. Experiments demonstrate that our method exhibits superior stability compared to existing approaches when optimizing initial poses with relatively high precision. Notably, the results indicate that our method effectively complements most existing pose estimation techniques, leading to improved performance in most cases. Furthermore, our method achieves promising results even in challenging scenarios involving textureless and symmetrical objects. Our source code is available at https://github.com/zhanhz/KRF.

Dual-Graph Attention Convolution Network for 3-D Point Cloud Classification.

Three-dimensional point cloud classification is fundamental but still challenging in 3-D vision. Existing graph-based deep learning methods fail to learn both low-level extrinsic and high-level intrinsic features together. These two levels of features are critical to improving classification accuracy. To this end, we propose a dual-graph attention convolution network (DGACN). The idea of DGACN is to use two types of graph attention convolution operations with a feedback graph feature fusion mechanism. Specifically, we exploit graph geometric attention convolution to capture low-level extrinsic features in 3-D space. Furthermore, we apply graph embedding attention convolution to learn multiscale low-level extrinsic and high-level intrinsic fused graph features together. Moreover, the points belonging to different parts in real-world 3-D point cloud objects are distinguished, which results in more robust performance for 3-D point cloud classification tasks than other competitive methods, in practice. Our extensive experimental results show that the proposed network achieves state-of-the-art performance on both the synthetic ModelNet40 and real-world ScanObjectNN datasets.

Multisensor Fusion for Vehicle-to-Vehicle Cooperative Localization With Object Detection and Point Cloud Matching

Multisensor Fusion for Vehicle-to-Vehicle Cooperative Localization With Object Detection and Point Cloud Matching

ODN-Pro: An Improved Model Based on YOLOv8 for Enhanced Instance Detection in Orchard Point Clouds

In an unmanned orchard, various tasks such as seeding, irrigation, health monitoring, and harvesting of crops are carried out by unmanned vehicles. These vehicles need to be able to distinguish which objects are fruit trees and which are not, rather than relying on human guidance. To address this need, this study proposes an efficient and robust method for fruit tree detection in orchard point cloud maps. Feature extraction is performed on the 3D point cloud to form a two-dimensional feature vector containing three-dimensional information of the point cloud and the tree target is detected through the customized deep learning network. The impact of various feature extraction methods such as average height, density, PCA, VFH, and CVFH on the detection accuracy of the network is compared in this study. The most effective feature extraction method for the detection of tree point cloud objects is determined. The ECA attention module and the EVC feature pyramid structure are introduced into the YOLOv8 network. The experimental results show that the deep learning network improves the precision, recall, and mean average precision by 1.5%, 0.9%, and 1.2%, respectively. The proposed framework is deployed in unmanned orchards for field testing. The experimental results demonstrate that the framework can accurately identify tree targets in orchard point cloud maps, meeting the requirements for constructing semantic orchard maps.

BEV-MAE: Bird’s Eye View Masked Autoencoders for Point Cloud Pre-training in Autonomous Driving Scenarios

Existing LiDAR-based 3D object detection methods for autonomous driving scenarios mainly adopt the training-from-scratch paradigm. Unfortunately, this paradigm heavily relies on large-scale labeled data, whose collection can be expensive and time-consuming. Self-supervised pre-training is an effective and desirable way to alleviate this dependence on extensive annotated data. In this work, we present BEV-MAE, an efficient masked autoencoder pre-training framework for LiDAR-based 3D object detection in autonomous driving. Specifically, we propose a bird's eye view (BEV) guided masking strategy to guide the 3D encoder learning feature representation in a BEV perspective and avoid complex decoder design during pre-training. Furthermore, we introduce a learnable point token to maintain a consistent receptive field size of the 3D encoder with fine-tuning for masked point cloud inputs. Based on the property of outdoor point clouds in autonomous driving scenarios, i.e., the point clouds of distant objects are more sparse, we propose point density prediction to enable the 3D encoder to learn location information, which is essential for object detection. Experimental results show that BEV-MAE surpasses prior state-of-the-art self-supervised methods and achieves a favorably pre-training efficiency. Furthermore, based on TransFusion-L, BEV-MAE achieves new state-of-the-art LiDAR-based 3D object detection results, with 73.6 NDS and 69.6 mAP on the nuScenes benchmark. The source code will be released at https://github.com/VDIGPKU/BEV-MAE.

Modeling Continuous Motion for 3D Point Cloud Object Tracking

The task of 3D single object tracking (SOT) with LiDAR point clouds is crucial for various applications, such as autonomous driving and robotics. However, existing approaches have primarily relied on appearance matching or motion modeling within only two successive frames, thereby overlooking the long-range continuous motion property of objects in 3D space. To address this issue, this paper presents a novel approach that views each tracklet as a continuous stream: at each timestamp, only the current frame is fed into the network to interact with multi-frame historical features stored in a memory bank, enabling efficient exploitation of sequential information. To achieve effective cross-frame message passing, a hybrid attention mechanism is designed to account for both long-range relation modeling and local geometric feature extraction. Furthermore, to enhance the utilization of multi-frame features for robust tracking, a contrastive sequence enhancement strategy is proposed, which uses ground truth tracklets to augment training sequences and promote discrimination against false positives in a contrastive manner. Extensive experiments demonstrate that the proposed method outperforms the state-of-the-art method by significant margins on multiple benchmarks.

Three-Dimensional Point Cloud Object Detection Based on Feature Fusion and Enhancement

With the continuous emergence and development of 3D sensors in recent years, it has become increasingly convenient to collect point cloud data for 3D object detection tasks, such as the field of autonomous driving. But when using these existing methods, there are two problems that cannot be ignored: (1) The bird’s eye view (BEV) is a widely used method in 3D objective detection; however, the BEV usually compresses dimensions by combined height, dimension, and channels, which makes the process of feature extraction in feature fusion more difficult. (2) Light detection and ranging (LiDAR) has a much larger effective scanning depth, which causes the sector to become sparse in deep space and the uneven distribution of point cloud data. This results in few features in the distribution of neighboring points around the key points of interest. The following is the solution proposed in this paper: (1) This paper proposes multi-scale feature fusion composed of feature maps at different levels made of Deep Layer Aggregation (DLA) and a feature fusion module for the BEV. (2) A point completion network is used to improve the prediction results by completing the feature points inside the candidate boxes in the second stage, thereby strengthening their position features. Supervised contrastive learning is applied to enhance the segmentation results, improving the discrimination capability between the foreground and background. Experiments show these new additions can achieve improvements of 2.7%, 2.4%, and 2.5%, respectively, on KITTI easy, moderate, and hard tasks. Further ablation experiments show that each addition has promising improvement over the baseline.

Mapping 3-D classroom seats based on partial object point cloud completion

ABSTRACT 3-D classroom seating information is of great significance in a host of planning contexts including indoor navigation and facility management. Mapping classroom 3-D seats based on a single-view LiDAR scan is challenging because of occlusions and viewpoint limitations. In essence, one is left with incomplete point clouds, making object recognition difficult. The aim of this paper is to acquire complete point clouds for objects in a room based on a single-view LiDAR scan, focusing in particular on chairs in indoor spaces such as classrooms, lecture halls, theaters, concert venues, and sports arenas. A spatial optimization enabled framework is proposed for 3-D classroom seat mapping from a single-view LiDAR scan. A two-step clustering method is proposed to segment each individual chair object in the LiDAR scan. A chair model is extracted and used to complete other chair point clouds via registration. Notably, a parameter estimation method is devised based on spatial priors and integrated in an iterative closest point optimization approach to refine registration. Applications to multiple classroom scans demonstrate the effectiveness and accuracy of the proposed methodology. The developed segmentation-registration framework harnesses spatial knowledge to complete a single-view LiDAR scan with many identical objects. The resultant object point cloud is critical for navigation, route guidance, facility management, pandemic response and planning, indoor mapping and venue utilization, among others, which require precise and complete spatial information.

3D Point Cloud Object Detection Method Based on Multi-Scale Dynamic Sparse Voxelization.

Perception plays a crucial role in ensuring the safety and reliability of autonomous driving systems. However, the recognition and localization of small objects in complex scenarios still pose challenges. In this paper, we propose a point cloud object detection method based on dynamic sparse voxelization to enhance the detection performance of small objects. This method employs a specialized point cloud encoding network to learn and generate pseudo-images from point cloud features. The feature extraction part uses sliding windows and transformer-based methods. Furthermore, multi-scale feature fusion is performed to enhance the granularity of small object information. In this experiment, the term "small object" refers to objects such as cyclists and pedestrians, which have fewer pixels compared to vehicles with more pixels, as well as objects of poorer quality in terms of detection. The experimental results demonstrate that, compared to the PointPillars algorithm and other related algorithms on the KITTI public dataset, the proposed algorithm exhibits improved detection accuracy for cyclist and pedestrian target objects. In particular, there is notable improvement in the detection accuracy of objects in the moderate and hard quality categories, with an overall average increase in accuracy of about 5%.

A Point Cloud Contour Extraction Method based on Plane Segmentation

A method based on plane segmentation and dimensionality reduction for extracting incomplete and slow contour features of object point clouds is proposed. The method consists of two main steps: plane segmentation and contour extraction. In plane segmentation, the random sample consensus (Random Sample Consensus, RANSAC) algorithm is optimized based on principal component analysis (Principal Component Analysis, PCA); the optimized planar point cloud is then subjected to dimensionality reduction, and the contour features are extracted using gradients. Experimental results show that the method can effectively segment point clouds and extract the contours of target surfaces, and has great potential for application in industrial inspection and other fields.

Enhancing pseudo label quality for pedestrian and cyclist in weakly supervised 3D object detection

Weakly supervised 3D object detection for autonomous driving primarily focuses on cars because of their distinct rectangle boundaries and abundant instances. However, detecting categories with ambiguous rectangle boundaries and fewer instances than cars, such as pedestrians and cyclists, remains challenging with limited research. Ambiguity in rectangle boundaries presents significant difficulties in generating accurate 3D pseudo labels, while the scarcity of instances often leads to convergence issues during detector training. Pedestrians and cyclists are dense inside the 3D bounding boxes but sparse at corners and boundaries. Density is a practical clue for locating and discriminating pedestrians and cyclists in point clouds. This paper proposes a density-based 3D pseudo-label generation module (DPL-3D), addressing the challenges of ambiguous rectangle boundaries. Ambiguity rectangle boundaries will lead to poor pseudo-label quality. Therefore, By leveraging the density information of 3D points, our DPL-3D improves the accuracy and localization quality of the generated pseudo labels. It effectively segments background points, improving the estimation of pseudo labels’ location, dimension, and orientation. Few training samples always lead to local optima. Introducing multi-modal data in the detector network could enhance the constraints of objects’ features, but 2D images and 3D point clouds have a resolution gap. A motivation for dealing with the resolution gap is that neighboring regions with similar colors and textures in 2D images may exhibit spatial proximity in 3D space. Therefore, a multi-modal network driven by superpixel segmentation is introduced. This network enables effective discrimination between objects in 2D images and 3D point clouds, bridging the resolution gap and leveraging complementary features from both modalities. Experimental results on the KITTI dataset demonstrate the effectiveness of the proposed methods in addressing the challenges associated with weakly-supervised 3D object detection, particularly for categories with ambiguous rectangle boundaries and few instances.