Local Point Cloud Research Articles

4D RADAR/IMU/GNSS INTEGRATED POSITIONING AND MAPPING FOR LARGE-SCALE ENVIRONMENTS

Abstract. 4D millimeter-wave radar can work in harsh weather conditions such as fog and snow, and measure the position and radial Doppler velocity of objects in three-dimensional space. Some existing methods perform positioning by fusing 3D velocity or point cloud matching information estimated by 4D radar with IMU information. However, the sparsity of radar point clouds and the interference of large amount of noise lead to low accuracy of odometry, and most of the existing work cannot achieve the global consistent mapping with radar point clouds. In this paper, a 4D radar/IMU/GNSS localization and mapping method, G-iRIOM, is proposed. We tightly coupled IMU measurement, Doppler observation information and point cloud matching information from 4D radar by iterative extended Kalman filtering (iEKF) method, and introduced GNSS RTK and loop closure information as global observation to correct the positioning drift of the odometry. The experimental results show that the tightly coupled Doppler velocity information can effectively improve the control of the pose on the local point cloud position, thus enhancing the mapping accuracy. Meanwhile, the introduction of GNSS and loop closure information can significantly improve the positioning accuracy of 4D radar odometry as a kind of global observation, and achieve global consistent mapping of large-scale outdoor scenes based on 4D radar point clouds.

3D point cloud global denoising algorithm based on different noise characteristics

When acquiring object point cloud data by three-dimensional scanning technology, noise is generated due to instrument accuracy and external factors. Existing algorithms rarely consider the characteristics of different noises and different regional noises when solving the point cloud denoising problem, this results in a limited effect on denoising. This paper presents an algorithm for denoising based on the characteristics of different types of noise and different regions in the point cloud. The algorithm includes large-scale noise removal and small-scale noise smoothing. Remove large-scale noise points by the relationship between the local point cloud and the global point cloud. For small-scale noise, the feature regions and non-feature regions are extracted according to the normal cosine information entropy. According to the characteristics of the small-scale noise in two regions, the noise distance distribution and the optimized bilateral filtering are used to deal with the small-scale noise in two regions respectively. Comparison experiments show that our algorithm can effectively remove the noise points mixed in the normal point cloud. The accuracy of large-scale noise removal reaches 99.1%. The proposed algorithm can protect the feature areas from being over-smoothed while smoothing the small-scale noise in non-featured areas.

Bone marrow sparing oriented multi-model image registration in cervical cancer radiotherapy

Cervical cancer poses a serious threat to the health of women and radiotherapy is one of the primary treatment methods for this condition. However, this treatment is associated with a high risk of causing acute hematologic toxicity. Delineating the bone marrow (BM) for sparing based on computer tomography (CT) images before radiotherapy can effectively avoid this risk. Unfortunately, compared to magnetic resonance (MR) images, CT images lack the ability to express the activity of BM. Therefore, medical practitioners currently manually delineate the BM on CT images by corresponding to MR images. However, the manual delineation of BM is time-consuming and cannot guarantee accuracy due to the inconsistency of the CT-MR multimodal images. This study proposes a multimodal image-oriented automatic registration method for pelvic BM sparing. The proposed method includes three-dimensional (3D) bone point clouds reconstruction and an iterative closest point registration based on a local spherical system for marking BM on CT images. By introducing a joint coordinate system that combines the global Cartesian coordinate system with the local point clouds’ spherical coordinate system, the increasement of point descriptive dimension avoids the local optimal registration and improves the registration accuracy. Experiments on the dataset of patients demonstrate that our proposed method can enhance the multimodal image registration accuracy and efficiency for medical practitioners in BM-sparing of cervical cancer radiotherapy. The method proposed in this contribution might also provide a solution to multimodal registration, especially in multimodal sequential images in other clinical applications, such as the diagnosis of cervical cancer and the preservation of normal organs during radiotherapy.

Detection of 3D Object in Point Cloud: Cloud Semantic Segmentation in Lane Marking

Managing a city efficiently and effectively is more important than ever as growing population and economic strain put a strain on infrastructure like transportation and public services like keeping urban green areas clean and maintained. For effective administration, knowledge of the urban setting is essential. Both portable and stationary laser scanners generate 3D point clouds that accurately depict the environment. These data points may be used to infer the state of the roads, buildings, trees, and other important elements involved in this decision-making process. Perhaps they would support "smart" or "smarter" cities in general. Unfortunately, the point clouds do not immediately supply this sort of data. It must be eliminated. This extraction is done either by human specialists or by sophisticated computer programmes that can identify objects. Because the point clouds might represent such large locations, relying on specialists to identify the things may be an unproductive use of time (streets or even whole cities). Automatic or nearly automatic discovery and recognition of essential objects is now possible with the help of object identification software. In this research, In this paper, we describe a unique approach to semantic segmentation of point clouds, based on the usage of contextual point representations to take use of both local and global features within the point cloud. We improve the accuracy of the point's representation by performing a single innovative gated fusion on the point and its neighbours, which incorporates the knowledge from both sets of data and enhances the representation of the point. Following this, we offer a new graph point net module that further develops the improved representation by composing and updating each point's representation inside the local point cloud structure using the graph attention block in real time. Finally, we make advantage of the global structure of the point cloud by using spatial- and channel-wise attention techniques to construct the ensuing semantic label for each point.

A Lidar-Assisted Self-Localization Technology for Indoor Wireless Sensor Networks

The self-localization of wireless sensor networks (WSNs) is facing the problem of insufficient positioning accuracy in indoor environment due to multipath and interference issues. At the same time, without external references, through mutual measurement nodes can calculate only a set of relative coordinates. Therefore, it is difficult to achieve the mapping of coordinate values to the physical world. The development of simultaneous localization and mapping (SLAM) technologies has provided new opportunities to solve the above problems by making it easier to obtain real-time indoor maps. This paper proposes a Lidar-assisted self-localization (LASL) technique to further improve the localization accuracy of WSNs in indoor scenes by combining spatial constraint information obtained from real-time maps, and to place the relative node coordinate network in the visualized maps. Based on the general assumption that the nodes are deployed on the surface of the object, the proposed technique combines the spatial constraints obtained by plane fitting of a local point cloud map (PCM) and finite area approximation of object surfaces with the distance constraints provided by radio ranging. Subsequently, the self-localization results under the joint constraints are solved by the alternating coordinate descent (ACD) method. Simulations and experiments demonstrate that the proposed technique can effectively combine the spatial constraints provided by the Lidar PCM to further improve the self-localization accuracy of the sensor nodes, and further optimize the relative position relationship between each node and the map environment to achieve better matching and integration of the WSN and the real-time map.

Collaborative 3D Object Detection for Autonomous Vehicles via Learnable Communications

3D object detection from LiDAR point cloud is a challenging task in autonomous driving systems. Collaborative perception can incorporate information from spatially diverse sensors and provide significant benefits for accurate 3D object detection from point clouds. In this work, we consider that the autonomous vehicle uses local point cloud data and combines information from neighboring infrastructures through wireless links for cooperative 3D object detection. However, information sharing among vehicles and infrastructures in predefined communication schemes may result in communication congestion and/or bring limited performance improvement. To this end, we propose a novel collaborative 3D object detection framework using an encoder-decoder network architecture and an attention-based learnable communications scheme. It consists of three components: a feature encoder network that maps point clouds into feature maps; an attention-based communication module that propagates compact and fine-grained query feature maps from the vehicle to support infrastructures, and optimizes attention weights between query and key to refine support feature maps; a region proposal network that fuses local feature maps and weighted support feature maps for 3D object detection. We evaluate the performance of the proposed framework on CARLA-3D, a new dataset that we synthesized using CARLA for 3D cooperative object detection. Experimental results and bandwidth consumption analysis show that the proposed collaborative 3D object detection framework achieves a better detection performance and communication bandwidth trade-off than five baseline 3D object detection models under different detection difficulties.

FSFNet: Foreground Score-Aware Fusion for 3-D Object Detector Under Unfavorable Conditions

Nowadays, various multimodal fusion based 3D object detectors appear to provide a potential opportunity to solve the failure cases in single modality methods. However, current fusion approaches still face some unfavorable factors, e.g., poor illumination driving conditions and crowded traffic scenarios, which will cause unsatisfying image quality and objects occlusion. To this end, we present a multimodal fusion network FSFNet consisting of local graph-aware point cloud backbone, foreground score-aware fusion network (FSFN), and the proposals refining loss for 3D object detection task in this paper. Concretely, the directed graph is built to generate edge-wise features for each point, and the point features are supplemented with graph information in local graph-aware point cloud backbone. To alleviate the defect of undesirable image quality features caused by poor illumination condition, FSFN is designed to produce an adaptive multi-modal feature by taking point-wise foreground scores into consideration. Hence, level-wise point features with high confidence are fully utilized and the imperfect image information is suppressed in fusion stage. We further introduce proposals refining loss to reduce false positive and false negative cases in crowded scenes by optimizing the location and scores of predicted 3D bounding boxes. Extensive experiments conducted on KITTI benchmark demonstrate that FSFNet owns its superiority over state-of-the-art networks. Moreover, FSFN is verified to be robust against the image inputs under poor illumination conditions.

A multi-modal garden dataset and hybrid 3D dense reconstruction framework based on panoramic stereo images for a trimming robot

Recovering an outdoor environment’s surface mesh is vital for an agricultural robot during task planning and remote visualization. Image-based dense 3D reconstruction is sensitive to large movements between adjacent frames and the quality of the estimated depth maps. Our proposed solution for these problems is based on a newly-designed panoramic stereo camera along with a hybrid novel software framework that consists of three fusion modules: disparity fusion, pose fusion, and volumetric fusion. The panoramic stereo camera with a pentagon shape consists of 5 stereo vision camera pairs to stream synchronized panoramic stereo images for the following three fusion modules. In the disparity fusion module, rectified stereo images produce the initial disparity maps using multiple stereo vision algorithms. Then, these initial disparity maps, along with the intensity images, are input into a disparity fusion network to produce refined disparity maps. Next, the refined disparity maps are converted into full-view (360°) point clouds or single-view (72°) point clouds for the pose fusion module. The pose fusion module adopts a two-stage global-coarse-to-local-fine strategy. In the first stage, each pair of full-view point clouds is registered by a global point cloud matching algorithm to estimate the transformation for a global pose graph’s edge, which effectively implements loop closure. In the second stage, a local point cloud matching algorithm is used to match single-view point clouds in different nodes. Next, we locally refine the poses of all corresponding edges in the global pose graph using three proposed rules, thus constructing a refined pose graph. The refined pose graph is optimized to produce a global pose trajectory for volumetric fusion. In the volumetric fusion module, the global poses of all the nodes are used to integrate the single-view point clouds into the volume to produce the mesh of the whole garden. The proposed framework and its three fusion modules are tested on a real outdoor garden dataset to show the superiority of the performance. The whole pipeline takes about 4 min on a desktop computer to process the real garden dataset, which is available at: https://github.com/Canpu999/Trimbot-Wageningen-SLAM-Dataset.

Point cloud segmentation of overhead contact systems with deep learning in high-speed rails

High-speed rails have strategic significance for political and economic development. As a critical part of the power supply system, overhead catenary systems (OCSs) power high-speed rails. OCSs are easily damaged due to artificial and natural factors. To ensure that OCSs work properly, researchers often use light detection and ranging (LiDAR) to recognize and inspect OCS components. However, recent OCS recognition methods relying on LiDAR need to enhance the ability to extract local features, integrate contextual features, and enhance crucial features while maintaining less latency and model complexity. To accomplish this objective, we proposed an accelerated point cloud segmentation algorithm for OCS recognition. The algorithm extracted local point cloud features based on machine learning methods. Then, it integrated the contextual features of point clouds using dot and accumulation operations. Additionally, we captured vital features and neglected unimportant features by constructing a feature enhancement algorithm. We accelerated the proposed algorithms by eliminating the run-time overhead of GPU scheduling. Experiments showed that our method had high precision with low model complexity and latency. For example, our precision was at least 0.64% better than comparison studies in recognizing steady arms. Our parameters were at least 43.59% fewer than others. Our optimized algorithm achieved a 2.47, 2.81, 3.24, 1.54, 7.12, and 7× speedup on the Nano, Tesla T4, RTX 2080 Ti, TX2, Tesla V100S GPU, and TITAN V, respectively. Our visualization effect also outperformed other methods in OCS recognition of high-speed rail scenarios.

MFTR-Net: A Multi-Level Features Network with Targeted Regularization for Large-Scale Point Cloud Classification.

There are some irregular and disordered noise points in large-scale point clouds, and the accuracy of existing large-scale point cloud classification methods still needs further improvement. This paper proposes a network named MFTR-Net, which considers the local point cloud's eigenvalue calculation. The eigenvalues of 3D point cloud data and the 2D eigenvalues of projected point clouds on different planes are calculated to express the local feature relationship between adjacent point clouds. A regular point cloud feature image is constructed and inputs into the designed convolutional neural network. The network adds TargetDrop to be more robust. The experimental result shows that our methods can learn more high-dimensional feature information, further improving point cloud classification, and our approach can achieve 98.0% accuracy with the Oakland 3D dataset.

A Pose Estimation Method Combining Instance Segmentation and Point Pair Features

For the traditional pose estimation method based on point pair features, both the preprocessing of scene point cloud and the construction of point pair features of scene point cloud have serious time-consuming problems, which cannot meet the needs of actual industry. Therefore, this paper proposes a pose estimation method that combines instance segmentation and point pair features. Therefore, this paper proposes a pose estimation method that combines instance segmentation and point pair features. First, use the Mask R-CNN-based instance segmentation network to obtain the location of the target object in the two-dimensional image of the scene; then, obtain the local point cloud data of the space where the target object is located from this position and the depth information of the scene; finally, the local the point cloud data is used as the scene point cloud based on point pair feature pose estimation to perform feature matching with the point cloud of the target object.

A model-free 6-DOF grasp detection method based on point clouds of local sphere area

ABSTRACT Grasping object is one of the basic tasks of robots in many scenarios. The main challenge is how to generate grasping poses for unknown objects in cluttered scenes. This paper proposes a model-free 6-DOF grasp detection framework based on single-view local point clouds. The whole process includes three stages: Candidate Generation Network(CGN), Reliable Adjustment Module(RAM), and Quality Assessment Network(QAN). CGN predicts the graspability and the initial grasp pose of the sampled points by the features extracted from the local sphere region based on the improved Pointnet. In order to better learn local area point clouds, we propose a progressive local region data learning mechanism, which can extract features from small to large scales efficiently. Candidate grasps are then consisting of graspable points and their grasping poses. RAM adjusts the position and width of the generated candidate grasps by using reliable heuristic rules. QAN uses a simpled-pointnet to evaluate the quality of the grasp candidates and filters out grasps with high confidence to execute. The proposed method not only achieves state-of-the-art results on GraspNet-1Billion but also shows high grasping success rates in real cluttered scenes.

FA-ResNet: Feature affine residual network for large-scale point cloud segmentation

Semantic segmentation for large-scale point clouds in 3D computer vision remains challenging. Most existing studies focus on creating complex local geometry extractors without considering the sparsity of point clouds and the multi-scale problem of objects in large-scale, resulting in networks that fail to efficiently extract local features and affect segmentation accuracy. In this study, we propose a novel Feature Affine Residual (FA-Res) learnable module for this problem to learn robust point cloud semantic information from the point cloud domain. First, we create a Local Relation-shape Learning module to learn local shape relationships, thereby supplementing the structural information. Second, we propose a lightweight Feature Affine module that can conduct adaptive modifications on local point features to reduce differences between local point clouds with varying densities and the K-nearest neighbor (KNN) algorithm’s domain determination. Finally, we design a Residual MLP Pooling module that can learn deep aggregation features to explore more sophisticated semantic data and provide better guidance for semantic segmentation. We compare our network with state-of-the-art networks on two separate datasets to show its effectiveness. Specifically, our method achieves 68.1% mIoU on S3DIS tested on Area 5, which is an improvement of 2.7% compared with the latest representative network.

A Multivariate Local Descriptor Registration Method for Surface Topography Evaluation

This paper illustrates a systematical surface topography measurement and evaluation method based on a 3D optical system. Firstly, the point cloud data of the workpiece are extracted by the use of a 3D structured light measurement system, and the STEP file of the design model is converted into point cloud data. Secondly, the local measurement point cloud (LMPC) and digital model point cloud (DMPC) are registered by a multivariate local descriptor registration scheme proposed in this study. Thirdly, the surface shapes extracted from the STEP file are applied as a reference to segment the measuring point cloud. Finally, an error analysis scheme is conducted on specific functional surfaces. An experiment was conducted to analyse the flatness, cylindricity and roughness to demonstrate the effectiveness and advantage of the method. The comparison results show that the proposed method outperforms other 3D optical surface topography analysis methods.

A Progressive Plane Detection Filtering Method for Airborne LiDAR Data in Forested Landscapes

Ground filtering is necessary in processing airborne light detection and ranging (LiDAR) point clouds for forestry applications. This study proposes a progressive plane detection filtering (PPDF) method. First, the method uses multi-scale planes to characterize terrain, i.e., the local terrain with large slope variations is represented by small-scale planes, and vice versa. The planes are detected in local point clouds by the random sample consensus method with decreasing plane sizes. The reliability of the planes to represent local terrain is evaluated and the planes with optimal sizes are selected according to evaluation results. Then, ground seeds are identified by selecting the interior points of the planes. Finally, ground points are iteratively extracted based on the reference terrain, which is constructed using evenly distributed neighbor ground points. These neighbor points are identified by selecting the nearest neighbor points of multiple subspaces, which are divided from the local space with an unclassified point as center point. PPDF was tested in six sites with various terrain and vegetation characteristics. Results showed that PPDF was more accurate and robust compared to the classic filtering methods including maximum slope, progressive morphology, cloth simulation, and progressive triangulated irregular network densification filtering methods, with the smallest average total error and standard deviation of 3.42% and 2.45% across all sites. Moreover, the sensitivity of PPDF to parameters was low and these parameters can be set as fixed values. Therefore, PPDF is effective and easy-to-use for filtering airborne LiDAR data.

Complex parts machining path planning through cross-source point cloud processing

When cleaning cast parts, production machinery such as robots and smart machines must be able to operate tools along appropriate machining paths. Planning the machining path using 3D point cloud processing techniques is a suitable approach. The local point clouds generated from in situ scans and the standard point clouds, however, differ in scale and accuracy, and these issues make the registration of cross-source point clouds a difficult academic topic. The study presented here suggests a novel cross-source point cloud registration technique that combines a point cloud slicing algorithm and a curve fitting algorithm to plan machining paths for complex cast parts. The results demonstrate the practicality of the proposed method, with the uniformity of the surface profile and the operational effectiveness of the machining process being 57.90% and 67.74% higher, respectively, for the machining paths generated using the proposed method compared to the manual method. By comparing trials of various machining path curve fitting algorithms, we find that the Fourier fitting approach performs best. In addition, the co-planar features chosen for coarse registration will be better registered if they are closer to the plane containing the intended machining path. The proposed method may be useful for online inspection by UAVs and interactive robots.

A Novel Method for Obstacle Detection in Front of Vehicles Based on the Local Spatial Features of Point Cloud

Obstacle detection is the primary task of the Advanced Driving Assistance System (ADAS). However, it is very difficult to achieve accurate obstacle detection in complex traffic scenes. To this end, this paper proposes an obstacle detection method based on the local spatial features of point clouds. Firstly, the local spatial point cloud of a superpixel is obtained through stereo matching and the SLIC image segmentation algorithm. Then, the probability of the obstacle in the corresponding area is estimated from the spatial feature information of the local plane normal vector and the superpixel point-cloud height, respectively. Finally, the detection results of the two methods are input into the Bayesian framework in the form of probabilities for the final decision. In order to describe the traffic scene efficiently and accurately, the detection results are further transformed into a multi-layer stixel representation. We carried out experiments on the KITTI dataset and compared several obstacle detection methods. The experimental results indicate that the proposed method has advantages in terms of its Pixel-wise True Positive Rate (PTPR) and Pixel-wise False Positive Rate (PFPR), particularly in complex traffic scenes, such as uneven roads.

E-LOAM: LiDAR Odometry and Mapping With Expanded Local Structural Information

This paper investigates the real time LiDAR odometry and mapping (LOAM) problem in unstructured environments. We propose E-LOAM (LOAM with Expanded Local Structural Information), a paradigm which expands the pre-extracted geometric information with local point cloud information around the geometric feature points. State-of-the-art approaches usually extract pointed geometric features as the only correspondence primitives for point cloud scan-to-scan and scan-to-map registration. We argue that, in unstructured environments, sometimes, the extracted geometric features are too sparse for adequate point cloud registration. Therefore, E-LOAM expands the ‘pointed’ geometric correspondence primitives with the point clouds around them, <i>i.e.,</i> the local point clouds in the voxel around the feature point. The local point clouds, approximated by a multivariate normal distribution, offer additional local structural information, on top of the <i>pointed </i>geometric information. Additionally, to enrich the sparse geometric features, we make use of the intensity information of point clouds, and extract the places with high intensity variations as additional feature points. Experimental results with KITTI dataset show the efficacy of E-LOAM, when compared with state of the arts. We further implement E-LOAM on a real robot platform, and evaluate E-LOAM with in-field tests.

Anatomical Feature Segmentation of Femur Point Cloud Based on Medical Semantics

Feature segmentation is an essential phase for geometric modeling and shape processing in anatomical study of human skeleton and clinical digital treatment of orthopedics. Due to various degrees of freedom of bone surface, the existing segmentation algorithms can hardly meet specific medical need. To address this, a novel segmentation methodology for anatomical features of femur model based on medical semantics is put forward. First, anatomical reference objects (ARO) are created to represent typical characteristics of femur anatomy by 3D point fitting in combination with medical priori knowledge. Then, local point clouds between adjacent anatomies are selected according to the AROs to extract boundary feature point (BFP)s. Finally, the complete model of femur is divided into anatomical regions by executing the enhanced watershed algorithm guided with BFPs. Experimental results show that the proposed method has the advantages of automatic segmentation of femoral head, neck and other complex areas, and the segmentation results have better medical semantics. In addition, the slight modification of segmentation results can be achieved by adjusting a few threshold parameter values, which improves the convenience of modification for ordinary users.

FG-Net: A Fast and Accurate Framework for Large-Scale LiDAR Point Cloud Understanding.

This work presents FG-Net, a general deep learning framework for large-scale point cloud understanding without voxelizations, which achieves accurate and real-time performance with a single NVIDIA GTX 1080 8G GPU and an i7 CPU. First, a novel noise and outlier filtering method is designed to facilitate the subsequent high-level understanding tasks. For effective understanding purpose, we propose a novel plug-and-play module consisting of correlated feature mining and deformable convolution-based geometric-aware modeling, in which the local feature relationships and point cloud geometric structures can be fully extracted and exploited. For the efficiency issue, we put forward a new composite inverse density sampling (IDS)-based and learning-based operation and a feature pyramid-based residual learning strategy to save the computational cost and memory consumption, respectively. Compared with current methods which are only validated on limited datasets, we have done extensive experiments on eight real-world challenging benchmarks, which demonstrates that our approaches outperform state-of-the-art (SOTA) approaches in terms of accuracy, speed, and memory efficiency. Moreover, weakly supervised transfer learning is also conducted to demonstrate the generalization capacity of our method.