Point Cloud Frame Research Articles

A Versatile Point Cloud Compressor Using Universal Multiscale Conditional Coding - Part I: Geometry.

A universal multiscale conditional coding framework, Unicorn, is proposed to compress the geometry and attribute of any given point cloud. Geometry compression is addressed in Part I of this paper, while attribute compression is discussed in Part II. We construct the multiscale sparse tensors of each voxelized point cloud frame and properly leverage lower-scale priors in the current and (previously processed) temporal reference frames to improve the conditional probability approximation or content-aware predictive reconstruction of geometry occupancy in compression. Unicorn is a versatile, learning-based solution capable of compressing static and dynamic point clouds with diverse source characteristics in both lossy and lossless modes. Following the same evaluation criteria, Unicorn significantly outperforms standard-compliant approaches like MPEG G-PCC, V-PCC, and other learning-based solutions, yielding state-of-the-art compression efficiency while presenting affordable complexity for practical implementations.

Unsupervised 3D Object Segmentation of Point Clouds by Geometry Consistency.

In this paper, we study the problem of 3D object segmentation from raw point clouds. Unlike existing methods which usually require a large amount of human annotations for full supervision, we propose the first unsupervised method, called OGC, to simultaneously identify multiple 3D objects in a single forward pass, without needing any type of human annotations. The key to our approach is to fully leverage the dynamic motion patterns over sequential point clouds as supervision signals to automatically discover rigid objects. Our method consists of three major components, 1) the object segmentation network to directly estimate multi-object masks from a single point cloud frame, 2) the auxiliary self-supervised scene flow estimator, and 3) our core object geometry consistency component. By carefully designing a series of loss functions, we effectively take into account the multi-object rigid consistency and the object shape invariance in both temporal and spatial scales. This allows our method to truly discover the object geometry even in the absence of annotations. We extensively evaluate our method on five datasets, demonstrating the superior performance for object part instance segmentation and general object segmentation in both indoor and the challenging outdoor scenarios.

Vehicle trajectory extraction with interacting multiple model for low-channel roadside LiDAR

High-precision and consistent vehicle trajectories encompass microscopic traffic parameters, mesoscopic traffic flow characteristics, and macroscopic traffic flow features, which is the cornerstone of innovation in data-driven traffic management and control applications. However, occlusion and trajectory interruption remain challenging in multivehicle tracking under complex traffic environments using low-channel roadside LiDAR. To address the challenge, a novel framework for vehicle trajectory extraction using low-channel roadside LiDAR was proposed. First, the geometric features of the cluster and its L-shape bounding box were used to address the over-segmentation in vehicle detection arising from occlusion and point cloud sparse. Then, objects within adjacent point cloud frames were associated by developing an improved Hungarian algorithm integrated with an adaptive distance threshold to solve the mismatching problem caused by objects entrancing and exiting in a new point cloud frame. Finally, an improved interacting multiple model by considering vehicle driving patterns was deployed to predict the location of missing vehicles and connect the interrupted trajectories. Experimental results showed that the proposed methods achieve 98.76 % of vehicle detection accuracy and 97.40 % of data association precision. The mean absolute error (MAE) and mean square error (MSE) of the vehicle position estimation are 0.2252 m and 0.0729 m2, respectively. The trajectory extraction precision outperforms most of the state-of-the-art algorithms.

Dense 3D Point Cloud Environmental Mapping Using Millimeter-Wave Radar

To address the challenges of sparse point clouds in current MIMO millimeter-wave radar environmental mapping, this paper proposes a dense 3D millimeter-wave radar point cloud environmental mapping algorithm. In the preprocessing phase, a radar SLAM-based approach is introduced to construct local submaps, which replaces the direct use of radar point cloud frames. This not only reduces data dimensionality but also enables the proposed method to handle scenarios involving vehicle motion with varying speeds. Building on this, a 3D-RadarHR cross-modal learning network is proposed, which uses LiDAR as the target output to train the radar submaps, thereby generating a dense millimeter-wave radar point cloud map. Experimental results across multiple scenarios, including outdoor environments and underground tunnels, demonstrate that the proposed method can increase the point cloud density of millimeter-wave radar environmental maps by over 50 times, with a point cloud accuracy better than 0.1 m. Compared to existing algorithms, the proposed method achieves superior environmental map reconstruction performance while maintaining a real-time processing rate of 15 Hz.

Fast Motion State Estimation Based on Point Cloud by Combing Deep Learning and Spatio-Temporal Constraints

Moving objects in the environment have a higher priority and more challenges in growing domains like unmanned vehicles and intelligent robotics. Estimating the motion state of objects based on point clouds in outdoor scenarios is currently a challenging area of research. This is due to factors such as limited temporal information, large volumes of data, extended network processing times, and the ego-motion. The number of points in a point cloud frame is typically 60,000–120,000 points, but most current motion state estimation methods for point clouds only downsample to a few thousand points for fast processing. The downsampling step will lead to the loss of scene information, which means these methods are far from being used in practical applications. Thus, this paper proposes a motion state estimation method that combines spatio-temporal constraints and deep learning. It starts by estimating and compensating the ego-motion of multi-frame point cloud data and mapping multi-frame data to a unified coordinate system; then the point cloud motion segmentation model on the multi-frame point cloud is proposed for motion object segmentation. Finally, spatio-temporal constraints are utilized to correlate the moving object at different moments and estimate the motion vectors. Experiments on KITTI, nuScenes, and real captured data show that the proposed method has good results, with an average vector deviation of only 0.036 m and 0.043 m in KITTI and nuScenes under a processing time of about 80 ms. The EPE3D error under the KITTI data is only 0.076 m, which proves the effectiveness of the method.

High precision rail surface obstacle detection algorithm based on 3D imaging LiDAR

Railway perimeter intrusion detection is of great significance for ensuring railway safety operations. In this paper, a Light Detection and Ranging (LiDAR) system is designed and developed based on the specific environmental requirements of railway applications. The system is capable of autonomously adjusting the imaging range and angular resolution, enabling high-precision imaging within a 50 m monitoring area. Additionally, based on registration and a series of filtering algorithms, a railway surface obstacle extraction algorithm is developed. The algorithm achieves a 100 % detection rate for obstacles with a minimum height of 15 cm at any position within the monitoring area, and a 70 % detection rate for obstacles with a height of 10 cm. In this algorithm, a coarse registration method based on global planar features combined with the Iterative Closest Point (ICP) registration algorithm achieves sub-centimeter registration accuracy for consecutive point cloud frames. The filtering algorithm based on local planar features enables precise extraction of geometric changes in the railway surface, with simple calculations and fast processing speed, making it highly valuable for extracting object edge features.

PatchDPCC: A Patchwise Deep Compression Framework for Dynamic Point Clouds

When compressing point clouds, point-based deep learning models operate points in a continuous space, which has a chance to minimize the geometric fidelity loss introduced by voxelization in preprocessing. But these methods could hardly scale to inputs with arbitrary points. Furthermore, the point cloud frames are individually compressed, failing the conventional wisdom of leveraging inter-frame similarity. In this work, we propose a patchwise compression framework called patchDPCC, which consists of a patch group generation module and a point-based compression model. Algorithms are developed to generate patches from different frames representing the same object, and more importantly, these patches are regulated to have the same number of points. We also incorporate a feature transfer module in the compression model, which refines the feature quality by exploiting the inter-frame similarity. Our model generates point-wise features for entropy coding, which guarantees the reconstruction speed. The evaluation on the MPEG 8i dataset shows that our method improves the compression ratio by 47.01% and 85.22% when compared to PCGCv2 and V-PCC with the same reconstruction quality, which is 9% and 16% better than that D-DPCC does. Our method also achieves the fastest decoding speed among the learning-based compression models.

Fast Inter-frame Motion Prediction for Compressed Dynamic Point Cloud Attribute Enhancement

Recent years have witnessed the success of deep learning methods in quality enhancement of compressed point cloud. However, existing methods focus on geometry and attribute enhancement of single-frame point cloud. This paper proposes a novel compressed quality enhancement method for dynamic point cloud (DAE-MP). Specifically, we propose a fast inter-frame motion prediction module (IFMP) to explicitly estimate motion displacement and achieve inter-frame feature alignment. To maintain motion continuity between consecutive frames, we propose a motion consistency loss for supervised learning. Furthermore, a frequency component separation and fusion module is designed to extract rich frequency features adaptively. To the best of our knowledge, the proposed method is the first deep learning-based work to enhance the quality for compressed dynamic point cloud. Experimental results show that the proposed method can greatly improve the quality of compressed dynamic point cloud and provide a fast and efficient motion prediction plug-in for large-scale point cloud. For dynamic point cloud attribute with severely compressed artifact, our proposed DAE-MP method achieves up to 0.52dB (PSNR) performance gain. Moreover, the proposed IFMP module has a certain real-time processing ability for calculating the motion offset between dynamic point cloud frame.

Louvain-Based Traffic Object Detection for Roadside 4D Millimeter-Wave Radar

Object detection is the fundamental task of vision-based sensors in environmental perception and sensing. To leverage the full potential of roadside 4D MMW radars, an innovative traffic detection method is proposed based on their distinctive data characteristics. First, velocity-based filtering and region of interest (ROI) extraction were employed to filter and associate point data by merging the point cloud frames to enhance the point relationship. Then, the Louvain algorithm was used to divide the graph into modularity by converting the point cloud data into graph structure and amplifying the differences with the Gaussian kernel function. Finally, a detection augmentation method is introduced to address the problems of over-clustering and under-clustering based on the object ID characteristics of 4D MMW radar data. The experimental results showed that the proposed method obtained the highest average precision and F1 score: 98.15% and 98.58%, respectively. In addition, the proposed method showcased the lowest over-clustering and under-clustering errors in various traffic scenarios compared with the other detection methods.

RESEARCH ON LIDAR SLAM METHOD WITH FUSED POINT CLOUD INTENSITY INFORMATION

Abstract. Under the current trend of intelligence and automation, simultaneous positioning and mapping technology has become one of the research hotspots. The main problems of SLAM technology research are to improve the robustness of mapping and positioning, establish an efficient back-end optimization system, and improve the generalization of SLAM technology. This paper proposes to fuse the intensity information of the point cloud and the geometric information of the environment scene to construct a globally consistent environment feature descriptor and use the non-iterative two-step method to perform the nearest neighbour search on the point cloud in the point cloud registration stage. Then use the globally consistent descriptor that has been constructed to extract the laser point cloud descriptor by using the ring partition method, combine the method based on domain search to search for the closest point cloud frame, and finally use Intensity-ICP to complete the loopback frame. Fine registration, outputs the optimal pose transformation, to complete the loop detection. We use our self-built mobile platform to verify the robustness and generalization of the improved laser SLAM algorithm in public datasets and campus datasets. Experimental results show that the improved algorithm reduces the trajectory drift of the mobile platform and improves the efficiency of point cloud registration and loop closure detection.

VPRNet: Voxel-based Efficient and Partial-to-Partial Point Cloud Registration on Mobile Devices

With the popularity of embedded devices such as LIDAR sensors and depth cameras, the resulting point clouds become the main data format for representing the 3D world and spawn various smart mobile applications. A key technology for enabling these applications to furnish high-quality services is real-time point cloud registration on mobile devices, which synthesizes a complete model or a large-scale scene from multiple partial scans. It aims to deliver increasing sensing range, faster 3D reconstruction and more robust robot navigation. Unfortunately, the performance of these applications is limited by the scale and partial loss of raw point cloud frame. The existing solutions for point cloud registration are difficult to deploy on mobile devices due to their complex models and assumptions about point cloud pairs with large overlap, which cause significant delay and inaccuracy. This paper proposes VPRNet - the first voxel-based registration solution that can achieve real-time partial-to-partial registration with competitive registration quality while being more advantageous for large-scale point clouds on mobile devices. We conduct real-world experiments and extensive simulations cross various datasets and platforms to validate the efficacy of VPRNet and further compare the performance with state-of-the-art approaches.

AI-Enabled Condition Monitoring Framework for Outdoor Mobile Robots Using 3D LiDAR Sensor

An automated condition monitoring (CM) framework is essential for outdoor mobile robots to trigger prompt maintenance and corrective actions based on the level of system deterioration and outdoor uneven terrain feature states. Vibration indicates system failures and terrain abnormalities in mobile robots; hence, five vibration threshold classes for CM in outdoor mobile robots were identified, considering both vibration source system deterioration and uneven terrain. This study proposes a novel CM approach for outdoor mobile robots using a 3D LiDAR, employed here instead of its usual use as a navigation sensor, by developing an algorithm to extract the vibration-indicated data based on the point cloud, assuring low computational costs without losing vibration characteristics. The algorithm computes cuboids for two prominent clusters in every point cloud frame and sets motion points at the corners and centroid of the cuboid. The three-dimensional vector displacement of these points over consecutive point cloud frames, which corresponds to the vibration-affected clusters, are compiled as vibration indication data for each threshold class. A simply structured 1D Convolutional Neural Network (1D CNN)-based vibration threshold prediction model is proposed for fast, accurate, and real-time application. Finally, a threshold class mapping framework is developed which fuses the predicted threshold classes on the 3D occupancy map of the workspace, generating a 3D CbM map in real time, fostering a Condition-based Maintenance (CbM) strategy. The offline evaluation test results show an average accuracy of vibration threshold classes of 89.6% and consistent accuracy during real-time field case studies of 89%. The test outcomes validate that the proposed 3D-LiDAR-based CM framework is suitable for outdoor mobile robots, assuring the robot’s health and operational safety.

Temporal Estimation of Non-Rigid Dynamic Human Point Cloud Sequence Using 3D Skeleton-Based Deformation for Compression.

This paper proposes an algorithm for transmitting and reconstructing the estimated point cloud by temporally estimating a dynamic point cloud sequence. When a non-rigid 3D point cloud sequence (PCS) is input, the sequence is divided into groups of point cloud frames (PCFs), and a key PCF is selected. The 3D skeleton is predicted through 3D pose estimation, and the motion of the skeleton is estimated by analyzing the joints and bones of the 3D skeleton. For the deformation of the non-rigid human PC, the 3D PC model is transformed into a mesh model, and the key PCF is rigged using the 3D skeleton. After deforming the key PCF into the target PCF utilizing the motion vector of the estimated skeleton, the residual PC between the motion compensation PCF and the target PCF is generated. If there is a key PCF, the motion vector of the target PCF, and a residual PC, the target PCF can be reconstructed. Just as compression is performed using pixel correlation between frames in a 2D video, this paper compresses 3D PCFs by estimating the non-rigid 3D motion of a 3D object in a 3D PC. The proposed algorithm can be regarded as an extension of the 2D motion estimation of a rigid local region in a 2D plane to the 3D motion estimation of a non-rigid object (human) in 3D space. Experimental results show that the proposed method can successfully compress 3D PC sequences. If it is used together with a PC compression technique such as MPEG PCC (point cloud compression) in the future, a system with high compression efficiency may be configured.

3D LiDAR Point Cloud Registration Based on IMU Preintegration in Coal Mine Roadways

Point cloud registration is the basis of real-time environment perception for robots using 3D LiDAR and is also the key to robust simultaneous localization and mapping (SLAM) for robots. Because LiDAR point clouds are characterized by local sparseness and motion distortion, the point cloud features of coal mine roadway environments show a weak texture and degradation. Therefore, for these environments, the traditional point cloud registration method to register directly will lead to problems, such as a decline in registration accuracy, z-axis drift, and map ghosting. To solve the above problems, we propose a point cloud registration method based on IMU preintegration with the sensor characteristics of LiDAR and IMU. The system framework of this method mainly consists of four modules: IMU preintegration, point cloud preprocessing, point cloud frame matching and point cloud registration. First, IMU sensor data are introduced, and IMU linear interpolation is used to correct the motion distortion in LiDAR scanning, and the IMU preintegration error function is constructed. Second, the point cloud segmentation is performed using the ground segmentation method of RANSAC to provide additional ground constraints for the z-axis displacement and to remove the unstable flawed points from the point cloud. On this basis, the LiDAR point cloud registration error function is constructed by extracting the feature corner points and feature plane points. Finally, the Gaussian Newton solution is used to optimize the constraint relationship between the LiDAR odometry frames to minimize the error function, complete the LiDAR point cloud registration and better estimate the position and pose of the mobile robot. The experimental results show that compared with the traditional point cloud registration method, the proposed method has a higher point cloud registration accuracy, success rate and computational efficiency. The LiDAR odometry constructed using this method can better reflect the authenticity of the robot trajectory and has higher trajectory accuracy and smaller absolute position and pose error.

Real-Time Neural Dense Elevation Mapping for Urban Terrain With Uncertainty Estimations

Having good knowledge of terrain information is essential for improving the performance of various downstream tasks on complex terrains, especially for the locomotion and navigation of legged robots. We present a novel framework for neural urban terrain reconstruction with uncertainty estimations. It generates dense robot-centric elevation maps online from sparse LiDAR observations. We design a novel pre-processing and point features representation approach that ensures high robustness and computational efficiency when integrating multiple point cloud frames. A generative Bayesian model then recovers the detailed terrain structures while simultaneously providing the pixel-wise reconstruction uncertainty. We evaluate the proposed pipeline through both simulation and real-world experiments. Our approach achieves high-quality terrain reconstruction with real-time performance on a mobile platform, and the uncertainty estimates may further benefit the downstream tasks of legged robots.

PointMapNet: Point Cloud Feature Map Network for 3D Human Action Recognition

3D human action recognition is crucial in broad industrial application scenarios such as robotics, video surveillance, autonomous driving, or intellectual education, etc. In this paper, we present a new point cloud sequence network called PointMapNet for 3D human action recognition. In PointMapNet, two point cloud feature maps symmetrical to depth feature maps are proposed to summarize appearance and motion representations from point cloud sequences. Specifically, we first convert the point cloud frames to virtual action frames using static point cloud techniques. The virtual action frame is a 1D vector used to characterize the structural details in the point cloud frame. Then, inspired by feature map-based human action recognition on depth sequences, two point cloud feature maps are symmetrically constructed to recognize human action from the point cloud sequence, i.e., Point Cloud Appearance Map (PCAM) and Point Cloud Motion Map (PCMM). To construct PCAM, an MLP-like network architecture is designed and used to capture the spatio-temporal appearance feature of the human action in a virtual action sequence. To construct PCMM, the MLP-like network architecture is used to capture the motion feature of the human action in a virtual action difference sequence. Finally, the two point cloud feature map descriptors are concatenated and fed to a fully connected classifier for human action recognition. In order to evaluate the performance of the proposed approach, extensive experiments are conducted. The proposed method achieves impressive results on three benchmark datasets, namely NTU RGB+D 60 (89.4% cross-subject and 96.7% cross-view), UTD-MHAD (91.61%), and MSR Action3D (91.91%). The experimental results outperform existing state-of-the-art point cloud sequence classification networks, demonstrating the effectiveness of our method.

Multi-Level Structure-Enhanced Network for 3D Single Object Tracking in Sparse Point Clouds

3D single object tracking is the task of localizing a target object in a search point cloud frame. In this letter, we present a multi-level structure-enhanced tracking model to improve the tracking performance in sparse 3D point clouds. Towards this end, we first encode the target and the search point clouds efficiently in two near-neighbors graphs, which allows structural information to flow between neighbors along graph edges. We then design a cross-graph attention mechanism, which associates similar nodes across the target graph and the search graph, and further dissimilar graph nodes for apart. Integrating the proposed mechanism into the above Siamese feature learning of both the target and the search frame, we strengthen the structural correlation between the target and the search frame. In that case, distinguishing the potential target from the background in the search frame would be much simpler. Finally, we design a U-shaped sparse convolutional block to aggregate the structural features of the potential target in the search frame. Integrating the proposed block into an existing target localization module from (Hui et al., 2021), we localize target centers accurately. Experiments on the KITTI benchmark demonstrate that our method outperforms some state-of-the-art models, achieving at least a 3.2% improvement in terms of average tracking precision.

SAM-Net: LiDAR Depth Inpainting for 3D Static Map Generation

Various sensors can be attached and added to autonomous vehicles, included visual cameras, radar, LiDAR (Light Detection And Ranging), and GNSS (Global Navigation Satellite System). These sensors have been studied in many research areas, in particular studies on building precise 3D maps. It is essential for autonomous driving to create an accurate 3D map of the surrounding scene. However, creating an accurate static 3D map is difficult due to changes in moving objects or dynamic environments. Spurious objects on the 3D map can be handled by removing or ignoring them for 3D mapping. Following this idea, we propose an object segmentation and inpainting network. The proposed network called SAM-Net, addresses the object duplication issue by segmenting the objects and inpainting them with the segmentation results. Conventional inpainting research has dealt with RGB images. No matter how well such approaches reconstruct holes or corrupted images, they do not establish 3D points’ relationship with the point cloud frame. Therefore, we suggest a depth inpainting method for outdoor object segmentation and inpainting tasks that utilizes a high-precision depth range sensor (Velodyne HDL-64E), which is not suggested before. Unfortunately, no dataset exists for the outdoor depth inpainting task. Thus, to train our model, we generate a new dataset by locating objects on a clean static background. Moreover, our proposed method shows outstanding depth performance compared to the previous visual inpainting method. Our dataset will be available at: “ <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/JunhyeopLee/lidar_inpainting</uri> ”.

Understanding the dynamic properties of trees using the motions constructed from multi-beam flash light detection and ranging measurements.

Measuring the three-dimensional motion of trees at every position remains challenging as it requires dynamic measurement technology with sufficient spatial and temporal resolution. Consequently, this study explores the use of a novel multi-beam flash light detection and ranging (LiDAR) sensor to tackle such a sensing barrier. A framework is proposed to record tree vibrations, to construct the motions of tree skeletons from the point-cloud frames recorded by the LiDAR sensor and to derive the dynamic properties of trees. The feasibility of the framework is justified through measurement on a Ficus microcarpa under pull-and-release tests. The relative differences for the first two modal frequencies between the LiDAR and linear variable differential transformer measurements in the displacement Fourier spectra are 0.1% and 2.5%, respectively. The framework is further adopted to study the dynamic response of different trees subjected to typhoons, including a Liquidambar formosana, three Araucaria heterophylla trees, a Sterculia lanceolata, a Celtis sinensis, a Tabebuia chrysantha and a Cinnamomum camphora. Results suggest that broadleaved trees might exhibit vibration in a wide frequency band, whereas the coniferous trees could follow a distinct dominant frequency.

Point cloud registration and localization based on voxel plane features

The 3D point cloud can directly provide accurate distance information, which facilitates many applications such as autonomous driving and environmental modeling. In order to construct more complete environmental information for these applications, it is necessary to register point clouds that are obtained from different poses. Registration after coarse localization is an effective method for understanding the pose of the device among the context of its environment. Here, a method for extracting plane features based on voxels is proposed and used for coarse registration and localization. The point cloud is divided into voxels by an octree, and voxels with the same plane characteristics are merged to obtain plane features. The candidate transformation matrix is calculated by using the corresponding plane set, and the RANSAC process with two-level transformation matrix verification, including quick verification and fine verification, is used to find the optimum transformation matrix from the candidates after clustering. Then, the coarse registration can be achieved. With the extracted plane features, a combined plane feature description of the point cloud frame is constructed to fulfill fast frame-level global localization. With the integration of the registration method presented in the current paper, pose localization can also be achieved. Experimental results show that the proposed method can achieve more than 85% successful registration rate with short time consumptions. This implies that the proposed method is more efficient than the benchmark method. Even when the map is large, frame-level localization is still fast, and has a successful localization rate of over 90%. The corresponding code are available at https://github.com/zhanjiawang/VPFBR-L.