LiDAR Frames Research Articles

Adaptive robot localization in dynamic environments through self-learnt long-term 3D stable points segmentation

In field robotics, particularly in the agricultural sector, precise localization presents a challenge due to the constantly changing nature of the environment. Simultaneous Localization and Mapping algorithms can provide an effective estimation of a robot’s position, but their long-term performance may be impacted by false data associations. Additionally, alternative strategies such as the use of RTK-GPS can also have limitations, such as dependence on external infrastructure. To address these challenges, this paper introduces a novel stability scan filter. This filter can learn and infer the motion status of objects in the environment, allowing it to identify the most stable objects and use them as landmarks for robust robot localization in a continuously changing environment. The proposed method involves an unsupervised point-wise labelling of LiDAR frames by utilizing temporal observations of the environment, as well as a regression network, called Long-Term Stability Network (LTS-NET) to learn and infer 3D LiDAR points long-term motion status. Experiments demonstrate the ability of the stability scan filter to infer the motion stability of objects on a real agricultural long-term dataset. Results show that by only utilizing points belonging to long-term stable objects, the localization system exhibits reliable and robust localization performance for long-term missions compared to using the entire LiDAR frame points.

Camera LiDAR calibration: an automatic and accurate method with novel PLE metrics

Abstract Camera-LiDAR calibration is a crucial aspect of perception systems and sensor fusion in various fields, facilitating the fusion of data from cameras and LiDAR sensors for applications such as autonomous vehicles, robotics, and augmented reality. In this paper, we presented a novel multi-modal fusion method that introduces an efficient camera-LiDAR joint calibration technique using a simple checkerboard. Our method estimates the 3D rigid transformation of the camera system with respect to the LiDAR frame by establishing 2D-to-3D point correspondences for geometric calibration. We improved the idea of the traditional PNP (Perspective-N-Point) algorithm by employing point cloud segmentation and clustering to re-match the checkerboard. When compared with the calibration methods in the Autoware and MATLAB calibration toolbox, the reprojection error was improved by 38.13\% and 58.30\% respectively. The average translation error was improved by 78.26\% and 51.61\% respectively, while the average rotation error is improved by 75\% and 60.78\% respectively, which has a great improvement in all three areas. At the same time, we propose a new evaluation metric for joint calibration results based on reprojected images: PLE (Pixel-Level Evaluation) index to reflect the accuracy of the joint calibration of the camera and LiDAR. An automatic calibration software has been developed for the calibration of the camera's intrinsic parameters as well as for the joint calibration of the camera-LIDAR extrinsic parameters. We have extensively validated our algorithm using the Intel RealSense Depth D435 Camera and LeiShen C16-151B LiDAR.

TSG-Seg: Temporal-selective guidance for semi-supervised semantic segmentation of 3D LiDAR point clouds

LiDAR-based semantic scene understanding holds a pivotal role in various applications, including remote sensing and autonomous driving. However, the majority of LiDAR segmentation models rely on extensive and densely annotated training datasets, which is extremely laborious to annotate and hinder the widespread adoption of LiDAR systems. Semi-supervised learning (SSL) offers a promising solution by leveraging only a small amount of labeled data and a larger set of unlabeled data, aiming to train robust models with desired accuracy comparable to fully supervised learning. A typical pipeline of SSL involves the initial use of labeled data to train segmentation models, followed by the utilization of predictions generated from unlabeled data, which are used as pseudo-ground truths for model retraining. However, the scarcity of labeled data limits the capture of comprehensive representations, leading to the constraints of these pseudo-ground truths in reliability. We observed that objects captured by LiDAR sensors from varying perspectives showcase diverse data characteristics due to occlusions and distance variation, and LiDAR segmentation models trained with limited labels prove susceptible to these viewpoint disparities, resulting in inaccurately predicted pseudo-ground truths across viewpoints and the accumulation of retraining errors. To address this problem, we introduce the Temporal-Selective Guided Learning (TSG-Seg) framework. TSG-Seg explores temporal cues inherent in LiDAR frames to bridge the cross-viewpoint representations, fostering consistent and robust segmentation predictions across differing viewpoints. Specifically, we first establish point-wise correspondences across LiDAR frames with different time stamps through point registration. Subsequently, reliable point predictions are selected and propagated to points from adjacent views to the current view, serving as strong and refined supervision signals for subsequent model re-training to achieve better segmentation. We conducted extensive experiments on various SSL labeling setups across multiple public datasets, including SemanticKITTI and SemanticPOSS, to evaluate the effectiveness of TSG-Seg. Our results demonstrate its competitive performance and robustness in diverse scenarios, from data-limited to data-abundant settings. Notably, TSG-Seg achieves a mIoU of 48.6% using only 5% of and 62.3% with 40% of labeled data in the sequential split on SemanticKITTI. This consistently outperforms state-of-the-art segmentation methods, including GPC and LaserMix. These findings underscore TSG-Seg’s superior capability and potential for real-world applications. The project can be found at https://tsgseg.github.io.

Global principal planes aided LiDAR-based mobile mapping method in artificial environments

3-D mapping of buildings is crucial for urban renewal, but traditional LiDAR-based mapping methods are often less effective for buildings with narrow spaces and limited geometric features. Current methods attempt to overcome this by integrating additional sensors, such as cameras, which increases cost and complexity. This paper proposes a novel LiDAR-based mobile mapping framework using global principal planes (GPPs) to address this challenge without additional sensors. GPPs are defined as unlimited planes characterized by principal normal vectors (PNVs). GPPs can provide stronger constraints than traditional small planes extracted from one or certain LiDAR frames because they are little affected by the accumulative error from point cloud matching. A PNV estimation method is also proposed based on an inertial measurement unit and polar histogram, and PNVs are axes of the natural cartesian XYZ coordinate system. Point clouds are transformed into the PNVs coordinate system to extract robust edge and plane feature points and GPPs. The proposed framework is tested in various environments. It achieves about 3 cm accuracy in corridors and similar accuracy in stairwells. Compared to five state-of-the-art mapping methods (Cartographer, etc.), its accuracy improves by over 76%, increasing at least an order of magnitude. In the outdoor KITTI dataset, it shows a reduction in absolute pose errors by 4% to 20%. Extensive experiments demonstrate its accuracy, robustness, and generalizability. Ablation experiments further validate the efficacy of different components in the framework.

Hierarchical fusion based high precision SLAM for solid-state lidar

Abstract Solid-state LiDARs have become an important perceptual device for simultaneous localization and mapping (SLAM) due to its low-cost and high-reliability compared to mechanical LiDARs. Nevertheless, existing solid-state LiDARs-based SLAM methods face challenges, including drift and mapping inconsistency, when operating in dynamic environments over extended periods and long distances. To this end, this paper proposes a robust, high-precision, real-time LiDAR-inertial SLAM method for solid-state LiDARs. At the front-end, the raw point cloud is segmented to filter dynamic points in preprocessing process. Subsequently, features are extracted using a combination of Principal Component Analysis (PCA) and Mean Clustering to reduce redundant points and improve data processing efficiency. At the back-end, a hierarchical fusion method is proposed to improve the accuracy of the system by fusing the feature information to iteratively optimize the LiDAR frames, and then adaptively selecting the LiDAR keyframes to be fused with the IMU. The proposed method is extensively evaluated using a Livox Avia solid-state LiDAR collecting datasets on two different platforms. In experiments, the end-to-end error is reduced by 35% and the single-frame operational efficiency is improved by 12% compared to LiLi-OM.

TOWARDS AUTONOMOUS HIGH-DEFINITION MAP CONSTRUCTION

Abstract. This paper presents a GPS constrained SLAM solution, which adds reliable GPS observations as key frames to the state-of-the-art SLAM algorithm Lightweight and Ground-Optimized Lidar Odometry and Mapping (LeGO-LOAM). As the GPS has a much higher frequency than that of lidar, we first assign each lidar frame with GPS time, then every 5 seconds, a reliable GPS observation is inserted as a key frame to the pose graph, then optimizes the pose estimation and updates the old key frames. We test our method with two platforms in real driving scene, and compare its performance with LeGO-LOAM, where LeGO-LOAM cannot acquire realtime efficiency. The difference of lidar odometry of our solution with compare to RTK-GPS is less than 0.4m, with the globally referenced map can be used for relocalization.

Tightly coupled laser-inertial pose estimation and map building based on B-spline curves

Simultaneous localization and mapping (SLAM) plays a key role in 3D environment modeling and mobile robot environment perception. However, the traditional discrete-time Laser-inertial SLAM methods are not robust due to the imbalanced registration steps between a single LiDAR frame and the global map. This paper proposes a tightly coupled laser-inertial pose estimation and map building method that uses B-spline curves to represent continuous-time trajectory and achieve high robustness of the registration steps. To ensure efficiency, the proposed method separates the SLAM task into an odometer module and a mapping module. The odometer module performs a coarse pose estimation, while the mapping module performs a fine one and builds a global map with 3D LiDAR points. B-spline curves are utilized to integrate both IMU measurement constraints and LiDAR point constraints in the proposed mapping module, which can enhance the association of consecutive LiDAR frames in the optimization step. Besides, the explicit expression of the Jacobi matrix derivation for B-spline-based laser residuals is also introduced to furtherly improve the computation efficiency. Both indoor and outdoor experiments are conducted on a self-collected dataset and a public dataset. Experimental results show that the proposed method can achieve superior performance than the baseline method LIO-mapping.

LidarMultiNet: Towards a Unified Multi-Task Network for LiDAR Perception

LiDAR-based 3D object detection, semantic segmentation, and panoptic segmentation are usually implemented in specialized networks with distinctive architectures that are difficult to adapt to each other. This paper presents LidarMultiNet, a LiDAR-based multi-task network that unifies these three major LiDAR perception tasks. Among its many benefits, a multi-task network can reduce the overall cost by sharing weights and computation among multiple tasks. However, it typically underperforms compared to independently combined single-task models. The proposed LidarMultiNet aims to bridge the performance gap between the multi-task network and multiple single-task networks. At the core of LidarMultiNet is a strong 3D voxel-based encoder-decoder architecture with a Global Context Pooling (GCP) module extracting global contextual features from a LiDAR frame. Task-specific heads are added on top of the network to perform the three LiDAR perception tasks. More tasks can be implemented simply by adding new task-specific heads while introducing little additional cost. A second stage is also proposed to refine the first-stage segmentation and generate accurate panoptic segmentation results. LidarMultiNet is extensively tested on both Waymo Open Dataset and nuScenes dataset, demonstrating for the first time that major LiDAR perception tasks can be unified in a single strong network that is trained end-to-end and achieves state-of-the-art performance. Notably, LidarMultiNet reaches the official 1 place in the Waymo Open Dataset 3D semantic segmentation challenge 2022 with the highest mIoU and the best accuracy for most of the 22 classes on the test set, using only LiDAR points as input. It also sets the new state-of-the-art for a single model on the Waymo 3D object detection benchmark and three nuScenes benchmarks.

Traversability analysis for autonomous driving in complex environment: A LiDAR‐based terrain modeling approach

AbstractFor autonomous driving, traversability analysis is one of the most basic and essential tasks. In this paper, we propose a novel LiDAR‐based terrain modeling approach, which could output stable, complete, and accurate terrain models and traversability analysis results. As terrain is an inherent property of the environment that does not change with different view angles, our approach adopts a multiframe information fusion strategy for terrain modeling. Specifically, a normal distributions transform mapping approach is adopted to accurately model the terrain by fusing information from consecutive LiDAR frames. Then the spatial‐temporal Bayesian generalized kernel inference and bilateral filtering are utilized to promote the stability and completeness of the results while simultaneously retaining the sharp terrain edges. Based on the terrain modeling results, the traversability of each region is obtained by performing geometric connectivity analysis between neighboring terrain regions. Experimental results show that the proposed method could run in real‐time and outperforms state‐of‐the‐art approaches.

MASS: Mobility-Aware Sensor Scheduling of Cooperative Perception for Connected Automated Driving

Timely and reliable environment perception is fundamental to safe and efficient automated driving. However, the perception of standalone intelligence inevitably suffers from occlusions. A new paradigm, Cooperative Perception (CP), comes to the rescue by sharing sensor data from another perspective, i.e., from a cooperative vehicle (CoV). Due to the limited communication bandwidth, it is essential to schedule the most beneficial CoV, considering both the viewpoints and communication quality. Existing methods rely on the exchange of meta-information, such as visibility maps, to predict the perception gains from nearby vehicles, which induces extra communication and processing overhead. In this paper, we propose a new approach, learning while scheduling, for distributed scheduling of CP. The solution enables CoVs to predict the perception gains using past observations, leveraging the temporal continuity of perception gains. Specifically, we design a mobility-aware sensor scheduling (MASS) algorithm based on the restless multi-armed bandit (RMAB) theory to maximize the expected average perception gain. An upper bound on the expected average learning regret is proved, which matches the lower bound of any online algorithm up to a logarithmic factor. Extensive simulations are carried out on realistic traffic traces. The results show that the proposed MASS algorithm achieves the best average perception gain and improves recall by up to 3.1 percentage points compared to other learning-based algorithms. Finally, a case study on a trace of LiDAR frames qualitatively demonstrates the superiority of adaptive exploration, the key element of the MASS algorithm.

Conception of a High-Level Perception and Localization System for Autonomous Driving

This paper describes the conception of a high level, compact, scalable, and long autonomy perception and localization system for autonomous driving applications. Our benchmark is composed of a high resolution lidar (128 channels), a stereo global shutter camera, an inertial navigation system, a time server, and an embedded computer. In addition, in order to acquire data and build multi-modal datasets, this system embeds two perception algorithms (RBNN detection, DCNN detection) and one localization algorithm (lidar-based localization) to provide real-time advanced information such as object detection and localization in challenging environments (lack of GPS). In order to train and evaluate the perception algorithms, a dataset is built from 10,000 annotated lidar frames from various drives carried out under different weather conditions and different traffic and population densities. The performances of the three algorithms are competitive with the state-of-the-art. Moreover, the processing time of these algorithms are compatible with real-time autonomous driving applications. By providing directly accurate advanced outputs, this system might significantly facilitate the work of researchers and engineers with respect to planning and control modules. Thus, this study intends to contribute to democratizing access to autonomous vehicle research platforms.

Semantic Lidar-Inertial SLAM for Dynamic Scenes

Over the past few years, many impressive lidar-inertial SLAM systems have been developed and perform well under static scenes. However, most tasks are under dynamic environments in real life, and the determination of a method to improve accuracy and robustness poses a challenge. In this paper, we propose a semantic lidar-inertial SLAM approach with the combination of a point cloud semantic segmentation network and lidar-inertial SLAM LIO mapping for dynamic scenes. We import an attention mechanism to the PointConv network to build an attention weight function to improve the capacity to predict details. The semantic segmentation results of the point clouds from lidar enable us to obtain point-wise labels for each lidar frame. After filtering the dynamic objects, the refined global map of the lidar-inertial SLAM sytem is clearer, and the estimated trajectory can achieve a higher precision. We conduct experiments on an UrbanNav dataset, whose challenging highway sequences have a large number of moving cars and pedestrians. The results demonstrate that, compared with other SLAM systems, the accuracy of trajectory can be improved to different degrees.

Real-Time Fast Channel Clustering for LiDAR Point Cloud

LiDAR sensors can produce point clouds with precise 3D depth information that is essential for autonomous vehicles and robotic systems. As a perception task, point cloud clustering algorithms can be applied to segment the points into object instances. In this brief, we propose a novel, hardware-friendly fast channel clustering (FCC) algorithm that achieves state-of-the-art performance when evaluated using KITTI panoptic segmentation benchmark. Furthermore, an efficient, pipeline hardware architecture is proposed to implement the FCC algorithm on an FPGA. Experiments show that the hardware design can process each LiDAR frame with 64 channels, 2048 horizontal resolution at various point sparsity in 1.93 ms, which is more than 471.5 times faster than running on the CPU. The code will be released to the public via GitHub.

An Effective Camera-to-Lidar Spatiotemporal Calibration Based on a Simple Calibration Target.

In this contribution, we present a simple and intuitive approach for estimating the exterior (geometrical) calibration of a Lidar instrument with respect to a camera as well as their synchronization shifting (temporal calibration) during data acquisition. For the geometrical calibration, the 3D rigid transformation of the camera system was estimated with respect to the Lidar frame on the basis of the establishment of 2D to 3D point correspondences. The 2D points were automatically extracted on images by exploiting an AprilTag fiducial marker, while the detection of the corresponding Lidar points was carried out by estimating the center of a custom-made retroreflective target. Both AprilTag and Lidar reflective targets were attached to a planar board (calibration object) following an easy-to-implement set-up, which yielded high accuracy in the determination of the center of the calibration target. After the geometrical calibration procedure, the temporal calibration was carried out by matching the position of the AprilTag to the corresponding Lidar target (after being projected onto the image frame), during the recording of a steadily moving calibration target. Our calibration framework was given as an open-source software implemented in the ROS platform. We have applied our method to the calibration of a four-camera mobile mapping system (MMS) with respect to an integrated Velodyne Lidar sensor and evaluated it against a state-of-the-art chessboard-based method. Although our method was a single-camera-to-Lidar calibration approach, the consecutive calibration of all four cameras with respect to the Lidar sensor yielded highly accurate results, which were exploited in a multi-camera texturing scheme of city point clouds.

Efficient Camera–LiDAR Calibration Using Accumulated LiDAR Frames

In autonomous driving, the camera and LiDAR are complementary to each other through their strengths, and many autonomous vehicles and robot recognition systems utilize cameras and LiDAR. In order to effectively fuse the data of the camera and LiDAR attached to different positions and angles, we must perform camera–LiDAR extrinsic calibration. Most existing camera–LiDAR calibration methods infer the results by constructing a camera–LiDAR feature pair using features acquired from a single frame. It has the disadvantage that it is challenging to draw results. In this paper, we used sequential LiDAR data to extract features by accumulating LiDAR frames effectively. By using the location information between the accumulated frame (global) and the single frame (local) to convert the features detected from the global to the local, the shortcomings of LiDAR’s single frame with few features were supplemented. Methods for detecting feature points in LiDAR are described step-by-step, and the advantages and excellence of this camera–LiDAR calibration system are shown through quantitative/qualitative evaluation methods. As a result, the proposed system outputs superior performance compared to other systems.

Canadian Adverse Driving Conditions dataset

The Canadian Adverse Driving Conditions (CADC) dataset was collected with the Autonomoose autonomous vehicle platform, based on a modified Lincoln MKZ. The dataset, collected during winter within the Region of Waterloo, Canada, is the first autonomous driving dataset that focuses on adverse driving conditions specifically. It contains 7,000 frames of annotated data from 8 cameras (Ximea MQ013CG-E2), lidar (VLP-32C), and a GNSS+INS system (Novatel OEM638), collected through a variety of winter weather conditions. The sensors are time synchronized and calibrated with the intrinsic and extrinsic calibrations included in the dataset. Lidar frame annotations that represent ground truth for 3D object detection and tracking have been provided by Scale AI.

INDOOR LIDAR RELOCALIZATION BASED ON DEEP LEARNING USING A 3D MODEL

Abstract. Indoor localization, navigation and mapping systems highly rely on the initial sensor pose information to achieve a high accuracy. Most existing indoor mapping and navigation systems cannot initialize the sensor poses automatically and consequently these systems cannot perform relocalization and recover from a pose estimation failure. For most indoor environments, a map or a 3D model is often available, and can provide useful information for relocalization. This paper presents a novel relocalization method for lidar sensors in indoor environments to estimate the initial lidar pose using a CNN pose regression network trained using a 3D model. A set of synthetic lidar frames are generated from the 3D model with known poses. Each lidar range image is a one-channel range image, used to train the CNN pose regression network from scratch to predict the initial sensor location and orientation. The CNN regression network trained by synthetic range images is used to estimate the poses of the lidar using real range images captured in the indoor environment. The results show that the proposed CNN regression network can learn from synthetic lidar data and estimate the pose of real lidar data with an accuracy of 1.9 m and 8.7 degrees.

Real-Time Streaming Point Cloud Compression for 3D LiDAR Sensor Using U-Net

Point cloud data from LiDAR sensors is the currently the basis of most L4 autonomous driving systems. Sharing and storing point clouds will also be important for future applications, such as accident investigation or V2V/V2X networks. Due to the huge volume of data involved, storing point clouds collected over long periods of time and transmitting point clouds in real-time are difficult tasks, making compression an indispensable step before storing or transmitting. Previous streaming point cloud compression methods, such as octree compression or video compression-based approaches, have difficulty compressing this data in real-time into very small volumes with low information loss. To reduce temporal redundancy efficiently and rapidly, in this paper we propose a real-time streaming point cloud data compression method using U-net. By utilizing raw packet data from LiDAR sensors, we can store 3D point cloud information losslessly in a 2D matrix, and convert streaming point cloud data into a video-like format. By designating some frames as reference frames and then using U-net to interpolate the remaining LiDAR frames, we can greatly reduce temporal redundancy. Our use of U-net was inspired by a video interpolation approach employed in another study. Noise in LiDAR data is a big issue which significantly affects network training and compression results. In this paper, we propose a padding strategy to alleviate the negative impact of this noise. As a result of these improvements, our proposed method can outperform octree compression, MPEG-based compression and our previously proposed SLAM-based compression method.

Relative continuous-time SLAM

Appearance-based techniques for simultaneous localization and mapping (SLAM) have been highly successful in assisting robot-motion estimation; however, these vision-based technologies have long assumed the use of imaging sensors with a global shutter, which are well suited to the traditional, discrete-time formulation of visual problems. In order to adapt these technologies to use scanning sensors, we propose novel methods for both outlier rejection and batch nonlinear estimation. Traditionally, the SLAM problem has been formulated in a single-privileged coordinate frame, which can become computationally expensive over long distances, particularly when a loop closure requires the adjustment of many pose variables. Recent discrete-time estimators have shown that a completely relative coordinate framework can be used to incrementally find a close approximation of the full maximum-likelihood solution in constant time. In order to use scanning sensors, we propose moving the relative coordinate formulation of SLAM into continuous time by estimating the velocity profile of the robot. We derive the relative formulation of the continuous-time robot trajectory and formulate an estimator using temporal basis functions. A motion-compensated outlier rejection scheme is proposed by using a constant-velocity model for the random sample consensus algorithm. Our experimental results use intensity imagery from a two-axis scanning lidar; due to the sensors’ scanning nature, it behaves similarly to a slow rolling-shutter camera. Both algorithms are validated using a sequence of 6880 lidar frames acquired over a 1.1 km traversal.

AEROTRIANGULATION SUPPORTED BY CAMERA STATION POSITION DETERMINED VIA PHYSICAL INTEGRATION OF LIDAR AND SLR DIGITAL CAMERA

Abstract. Nowadays lidar and photogrammetric surveys have been used together in many mapping procedures due to their complementary characteristics. Lidar survey is capable to promptly acquire reliable elevation information that is sometimes difficult via photogrammetric procedure. On the other hand, photogrammetric survey is easily able to get semantic information of the objects. Accessibility, availability, the increasing sensor size and quick image acquisition and processing are properties that have raised the use of SLR digital cameras in photogrammetry. Orthoimage generation is a powerful photogrammetric mapping procedure, where the advantages of the integration of lidar and image datasets are very well characterized. However, to perform this application both datasets must be within a common reference frame. In this paper, a procedure to have digital images positioned and oriented in the same lidar frame via a combination of direct and indirect georeferencing is studied. The SLR digital camera was physically connected with the lidar system to calculate the camera station's position in lidar frame. After that, the aerotriangulation supported by camera station's position is performed to get image's exterior orientation parameters (EOP).