3D LiDAR Research Articles

UPKD: Unsupervised pylon keypoint detection from 3D LiDAR data for autonomous UAV power inspection

The automatic extraction of inspection points for pylons is essential for intelligent Unmanned Aerial Vehicle (UAV) power inspections, especially in generating inspection route. However, current researches primarily focus on power scene perception and power component fault detection, with little attention paid to the inspection point detection. The primary reason are the lack of public pylon datasets for keypoint detection and the neglect of distinguish the inspection points from the keypoints used in feature extraction. Regarding that, this paper proposes a two-stage unsupervised pylon keypoint detection (UPKD) method to improve the efficiency of power inspections. In the first stage, the UPKD Network processes the point cloud to generate candidate keypoints, which comprises two main components: a data normalization module and an unsupervised keypoint detection network (UKD-Net). The data normalization module compresses information based on the symmetric structure of pylons, thereby reducing instability in inspection point detection. The UKD-Net incorporates a Point Transformer layer that uses self-attention mechanisms to extract features from the point cloud. In the second stage, a convex optimization strategy is applied to filter and acquire inspection points. These inspection points are then interconnected using a shortest-path strategy to generate the UAV inspection route. Our dataset is obtained using the Riegl VUX-1 laser measurement system and comprises 3,296 pylons of 10 types. Each pylon’s point cloud contains up to 25,000 points, with a high point density of 100 pts/m2. Extensive experiments show that the UPKD Network achieved state-of-the-art performance on our dataset, with repeatability achieving 90.39%, effectiveness (the ratio of effective keypoints to annotation points) achieving 69.95%, and completeness (the ratio of detected annotation points to keypoints) achieving 79.55%.

Adaptive robot localization in dynamic environments through self-learnt long-term 3D stable points segmentation

In field robotics, particularly in the agricultural sector, precise localization presents a challenge due to the constantly changing nature of the environment. Simultaneous Localization and Mapping algorithms can provide an effective estimation of a robot’s position, but their long-term performance may be impacted by false data associations. Additionally, alternative strategies such as the use of RTK-GPS can also have limitations, such as dependence on external infrastructure. To address these challenges, this paper introduces a novel stability scan filter. This filter can learn and infer the motion status of objects in the environment, allowing it to identify the most stable objects and use them as landmarks for robust robot localization in a continuously changing environment. The proposed method involves an unsupervised point-wise labelling of LiDAR frames by utilizing temporal observations of the environment, as well as a regression network, called Long-Term Stability Network (LTS-NET) to learn and infer 3D LiDAR points long-term motion status. Experiments demonstrate the ability of the stability scan filter to infer the motion stability of objects on a real agricultural long-term dataset. Results show that by only utilizing points belonging to long-term stable objects, the localization system exhibits reliable and robust localization performance for long-term missions compared to using the entire LiDAR frame points.

Calibration and validation of a multi-beam LiDAR onboard China Chang’e lunar probe for landing obstacle avoidance

For automatic exploration of specific areas with high scientific value on the moon and other planets, high-precision obstacle detection and hazard avoidance capabilities are required. Three-dimensional (3D) LiDAR can obtain a high-resolution 3D point cloud, offering significant advantages in landing obstacle avoidance. During the landing of the Chang’e lander, a laser 3D imaging sensor (L3DIS) is used to measure and generate accurate topographic maps of the candidate landing area in real-time to help find a safe landing zone. The L3DIS employs 16 beamlets split from a Gaussian laser beam and 16 channels of linear-array avalanche photodiodes within the same optical path and scans the object with a two-axis galvanometer in a field of view of 29°×33°. The calibration of the systematic errors and the performance of obstacle detection on this sensor are conducted in this paper. First, the instrument components and main error sources of the sensor are introduced, and the systematic errors are calibrated based on planar targets, indicating that the sensor’s accuracy is greater than 4 cm, and the corresponding obstacle detection accuracy is better than 12 cm. Second, the ground validation of obstacle detection is demonstrated, and the obtained point cloud is compared with that obtained by a terrestrial laser scanner, indicating that the sensor has good performance. Finally, the performance of onboard measurement and obstacle detection is analyzed, showing that the sensor can identify obvious craters and rocks, i.e., nine major craters with a maximum and minimum depth of 1.36 m and 0.16 m, respectively, and major rocks with a maximum and minimum height of 0.1 m and 0.05 m, respectively, and main landing area with slop less than 10° except for the edges of craters and rocks.

Pose Estimation Based on Bidirectional Visual–Inertial Odometry with 3D LiDAR (BV-LIO)

Due to the limitation of a single sensor such as only camera or only LiDAR, the Visual SLAM detects few effective features in the case of poor lighting or no texture. The LiDAR SLAM will also degrade in an unstructured environment and open spaces, which reduces the accuracy of pose estimation and the quality of mapping. In order to solve this problem, on account of the high efficiency of Visual odometry and the high accuracy of LiDAR odometry, this paper investigates the multi-sensor fusion of bidirectional visual–inertial odometry with 3D LiDAR for pose estimation. This method can couple the IMU with the bidirectional vision respectively, and the LiDAR odometry is obtained assisted by the bidirectional visual inertial. The factor graph optimization is constructed, which effectively improves the accuracy of pose estimation. The algorithm in this paper is compared with LIO-LOAM, LeGO-LOAM, VINS-Mono, and so on using challenging datasets such as KITTI and M2DGR. The results show that this method effectively improves the accuracy of pose estimation and has high application value for mobile robots.

MosViT: towards vision transformers for moving object segmentation based on Lidar point cloud

Abstract Moving object segmentation is fundamental for various downstream tasks in robotics and autonomous driving, providing crucial information for them. Effectively extracting spatial-temporal information from consecutive frames and addressing the scarcity of dataset is paramount for learning-based 3D LiDAR Moving Object Segmentation (LIDAR-MOS). In this work, we propose a novel deep neural network based on Vision Transformers (ViTs) to tackle this problem. We first validate the feasibility of Transformer networks for this task, offering an alternative to CNNs. Specifically, we utilize a dual-branch structure based on range-image data to extract spatial-temporal information from consecutive frames and fuse it using a motion-guided attention mechanism. Furthermore, we employ the ViT as the backbone, keeping its architecture unchanged from what is used for RGB images. This enables us to leverage pre-trained models from RGB images to improve results, addressing the issue of limited LIDAR point cloud data, which is cheaper compared to acquiring and annotating point cloud data. We validate the effectiveness of our approach on the LIDAR-MOS benchmark of SemanticKitti and achieve comparable results to methods that use CNNs on range image data. The source code and trained models are available at https://github.com/mafangniu/MOSViT.git.

Research on agricultural monitoring and progressive map creation in the context of ground-air coordination

Abstract By combining network RTK technology, 2D LiDAR, and UAV, this study investigates how to achieve agricultural monitoring effectively. A collaborative agricultural monitoring scheme was designed to accurately and comprehensively gather agricultural information in response to the demand for targeted sequential monitoring at specified agricultural locations. To establish a low-cost and low computational power-dependent 3D map of obstacle point clouds, a progressive 3D mapping scheme based on 2D LiDAR was designed and implemented. The effectiveness of the design was validated through simulation experiments and partly through real-world testing.

Drivable Region Completion via a 3D LiDAR

Drivable Region Completion via a 3D LiDAR

Seismic Deformation Analysis of Precast Concrete Pipe Hybrid Based on 3D LiDAR and Unmanned Aerial Vehicle Photogrammetry

Seismic Deformation Analysis of Precast Concrete Pipe Hybrid Based on 3D LiDAR and Unmanned Aerial Vehicle Photogrammetry

Real-time testing of AI enabled automatic emergency braking system for ADAS vehicle using 3D point cloud and precise depth information

At the forefront of automobile safety technology, Automatic Emergency Braking (AEB) represents a major advancement in collision avoidance systems. The cutting-edge technology provides an additional layer of security at pivotal times, making it a vital part of the changing landscape of vehicle safety. This research introduces an effective system for automatic emergency braking in ADAS-equipped or Autonomous vehicles using a combination of 3d lidar and stereo vision camera for a swift and robust system in the vehicle that is faster than human drivers in cases of unexpected emergencies. Utilizing the power of clustering algorithms on 3d point clouds and state-of-the-art computer vision algorithms on an RGB image mapped to a depth frame from the stereo vision camera, the system as a whole provides a comprehensive system adding to the safety of the vehicle and the passengers. Further, the efficiency of the system is studied based on various parameters. The data from an external inertial measurement unit is also utilized to derive results that support the claims of the study. The system has been developed and implemented on a passenger car that has been modified into an electric vehicle and further tested in real-world traffic conditions in autonomous driving mode. The findings of the study were having exceptionally good precision in split-second decision-making in emergency maneuvers.

Assessing Normalized Difference Vegetation Index as a proxy of urban greenspace exposure

The Normalized Difference Vegetation Index (NDVI) is a popular proxy of urban greenspace (UGS). However, it’s unclear how NDVI approximates physical characteristics of UGS in the context of urban health studies, causing ambiguities in translating research findings to UGS management. Therefore, we collected data from Landsat and MODIS satellites and Lidar 3D scans in New York City as of circa 2013, and we evaluated linear and non-linear relationships between NDVI and UGS characteristics. We found that: (1) % UGS was the best predicted UGS characteristic by NDVI (R2: 0.35–0.90, varies by data source and unit of analysis), whereas average tree height was the worst (R2: 0.09–0.46). The predictive power on % canopy cover, tree density, and crown volume density was in a similar range (R2: 0.10–0.67). Prediction improved with finer-resolution NDVI sources and larger units of analysis at the cost of losing useful variations; (2) There was a saturation effect where a linear relationship underestimated UGS characteristics in areas of high NDVI. These areas typically had NAIP-NDVI greater than the range of 0.08–0.25, Landsat-NDVI greater than the range of 0.42–0.65, and MODIS-NDVI greater than the range of 0.49–0.75; (3) Smaller absolute errors from a linear NDVI-UGS relationship were often found in more developed locations. We therefore recommend NDVI as a reliable predictor of UGS coverage and its use in longitudinal studies. Future studies should also consider fine resolution land cover maps and Lidar, which are increasingly available to derive detailed UGS characteristics.

Unveiling gap acceptance behaviour during lane change with EDIV data: A deep dive into driving behaviour on expressway using a three level mixed effect linear regression approach

Lane change has a potential significance in road safety. Gap acceptance phenomena serves as a primary and critical phase in lane change maneuver. This study aims to investigate the gap acceptance behaviour of drivers during lane changes on expressways, with a focus on understanding how various factors influence drivers' decisions to change lanes. An extensive dataset collected through various sensors tailored for expressway driving, known as the ‘Expressway Drive: Instrumented Vehicle (EDIV) Dataset’ is utilized. Driving data from 59 drivers covering a distance of around 4000 km was used in the current study. Total 2578 lane changing events are identified through computing lateral deviations measured through 3D LiDAR sensor. Substantial differences are observed within the groups in primary analysis which suggest that lane-change direction significantly affect gap acceptance. To effectively manage both intra- and inter-cluster variances, this study employs two separate three levels mixed-effects linear models. These models account for the interdependence of gap acceptance characteristics within individual drivers and for different directions of lane changes by incorporating random effects. Furthermore, these models examine relationships between lead/ lag gap acceptance and the various influencing factors as fixed effects. It was found that factors such as speed of the subject vehicle, gap position, relative speeds, and surrounding vehicle types had influence on gap acceptance during lane changes on expressways. The insights gained from this study could inform the development of advanced driver assistance systems (ADAS) as well as development of autonomous vehicles, contributing to improved road safety and traffic flow management in high-speed environments.

A novel 3D LiDAR deep learning approach for uncrewed vehicle odometry.

Self-localization and pose registration are required for sound operation of next generation autonomous vehicles under uncertain environments. Thus, precise localization and mapping are crucial tasks in odometry, planning and other downstream processing. In order to reduce information loss in preprocessing, we propose leveraging LiDAR-based localization and mapping (LOAM) with point cloud-based deep learning instead of convolutional neural network (CNN) based methods that require cylindrical projection. The normal distribution transform (NDT) algorithm is then used to refine the former coarse pose estimation from the deep learning model. The results demonstrate that the proposed method is comparable in performance to recent benchmark studies. We also explore the possibility of using Product Quantization to improve NDT internal neighborhood searching by using high-level features as fingerprints.

Cascaded-prism multi-mode beam scanning method for three-dimensional imaging lidar

Light detection and ranging (lidar) has emerged as an indispensable approach to three-dimensional (3D) perception, which probably suffers from performance limitations with traditional beam steering devices. In this paper, we investigate a cascaded-prism beam scanning method to enhance the versatility of 3D imaging lidar systems. The cascaded-prism-based 3D lidar architecture is theoretically developed with an emphasis on a rigorous beam scan model. By exploiting the additional flexibility of cascaded prisms, the lidar can achieve beam scanning through prism rotation in either step-motion or constant-speed mode, which exhibits superior agility and adaptability in multi-mode pattern analysis. Moreover, the cascaded-prism beam scanning lidar is demonstrated with a 3D imaging performance evaluation in terms of field of view, angular resolution and sampling density. It proves that the cascaded-prism beam scanner can offer lidar systems with flexible and configurable 3D imaging capability while balancing between a wide field of view and high angular resolution.

Design and Experiment of Ordinary Tea Profiling Harvesting Device Based on Light Detection and Ranging Perception

Due to the complex shape of the tea tree canopy and the large undulation of a tea garden terrain, the quality of fresh tea leaves harvested by existing tea harvesting machines is poor. This study proposed a tea canopy surface profiling method based on 2D LiDAR perception and investigated the extraction and fitting methods of canopy point clouds. Meanwhile, a tea profiling harvester prototype was developed and field tests were conducted. The tea profiling harvesting device adopted a scheme of sectional arrangement of multiple groups of profiling tea harvesting units, and each unit sensed the height information of its own bottom canopy area through 2D LiDAR. A cross-platform communication network was established, enabling point cloud fitting of tea plant surfaces and accurate estimation of cutter profiling height through the RANSAC algorithm. Additionally, a sensing control system with multiple execution units was developed using rapid control prototype technology. The results of field tests showed that the bud leaf integrity rate was 84.64%, the impurity rate was 5.94%, the missing collection rate was 0.30%, and the missing harvesting rate was 0.68%. Furthermore, 89.57% of the harvested tea could be processed into commercial tea, with 88.34% consisting of young tea shoots with one bud and three leaves or fewer. All of these results demonstrated that the proposed device effectively meets the technical standards for machine-harvested tea and the requirements of standard tea processing techniques. Moreover, compared to other commercial tea harvesters, the proposed tea profiling harvesting device demonstrated improved performance in harvesting fresh tea leaves.

Semantic enrichment of BIM with IndoorGML for quadruped robot navigation and automated 3D scanning

Planning scan routes with prior knowledge can improve scan data quality and completeness. This paper presents a BIM-enabled approach to optimize quadruped robot navigation for automated 3D scanning. The BIM data schema is enriched with IndoorGML, integrating building geometry with spatial data to establish an indoor navigation model describing multi-scale spatial topological networks. This navigation model, which includes an enhanced greedy algorithm, optimizes quadruped robot scanning positions and traversal sequences. The scan planning optimization outperforms existing heuristic algorithms in computational efficiency, coverage, and scan point count. The BIM-enabled approach is validated on ROS and in real-world conditions with a 3D LiDAR sensor integrated with a quadruped robot. The robotic scans achieve visible coverage of 70–90% of the structure, with a fluctuation of 0.006–0.021 mm compared to traditional laser scans. The findings demonstrate robotic scans as a viable way of obtaining complete and accurate point clouds, reducing human effort in traditional scanning.

Enhancing point cloud data fusion through 2D thermal infrared camera and 2D lidar scanning

Sensor data fusion from multiple sources plays a pivotal role in advancing our understanding of real-world phenomena. This paper focuses on the integration of a thermal imager and a lidar to simultaneously capture four-dimensional (XYZT) space coordinates and temperatures. The fusion of data relies on establishing correspondence between overlapping lines in the lidar’s 3D point cloud and the infrared (IR) camera’s 2D thermal map. The presented combination method offers several key advantages. Firstly, it simplifies the calibration process. Furthermore, it can achieve near real-time performance, enabling prompt data analysis. Moreover, the method demonstrates high accuracy in capturing spatial information and temperature measurements. The thermal IR camera used in the experiment features a 1280 × 1024 pixel cooled indium antimonide (InSb) detector, operating in the short wavelength infrared (MWIR) spectral range of 1.5 to 5.5 µm. It provides temperature readings with ± 1 K accuracy and sub-30 mK resolution. On the other hand, the lidar system delivers centimeter-level precision within a range of approximately 1 to 12 m, even in well-lit outdoor environments. The experimental results validate the effectiveness of our combined system, successfully generating high-quality four-dimensional (4D) point clouds. The integration of the thermal IR camera and profilometer presents an intriguing tool with significant potential in various industrial applications. For instance, it can be employed to accurately assess gas leaks, enhance metallurgical processes, and improve safety on construction sites.

Attention-guided LiDAR segmentation and odometry using image-to-point cloud saliency transfer

LiDAR odometry estimation and 3D semantic segmentation are crucial for autonomous driving, which has achieved remarkable advances recently. However, these tasks are challenging due to the imbalance of points in different semantic categories for 3D semantic segmentation and the influence of dynamic objects for LiDAR odometry estimation, which increases the importance of using representative/salient landmarks as reference points for robust feature learning. To address these challenges, we propose a saliency-guided approach that leverages attention information to improve the performance of LiDAR odometry estimation and semantic segmentation models. Unlike in the image domain, only a few studies have addressed point cloud saliency information due to the lack of annotated training data. To alleviate this, we first present a universal framework to transfer saliency distribution knowledge from color images to point clouds, and use this to construct a pseudo-saliency dataset (i.e. FordSaliency) for point clouds. Then, we adopt point cloud based backbones to learn saliency distribution from pseudo-saliency labels, which is followed by our proposed SalLiDAR module. SalLiDAR is a saliency-guided 3D semantic segmentation model that integrates saliency information to improve segmentation performance. Finally, we introduce SalLONet, a self-supervised saliency-guided LiDAR odometry network that uses the semantic and saliency predictions of SalLiDAR to achieve better odometry estimation. Our extensive experiments on benchmark datasets demonstrate that the proposed SalLiDAR and SalLONet models achieve state-of-the-art performance against existing methods, highlighting the effectiveness of image-to-LiDAR saliency knowledge transfer. Source code will be available at https://github.com/nevrez/SalLONet

3D ToF LiDAR for Mobile Robotics in Harsh Environments: A Review

Over the past decade, the use of 3D Time-of-Flight (ToF) LiDARs in mobile robotics has grown rapidly. Based on our accumulation of relevant research, this paper systematically reviews and analyzes the use of 3D ToF LiDARs for mobile robotics under harsh conditions such as adverse weather, GPS-denied, and highly dynamic environments for both research and industrial applications. The former include LiDAR data processing in adverse weather, object detection, and autonomous navigation. The latter encompasses autonomous driving, service robotics, and public health crises applications. We hope that our efforts can effectively provide readers with a reference based on our hands-on experiences and promote the deployment of existing mature technologies in real-world systems.

Accurate 3D LiDAR SLAM System Based on Hash Multi-Scale Map and Bidirectional Matching Algorithm.

Simultaneous localization and mapping (SLAM) is a hot research area that is widely required in many robotics applications. In SLAM technology, it is essential to explore an accurate and efficient map model to represent the environment and develop the corresponding data association methods needed to achieve reliable matching from measurements to maps. These two key elements impact the working stability of the SLAM system, especially in complex scenarios. However, previous literature has not fully addressed the problems of efficient mapping and accurate data association. In this article, we propose a novel hash multi-scale (H-MS) map to ensure query efficiency with accurate modeling. In the proposed map, the inserted map point will simultaneously participate in modeling voxels of different scales in a voxel group, enabling the map to represent objects of different scales in the environment effectively. Meanwhile, the root node of the voxel group is saved to a hash table for efficient access. Secondly, considering the one-to-many (1 ×103 order of magnitude) high computational data association problem caused by maintaining multi-scale voxel landmarks simultaneously in the H-MS map, we further propose a bidirectional matching algorithm (MSBM). This algorithm utilizes forward-reverse-forward projection to balance the efficiency and accuracy problem. The proposed H-MS map and MSBM algorithm are integrated into a completed LiDAR SLAM (HMS-SLAM) system. Finally, we validated the proposed map model, matching algorithm, and integrated system on the public KITTI dataset. The experimental results show that, compared with the ikd tree map, the H-MS map model has higher insertion and deletion efficiency, both having O(1) time complexity. The computational efficiency and accuracy of the MSBM algorithm are better than that of the small-scale priority matching algorithm, and the computing speed of the MSBM achieves 49 ms/time under a single CPU thread. In addition, the HMS-SLAM system built in this article has also reached excellent performance in terms of mapping accuracy and memory usage.

Precision Inter-Row Relative Positioning Method by Using 3D LiDAR in Planted Forests and Orchards

Precision Inter-Row Relative Positioning Method by Using 3D LiDAR in Planted Forests and Orchards