Lidar Fusion Research Articles

A Reliable Robot Localization Method Using LiDAR and GNSS Fusion Based on a Two-Step Particle Adjustment Strategy

A Reliable Robot Localization Method Using LiDAR and GNSS Fusion Based on a Two-Step Particle Adjustment Strategy

Moving Object Segmentation Network for Multiview Fusion of Vehicle-Mounted LiDAR

Moving Object Segmentation Network for Multiview Fusion of Vehicle-Mounted LiDAR

Strong Robust LIO System Based on Event Camera Assistance

Abstract. At present, the LiDAR system can achieve high precision and is extensively used for indoor and outdoor mobile positioning and mapping. However, LiDAR systems still face issues in cluttered environments where strong features are absent, leading to a degradation of the LiDAR-based solution. When the carrier movement involves high-speed or prolonged exposure to the mirror wall, it can cause severe degradation issues or even positioning failures in the laser slam system. Event cameras are vision sensors inspired by biology that exhibit strong robustness in high dynamic and low texture environments, potentially leading to better performance in such environments. However, some issues with event cameras remain unresolved. In this paper, a multi-source fusion method based on EVIO, LIO and IMU trajectory layer post-processing method is proposed, which will fully consider the robustness of event camera in high dynamic environment and the high precision advantage of Lidar in conventional environment, and use an algorithm based on normalized uncertainty. The elastic multi-source fusion of event camera and LiDAR is realized and tested in realde environment. Experimental results show that the proposed algorithm can effectively improve the accuracy of event camera and LiDAR. Compared to the current more advanced algorithms, the proposed algorithm effectively addresses the problem of LIO in degraded environments. Additionally, it mitigates the scale inaccuracy and divergence of the EVIO trajectory to some extent. Compared to LIO, the algorithm can reduce the maximum position error by approximately 30% and increase the overall position accuracy by 32%. Additionally, it can significantly constrain the divergence of errors in the Y direction, improving its accuracy by about 75% and 65% compared to the LIO and EVIO algorithms, respectively.

Optimized Factor Graph for Tightly-Coupled LiDAR/IMU Localization in Underground Parking Garages

Abstract. To address the challenge of intelligent vehicle localization in underground parking structures due to the loss of GNSS signals, this paper introduces a method to address this issue by developing a novel localization framework known as GF-LIO, which denotes a tightly-coupled fusion of LiDAR and IMU data, innovatively combining the Interactive Extended State Kalman Filter (IESKF) with a factor graph to enhance the localization process and solve the problem of GNSS signal loss in underground parking lots. The GF-LIO model commences with a strategic feature selection process, facilitated by a greedy algorithm that prioritizes environmental cues within the point cloud data. This method effectively filters out redundant features, thereby enhancing the saliency of retained features and subsequently improving the robustness of the localization process. Following feature selection, the model integrates LiDAR and IMU measurements utilizing the IESKF algorithm, ensuring a cohesive fusion of sensor data and bolstering attitude estimation accuracy. The culmination of the GF-LIO framework involves factor graph optimization, a sophisticated technique that synthesizes LiDAR odometry, IMU pre-integration factors, and loop closure detection factors. This optimization step enhances the overall precision and consistency of the localization process, resulting in superior performance compared to existing methodologies. Experimental evaluations conducted within underground parking environments corroborate the efficacy of the GF-LIO model. Comparative analyses against established approaches such as A-LOAM, LeGO-LOAM, LIO-LOAM, and FAST-LIO demonstrate a notable performance improvement exceeding 12.53%. The proposed model adeptly integrates domain-specific environmental characteristics with multi-sensor data, thereby facilitating precise localization and map construction tasks for intelligent vehicle navigating within the intricate confines of subterranean parking structures.

Data Fusion of Roadside Camera, LiDAR, and Millimeter-Wave Radar

Data Fusion of Roadside Camera, LiDAR, and Millimeter-Wave Radar

A NOVEL HYBRID DEEP LEARNING APPROACH FOR 3D OBJECT DETECTION AND TRACKING IN AUTONOMOUS DRIVING

Recently Object detection and tracking using fusion of LiDAR and RGB camera for the autonomous vehicle environment is a challenging task. The existingworks initiates several object detection and tracking frameworks using ArtificialIntelligence (AI) algorithms. However, they were limited with high false positives and computation time issues thus lacking the performance of autonomousdriving environment. The existing issues are resolved by proposing HybridDeep Learning based Multi Object Detection and Tracking (HDL-MODT) using sensor fusion methods. The proposed work performs fusion of solid stateLiDAR, Pseudo LiDAR, and RGB camera for improving detection and trackingquality. At first, the multi-stage preprocessing is done in which noise removal isperformed using Adaptive Fuzzy Filter (A-Fuzzy). The pre-processed fused image is then provided for instance segmentation to reduce the classification andtracking complexity. For that, the proposed work adopts Lightweight GeneralAdversarial Networks (LGAN). The segmented image is provided for objectdetection and tracking using HDL. For reducing the complexity, the proposedwork utilized VGG-16 for feature extraction which forms the feature vectors.The features vectors are then provided for object detection using YOLOv4.Finally, the detected objects were tracked using Improved Unscented KalmanFilter (IUKF) and mapping the vehicles using time based mapping by considering their RFID, velocity, location, dimension and unique ID. The simulation ofthe proposed work is carried out using MATLAB R2020a simulation tool andperformance of the proposed work is compared with several metrics that showthat the proposed work outperforms than the existing works.

Research on Corn Leaf and Stalk Recognition and Ranging Technology Based on LiDAR and Camera Fusion.

Corn, as one of the three major grain crops in China, plays a crucial role in ensuring national food security through its yield and quality. With the advancement of agricultural intelligence, agricultural robot technology has gained significant attention. High-precision navigation is the basis for realizing various operations of agricultural robots in corn fields and is closely related to the quality of operations. Corn leaf and stalk recognition and ranging are the prerequisites for achieving high-precision navigation and have attracted much attention. This paper proposes a corn leaf and stalk recognition and ranging algorithm based on multi-sensor fusion. First, YOLOv8 is used to identify corn leaves and stalks. Considering the large differences in leaf morphology and the large changes in field illumination that lead to discontinuous identification, an equidistant expansion polygon algorithm is proposed to post-process the leaves, thereby increasing the average recognition completeness of the leaves to 86.4%. Secondly, after eliminating redundant point clouds, the IMU data are used to calculate the confidence of the LiDAR and depth camera ranging point clouds, and point cloud fusion is performed based on this to achieve high-precision ranging of corn leaves. The average ranging error is 2.9 cm, which is lower than the measurement error of a single sensor. Finally, the stalk point cloud is processed and clustered using the FILL-DBSCAN algorithm to identify and measure the distance of the same corn stalk. The algorithm combines recognition accuracy and ranging accuracy to meet the needs of robot navigation or phenotypic measurement in corn fields, ensuring the stable and efficient operation of the robot in the corn field.

LOFF: LiDAR and Optical Flow Fusion Odometry

LOFF: LiDAR and Optical Flow Fusion Odometry

Object Detection and Information Perception by Fusing YOLO-SCG and Point Cloud Clustering.

Robots need to sense information about the external environment before moving, which helps them to recognize and understand their surroundings so that they can plan safe and effective paths and avoid obstacles. Conventional algorithms using a single sensor cannot obtain enough information and lack real-time capabilities. To solve these problems, we propose an information perception algorithm with vision as the core and the fusion of LiDAR. Regarding vision, we propose the YOLO-SCG model, which is able to detect objects faster and more accurately. When processing point clouds, we integrate the detection results of vision for local clustering, improving both the processing speed of the point cloud and the detection effectiveness. Experiments verify that our proposed YOLO-SCG algorithm improves accuracy by 4.06% and detection speed by 7.81% compared to YOLOv9, and our algorithm excels in distinguishing different objects in the clustering of point clouds.

Multi-sensor feature fusion for vehicle detection based on the fuzzy longest common subsequence

To mitigate the impact of sensor misdetection and omission on the multi-sensor feature fusion algorithm, this paper proposes a vehicle detection method for multi-sensor feature fusion based on the fuzzy longest common subsequence algorithm. Firstly, the lightweight YOLOv4 algorithm is utilized to obtain image features. Secondly., an adaptive threshold-based clustering method processes point cloud data, extracting trajectory information from both image and point cloud sources. Then, an enhanced longest common subsequence algorithm, incorporating a fuzzy membership function, is introduced to assess the similarity between the lidar target trajectory and the camera target trajectory. Next, the fusion of lidar and camera detection results is carried out based on the computed similarity scores. Ultimately, experimental validation affirms that this algorithm significantly improves vehicle detection accuracy. This enhancement contributes to a more robust and reliable vehicle environment sensing system.

A method for measuring banana pseudo-stem phenotypic parameters based on handheld mobile LiDAR and IMU fusion.

Diameter and height are crucial morphological parameters of banana pseudo-stems, serving as indicators of the plant's growth status. Currently, in densely cultivated banana plantations, there is a lack of applicable research methods for the scalable measurement of phenotypic parameters such as diameter and height of banana pseudo-stems. This paper introduces a handheld mobile LiDAR and Inertial Measurement Unit (IMU)-fused laser scanning system designed for measuring phenotypic parameters of banana pseudo-stems within banana orchards. To address the challenges posed by dense canopy cover in banana orchards, a distance-weighted feature extraction method is proposed. This method, coupled with Lidar-IMU integration, constructs a three-dimensional point cloud map of the banana plantation area. To overcome difficulties in segmenting individual banana plants in complex environments, a combined segmentation approach is proposed, involving Euclidean clustering, Kmeans clustering, and threshold segmentation. A sliding window recognition method is presented to determine the connection points between pseudo-stems and leaves, mitigating issues caused by crown closure and heavy leaf overlap. Experimental results in banana orchards demonstrate that, compared with manual measurements, the mean absolute errors and relative errors for banana pseudo-stem diameter and height are 0.2127 cm (4.06%) and 3.52 cm (1.91%), respectively. These findings indicate that the proposed method is suitable for scalable measurements of banana pseudo-stem diameter and height in complex, obscured environments, providing a rapid and accurate inter-orchard measurement approach for banana plantation managers.

Fusion of Google Street View, LiDAR, and Orthophoto Classifications Using Ranking Classes Based on F1 Score for Building Land-Use Type Detection

Building land-use type classification using earth observation data is essential for urban planning and emergency management. Municipalities usually do not hold a detailed record of building land-use types in their jurisdictions, and there is a significant need for a detailed classification of this data. Earth observation data can be beneficial in this regard, because of their availability and requiring a reduced amount of fieldwork. In this work, we imported Google Street View (GSV), light detection and ranging-derived (LiDAR-derived) features, and orthophoto images to deep learning (DL) models. The DL models were trained on building land-use type data for the Greater Toronto Area (GTA). The data was created using building land-use type labels from OpenStreetMap (OSM) and web scraping. Then, we classified buildings into apartment, house, industrial, institutional, mixed residential/commercial, office building, retail, and other. Three DL-derived classification maps from GSV, LiDAR, and orthophoto images were combined at the decision level using the proposed ranking classes based on the F1 score method. For comparison, the classifiers were combined using fuzzy fusion as well. The results of two independent case studies, Vancouver and Fort Worth, showed that the proposed fusion method could achieve an overall accuracy of 75%, up to 8% higher than the previous study using CNNs and the same ground truth data. Also, the results showed that while mixed residential/commercial buildings were correctly detected using GSV images, the DL models confused many houses in the GTA with mixed residential/commercial because of their similar appearance in GSV images.

Vehicle Detection Algorithms for Autonomous Driving: A Review.

Autonomous driving, as a pivotal technology in modern transportation, is progressively transforming the modalities of human mobility. In this domain, vehicle detection is a significant research direction that involves the intersection of multiple disciplines, including sensor technology and computer vision. In recent years, many excellent vehicle detection methods have been reported, but few studies have focused on summarizing and analyzing these algorithms. This work provides a comprehensive review of existing vehicle detection algorithms and discusses their practical applications in the field of autonomous driving. First, we provide a brief description of the tasks, evaluation metrics, and datasets for vehicle detection. Second, more than 200 classical and latest vehicle detection algorithms are summarized in detail, including those based on machine vision, LiDAR, millimeter-wave radar, and sensor fusion. Finally, this article discusses the strengths and limitations of different algorithms and sensors, and proposes future trends.

Real-Time Perception Enhancement in Obscured Environments for Underground Mine Search and Rescue Teams

First responders in underground mines face a myriad of challenges when searching for personnel in a disaster scenario. Possibly, the most acute challenge is the complete lack of visibility owing to a combination of dust, smoke, and pitch-black conditions. Moreover, the complex environment compounds the difficulty of navigating and searching the area as well as identifying hazardous conditions until in close proximity. Enhanced perception and localization technologies that enable rapid and safe disaster response could mitigate the mine rescue team’s risk and reduce response times. We developed a proof of concept perception enhancement tool for mine rescue personnel in pitch-black conditions by leveraging LiDAR, thermal camera, and data fusion to reconstruct a 3D representation of the space in real-time. This enhancement is visualized on HoloLens, allowing the responders real-time situational awareness of personnel, walls, obstacles, or fires in otherwise opaque environments. The technology is a first step towards faster, safer, and more effective disaster response for mine rescue operations, including detection of unexpected hazards before they become imminent threats.

LVI-Fusion: A Robust Lidar-Visual-Inertial SLAM Scheme

With the development of simultaneous positioning and mapping technology in the field of automatic driving, the current simultaneous localization and mapping scheme is no longer limited to a single sensor and is developing in the direction of multi-sensor fusion to enhance the robustness and accuracy. In this study, a localization and mapping scheme named LVI-fusion based on multi-sensor fusion of camera, lidar and IMU is proposed. Different sensors have different data acquisition frequencies. To solve the problem of time inconsistency in heterogeneous sensor data tight coupling, the time alignment module is used to align the time stamp between the lidar, camera and IMU. The image segmentation algorithm is used to segment the dynamic target of the image and extract the static key points. At the same time, the optical flow tracking based on the static key points are carried out and a robust feature point depth recovery model is proposed to realize the robust estimation of feature point depth. Finally, lidar constraint factor, IMU pre-integral constraint factor and visual constraint factor together construct the error equation that is processed with a sliding window-based optimization module. Experimental results show that the proposed algorithm has competitive accuracy and robustness.

A Real-Time Method for Railway Track Detection and 3D Fitting Based on Camera and LiDAR Fusion Sensing

A Real-Time Method for Railway Track Detection and 3D Fitting Based on Camera and LiDAR Fusion Sensing

UAV-Based Wetland Monitoring: Multispectral and Lidar Fusion with Random Forest Classification

As sea levels rise and temperatures increase, vegetation communities in tropical and sub-tropical coastal areas will be stressed; some will migrate northward and inland. The transition from coastal marshes and scrub–shrubs to woody mangroves is a fundamental change to coastal community structure and species composition. However, this transition will likely be episodic, complicating monitoring efforts, as mangrove advances are countered by dieback from increasingly impactful storms. Coastal habitat monitoring has traditionally been conducted through satellite and ground-based surveys. Here we investigate the use of UAV-LiDAR (unoccupied aerial vehicle–light detection and ranging) and multispectral photogrammetry to study a Florida coastal wetland. These data have higher resolution than satellite-derived data and are cheaper and faster to collect compared to crewed aircraft or ground surveys. We detected significant canopy change in the period between our survey (2020–2022) and a previous survey (2015), including loss at the scale of individual buttonwood trees (Conocarpus erectus), a woody mangrove associate. The UAV-derived data were collected to investigate the utility of simplified processing and data inputs for habitat classification and were validated with standard metrics and additional ground truth. UAV surveys combined with machine learning can streamline coastal habitat monitoring, facilitating repeat surveys to assess the effects of climate change and other change agents.

Shovel point optimization for unmanned loader based on pile reconstruction

AbstractThis study details an advanced shovel point optimization system for unmanned loaders, crucial for efficient shovelling operations. First, the shovel point evaluation index is established with reference to the driver's experience. Second, a novel method for pile profile reconstruction is proposed, utilizing a trained neural network to detect piles and extracting the point cloud using LiDAR and camera fusion. Subsequently, the system employs optimization algorithm to identify the best shovel point. Finally, 62 consecutive working experiments are successfully conducted. The system's performance closely approximates the driver's choices and achieves an average bucket fill factor of 97.7% for four materials. Results demonstrate the proposed method is reliable and efficient and contributes to the development of automated construction machinery.

Improving urban tree species classification by deep-learning based fusion of digital aerial images and LiDAR

Accurate information on tree species distribution in urban areas can offer insights into how street trees provide ecosystem services, such as air pollution mitigation and surface cooling. This article presents a method to improve tree species classification in a tropical urban area using LiDAR-derived structural properties of individual tree crowns (ITCs) and digital aerial images. We extracted four LiDAR features, including surface normals of tree leaves, intensity, tree height, and leaf area index (LAI). We conducted two experiments: In the first, we trained encoder-decoder convolutional neural networks using a stack of optical bands and one LiDAR feature at a time. In the second, we developed an optical-LiDAR fusion strategy that combined feature maps from two encoder-decoder networks. One network was trained with optical bands only, while the other was trained with the LiDAR features that improved classification accuracy in the first experiment. Our experiment results demonstrated the usefulness of surface normals and intensity in discriminating among tree species. We found that the optical-LiDAR fusion strategy increased the average F1-score by 12.6 percentage points compared to only optical bands. We also employed the new segment anything (SAM) model to automatically delineate ITCs. SAM outlined ITCs with a boundary F1-score of 98%. The SAM-delineated ITCs were used to improve raw model predictions and produce reliable species maps. This study contributes to mapping and monitoring urban tree species in tropical areas.

Deep learning-based fusion networks with high-order attention mechanism for 3D object detection in autonomous driving scenarios

In the realm of autonomous driving, accurately detecting 3D objects is both vital and challenging. Recently, deep convolutional networks have been successfully implemented in the fusion of LiDAR and camera data, delivering impressive results. However, prevailing approaches tend to concentrate on basic architectural designs and the use of fixed 3D bounding boxes, overlooking the exploration of feature interrelations and the varying scales of 3D objects. In this paper, we propose High-order Attention Mechanism Fusion Networks (HAMFNs) for image expression and multi-scale learning, based on a novel high-order attention mechanism with multi-scale detection and scale linear regression. High-order convolution layers are built for tenser filtering with discriminative representations of the holistic image. Multi-scale query module further characterizes the saliency properties of the 3D objects. Our tests on the nuScenes dataset show that HAMFNs outperform the latest top-performing methods, achieving a 0.7% increase in mean Average Precision (mAP). We further integrated high-order convolutional layers into ResNet-50, ResNet-101, and ResNet-152 architectures, enhancing their performance with minimal parameter increase. The Top-1 error rates were reduced by 1.65%, 1.63%, and 1.60% for each network, respectively.