Extrinsic Calibration Research Articles

Low-Cost Thermal Point Clouds of Indoor Environments

Abstract. The integration of low-cost thermal sensors with Apple Smart Devices supports the generation of 3D point clouds that include temperature indicators. This generates a new perspective for the study of buildings, allowing for fast and reliable examination of physical building structures. To the best of our knowledge, this study is the first to demonstrate the use of affordable sensors for 3D thermal point cloud generation. The study involved capturing data from the LiDAR and thermal sensors, followed by an extrinsic calibration process to align the datasets. Subsequently, the point cloud was segmented based on different acquisition poses of the device and finally, the thermal data was projected onto the 3D model, integrating temperature information with spatial coordinates. Our results demonstrate the effectiveness of the approach for three-dimensional point cloud generation in indoor environments, highlighting significant thermal variations and enabling thermal mapping of building structures. Furthermore, our findings underscore the feasibility of employing low-cost sensors for generating detailed thermal models, opening possibilities for widespread adoption in various building analysis applications. This approach provides a comprehensive and cost-effective solution for building monitoring, democratizing access to advanced evaluation tools.

Automated Pipeline for Multi-LiDAR Extrinsic Calibration: Experimental Evaluation and Performance Analysis

Abstract. In recent years, 3D LiDAR (Light Detection and Ranging) has become a crucial sensor in various applications such as autonomous vehicles, robotics, object detection, precision forestry, and agriculture. However, specific LiDAR sensors, such as Velodyne VLP16, exhibit some drawbacks, such as a limited field of view and sparse data density, making them inadequate for certain specific applications. Hence, this research proposes a method for calibrating two VLP16 LiDAR sensors to improve coverage and reduce blind spots. The pipeline for performing the calibration begins with mounting LiDARs correctly in a rod at a specific orientation and distance, followed by the selection of multiple sites for data collection, then performing a registration algorithm for estimating calibration parameters, and then an accuracy assessment of the calibrated point clouds. The registration algorithm used here is a modified version of ICP (Iterative Closest Point), which overcomes the need for initialization and eliminates manual intervention in installing targets or retro-reflectors. Finally, we evaluated the accuracy of the fused point cloud collected in an open environment using two calibrated Velodyne VLP16 sensors. For accuracy assessment, we used the PCA eigenvalue and RMSE value to observe how tightly point clouds are fused. As a calibration result, we got the orientation and translation parameters, which are used to achieve the common coordinate system, and accomplished calibration accuracy up to single-digit precision from all the experimental sites. Now the system of two calibrated VLP16 sensors will provide higher coverage and increased data density and might be useful for forest applications.

Joint Intrinsic and Extrinsic Calibration of Perception Systems Utilizing a Calibration Environment

Joint Intrinsic and Extrinsic Calibration of Perception Systems Utilizing a Calibration Environment

Automated extrinsic calibration of solid-state frame LiDAR sensors with non-overlapping field of view for monitoring indoor stockpile storage facilities

Several industrial and commercial bulk material management applications rely on accurate, current stockpile volume estimation. Proximal imaging and LiDAR sensing modalities can be used to derive stockpile volume estimates in outdoor and indoor storage facilities. Among available imaging and LiDAR sensing modalities, the latter is more advantageous for indoor storage facilities due to its ability to capture scans under poor lighting conditions. Evaluating volumes from such sensing modalities requires the pose (i.e., position and orientation) parameters of the used sensors relative to a common reference frame. For outdoor facilities, a Global Navigation Satellite System (GNSS) combined with an Inertial Navigation System (INS) can be used to derive the sensors’ pose relative to a global reference frame. For indoor facilities, GNSS signal outages will not allow for such capability. Prior research has developed strategies for establishing the sensor position and orientation for stockpile volume estimation while relying on multi-beam spinning LiDAR units. These approaches are feasible due to the large range and Field of View (FOV) of such systems that can capture the internal surfaces of indoor storage facilities.The mechanical movement of multi-beam spinning LiDAR units together with the harsh conditions within indoor facilities (e.g., excessive humidity, wide range of temperature variation, dust, and corrosive environment in deicing salt storage facilities) limit the use of such systems. With the increasing availability of solid-state LiDAR units, there is an interest in exploring their potential for stockpile volume estimation. Despite their higher robustness to harsh conditions, solid-state LiDAR units have shorter distance measurement range and limited FOV when compared with multi-beam spinning LiDAR. This research presents a strategy for the extrinsic calibration (i.e., estimating the relative pose parameters) of installed solid-state LiDAR units inside stockpile storage facilities. The extrinsic calibration is made possible using deployed spherical targets and a complete, reference scan of the facility from another LiDAR sensing modality. The proposed research introduces strategies for: 1) automated extraction of the spherical targets; 2) automated matching of these targets in the solid-state LiDAR and reference scans using invariant relationships among them; and 3) coarse-to-fine estimation of the calibration parameters. Experimental results in several facilities have shown the feasibility of using the proposed methodology to conduct the extrinsic calibration and volume evaluation with an error percentage less than 3.5% even with occlusion percentages reaching up to 50%.

FlockSeer: A portable stereo vision observer for bird flocking

AbstractBird flocking is a paradigmatic case of self‐organised collective behaviours in biology. Stereo camera systems are employed to observe flocks of starlings, jackdaws, and chimney swifts, mainly on a spot‐fixed basis. A portable non‐fixed stereo vision‐based flocking observation system, namely FlockSeer, is developed by the authors for observing more species of bird flocks within field scenarios. The portable flocking observer, FlockSeer, responds to the challenges in extrinsic calibration, camera synchronisation and field movability compared to existing spot‐fixed observing systems. A measurement and sensor fusion approach is utilised for rapid calibration, and a light‐based synchronisation approach is used to simplify hardware configuration. FlockSeer has been implemented and tested across six cities in three provinces and has accomplished diverse flock‐tracking tasks, accumulating behavioural data of four species, including egrets, with up to 300 resolvable trajectories. The authors reconstructed the trajectories of a flock of egrets under disturbed conditions to verify the practicality and reliability. In addition, we analysed the accuracy of identifying nearest neighbours, and then examined the similarity between the trajectories and the Couzin model. Experimental results demonstrate that the developed flocking observing system is highly portable, more convenient and swift to deploy in wetland‐like or coast‐like fields. Its observation process is reliable and practical and can effectively support the study of understanding and modelling of bird flocking behaviours.

Accurate extrinsic calibration for the invisible-light 1D laser rangefinder and camera

Abstract A combined sensor, comprising a camera and a one-dimensional laser rangefinder (1D LRF), has wide application across engineering sectors, notably in aerospace. This combined sensor is pivotal for earth observation and deep space exploration. To achieve precise and stable external parameters for this combined sensor, an accurate external calibration method is proposed. Initially, a technique for localized registration of laser spots is introduced to ensure precise determination of their positions, addressing the challenge of laser invisibility in a 1D LRF. Subsequently, a data evaluation criterion known as the data synthesis criterion is presented, addressing the issue of limited constraints in traditional calibration methods. This criterion evaluates relative errors encompassing 1D LRF ranging values, camera external parameters, and laser spot positions. Finally, based on the proposed criteria, a robust extrinsic calibration method is introduced that automatically filters observation data with significant errors and utilizes the growth rate of camera spatial resolution as the termination condition. The efficacy of the proposed method is confirmed through simulation experiments and real-world data experiments.

Probability-Based LIDAR-Camera Calibration Considering Target Positions and Parameter Evaluation Using a Data Fusion Map.

The data fusion of a 3-D light detection and ranging (LIDAR) point cloud and a camera image during the creation of a 3-D map is important because it enables more efficient object classification by autonomous mobile robots and facilitates the construction of a fine 3-D model. The principle behind data fusion is the accurate estimation of the LIDAR-camera's external parameters through extrinsic calibration. Although several studies have proposed the use of multiple calibration targets or poses for precise extrinsic calibration, no study has clearly defined the relationship between the target positions and the data fusion accuracy. Here, we strictly investigated the effects of the deployment of calibration targets on data fusion and proposed the key factors to consider in the deployment of the targets in extrinsic calibration. Thereafter, we applied a probability method to perform a global and robust sampling of the camera external parameters. Subsequently, we proposed an evaluation method for the parameters, which utilizes the color ratio of the 3-D colored point cloud map. The derived probability density confirmed the good performance of the deployment method in estimating the camera external parameters. Additionally, the evaluation quantitatively confirmed the effectiveness of our deployments of the calibration targets in achieving high-accuracy data fusion compared with the results obtained using the previous methods.

A Review of Deep Learning-Based LiDAR and Camera Extrinsic Calibration

Extrinsic parameter calibration is the foundation and prerequisite for LiDAR and camera data fusion of the autonomous system. This technology is widely used in fields such as autonomous driving, mobile robots, intelligent surveillance, and visual measurement. The learning-based method is one of the targetless calibrating methods in LiDAR and camera calibration. Due to its advantages of fast speed, high accuracy, and robustness under complex conditions, it has gradually been applied in practice from a simple theoretical model in just a few years, becoming an indispensable and important method. This paper systematically summarizes the research and development of this type of method in recent years. According to the principle of calibration parameter estimation, learning-based calibration algorithms are divided into two categories: accurate calibrating estimation and relative calibrating prediction. The evolution routes and algorithm frameworks of these two types of algorithms are elaborated, and the methods used in the algorithms’ steps are summarized. The algorithm mechanism, advantages, limitations, and applicable scenarios are discussed. Finally, we make a summary, pointing out existing research issues and trends for future development.

Extrinsic calibration method for integrating infrared thermal imaging camera and 3D LiDAR

PurposeIn the context of fire incidents within buildings, efficient scene perception by firefighting robots is particularly crucial. Although individual sensors can provide specific types of data, achieving deep data correlation among multiple sensors poses challenges. To address this issue, this study aims to explore a fusion approach integrating thermal imaging cameras and LiDAR sensors to enhance the perception capabilities of firefighting robots in fire environments.Design/methodology/approachPrior to sensor fusion, accurate calibration of the sensors is essential. This paper proposes an extrinsic calibration method based on rigid body transformation. The collected data is optimized using the Ceres optimization algorithm to obtain precise calibration parameters. Building upon this calibration, a sensor fusion method based on coordinate projection transformation is proposed, enabling real-time mapping between images and point clouds. In addition, the effectiveness of the proposed fusion device data collection is validated in experimental smoke-filled fire environments.FindingsThe average reprojection error obtained by the extrinsic calibration method based on rigid body transformation is 1.02 pixels, indicating good accuracy. The fused data combines the advantages of thermal imaging cameras and LiDAR, overcoming the limitations of individual sensors.Originality/valueThis paper introduces an extrinsic calibration method based on rigid body transformation, along with a sensor fusion approach based on coordinate projection transformation. The effectiveness of this fusion strategy is validated in simulated fire environments.

Structured light for touchless 3D registration in video-based surgical navigation

PurposeArthroscopic surgery, with its inherent difficulties on visibility and maneuverability inside the joint, poses significant challenges to surgeons. Video-based surgical navigation (VBSN) has proven to have clinical benefits in arthroscopy but relies on a time-consuming and challenging surface digitization using a touch probe to accomplish registration of intraoperative data with preoperative anatomical models. This paper presents an off-the-shelf laser scanner for noninvasive registration that enables an increased area of reachable region.MethodsOur solution uses a standard arthroscope and a light projector with visual markers for real-time extrinsic calibration. Nevertheless, the shift from a touch probe to a laser scanner introduces a new challenge—the presence of a significant amount of outliers resulting from the reconstruction of nonrigid structures. To address this issue, we propose to identify the structures of interest prior to reconstruction using a deep learning-based semantic segmentation technique.ResultsExperimental validation using knee and hip phantoms, as well as ex-vivo data, assesses the laser scanner’s effectiveness. The integration of the segmentation model improves results in ex-vivo experiments by mitigating outliers. Specifically, the laser scanner with the segmentation model achieves registration errors below 2.2 mm, with the intercondylar region exhibiting errors below 1 mm. In experiments with phantoms, the errors are always below 1 mm.ConclusionThe results show the viability of integrating the laser scanner with VBSN as a noninvasive and potential alternative to traditional methods by overcoming surface digitization challenges and expanding the reachable region. Future efforts aim to improve hardware to further optimize performance and applicability in complex procedures.

Camera-IMU extrinsic calibration method based on intermittent sampling and RANSAC optimization

Accurately calibrate the extrinsic parameters between a camera and an Inertial Measurement Unit (IMU) is a prerequisite for achieving sensor data fusion in visual-inertial navigation systems. Due to delays introduced by triggering, transmission, and other factors, the sampled times of the camera and IMU do not align with the system timestamps, leading to a decrease in the accuracy of extrinsic calibration. Existing calibration methods are unable to avoid the temporal misalignment between the camera and IMU. This paper introduces a novel intermittent calibration sampling method to address temporal misalignment challenges. The approach entails detecting motion-stillness transition points in the IMU data as keyframe segmentation criteria, thereby decoupling data acquisition from system timestamps and aligning it with the sensor’s motion process. Subsequently, the robot hand-eye calibration method is applied to the extrinsic calibration of the Camera-IMU, combining the Random Sample Consensus algorithm to filter the poses of the camera and IMU. This process achieves precise calibration of extrinsic parameters. Experimental validation shows that the algorithm proposed in this paper enhances calibration accuracy when compared to the widely used Kalibr Camera-IMU calibration toolbox. The rotational extrinsic parameter accuracy increased by 181.25%, and the translational extrinsic parameter accuracy improved by 168.33%. These findings provide effective technical means for precise measurement of Camera-IMU extrinsic parameters.

Study of Underwater Large-Target Localization Based on Binocular Camera and Laser Rangefinder

Currently, for underwater close-range large-target localization, visual localization techniques fail since large targets completely occupy the camera’s field of view at ultraclose ranges. To address the issue, a multi-stage optical localization method combining a binocular camera and a single-point laser rangefinder is proposed in this paper. The proposed method comprises three parts. First, the imaging model of the underwater camera is modified, and a laser rangefinder is used to further correct the underwater calibration results of the binocular camera. Second, YOLOv8 is applied to recognize the targets to prepare for target localization. Third, extrinsic calibration of the binocular camera and laser rangefinder is performed, and a Kalman filter is employed to fuse the target position information measured by the binocular camera and laser rangefinder. The experimental results show that, compared with using a binocular camera alone, the proposed method can accurately and stably locate the target at close ranges with an average error of only 2.27 cm, without the risk of localization failure, and reduces binocular localization error by 90.57%.

MUN-FRL: A Visual-Inertial-LiDAR Dataset for Aerial Autonomous Navigation and Mapping

This paper presents a unique outdoor aerial visual-inertial-LiDAR dataset captured using a multi-sensor payload to promote the global navigation satellite system (GNSS)-denied navigation research. The dataset features flight distances ranging from 300 m to 5 km, collected using a DJI-M600 hexacopter drone and the National Research Council (NRC) Bell412 Advanced Systems Research Aircraft (ASRA). The dataset consists of hardware-synchronized monocular images, inertial measurement unit (IMU) measurements, 3D light detection and ranging (LiDAR) point-clouds, and high-precision real-time kinematic (RTK)-GNSS based ground truth. Nine data sequences were collected as robot operating system (ROS) bags over 100 mins of outdoor environment footage ranging from urban areas, highways, airports, hillsides, prairies, and waterfronts. The dataset was collected to facilitate the development of visual-inertial-LiDAR odometry and mapping algorithms, visual-inertial navigation algorithms, object detection, segmentation, and landing zone detection algorithms based on real-world drone and full-scale helicopter data. All the data sequences contain raw sensor measurements, hardware timestamps, and spatio-temporally aligned ground truth. The intrinsic and extrinsic calibrations of the sensors are also provided, along with raw calibration datasets. A performance summary of state-of-the-art methods applied on the data sequences is also provided.

High-recall calibration monitoring for stereo cameras

Cameras are the prevalent sensors used for perception in autonomous robotic systems, but their initial calibration may degrade over time due to dynamic factors. This may lead to a failure of downstream tasks, such as simultaneous localization and mapping (SLAM) or object recognition. Hence, a computationally lightweight process that detects the decalibration is of interest. We describe a modification of StOCaMo, an online calibration monitoring procedure for a stereoscopic system. The method uses robust kernel correlation based on epipolar constraints; it validates extrinsic calibration parameters on a single frame with no temporal tracking. In this paper, we present a modified StOCaMo with an improved recall rate on small decalibrations through a confirmation technique based on resampled variance. With fixed parameters learned on a realistic synthetic dataset from CARLA, StOCaMo and its proposed modification were tested on multiple sequences from two real-world datasets: KITTI and EuRoC MAV. The modification improved the recall of StOCaMo by 25 % (to 91 % and 82 %, respectively), and the accuracy by 12 % (to 94.7 % and 87.5 %, respectively), while labeling at most one-third of the input data as uninformative. The upgraded method achieved the rank correlation between StOCaMo V-index and downstream SLAM error of 0.78 (Spearman).

Automatic extrinsic calibration for structured light camera and repetitive LiDARs

AbstractThe integration of camera and LiDAR technologies has the potential to significantly enhance construction robots’ perception capabilities by providing complementary construction information. Structured light cameras (SLCs) are a desirable alternative as they provide comprehensive information on construction defects. However, fusing these two types of information depends largely on the sensors’ relative positions, which can only be established through extrinsic calibration. This paper introduces a novel calibration algorithm considering a customized board for SLCs and repetitive LiDARs, which are designed to facilitate the automation of construction robots. The calibration board is equipped with four symmetrically distributed hemispheres, whose centers are obtained by fitting the spheres and adoption with the geometric constraints. Subsequently, the spherical centers serve as reference features to estimate the relationship between the sensors. These distinctive features enable our proposed method to only require one calibration board pose and minimize human intervention. We conducted both simulation and real-world experiments to assess the performance of our algorithm. And the results demonstrate that our method exhibits enhanced accuracy and robustness.

ZUST Campus: A Lightweight and Practical LiDAR SLAM Dataset for Autonomous Driving Scenarios

This research proposes a lightweight and applicable dataset with a precise elevation ground truth and extrinsic calibration toward the LiDAR (Light Detection and Ranging) SLAM (Simultaneous Localization and Mapping) task in the field of autonomous driving. Our dataset focuses on more cost-effective platforms with limited computational power and low-resolution three-dimensional LiDAR sensors (16-beam LiDAR), and fills the gaps in the existing literature. Our data include abundant scenarios that include degenerated environments, dynamic objects, and large slope terrain to facilitate the investigation of the performance of the SLAM system. We provided the ground truth pose from RTK-GPS and carefully rectified its elevation errors, and designed an extra method to evaluate the vertical drift. The module for calibrating the LiDAR and IMU was also enhanced to ensure the precision of point cloud data. The reliability and applicability of the dataset are fully tested through a series of experiments using several state-of-the-art LiDAR SLAM methods.

Analysis of the Photogrammetric Use of 360-Degree Cameras in Complex Heritage-Related Scenes: Case of the Necropolis of Qubbet el-Hawa (Aswan Egypt).

This study shows the results of the analysis of the photogrammetric use of 360-degree cameras in complex heritage-related scenes. The goal is to take advantage of the large field of view provided by these sensors and reduce the number of images used to cover the entire scene compared to those needed using conventional cameras. We also try to minimize problems derived from camera geometry and lens characteristics. In this regard, we used a multi-sensor camera composed of six fisheye lenses, applying photogrammetric procedures to several funerary structures. The methodology includes the analysis of several types of spherical images obtained using different stitching techniques and the comparison of the results of image orientation processes considering these images and the original fisheye images. Subsequently, we analyze the possible use of the fisheye images to model complex scenes by reducing the use of ground control points, thus minimizing the need to apply surveying techniques to determine their coordinates. In this regard, we applied distance constraints based on a previous extrinsic calibration of the camera, obtaining results similar to those obtained using a traditional schema based on points. The results have allowed us to determine the advantages and disadvantages of each type of image and configuration, providing several recommendations regarding their use in complex scenes.

Implementation and observability analysis of visual-inertial-wheel odometry with robust initialization and online extrinsic calibration

Combining camera, IMU and wheel encoder is a wise choice for car positioning because of the low cost and complementarity of the sensors. We propose a novel extended visual-inertial odometry algorithm based on sliding window tightly fusing data from the above three sensors. Firstly we propose an IMU-odometer pre-integration approach utilizing complete IMU measurements and wheel encoder readings, to make scale estimation more accurate in subsequent 4-degrees of freedom (DoF) optimization. Secondly we develop an original initialization module where encoder readings are fully utilized to refine gravity direction and provide an initial value for camera pose in real scale. Thirdly, we design a computationally efficient online extrinsic calibration method by fixing the linearization point for the rotational component of IMU-odometer extrinsic parameters, which is deployed depending on the convergence of accelerometer bias. Fourthly, we give an observability analysis of our optimization based approach under more general assumption. Extensive experiments are performed on two sets of data in various scenes, bringing the state-of-the-art visual odometry and visual-inertial odometry algorithms into comparison. Experimental results prove the overwhelmingly better performance of our proposed approach on the above two sets of data, as well as the robustness of our initialization module and the improvement resulted from online extrinsic calibration.

Extrinsic calibration for large FOV based on inverse depth parameterized bundle adjustment

To overcome the reliance on high-precision calibration plates in camera calibration, this paper proposes an extrinsic calibration method suitable for three-dimensional digital image correlation (3D-DIC) in large field of view (FOV). The method combines image feature algorithms with DIC techniques to extract matching point pairs (MPPs) from the left and right images of binocular cameras. These MPPs are then homogenized within the FOV. Next, initial values of the extrinsic parameters are solved based on epipolar constraint theory. Finally, the calibration parameters are nonlinearly optimized using the bundle adjustment method. To achieve stable and reliable numerical optimization in large FOV applications, the calibration control points (CCPs)’ spatial coordinates are represented using inverse depth parameterization. In scenarios where there might be a lack of sufficient CCPs, speckle patterns are artificially introduced to supplement the scene features. However, there is a lack of reliable experimental basis on how to add CCPs within the FOV. Therefore, through simulations, the factors affecting calibration accuracy are analyzed to guide the actual calibration process. The proposed method’s reliability and accuracy in large FOV 3D measurement are verified through experiments.

LiDAR-Link: Observability-Aware Probabilistic Plane-Based Extrinsic Calibration for Non-Overlapping Solid-State LiDARs

LiDAR-Link: Observability-Aware Probabilistic Plane-Based Extrinsic Calibration for Non-Overlapping Solid-State LiDARs