Visual-inertial Navigation System Research Articles

A Real-Time, Robust Visual-Inertial Navigation System Tightly Coupled With GNSS and Barometer

A Real-Time, Robust Visual-Inertial Navigation System Tightly Coupled With GNSS and Barometer

Camera-IMU extrinsic calibration method based on intermittent sampling and RANSAC optimization

Accurately calibrate the extrinsic parameters between a camera and an Inertial Measurement Unit (IMU) is a prerequisite for achieving sensor data fusion in visual-inertial navigation systems. Due to delays introduced by triggering, transmission, and other factors, the sampled times of the camera and IMU do not align with the system timestamps, leading to a decrease in the accuracy of extrinsic calibration. Existing calibration methods are unable to avoid the temporal misalignment between the camera and IMU. This paper introduces a novel intermittent calibration sampling method to address temporal misalignment challenges. The approach entails detecting motion-stillness transition points in the IMU data as keyframe segmentation criteria, thereby decoupling data acquisition from system timestamps and aligning it with the sensor’s motion process. Subsequently, the robot hand-eye calibration method is applied to the extrinsic calibration of the Camera-IMU, combining the Random Sample Consensus algorithm to filter the poses of the camera and IMU. This process achieves precise calibration of extrinsic parameters. Experimental validation shows that the algorithm proposed in this paper enhances calibration accuracy when compared to the widely used Kalibr Camera-IMU calibration toolbox. The rotational extrinsic parameter accuracy increased by 181.25%, and the translational extrinsic parameter accuracy improved by 168.33%. These findings provide effective technical means for precise measurement of Camera-IMU extrinsic parameters.

A Vision/Inertial Navigation/Global Navigation Satellite Integrated System for Relative and Absolute Localization in Land Vehicles.

This paper presents an enhanced ground vehicle localization method designed to address the challenges associated with state estimation for autonomous vehicles operating in diverse environments. The focus is specifically on the precise localization of position and orientation in both local and global coordinate systems. The proposed approach integrates local estimates generated by existing visual-inertial odometry (VIO) methods into global position information obtained from the Global Navigation Satellite System (GNSS). This integration is achieved through optimizing fusion in a pose graph, ensuring precise local estimation and drift-free global position estimation. Considering the inherent complexities in autonomous driving scenarios, such as the potential failures of a visual-inertial navigation system (VINS) and restrictions on GNSS signals in urban canyons, leading to disruptions in localization outcomes, we introduce an adaptive fusion mechanism. This mechanism allows seamless switching between three modes: utilizing only VINS, using only GNSS, and normal fusion. The effectiveness of the proposed algorithm is demonstrated through rigorous testing in the Carla simulation environment and challenging UrbanNav scenarios. The evaluation includes both qualitative and quantitative analyses, revealing that the method exhibits robustness and accuracy.

Exploration of actual sky polarization patterns: From influencing factor analyses to polarized light-aided navigation

The polarized light scattered by the sunlight and atmosphere is a general natural property of the atmospheric space of Earth. This study is devoted to an exploration of actual sky polarization pattern-oriented navigation. For the polarized light-aided VINS (abbreviated ‘visual-inertial navigation system’), one of our major concerns is to prove the heading (orientation) reference stays stable under the actual complex atmospheric environments. The influencing factors that contribute to the variation of the polarization pattern symmetry have been taken into account. The analytic AOP (abbreviated ‘angle of polarization’) and DOP (abbreviated ‘degree of polarization’) maps and quantitative analyses that concern spectrum character, cloudage, and aerosol optical depth relevant to light scattering effects are simultaneously presented. Given the globally referenced heading, one intuitive way of modeling multi-sensor fusion is to use a graph structure. Following the pipeline design of VINS Fusion, a polarization imaging factor constraint (expressed by the polarization measurement residual-encoded edges) is added in the state estimator and the cost function to be minimized during optimization is formulated. Via campus road experiments on the built robot platform, our proposal is compared against VINS Fusion under different weather conditions, revealing that over the entire closed path, our polarized light-aided VINS achieves high rate of both locally and globally consistent estimates. To our best knowledge, this is the first work to illuminate the insights into a practical, polarization imager-integrated navigating application in which the localization results can be optically explained, evaluated, and optimized.

Online Self-Calibration for Visual-Inertial Navigation: Models, Analysis, and Degeneracy

As sensor calibration plays an important role in visual-inertial sensor fusion, this article performs an in-depth investigation of online self-calibration for robust and accurate visual-inertial state estimation. To this end, we first conduct complete observability analysis for visual-inertial navigation systems (VINS) with full calibration of sensing parameters, including inertial measurement unit (IMU)/camera intrinsics and IMU-camera spatial-temporal extrinsic calibration, along with readout time of rolling shutter (RS) cameras (if used). We study different inertial model variants containing intrinsic parameters that encompass most commonly used models for low-cost inertial sensors. With these models, the observability analysis of linearized VINS with full sensor calibration is performed. Our analysis theoretically proves the intuition commonly assumed in the literature—that is, VINS with full sensor calibration has four unobservable directions, corresponding to the system's global yaw and position, while all sensor calibration parameters are observable given fully excited motions. Moreover, we, for the first time, identify degenerate motion primitives for IMU and camera intrinsic calibration, which, when combined, may produce complex degenerate motions. We compare the proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">online</i> self-calibration on commonly used IMUs against the state-of-art <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">offline</i> calibration toolbox Kalibr, showing that the proposed system achieves better consistency and repeatability. Based on our analysis and experimental evaluations, we also offer practical guidelines to effectively perform online IMU-camera self-calibration in practice.

Semantic Proximity Update of GNSS/INS/VINS for Seamless Vehicular Navigation Using Smartphone Sensors

The advancement of microelectromechanical systems (MEMS) and the Internet of Things (IoT) have enabled a wide range of applications based on smartphones. However, the existing navigation methods using these low-cost MEMS sensors cannot provide acceptable information for location-based applications in various environments. Their technical limitations, such as severe signal attenuation, reflections, blockages, error accumulation, and low quality of images degrade the performance of GNSS, INS, and camera. To mitigate these limitations, especially in indoor vehicle navigation, we first analyze the performance of the existing fusion algorithm, then we propose Semantic Proximity Update (SPU) based on a pre-trained model of real-time object detection to enhance the integration of Global Navigation Satellite System (GNSS), Inertial Navigation System (INS) and Visual Inertial Navigation System (VINS). SPU consists of the detection of geo-referenced objects and the relative movement to infer the absolute position. The proposed INS/GNSS/VINS/SPU can maintain long-term acceptable accuracy regardless of the indoor/outdoor environment. It only requires the use of smartphone sensors; thus, this scheme has no additional cost for users. Experimental results indicated that the errors of this scheme in horizontal positioning and three-dimensional positioning were 51.6% and 86.8% lower, respectively than those of a conventional integration.

IR-VIO: Illumination-Robust Visual-Inertial Odometry Based on Adaptive Weighting Algorithm With Two-Layer Confidence Maximization

Illumination change, image blur, and fast motion dramatically decrease the performance of visual-inertial navigation systems (VINS). This article presents a new illumination-robust visual-inertial odometry (IR-VIO) based on adaptive weighting algorithm with two-layer confidence maximization. First, to prevent the VIO performance degradation caused by poor image quality in complex scenes and ignoring the confidence differences of feature points, we develop a novel adaptive weighting algorithm on the multisensor layer and visual feature layer to better fuse multisensor information and maximize the overall confidence of VIO. Second, to solve the problems of image feature tracking difficulty and excessive image noise in illumination-changing scenes, an image enhancement algorithm is introduced to enhance consecutive images to the same brightness level, while a block noise removal algorithm with constraint protection mechanism is proposed to dynamically remove noise points. Finally, experimental results in the public dataset and real-world environments demonstrate that IR-VIO has superior performance in terms of accuracy and robustness compared with the state-of-the-art methods.

Visual–Inertial Navigation System Based on Virtual Inertial Sensors

In this paper, we propose a novel visual–inertial simultaneous localization and mapping (SLAM) method for intelligent navigation systems that aims to overcome the challenges posed by dynamic or large-scale outdoor environments. Our approach constructs a visual–inertial navigation system by utilizing virtual inertial sensor components that are mapped to the torso IMU under different gait patterns through gait classification. We apply a zero-velocity update (ZUPT) to initialize the system with the original visual–inertial information. The pose information is then iteratively updated through nonlinear least squares optimization, incorporating additional constraints from the ZUPT to improve the accuracy of the system’s positioning and mapping capabilities in degenerate environments. Finally, the corrected pose information is fed into the solution. We evaluate the performance of our proposed SLAM method in three typical environments, demonstrating its applicability and high precision across various scenarios. Our method represents a significant advancement in the field of intelligent navigation systems and offers a promising solution to the challenges posed by degenerate environments.

Analytic IMU Preintegration That Associates Uncertainty on Matrix Lie Groups for Consistent Visual–Inertial Navigation Systems

We present a novel right-invariant inertial measurement unit (RI IMU) preintegration model and apply it to an optimization-based visual–inertial navigation system (VINS). We find that the unobservable subspace of the proposed estimator was only affected by landmarks from the standpoint of an observability analysis. We highlight that the proposed VINS utilizing the RI IMU preintegration model is consistent with Jacobians evaluated using the most recent estimates of extended poses. Monte Carlo simulations and real-world experiments using the 4Seasons data sets validated our analysis and demonstrated the effectiveness of the proposed VINS by comparing its performance with VINS using different IMU preintegration models and other state-of-the-art VINS.

Covariance Estimation for Pose Graph Optimization in Visual-Inertial Navigation Systems

Pose graph optimization helps reduce drift accumulated in pure odometry of visual simultaneous localization and mapping (SLAM) systems by solving a nonlinear least square problem, including both sequential constraints and loop-closing constraints. However, the covariances of all constraints are set to constant matrices or by manual setting. In this paper, we propose a novel approach to approximate covariances of constraints in pose graph optimization to better represent the true uncertainty of the underlying visual-inertial navigation system (VINS) that fuses inertial measurements and visual observations. Specifically, for sequential constraints, we propose to utilize nonlinear factor recovery to optimally extract covariance matrices from the accumulated visual-inertial odometry (VIO). For loop-closing constraints, we propose a dynamic scale estimation method to approximate the scales of the information matrices. To evaluate the effectiveness and robustness of the proposed method, we conduct extensive experiments on public and self-collected datasets in various environments. Results show that our proposed method achieves higher accuracy compared with naively-formulated pose graph optimization adopted by several state-of-the-art visual-inertial navigation systems.

Design of automatic operation system of trowel based on stereo vision-inertial

The trowel is a construction machinery used for slurry lifting and trowelling cement concrete surfaces. The machine is noisy and has certain dangers during operation. This paper focuses on the automatic operation of the trowel and takes the driving hydraulically driven trowel as the research platform. According to the overall requirements of automatic operation, with the stereo visual-inertial navigation system as the core, a positioning system and a full coverage path planning scheme are designed. The real data was collected in the test site for the experiment, and the results show that the stereo visual-inertial navigation system can provide the required positioning accuracy for the trowel.

LE-VINS: A Robust Solid-State-LiDAR-Enhanced Visual-Inertial Navigation System for Low-Speed Robots

Accurate and long-distance depth estimation for visual landmarks is challenging in visual-inertial navigation systems (VINS). In visual-degenerated scenes with illumination changes, moving objects, or weak texture, depth estimation may be more difficult, resulting in poor robustness and accuracy. For low-speed robot navigation, we present a solid-state-LiDAR-enhanced VINS (LE-VINS) to improve the system robustness and accuracy in challenging environments. The point clouds from the solid-state LiDAR are projected to the visual keyframe with the inertial navigation system (INS) pose for depth association while compensating for the motion distortion. A robust depth-association method with an effective plane-checking algorithm is proposed to estimate the landmark depth. With the estimated depth, we present a LiDAR depth factor to construct accurate depth measurements for visual landmarks in factor graph optimization (FGO). The visual feature, LiDAR depth, and IMU measurements are tightly fused within the FGO framework to achieve maximum-a-posterior estimation. Field tests were conducted on a low-speed robot in large-scale challenging environments. The results demonstrate that the proposed LE-VINS yields significantly improved robustness and accuracy compared to the original VINS. Besides, LE-VINS exhibits superior accuracy than the state-of-the-art LiDAR-visual-inertial navigation system. LE-VINS also outperforms the existing LiDAR-enhanced method, benefiting from the robust depth-association algorithm and the effective LiDAR depth factor.

Visual Inertial Navigation System Aided by Polarized Light

Visual Inertial Navigation System Aided by Polarized Light

IC-GVINS: A Robust, Real-Time, INS-Centric GNSS-Visual-Inertial Navigation System

Visual navigation systems are susceptible to complex environments, while inertial navigation systems (INS) are not affected by external factors. Hence, we present IC-GVINS, a robust, real-time, INS-centric global navigation satellite system (GNSS)-visual-inertial navigation system to fully utilize the INS advantages. The Earth rotation has been compensated in the INS to improve the accuracy of high-grade inertial measurement units (IMUs). To promote the system robustness in high-dynamic conditions, the precise INS information is employed to assist the feature tracking and landmark triangulation. With a GNSS-aided initialization, the IMU, visual, and GNSS measurements are tightly fused in a unified world frame within the factor graph optimization framework. Dedicated experiments were conducted in the public vehicle and private robot datasets to evaluate the proposed method. The results demonstrate that IC-GVINS exhibits superior robustness and accuracy in complex environments. The proposed method with the INS-centric architecture yields improved robustness and accuracy compared to the state-of-the-art methods.

Efficient and Consistent Two Key-Frame Visual-Inertial Navigation Using Matrix Lie Groups

Abstract This paper presents the design of a two key-frame visual-inertial navigation system (2KF-VINS) using a combined Lie group SE2(3) extended Kalman filter (EKF) design framework. The conventional 2KF-VINS filter is unobservable for translations along all three axes and rotation about the gravity direction. As a result, the filter suffers from estimation inconsistencies related to unobservable transformations of the estimation problem. The proposed combined Lie group SE2(3) framework remedies this issue by implicitly preserving the observability consistency property of the filter. Monte Carlo numerical simulations are used to validate the theoretical performance of the right−SE2(3) 2KF-VINS, along with experimental validation using the EuRoC micro aerial vehicle (MAV) dataset to evaluate the performance in real-world scenarios. Additionally, the proposed algorithm is compared with state-of-the-art MSCKF, RI-MSCKF, left−SO(3), and right−SO(3) 2KF-VINS versions with identical and realistic tuning parameters to validate the performance related to the accuracy, consistency, and computational speed of the method.

Pedestrian Gait Information Aided Visual Inertial SLAM for Indoor Positioning Using Handheld Smartphones

Simultaneous localization and mapping (SLAM) is currently a widely used technology for indoor positioning. Many studies have focused on the smartphone-based localization and navigation for its portability and feature-rich sensors. But due to the low-quality sensors and complex environments, current SLAM-based positioning technology using smartphones still poses great challenges, such as inherent cumulative global drift and potential divergence facing texture-less indoor regions. For the regular gait characteristics of pedestrians when naturally walking, the gait motion model can certainly provide effective observation of the pedestrian motion state. We propose a visual-inertial odometry (VIO) assisted by pedestrian gait information for smartphone-based indoor positioning. This work mainly builds two additional state constraints, pedestrian velocity, and step displacement, obtained by the pedestrian dead reckoning (PDR) algorithm for the visual-inertial tracking system. For each step, the corresponding residual term of step length and velocity constraints is constructed and added to the cost function for nonlinear sliding-window optimization. Furthermore, the step displacement is applied again in the four-degree-of-freedom (4-DOF) graph-based optimization to refine the trajectory. VIO system also assists the PDR algorithm in mode switching, to improve the accuracy of gait information by applying the adaptive step length formula. Field experiments were conducted, and the results indicate that with the aiding of the pedestrian gait information, the accuracy and robustness of the visual-inertial pedestrian tracking system using smartphones have been significantly improved. Compared with the state-of-the-art algorithm monocular visual-inertial navigation system (VINS-MONO), our method improves the accuracy by 54.6% on average in our field tests in challenging environments.

Automated wall‐climbing robot for concrete construction inspection

AbstractHuman‐made concrete structures require cutting‐edge inspection tools to ensure the quality of the construction to meet the applicable building codes and to maintain the sustainability of the aging infrastructure. This paper introduces a wall‐climbing robot for metric concrete inspection that can reach difficult‐to‐access locations with a close‐up view for visual data collection and real‐time flaws detection and localization. The wall‐climbing robot is able to detect concrete surface flaws (i.e., cracks and spalls) and produce a defect‐highlighted 3D model with extracted location clues and metric measurements. The system encompasses four modules, including a data collection module to capture RGB‐D frames and inertial measurement unit data, a visual–inertial navigation system module to generate pose‐coupled keyframes, a deep neural network module (namely InspectionNet) to classify each pixel into three classes (background, crack, and spall), and a semantic reconstruction module to integrate per‐frame measurement into a global volumetric model with defects highlighted. We found that commercial RGB‐D camera output depth is noisy with holes, and a Gussian‐Bilateral filter for depth completion is introduced to inpaint the depth image. The method achieves the state‐of‐the‐art depth completion accuracy even with large holes. Based on the semantic mesh, we introduce a coherent defect metric evaluation approach to compute the metric measurement of crack and spall area (e.g., length, width, area, and depth). Field experiments on a concrete bridge demonstrate that our wall‐climbing robot is able to operate on a rough surface and can cross over shallow gaps. The robot is capable to detect and measure surface flaws under low illuminated environments and texture‐less environments. Besides the robot system, we create the first publicly accessible concrete structure spalls and cracks data set that includes 820 labeled images and over 10,000 field‐collected images and release it to the research community.

Rotate artificial potential field algorithm toward 3D real-time path planning for unmanned aerial vehicle

As one of the key technologies in the development of Unmanned Aerial Vehicle (UAV), the path planning method has an intuitive impact on the real-time performance of UAV. Most existing path planning algorithms have drawbacks in global optimization ability or optimization speed, and cannot meet the requirements of practical applications. To improve the effectiveness and practicability of the path planning algorithm for UAV obstacle avoidance, this paper proposes the Rotate Artificial Potential Field (R-APF) method based on the traditional artificial potential field method. By introducing a new potential field function and stimulating rotating repulsive force, the R-APF solves the problems of local minimum as well as inaccessibility to targets near obstacles, respectively. Convergence of the algorithm is discussed to demonstrate the feasibility of R-APF. This paper also designs and implements algorithm feasibility verification experiments, obstacle shuttle experiments, as well as running time, and efficiency comparison experiments. In addition, the improved Visual-Inertial Navigation System obstacle detection algorithm is introduced to carry out physical verification experiments. Theoretical proof and experimental results show the proposed R-APF method has higher obstacle avoidance capability, shorter running time, and higher success rate compared to other algorithms, and is promising in practical application.

Photometric Visual-Inertial Navigation With Uncertainty-Aware Ensembles

In this article, we propose a visual-inertial navigation system that directly minimizes a photometric error without an explicit data-association. We focus on the photometric error parametrized by pose and structure parameters that is highly nonconvex due to the nonlinearity of image intensity. The key idea is to introduce an <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">optimal intensity gradient</i> that accounts for a projective uncertainty of a pixel. Ensembles sampled from the state uncertainty contribute to the proposed gradient and yield a correct update direction even in a bad initialization point. We present two sets of experiments to demonstrate the strengths of our framework. First, a thorough Monte Carlo simulation in a virtual trajectory is designed to reveal robustness to large initial uncertainty. Second, we show that the proposed framework can achieve superior estimation accuracy with efficient computation time over state-of-the-art visual-inertial fusion methods in a real-world UAV flight test, where most scenes are composed of a featureless floor.

Robust Slip-Aware Fusion for Mobile Robots State Estimation

A novel robust and slip-aware speed estimation framework is developed and experimentally verified for mobile robot navigation by designing proprioceptive robust observers at each wheel. The observer for each corner is proved to be consistent, in the sense that it can provide an upper bound of the mean square estimation error (MSE) timely. Under proper conditions, the MSE is proved to be uniformly bounded. A covariance intersection fusion method is used to fuse the wheel-level estimates, such that the updated estimate remains consistent. The estimated slips at each wheel are then used for a robust consensus to improve the reliability of speed estimation in harsh and combined-slip scenarios. As confirmed by indoor and outdoor experiments under different surface conditions, the developed framework addresses state estimation challenges for mobile robots that experience uneven torque distribution or large slip. The novel proprioceptive observer can also be integrated with existing tightly-coupled visual-inertial navigation systems.