Visual-inertial State Estimation Research Articles

EVI‐SAM: Robust, Real‐Time, Tightly‐Coupled Event–Visual–Inertial State Estimation and 3D Dense Mapping

EVI‐SAM: Robust, Real‐Time, Tightly‐Coupled Event–Visual–Inertial State Estimation and 3D Dense Mapping

Design of visual inertial state estimator for autonomous systems via multi-sensor fusion approach

Design of visual inertial state estimator for autonomous systems via multi-sensor fusion approach

Online Self-Calibration for Visual-Inertial Navigation: Models, Analysis, and Degeneracy

As sensor calibration plays an important role in visual-inertial sensor fusion, this article performs an in-depth investigation of online self-calibration for robust and accurate visual-inertial state estimation. To this end, we first conduct complete observability analysis for visual-inertial navigation systems (VINS) with full calibration of sensing parameters, including inertial measurement unit (IMU)/camera intrinsics and IMU-camera spatial-temporal extrinsic calibration, along with readout time of rolling shutter (RS) cameras (if used). We study different inertial model variants containing intrinsic parameters that encompass most commonly used models for low-cost inertial sensors. With these models, the observability analysis of linearized VINS with full sensor calibration is performed. Our analysis theoretically proves the intuition commonly assumed in the literature—that is, VINS with full sensor calibration has four unobservable directions, corresponding to the system's global yaw and position, while all sensor calibration parameters are observable given fully excited motions. Moreover, we, for the first time, identify degenerate motion primitives for IMU and camera intrinsic calibration, which, when combined, may produce complex degenerate motions. We compare the proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">online</i> self-calibration on commonly used IMUs against the state-of-art <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">offline</i> calibration toolbox Kalibr, showing that the proposed system achieves better consistency and repeatability. Based on our analysis and experimental evaluations, we also offer practical guidelines to effectively perform online IMU-camera self-calibration in practice.

Tunable Impact and Vibration Absorbing Neck for Robust Visual-Inertial State Estimation for Dynamic Legged Robots

We propose a new neck design for legged robots to achieve robust visual-inertial state estimation in dynamic locomotion. While visual-inertial state estimation is widely used in robotics, it has a problem of being disturbed by the impacts and vibration generated when legged robots move dynamically. The use of rubber dampers may be a solution, but even if the dampers are proper for some gaits, they may be excessively deformed or resonated at certain frequencies during other gait locomotion since they are not tunable. To address this problem, we develop a tunable neck system that absorbs the impacts and vibration during diverse gait locomotions. This neck system consists of two components: 1) a suspension mechanism that compensates for the weight of the head equipped with a camera and IMU (inertial measurement unit), absorbs the impacts and the head motion of high frequencies including vibration as a fixed low-pass filter; and 2) a dynamic vibration absorber (DVA) that can be reactively-adjusted to diverse gait frequencies to alleviate excessive head movements. We present a dynamics analysis of the neck system and show how to adjust the target frequency of the system. Simulation and experimental validation are performed to verify the effect of the proposed neck design, manifesting superior estimation performance and robustness across diverse gaits.

G-VIDO: A Vehicle Dynamics and Intermittent GNSS-Aided Visual-Inertial State Estimator for Autonomous Driving

This paper proposes G-VIDO, a vehicle dynamics, and intermittent Global Navigation Satellite System (GNSS)-aided visual-inertial state estimator, to address the state estimation problem of autonomous vehicle localization (i.e., position and orientation estimation in the global coordinate system) under various GNSS states. A dynamics pre-integration theory is proposed on the basis of a two-degree-of-freedom (DOF) vehicle dynamics model, and dynamics constraints are built in the optimization back-end, considering the unobservable problem of the monocular visual-inertial system under degenerate motions. The proposed highly nonlinear system can be robustly initialized by loosely aligning the monocular structure from motion (SfM) results, pre-integrated IMU measurements, and vehicle motion information. GNSS is used for reference frame transformation and constraint construction in the sliding window. The cumulative error can be corrected with the aid of GNSS, and the vehicle’s position in the global coordinate system can be determined. A GNSS anomaly detection algorithm is proposed to improve the system robustness under intermittent GNSS. Experiments have shown that G-VIDO can provide real-time, robust, and seamless localization in multiple GNSS states, with an RMSE of less than 30 cm (with GNSS). Moreover, we proved that the initialization and local odometry modules in G-VIDO outperform several state-of-the-art VIO systems and our preliminary work VINS-Vehicle.

Fast visual inertial odometry with point–line features using adaptive EDLines algorithm

In mainstream visual inertial odometry (VIO) systems, the method of positional solution by feature point extraction and matching in the image is widely used. However, the tracking accuracy of point features is dependent on the texture richness in the environment. Although many existing algorithms introduce line features in the front end to improve the system’s environmental adaptability, most of them sacrifice system real-time in exchange for higher positioning accuracy. The extraction and matching of line features often require more time, thus failing to meet the real-time requirements of the system for localization. In this paper, we therefore propose a fast VIO fused with point and line features, which enables the system to maintain a high level of positioning robustness in dim and changing light environments with low time cost. The point–line features VIO algorithm is based on adaptive thresholding of EDLines. By adding an adaptive thresholding component to the EDLines algorithm, the robustness of line feature extraction is enhanced to better adapt to changes in ambient lighting. The time needed for line feature extraction is also significantly reduced. A line feature matching algorithm based on geometric information and structural similarity is proposed, which enables fast and accurate line feature matching. The algorithm is compared with point-line visual-inertial odometry and monocular visual-inertial state estimator algorithms on the European robotics challenge dataset and real-world scenes. Many experiments prove that the algorithm has improved in both real time and accuracy.

PLS-VINS: Visual Inertial State Estimator With Point-Line Features Fusion and Structural Constraints

One of the limitations of visual Simultaneous Localization and Mapping (VSLAM) of point-based is that it relies too much on scene features. When the texture information in the scene is missing or the image is blurred due to the rapid movement of the camera, the number of point features is often small, which affects the accuracy of pose estimation. Line feature information contains more geometric elements than point feature information. We propose a visual-inertial state estimator system based on point-line features and structure constraints (PLS-VINS), which combines both of points and line segments to enhance the performance of feature extraction in a wider variety of scenarios and optimizes the system states by jointly minimizing the pre-integration constraints of inertial measurement unit (IMU). Particularly in those where point features are scarce or not well-distributed in the image, PLS-VINS can extract many line features from artificial objects, which can provide more structural constraints (collinear, vertical, and parallel) for visual navigation, such as vertical and parallel constraints. Compared with the classic SLAM methods which use point-line fusion algorithm, we have adopted a new line feature error reprojection model and added structural constraints of man-made buildings in the back-end optimization to improve the robustness and accuracy in an illumination-changing environment. The proposed PLS-VINS has been extensively tested in both synthetic and real-world experiments whose results demonstrate that the proposed system outperforms the state-of-the-art methods in various scenarios. Compared with the VINS-Mono, the positioning RMSE in this study increases by 29.46% and 23.79% in rotation and translation for Dataset 1–1 with EuRoc Dataset, and by 55.64% in translation for Dataset 2–1 of actual scene with weak texture environment, respectively.

Robust Visual-Inertial Navigation System for Low Precision Sensors under Indoor and Outdoor Environments

Simultaneous Localization and Mapping (SLAM) has always been the focus of the robot navigation for many decades and becomes a research hotspot in recent years. Because a SLAM system based on vision sensor is vulnerable to environment illumination and texture, the problem of initial scale ambiguity still exists in a monocular SLAM system. The fusion of a monocular camera and an inertial measurement unit (IMU) can effectively solve the scale blur problem, improve the robustness of the system, and achieve higher positioning accuracy. Based on a monocular visual-inertial navigation system (VINS-mono), a state-of-the-art fusion performance of monocular vision and IMU, this paper designs a new initialization scheme that can calculate the acceleration bias as a variable during the initialization process so that it can be applied to low-cost IMU sensors. Besides, in order to obtain better initialization accuracy, visual matching positioning method based on feature point is used to assist the initialization process. After the initialization process, it switches to optical flow tracking visual positioning mode to reduce the calculation complexity. By using the proposed method, the advantages of feature point method and optical flow method can be fused. This paper, the first one to use both the feature point method and optical flow method, has better performance in the comprehensive performance of positioning accuracy and robustness under the low-cost sensors. Through experiments conducted with the EuRoc dataset and campus environment, the results show that the initial values obtained through the initialization process can be efficiently used for launching nonlinear visual-inertial state estimator and positioning accuracy of the improved VINS-mono has been improved by about 10% than VINS-mono.

Visual–inertial state estimation with camera and camera–IMU calibration

In the last two decades, the Visual–Inertial Odometry (VIO) has recently received much attention for efficient and accurate ego-motion estimation of unmanned vehicle systems. In particular, the VIO includes only an Inertial Measurement Unit (IMU) and a camera. In this paper, we present a novel calibration approach for accurate deployment of monocular VIO. For this purpose, the hybrid optimization algorithm is used for calibrating the camera intrinsic and camera–IMU extrinsic calibration, automatically and without knowing the mechanical configuration. It is a professional work to carefully calibrate the intrinsic and extrinsic parameters, and it is required to repeat this work when the mechanical configuration of the sensor suite changes. Quantitative comparisons our method with the offline conventional calibration method on the KITTI dataset verify the efficacy and accuracy of the proposed method. We also demonstrate the performance of the proposed approach in large scale outdoor experiments.

Online Spatial and Temporal Calibration for Monocular Direct Visual-Inertial Odometry.

Owing to the nonlinearity in visual-inertial state estimation, sufficiently accurate initial states, especially the spatial and temporal parameters between IMU (Inertial Measurement Unit) and camera, should be provided to avoid divergence. Moreover, these parameters are required to be calibrated online since they are likely to vary once the mechanical configuration slightly changes. Recently, direct approaches have gained popularity for their better performance than feature-based approaches in little-texture or low-illumination environments, taking advantage of tracking pixels directly. Based on these considerations, we perform a direct version of monocular VIO (Visual-inertial Odometry), and propose a novel approach to initialize the spatial-temporal parameters and estimate them with all other variables of interest (IMU pose, point inverse depth, etc.). We highlight that our approach is able to perform robust and accurate initialization and online calibration for the spatial and temporal parameters without utilizing any prior information, and also achieves high-precision estimates even when large temporal offset occurs. The performance of the proposed approach was verified through the public UAV (Unmanned Aerial Vehicle) dataset.

On the Comparison of Gauge Freedom Handling in Optimization-Based Visual-Inertial State Estimation

It is well known that visual-inertial state estimation is possible up to a four degrees-of-freedom (DoF) transformation (rotation around gravity and translation), and the extra DoFs (“gauge freedom”) have to be handled properly. While different approaches for handling the gauge freedom have been used in practice, no previous study has been carried out to systematically analyze their differences. In this letter, we present the first comparative analysis of different methods for handling the gauge freedom in optimization-based visual-inertial state estimation. We experimentally compare three commonly used approaches: fixing the unobservable states to some given values, setting a prior on such states, or letting the states evolve freely during optimization. Specifically, we show that 1) the accuracy and computational time of the three methods are similar, with the free gauge approach being slightly faster; 2) the covariance estimation from the free gauge approach appears dramatically different, but is actually tightly related to the other approaches. Our findings are validated both in simulation and on real-world data sets and can be useful for designing optimization-based visual-inertial state estimation algorithms.