Visual-inertial Odometry Research Articles

Observability Analysis of IMU Intrinsic Parameters in Stereo Visual–Inertial Odometry

In this article, we present self-calibrated visual–inertial odometry (VIO) that estimates inertial measurement unit (IMU) intrinsic parameters (scale factor and misalignment) using a stereo camera without any calibration boards. Most of the visual–inertial navigation algorithms assume that the visual and inertial sensors are well-calibrated before operating. However, this could degrade the navigation performance due to unmodeled errors. Specifically, we employ an extended Kalman filter (EKF)-based pose estimator in which the filter state is augmented by the IMU intrinsic parameter to model the egomotion more precisely. Since raw IMU readings are transformed by the intrinsic parameter, these are key factors that determine the performance of the egomotion tracking. The main contribution of this article is an analytic observability analysis of the self-calibrated VIO that is a nonlinear time-varying system. We inspect the rank of the observability matrix formed by Lie derivatives of the nonlinear system. Our theoretical result reveals that the IMU intrinsic parameter is fully observable when all six axes of an IMU are excited. This is further confirmed by our simulation experiments by examining state uncertainties. Moreover, the real-world experiment using a publicly available and author-collected data set reveals that the pose tracking performance is improved by modeling IMU intrinsic parameters.

ROVINS: Robust Omnidirectional Visual Inertial Navigation System

Visual odometry is an essential component in robot navigation and autonomous driving; however, visual sensors are vulnerable in fast motion or sudden illumination changes. This weakness can be compensated with inertial measurement units (IMUs), which maintain the short-term motion when visual sensing becomes unstable and enhance the quality of estimated motion with inertial information. An omnidirectional multi-view visual odometry (ROVO) has recently demonstrated superior performance and stability with the unceasing feature observation of the omnidirectional setup; however, the shortcomings of visual odometry remain. This letter introduced an omnidirectional visual-inertial odometry system (ROVINS) that could seamlessly integrate the inertial information into the omnidirectional visual odometer algorithm: (a) The soft relative pose constraints from the inertial measurement are first added to the pose optimization formulation, which enables blind motion estimation when all visual features are lost; (b) Using the prediction results from the estimated velocity, the visual features in tracking are initialized, resulting in feature tracking that is more robust to visual disturbances. The experimental results showed that the proposed ROVINS algorithm outperforms the vision-only algorithm by a significant margin.

Accurate and robust odometry by fusing monocular visual, inertial, and wheel encoder

Tracking the pose of a robot has been gaining importance in the field of Robotics, e.g., paving the way for robot navigation. In recent years, monocular visual–inertial odometry (VIO) is widely used to do the pose estimation due to its good performance and low cost. However, VIO cannot estimate the scale or orientation accurately when robots move along straight lines or circular arcs on the ground. To address the problem, in this paper we take the wheel encoder into account, which can provide us with stable translation information as well as small accumulated errors and momentary slippage errors. By jointly considering the kinematic constraints and the planar moving features, an odometry algorithm tightly coupled with monocular camera, IMU, and wheel encoder is proposed to get robust and accurate pose sensing for mobile robots, which mainly contains three steps. First, we present the wheel encoder preintegration theory and noise propagation formula based on the kinematic mobile robot model, which is the basis of accurate estimation in backend optimization. Second, we adopt a robust initialization method to obtain good initial values of gyroscope bias and visual scale in reality, by making full use of the camera, IMU and wheel encoder measurements. Third, we bound the high computation complexity with a marginalization strategy that conditionally eliminates unnecessary measurements in the sliding window. We implement a prototype and several extensive experiments showing that our system can achieve robust and accurate pose estimation, in terms of the scale, orientation and location, compared with the state-of-the-art.

Improved Point–Line Visual–Inertial Odometry System Using Helmert Variance Component Estimation

Mobile platform visual image sequence inevitably has large areas with various types of weak textures, which affect the acquisition of accurate pose in the subsequent platform moving process. The visual–inertial odometry (VIO) with point features and line features as visual information shows a good performance in weak texture environments, which can solve these problems to a certain extent. However, the extraction and matching of line features are time consuming, and reasonable weights between the point and line features are hard to estimate, which makes it difficult to accurately track the pose of the platform in real time. In order to overcome the deficiency, an improved effective point–line visual–inertial odometry system is proposed in this paper, which makes use of geometric information of line features and combines with pixel correlation coefficient to match the line features. Furthermore, this system uses the Helmert variance component estimation method to adjust weights between point features and line features. Comprehensive experimental results on the two datasets of EuRoc MAV and PennCOSYVIO demonstrate that the point–line visual–inertial odometry system developed in this paper achieved significant improvements in both localization accuracy and efficiency compared with several state-of-the-art VIO systems.

Robust and Autonomous Stereo Visual-Inertial Navigation for Non-Holonomic Mobile Robots

Unlike micro aerial vehicles, most mobile robots have non-holonomic constraints, which makes lateral movement impossible. Consequently, the vision-based navigation systems that perform accurate visual feature initialization by moving the camera to the side to ensure a sufficient parallax of the image are degraded when applied to mobile robots. Generally, to overcome this difficulty, a motion model based on wheel encoders mounted on a mobile robot is used to predict the pose of a robot, but it is difficult to cope with errors caused by wheel slip or inaccurate wheel calibration. In this study, we propose a robust autonomous navigation system that uses only a stereo inertial sensor and does not rely on wheel-based dead reckoning. The observation model of the line feature modified with vanishing-points is applied to the visual-inertial odometry along with the point features so that a mobile robot can perform robust pose estimation during autonomous navigation. The proposed algorithm, i.e., keyframe-based autonomous visual-inertial navigation (KAVIN) supports the entire navigation system and can run onboard without an additional graphics processing unit. A series of experiments in a real environment indicated that the KAVIN system provides robust pose estimation without wheel encoders and prevents the accumulation of drift error during autonomous driving.

Improving S-MSCKF With Variational Bayesian Adaptive Nonlinear Filter

In recent years, multi-sensor fusion has been paid a great attention and is widely applied to simultaneous localization and mapping (SLAM), e.g. Visual-Inertial Odometry (VIO). However, in the most of previous works the observation noise of sensor is assumed to be Gaussian with invariant constant mean and covariance, which is not the case actually. In this paper, a variational Bayesian adaptive algorithm is proposed to improve the performance of visual-inertial odometry, where covariance of observation noise is modeled as inverse-Wishart distribution with harmonic mean so that observation noise is non-Gaussian. Besides, unscented transformation is utilized to handle the nonlinear system problem. The proposed algorithm is applied to the framework of stereo multi-state constraint Kalman filter system (S-MSCKF), which is a state-of-the-art VIO system with low computational cost and good robustness. Extensive experiments are performed on simulation data and open source datasets. The results demonstrate that our proposed algorithm is more robust and accurate compared with S-MSCKF.

Autonomous drone race: A computationally efficient vision-based navigation and control strategy

Drone racing is becoming a popular sport where human pilots have to control their drones to fly at high speed through complex environments and pass a number of gates in a pre-defined sequence. In this paper, we develop an autonomous system for drones to race fully autonomously using only onboard resources. Instead of commonly used visual navigation methods, such as simultaneous localization and mapping and visual inertial odometry, which are computationally expensive for micro aerial vehicles (MAVs), we developed the highly efficient snake gate detection algorithm for visual navigation, which can detect the gate at 20 HZ on a Parrot Bebop drone. Then, with the gate detection result, we developed a robust pose estimation algorithm which has better tolerance to detection noise than a state-of-the-art perspective-n-point method. During the race, sometimes the gates are not in the drone’s field of view. For this case, a state prediction-based feed-forward control strategy is developed to steer the drone to fly to the next gate. Experiments show that the drone can fly a half-circle with 1.5 m radius within 2 s with only 30cm error at the end of the circle without any position feedback. Finally, the whole system is tested in a complex environment (a showroom in the faculty of Aerospace Engineering, TU Delft). The result shows that the drone can complete the track of 15 gates with a speed of 1.5m∕s which is faster than the speeds exhibited at the 2016 and 2017 IROS autonomous drone races.

ROBUST VISUAL-INERTIAL ODOMETRY IN DYNAMIC ENVIRONMENTS USING SEMANTIC SEGMENTATION FOR FEATURE SELECTION

Abstract. Camera based navigation in dynamic environments with high content of moving objects is challenging. Keypoint-based localization methods need to reliably reject features that do not belong to the static background. Here, traditional statistical methods for outlier rejection quickly reach their limits. A common approach is the combination with an inertial measurement unit for visual-inertial odometry. Also, deep learning based semantic segmentation was recently successfully applied in camera based localization to identify features on common objects. In this work, we study the application of mask-based feature selection based on semantic segmentation for robust localization in high dynamic environments. We focus on visual-inertial odometry, but similarly investigate a state-of-the-art pure vision-based method as baseline. For a versatile evaluation, we use challenging self-recorded datasets based on different sensor systems. This includes a combined dataset of a real world system and its synthetic clone with a large number of humans for in-depth analysis. We further deploy large-scale datasets from pedestrian navigation in a mall with escalator scenes and vehicle navigation during the day and at night. Our results show that visual-inertial odometry performs generally well in dynamic environments itself, but also shows significant failures in challenging scenes, which are prevented by using the segmentation aid.

LEARNING MAPS FOR OBJECT LOCALIZATION USING VISUAL-INERTIAL ODOMETRY

Abstract. Objects follow designated path on maps, such as vehicles travelling on a road. This observation signifies topological representation of objects’ motion on the map. Considering the position of object is unknown initially, as it traverses the map by moving and turning, the spatial uncertainty of its whereabouts reduces to a single location as the motion trajectory would fit only to a certain map trajectory. Inspired by this observation, we propose a novel end-to-end localization approach based on topological maps that exploits the object motion and learning the map using an recurrent neural network (RNN) model. The core of the proposed method is to learn potential motion patterns from the map and perform trajectory classification in the map’s edge-space. Two different trajectory representations, namely angle representation and augmented angle representation (incorporates distance traversed) are considered and an RNN is trained from the map for each representation to compare their performances. The localization accuracy in the tested map for the angle and augmented angle representations are 90.43% and 96.22% respectively. The results from the actual visual-inertial odometry have shown that the proposed approach is able to learn the map and localize objects based on their motion.

An Online Initialization and Self-Calibration Method for Stereo Visual-Inertial Odometry

Most online initialization and self-calibration methods for visual-inertial odometry (VIO) are only able to estimate the extrinsic parameters (orientation and translation) between one camera and inertial measurement unit (IMU) pair. They are not applicable to stereo VIO where both camera-IMU and camera-camera pairs exist. In this article, we address the issue by taking advantage of the geometric constraints among the multiple sensors. An online method is proposed to estimate the initial values of velocity, gravity, IMU biases, and simultaneously calibrate the extrinsic parameters of camera-camera and camera-IMU pairs for bootstrapping a smoothing-based stereo VIO system. The method includes a three-step process to incrementally solve several linear equations in a coarse-to-fine manner. It back-propagates historically estimated results to update weight factors and remove outliers, and employs a convergence criterion to monitor and terminate the process. It also includes an optional global optimization for further refinement. The method is evaluated in terms of accuracy, robustness, convergence, consistency, and tunable parameters using both simulated and public datasets. Experimental results show that the proposed method can accurately estimate the initial values and the extrinsic parameters.

Toward Autonomy of Micro Aerial Vehicles in Unknown and Global Positioning System Denied Environments

In this article, we present a comprehensive design and implementation for a micro aerial vehicle (MAV) that is able to perform 3-D autonomous navigation and obstacle avoidance in cluttered and realistic unknown environments without the aid of global positioning system and other external sensors or markers. To achieve these autonomous missions, modularized components are developed for the MAV, including visual-inertial odometry, 3-D occupancy mapping, and motion planning. The proposed system is implemented to run on a small embedded computer in real time. It is demonstrated to be robust in both simulation and real flight experiments. The demonstration video is available at: https://youtu.be/KUKzsnORm-4.

Cooperative Relative Navigation of Multiple Aircraft in Global Positioning System-Denied/Degraded Environments

This paper introduces a method for enabling multiple small unmanned aircraft to improve navigational accuracy in Global Positioning System (GPS)-denied environments by cooperatively sharing information. The method uses a multilevel framework called relative navigation and visual-inertial odometry: both of which have been used for single-vehicle GPS-denied navigation. This work modifies the relative navigation architecture by adding the necessary elements to enable decentralized, cooperative operations. The proposed framework also includes the ability to incorporate intervehicle measurements and uses a new concept called the coordinated reset, which is necessary for optimizing the cooperative odometry and improving localization. A simple communication protocol is presented, as well as flight-test results that show its effectiveness. The method is first demonstrated in simulation, including Monte Carlo testing, to show the expected improvement gained from cooperation. Finally, results from hardware testing are presented that show the ability of the system to reject initialization errors, to operate with temporary communications dropouts, and to greatly reduce the total amount of shared data compared to a naive approach. The proposed method is able to improve both global navigation accuracy and the accuracy of the relative aircraft positions for missions that require formation flying.

The UMA-VI dataset: Visual–inertial odometry in low-textured and dynamic illumination environments

This article presents a visual–inertial dataset gathered in indoor and outdoor scenarios with a handheld custom sensor rig, for over 80 min in total. The dataset contains hardware-synchronized data from a commercial stereo camera (Bumblebee®2), a custom stereo rig, and an inertial measurement unit. The most distinctive feature of this dataset is the strong presence of low-textured environments and scenes with dynamic illumination, which are recurrent corner cases of visual odometry and simultaneous localization and mapping (SLAM) methods. The dataset comprises 32 sequences and is provided with ground-truth poses at the beginning and the end of each of the sequences, thus allowing the accumulated drift to be measured in each case. We provide a trial evaluation of five existing state-of-the-art visual and visual–inertial methods on a subset of the dataset. We also make available open-source tools for evaluation purposes, as well as the intrinsic and extrinsic calibration parameters of all sensors in the rig. The dataset is available for download at http://mapir.uma.es/work/uma-visual-inertial-dataset

A Binocular MSCKF-Based Visual Inertial Odometry System Using LK Optical Flow

The odometry is an important part of intelligent mobile robots to achieve positioning and navigation functions. At present, the mainstream visual odometry locates only through the visual information obtained by camera sensors. Therefore, in the case of insufficient light, texture missing and camera jitter, the visual odometry is difficult to locate accurately. To solve the problem, we propose a binocular MSCKF-based visual inertial odometry system using Lucas-Kanade (LK) optical flow. Firstly, the Inertial Measurement Unit (IMU) is introduced to overcome the above problems. Moreover, LK optical flow algorithm is utilized to process the visual information obtained by the binocular camera, and MSCKF algorithm is employed to realize the fusion of visual information and inertial information, which improves the accuracy and efficiency of the visual inertial odometry system positioning. Finally, the proposed method is simulated on the European Robotics Challenge (EuRoc) dataset by Robot Operating System (ROS), and compared with two other advanced visual inertial odometry systems, ROVIO and MSCKF-mono. A large number of simulations verify that the proposed method can achieve accurate pose estimation, which is superior to the two existing advanced visual inertial odometry systems.

Accurate IMU Factor Using Switched Linear Systems for VIO

Accurate motion estimation plays a crucial role in state estimation of an unmanned aerial vehicle (UAV). This is usually carried out by fusing the kinematics of an inertial measurement unit (IMU) with the video output of a camera. However, the accuracy of existing approaches is hindered by the discretization effect of the model even at a high IMU sampling rate. In order to improve the accuracy, we propose a new IMU motion integration model for the IMU kinematics in continuous time. The kinematics are modeled using a switched linear system. A closed-form discrete formulation is derived to compute the mean measurement, the covariance matrix, and the Jacobian matrix. Thus, it is more accurate and more efficient for online estimation of visual-inertial odometry (VIO), particularly when there is a high dynamic change in the agent's motion or the agent travels with high speed. The proposed IMU factor framework is evaluated using both real public datasets and indoor environment under different scenarios of motion capture. Our evaluation shows that the proposed framework outperforms the state-of-the-art VIO approach by up to 22.71% accuracy improvement on the EuRoc dataset and 38.15% accuracy improvement for motion estimation under the indoor environment.

Lightweight hybrid visual-inertial odometry with closed-form zero velocity update

Visual-Inertial Odometry (VIO) fuses measurements from camera and Inertial Measurement Unit (IMU) to achieve accumulative performance that is better than using individual sensors. Hybrid VIO is an extended Kalman filter-based solution which augments features with long tracking length into the state vector of Multi-State Constraint Kalman Filter (MSCKF). In this paper, a novel hybrid VIO is proposed, which focuses on utilizing low-cost sensors while also considering both the computational efficiency and positioning precision. The proposed algorithm introduces several novel contributions. Firstly, by deducing an analytical error transition equation, one-dimensional inverse depth parametrization is utilized to parametrize the augmented feature state. This modification is shown to significantly improve the computational efficiency and numerical robustness, as a result achieving higher precision. Secondly, for better handling of the static scene, a novel closed-form Zero velocity UPdaTe (ZUPT) method is proposed. ZUPT is modeled as a measurement update for the filter rather than forbidding propagation roughly, which has the advantage of correcting the overall state through correlation in the filter covariance matrix. Furthermore, online spatial and temporal calibration is also incorporated. Experiments are conducted on both public dataset and real data. The results demonstrate the effectiveness of the proposed solution by showing that its performance is better than the baseline and the state-of-the-art algorithms in terms of both efficiency and precision. A related software is open-sourced to benefit the community.①https://github.com/PetWorm/LARVIO①

CKF-Based Visual Inertial Odometry for Long-Term Trajectory Operations

The estimation error accumulation in the conventional visual inertial odometry (VIO) generally forbids accurate long-term operations. Some advanced techniques such as global pose graph optimization and loop closure demand relatively high computation and processing time to execute the optimization procedure for the entire trajectory and may not be feasible to be implemented in a low-cost robotic platform. In an attempt to allow the VIO to operate for a longer duration without either using or generating a map, this paper develops iterated cubature Kalman filter for VIO application that performs multiple corrections on a single measurement to optimize the current filter state and covariance during the measurement update. The optimization process is terminated using the maximum likelihood estimate based criteria. For comparison, this paper also develops a second solution to integrate VIO estimation with ranging measurements. The wireless communications between the vehicle and multiple beacons produce the ranging measurements and help to bound the accumulative errors. Experiments utilize publicly available dataset for validation, and a rigorous comparison between the two solutions is presented to determine the application scenario of each solution.

The First Attempt of SAR Visual-Inertial Odometry

This article proposes a novel synthetic aperture radar visual-inertial odometry (SAR-VIO) consisting of an SAR and an inertial measurement unit (IMU), which aims to enable the observation platform to complete successfully a continuous observation mission in the context of low-cost demand and lack of enough navigation information. First, we establish the observation models of the SAR in a continuous observation process based on the SAR frequency-domain imaging algorithm and the SAR time-domain imaging algorithm, respectively. With the preintegrated IMU data, we then propose a method for estimating the geographic locations of the matched targets in the SAR images and verify the condition and correctness of the method. The optimization of the track and the locations of the targets is achieved by bundle adjustment according to the minimum reprojection error criterion, and a sparse point-cloud map can be obtained. Finally, these methods and models are organized into a complete SAR-VIO framework, and the feasibility of the framework is verified through experiments.

Keyframe-Based RGB-D Visual-Inertial Odometry and Camera Extrinsic Calibration Using Extended Kalman Filter

This paper proposes a novel method for Visual-Inertial Odometry (VIO) based on a RGB-D camera and an Inertial Measurement Unit (IMU). To fuse the data from visual and inertial measurements, an Extended Kalman Filter (EKF) is deployed, while we propose an iterative procedure to reduce the linearization error. Our method not only estimates the trajectory of the camera, but also calibrates the gravity and the camera extrinsics, i.e. the relative pose between camera and IMU. For what concerns visual odometry, a KeyFrame strategy is employed, as it tends to outperform frame-to-frame alignment. Moreover, we combine feature- and ICP-based visual odometry, so to provide accurate and robust estimation. Evaluation results on simulation and real data show the effectiveness of our approach.

Camera Intrinsic Parameters Estimation by Visual–Inertial Odometry for a Mobile Phone With Application to Assisted Navigation

The increasing computing and sensing capabilities of modern mobile phones have spurred research interests in developing new visual–inertial odometry (VIO) techniques to turn a smartphone into a self-contained vision-aided inertial navigation system for various applications. Smartphones nowadays use cameras with optical image stabilization (OIS) technology to reduce image blurs. However, the mechanism may result in varying camera intrinsic parameters (CIP), which must be taken into account in VIO computation. In this article, we first develop a linear model to relate the CIP with the inertial measurement unit measured acceleration. Based on the model, we introduce a new VIO method, called CIP-VMobile, which treats CIP as state variables and tightly couples them with other state variables in a graph optimization process to estimate the optimal state. The method uses the linear model to construct a factor graph and uses the linear-model-computed values as initial CIP estimates to speed up the VIO computation and attain a better pose estimation result. Simulation and experimental results with an iPhone 7 validate the method's efficacy. Based on CIP-VMobile, we fabricated a robotic navigation aid (RNA) based on an iPhone 7 for assisted navigation. Experimental results with the RNA demonstrate CIP-VMobile's promise in real-world navigation applications.