Stereo Visual Inertial Odometry Research Articles

UAV navigation in large-scale GPS-denied bridge environments using fiducial marker-corrected stereo visual-inertial localisation

The use of Unmanned Aerial Vehicles (UAVs) for bridge inspection has gained popularity recently; however, accurately localising the UAV in GPS-denied areas is still challenging, which hinders the development of fully autonomous UAV-assisted bridge inspection solutions. This paper proposes a fiducial marker-corrected stereo visual-inertial localisation (FMC-SVIL) method, running on a resource-constrained onboard computer, to estimate UAV's global pose underneath bridge girders. The proposed FMC-SVIL utilises an optimised stereo visual-inertial odometry for continuous relative pose estimation between consecutive camera frames and an improved AprilTag2-based measurement algorithm for accurate global referencing and periodic pose corrections. The method is validated through extensive experiments, and the results show that the FMC-SVIL achieved UAV localisation with a root mean square error of 0.416 m in sunny conditions and 0.340 m in cloudy conditions. FMC-SVIL outperforms the leading vision-based simultaneous localisation and mapping (SLAM) algorithms for flights over multiple bridge spans.

ESVIO: Event-Based Stereo Visual Inertial Odometry

Event cameras that asynchronously output low-latency event streams provide great opportunities for state estimation under challenging situations. Despite event-based visual odometry having been extensively studied in recent years, most of them are based on the monocular, while few research on stereo event vision. In this paper, we present ESVIO, the first event-based stereo visual-inertial odometry, which leverages the complementary advantages of event streams, standard images, and inertial measurements. Our proposed pipeline includes the ESIO (purely event-based) and ESVIO (event with image-aided), which achieves spatial and temporal associations between consecutive stereo event streams. A well-design back-end tightly-coupled fused the multi-sensor measurement to obtain robust state estimation. We validate that both ESIO and ESVIO have superior performance compared with other image-based and event-based baseline methods on public and self-collected datasets. Furthermore, we use our pipeline to perform onboard quadrotor flights under low-light environments. Autonomous driving data sequences and real-world large-scale experiments are also conducted to demonstrate long-term effectiveness. We highlight that this work is a real-time, accurate system that is aimed at robust state estimation under challenging environments.

An online temporal calibration method for nonlinear optimization-based stereo visual-inertial odometry

Mobile robots have been widely used in geological exploration, accident rescue, and other fields. As the core technology of these robots, simultaneous localization and mapping is responsible for estimating the robot’s position and building a scene map. With the rapid development of simultaneous localization and mapping technology in multi-sensor fusion, many visual-inertial odometry systems based on camera and inertial measurement unit fusion have been proposed. Although visual-inertial odometry has advantages over visual simultaneous localization and mapping in accuracy and robustness, they still cannot meet the requirements of robots to work stably for a long time. The time misalignment (time offset) between visual and inertial on the timeline is one of the main reasons for that problem. Therefore, this article proposes an online temporal calibration method based on nonlinear optimization, suitable for low-cost, self-assembled stereo visual-inertial odometry systems. We take full advantage of stereo camera, which constructs a new error factor concerning time offset using the epipolar constraint in the stereo camera. The proposed method could reduce the negative influence of features’ velocity errors during temporal calibration and improve the system’s robustness and accuracy. Experiments on the EuRoC data set show that our method’s temporal calibration results are closer to the actual value than the state-of-the-art calibration methods. Our method can significantly improve the system’s accuracy and robustness.

ESVIO: Event-Based Stereo Visual-Inertial Odometry.

The emerging event cameras are bio-inspired sensors that can output pixel-level brightness changes at extremely high rates, and event-based visual-inertial odometry (VIO) is widely studied and used in autonomous robots. In this paper, we propose an event-based stereo VIO system, namely ESVIO. Firstly, we present a novel direct event-based VIO method, which fuses events' depth, Time-Surface images, and pre-integrated inertial measurement to estimate the camera motion and inertial measurement unit (IMU) biases in a sliding window non-linear optimization framework, effectively improving the state estimation accuracy and robustness. Secondly, we design an event-inertia semi-joint initialization method, through two steps of event-only initialization and event-inertia initial optimization, to rapidly and accurately solve the initialization parameters of the VIO system, thereby further improving the state estimation accuracy. Based on these two methods, we implement the ESVIO system and evaluate the effectiveness and robustness of ESVIO on various public datasets. The experimental results show that ESVIO achieves good performance in both accuracy and robustness when compared with other state-of-the-art event-based VIO and stereo visual odometry (VO) systems, and, at the same time, with no compromise to real-time performance.

Stereo Visual-Inertial Odometry With Online Initialization and Extrinsic Self-Calibration

Reliable extrinsic calibration is critical to fuse camera and inertial measurement unit (IMU), which are usually used in stereo visual-inertial odometry (VIO), however, it is difficult to obtain extrinsic parameters in practice. This paper proposes a stereo VIO with the capability of calibrating the unknown extrinsic parameters online. The initial values of IMU-camera and camera-camera transformations are estimated during the initial process, which fully leverages commonly observed features between stereo cameras. The pose and velocity are estimated and the extrinsic parameters are jointly refined. Additionally, the accelerometer and gyroscope biases, and gravity direction are taken into account. The proposed VIO scheme have been demonstrated to be effective without prior knowledge of extrinsic parameters.

Stereo visual-inertial odometry using structural lines for localizing indoor wheeled robots

This paper proposes optimization-based stereo visual-inertial odometry (VIO) to locate indoor wheeled robots. The multiple Manhattan worlds (MWs) assumption is adopted to model the interior environment. Instead of treating these worlds as isolated ones, we fuse the latest MW with the previous ones if they are in the same direction, reducing the calculated errors on the orientation of the latest MW. Then, the structural lines that encode the orientation information of these worlds are taken as additional landmarks to improve the positioning accuracy and reduce the accumulated drift of the system, especially when the system is in a challenging environment (i.e. scenes with continuous turning and low textures). In addition, the structural lines are parameterized by only two variables, which improves the computational efficiency and simplifies the initialization of lines. Experiments on public benchmark datasets and in real-world environments demonstrate that the proposed VIO system can accurately position the wheeled robot in a complex indoor environment.

Human localization in robotized warehouses based on stereo odometry and ground-marker fusion

Modern logistic solutions for large warehouses consist of a fleet of robots that transfer goods, move racks, and perform other physically difficult and repetitive tasks. The shopfloor is usually enclosed with a safety fence and if a human needs to enter the warehouse all the robots are stopped, as opposed to only the ones in the most immediate vicinity of the human, thus significantly limiting the warehouse efficiency. To tackle this challenge, an integrated safety system is needed with human localization as one of its essential components. In this paper, we propose a novel human localization method for robotized warehouses that is based on a suite of wearable visual sensors installed on a vest worn by humans. The proposed method does not require any modifications of the warehouse environment and relies on the already existing infrastructure. Specifically, we estimate the human location by fusing stereo visual-inertial odometry data and distances to the known absolute poses of the detected ground-markers which robots use for their localization. Fusion is performed by building a pose graph, where we treat estimated human poses relative to markers as graph nodes and odometry estimates as graph edges. We conducted extensive laboratory and warehouse facility experiments, where we tested the reliability and accuracy of the proposed method and compared its performance to a state-of-the-art visual SLAM solution, namely ORB-SLAM2. The results indicate that our method can track absolute position in real-time and has competitive accuracy with respect to ORB-SLAM2, while ensuring higher localization reliability when faced with structural changes in the environment. Furthermore, we provide publicly the experimental datasets to the research community.

SLC-VIO: a stereo visual-inertial odometry based on structural lines and points belonging to lines

AbstractTo improve mobile robot positioning accuracy in building environments and construct structural three-dimensional (3D) maps, this paper proposes a stereo visual-inertial odometry (VIO) system based on structural lines and points belonging to lines. The 2-degree-of-freedom (DoF) spatial structural lines based on the Manhattan world assumption are used to establish visual measurement constraints. The property of point belonging to a line (PPBL) is used to initialize the structural lines and establish spatial distance-residual constraints between point and line landmarks in the reconstructed 3D map. Compared with the 4-DoF spatial straight line, the 2-DoF structural line reduces the variables to be estimated and introduces the orientation information of scenes to the VIO system. The utilization of PPBL makes the proposed system fully exploit the prior geometric information of environments and then achieves better performance. Tests on public data sets and real-world experiments show that the proposed system can achieve higher positioning accuracy and construct 3D maps that better reflect the structure of scenes than existing VIO approaches.

UniVIO: Unified Direct and Feature-Based Underwater Stereo Visual-Inertial Odometry

Providing an accurate trajectory in an underwater scene is challenging for visual odometry due to weak texture and varying illumination. In this article, we propose a novel underwater visual-inertial odometry (VIO) with the unifying direct method and feature-based method. Our approach starts with a new rectification method for underwater images. The new underwater image rectification method separately eliminates water-air refraction distortion and lens distortion with an approximate single viewpoint (SVP) camera model. Meanwhile, a unified optimization method is used in our approach to obtain robust and highly accurate odometry results. This method jointly optimizes projection errors and photometric errors. Moreover, the robust data association processing, which combines <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">keyframe</i> / <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">keypoint</i> selection, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">keypoint</i> position prediction, and robust local optical flow, was designed in our approach. The robust data association processing greatly improves the success rate of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">keypoints</i> tracking. To evaluate the precision of underwater visual odometry, an underwater dataset with complete and highly accurate ground truth trajectories is provided in this work. The performance of our system is compared with other state-of-the-art algorithms on multiple datasets, including visual odometry datasets and our proposed underwater dataset. The evaluation results show that the proposed system can achieve better results than the state-of-the-art algorithms. Furthermore, the proposed system is validated on public natural underwater datasets to show the performance of the system in a real underwater environment. The qualitative results demonstrate that the proposed system can work well in the natural underwater environment.

X-RIO: Radar Inertial Odometry with Multiple Radar Sensors and Yaw Aiding

A robust and accurate real-time navigation system is crucial for autonomous robotics. In particular, GNSS denied and poor visual conditions are still very challenging as vision based approaches tend to fail in darkness, direct sunlight, fog or smoke. Therefore, we are taking advantage of inertial data and FMCW radar sensors as both are not affected by such conditions. In this work, we propose a framework, which uses several 4D mmWave radar sensors simultaneously. The extrinsic calibration of each radar sensor is estimated online. Based on a single radar scan, the 3D ego velocity and optionally yaw measurements based on Manhattan world assumptions are fused. An extensive evaluation with real world datasets is presented. We achieve even better accuracies than state of the art stereo Visual Inertial Odometry (VIO) while being able to cope with degraded visual conditions and requiring only very little computational resources.

An Online Initialization and Self-Calibration Method for Stereo Visual-Inertial Odometry

Most online initialization and self-calibration methods for visual-inertial odometry (VIO) are only able to estimate the extrinsic parameters (orientation and translation) between one camera and inertial measurement unit (IMU) pair. They are not applicable to stereo VIO where both camera-IMU and camera-camera pairs exist. In this article, we address the issue by taking advantage of the geometric constraints among the multiple sensors. An online method is proposed to estimate the initial values of velocity, gravity, IMU biases, and simultaneously calibrate the extrinsic parameters of camera-camera and camera-IMU pairs for bootstrapping a smoothing-based stereo VIO system. The method includes a three-step process to incrementally solve several linear equations in a coarse-to-fine manner. It back-propagates historically estimated results to update weight factors and remove outliers, and employs a convergence criterion to monitor and terminate the process. It also includes an optional global optimization for further refinement. The method is evaluated in terms of accuracy, robustness, convergence, consistency, and tunable parameters using both simulated and public datasets. Experimental results show that the proposed method can accurately estimate the initial values and the extrinsic parameters.

Joint optimization based on direct sparse stereo visual-inertial odometry

This paper proposes a novel fusion of an inertial measurement unit (IMU) and stereo camera method based on direct sparse odometry (DSO) and stereo DSO. It jointly optimizes all model parameters within a sliding window, including the inverse depth of all selected pixels and the internal or external camera parameters of all keyframes. The vision part uses a photometric error function that optimizes 3D geometry and camera pose in a combined energy functional. The proposed algorithm uses image blocks to extract neighboring image features and directly forms measurement residuals in the image intensity space. A fixed-baseline stereo camera solves scale drift. IMU information is accumulated between several frames using manifold pre-integration and is inserted into the optimization as additional constraints between keyframes. The scale and gravity inserted are incorporated into the stereo visual inertial odometry model and are optimized together with other variables such as poses. The experimental results show that the tracking accuracy and robustness of the proposed method are superior to those of the state-of-the-art fused IMU method. In addition, compared with previous semi-dense direct methods, the proposed method displays a higher reconstruction density and scene recovery.

Robust Stereo Visual-Inertial Odometry Using Nonlinear Optimization.

The fusion of visual and inertial odometry has matured greatly due to the complementarity of the two sensors. However, the use of high-quality sensors and powerful processors in some applications is difficult due to size and cost limitations, and there are also many challenges in terms of robustness of the algorithm and computational efficiency. In this work, we present VIO-Stereo, a stereo visual-inertial odometry (VIO), which jointly combines the measurements of the stereo cameras and an inexpensive inertial measurement unit (IMU). We use nonlinear optimization to integrate visual measurements with IMU readings in VIO tightly. To decrease the cost of computation, we use the FAST feature detector to improve its efficiency and track features by the KLT sparse optical flow algorithm. We also incorporate accelerometer bias into the measurement model and optimize it together with other variables. Additionally, we perform circular matching between the previous and current stereo image pairs in order to remove outliers in the stereo matching and feature tracking steps, thus reducing the mismatch of feature points and improving the robustness and accuracy of the system. Finally, this work contributes to the experimental comparison of monocular visual-inertial odometry and stereo visual-inertial odometry by evaluating our method using the public EuRoC dataset. Experimental results demonstrate that our method exhibits competitive performance with the most advanced techniques.

Experimental Comparison of Visual-Aided Odometry Methods for Rail Vehicles

Today, rail vehicle localization is based on infrastructure-side Balises (beacons) together with on-board odometry to determine whether a rail segment is occupied. Such a coarse locking leads to a sub-optimal usage of the rail networks. New railway standards propose the use of moving blocks centered around the rail vehicles to increase the capacity of the network. However, this approach requires accurate and robust position and velocity estimation of all vehicles. In this work, we investigate the applicability, challenges and limitations of current visual and visual-inertial motion estimation frameworks for rail applications. An evaluation against RTK-GPS ground truth is performed on multiple datasets recorded in industrial, sub-urban, and forest environments. Our results show that stereo visual-inertial odometry has a great potential to provide a precise motion estimation because of its complementing sensor modalities and shows superior performance in challenging situations compared to other frameworks.

Stereo visual-inertial odometry with an online calibration and its field testing

In this paper, we present a visual-inertial odometry (VIO) with an online calibration using a stereo camera in planetary rover localization. We augment the state vector with extrinsic (rigid body transformation) and temporal (time-offset) parameters of a camera-IMU system in a framework of an extended Kalman filter. This is motivated by the fact that when fusing independent systems, it is practically crucial to obtain precise extrinsic and temporal parameters. Unlike the conventional calibration procedures, this method estimates both navigation and calibration states from naturally occurred visual point features during operation. We describe mathematical formulations of the proposed method, and it is evaluated through the author-collected dataset which is recorded by the commercially available visual-inertial sensor installed on the testing rover in the environment lack of vegetation and artificial objects. Our experimental results showed that 3D return position error as 1.54m of total 173m traveled and 10ms of time-offset with the online calibration, while 6.52m of return position error without the online calibration.

Optimization-Based, Simplified Stereo Visual-Inertial Odometry With High-Accuracy Initialization

This paper proposes a robust and highly efficient feature-based visual-inertial odometry (VIO) approach. In order to save the computational resource, a simplified stereo visual model is applied to reduce the dimension of visual measurements. Moreover, the speed of feature matching is improved by using prior information from the inertial sensor. And through the marginalization, optimization is limited in two sliding windows, which can meet the need for the real-time application. In addition, a high accuracy initialization method and the homogeneous extraction of image features are introduced to ensure excellent pose tracking. The proposed VIO system is implemented on open datasets to show its merit compared with other state-of-the-art algorithms. In addition, we also perform this system on a low-cost stereo visual-inertial sensor and validate its practicability and superiority. Furthermore, the comparative experiment shows that the proposed algorithm has a higher accuracy than the monocular VIO and a shorter running time than the stereo VIO.

Vehicle Positioning in GNSS-Deprived Urban Areas by Stereo Visual-Inertial Odometry

Accurate and continuous positioning in global navigation satellite system (GNSS) deprived urban areas is crucial for autonomous vehicles and mobile mapping systems. To achieve this goal, we propose a stereo visual-inertial odometry approach using a multistate constraint Kalman filter (MSCKF). In contrast with the conventional MSCKF, in which an inertial navigation system (INS) propagates the vehicle motion, and then the propagation is corrected by measurements of salient features extracted from images of a single camera, we update the propagation with observations extracted from images of a stereo pair of cameras. This way, additional constraints across the stereo pairs of images are exploited to improve the pose estimation. Experimental results on several KITTI datasets show that the stereo MSCKF outperforms the mono one achieving an average positioning error of 0.9% of the trajectory length compared to 1.3% for the mono approach and 2.5% for INS-only integration. These results show that visual-inertial odometry has a promising potential for vehicle positioning in short periods of GNSS signal outage.

Robust Stereo Visual Inertial Odometry for Fast Autonomous Flight

In recent years, vision-aided inertial odometry for state estimation has matured significantly. However, we still encounter challenges in terms of improving the computational efficiency and robustness of the underlying algorithms for applications in autonomous flight with microaerial vehicles, in which it is difficult to use high-quality sensors and powerful processors because of constraints on size and weight. In this letter, we present a filter-based stereo visual inertial odometry that uses the multistate constraint Kalman filter. Previous work on the stereo visual inertial odometry has resulted in solutions that are computationally expensive. We demonstrate that our stereo multistate constraint Kalman filter (S-MSCKF) is comparable to state-of-the-art monocular solutions in terms of computational cost, while providing significantly greater robustness. We evaluate our S-MSCKF algorithm and compare it with state-of-the-art methods including OKVIS, ROVIO, and VINS-MONO on both the EuRoC dataset and our own experimental datasets demonstrating fast autonomous flight with a maximum speed of 17.5 m/s in indoor and outdoor environments. Our implementation of the S-MSCKF is available at https://github.com/KumarRobotics/msckf_vio.

Stereo Visual-Inertial Odometry With Multiple Kalman Filters Ensemble

In this paper, we present a stereo visual-inertial odometry algorithm assembled with three separated Kalman filters, i.e., attitude filter, orientation filter, and position filter. Our algorithm carries out the orientation and position estimation with three filters working on different fusion intervals, which can provide more robustness even when the visual odometry estimation fails. In our orientation estimation, we propose an improved indirect Kalman filter, which uses the orientation error space represented by unit quaternion as the state of the filter. The performance of the algorithm is demonstrated through extensive experimental results, including the benchmark KITTI datasets and some challenging datasets captured in a rough terrain campus.