Visual Inertial Odometry System Research Articles

A novel online time calibration framework via double-stage EKF for visual-inertial odometry

Abstract There is an unavoidable time offset between the camera stream and the inertial measurement unit (IMU) data due to the sensor triggering and transmission delays, which will seriously affect the accuracy of visual-inertial odometry (VIO). A novel online time calibration framework via double-stage EKF for VIO is proposed in this paper. First, the first-stage complementary Kalman filter is constructed by adapting the complementary characteristics between the accelerometer and the gyroscope in the IMU, where the rotation result predicted by the gyroscope is corrected through the measurement of the accelerometer so that the IMU can output a more accurate initial pose. Second, the unknown time offset is added to the state vector of the VIO system. The estimated pose of IMU is used as the prediction information, and the reprojection error of multiple cameras on the same feature point is used as the constraint information. During the operation of the VIO system, the time offset is continuously calculated and superimposed on the IMU timestamp to obtain the data synchronized by the IMU and the camera. The Schur complement model is used to marginalize the camera state that carries less information in the system state, avoiding the loss of prior information between images, and improving the accuracy of camera pose estimation. Finally, the effectiveness of proposed algorithm is verified using the EuRoC dataset and the real experimental data.

Asynchronous Visual-Inertial Odometry for Event Cameras

Abstract. Event cameras are bio-inspired visual sensors that output changes in pixel-level brightness asynchronously instead of standard intensity frames at a fixed rate. These cameras offer reliable visual information in high-speed motion and high dynamic range (HDR) scenes, addressing the limitations of traditional cameras in such scenarios. Therefore, research of integrating event cameras into established visual algorithms holds significant value. In this study, based on traditional Visual-Inertial Odometry (VIO) frameworks, we proposed an innovative asynchronous monocular event-based inertial odometry method to fully exploit the benefits of event cameras. First, the corner features are extracted separately from the raw event stream and the time surface map, followed by uniform feature selection to accurately describe three-dimensional spatial geometry. Then, feature tracking is achieved by integrating these two event representation methods. In addition, our method obtains stable and high frequency state estimation by fusing event and IMU measurements through graph optimization. We validate the effectiveness of our proposed approach, comparing with several state-of-the-art EVIO systems and VIO systems.

RD-VIO: Robust Visual-Inertial Odometry for Mobile Augmented Reality in Dynamic Environments.

It is typically challenging for visual or visual-inertial odometry systems to handle the problems of dynamic scenes and pure rotation. In this work, we design a novel visual-inertial odometry (VIO) system called RD-VIO to handle both of these two problems. First, we propose an IMU-PARSAC algorithm which can robustly detect and match keypoints in a two-stage process. In the first state, landmarks are matched with new keypoints using visual and IMU measurements. We collect statistical information from the matching and then guide the intra-keypoint matching in the second stage. Second, to handle the problem of pure rotation, we detect the motion type and adapt the deferred-triangulation technique during the data-association process. We make the pure-rotational frames into the special subframes. When solving the visual-inertial bundle adjustment, they provide additional constraints to the pure-rotational motion. We evaluate the proposed VIO system on public datasets and online comparison. Experiments show the proposed RD-VIO has obvious advantages over other methods in dynamic environments.

Visual–Inertial Odometry of Structured and Unstructured Lines Based on Vanishing Points in Indoor Environments

In conventional point-line visual–inertial odometry systems in indoor environments, consideration of spatial position recovery and line feature classification can improve localization accuracy. In this paper, a monocular visual–inertial odometry based on structured and unstructured line features of vanishing points is proposed. First, the degeneracy phenomenon caused by a special geometric relationship between epipoles and line features is analyzed in the process of triangulation, and a degeneracy detection strategy is designed to determine the location of the epipoles. Then, considering that the vanishing point and the epipole coincide at infinity, the vanishing point feature is introduced to solve the degeneracy and direction vector optimization problem of line features. Finally, threshold constraints are used to categorize straight lines into structural and non-structural features under the Manhattan world assumption, and the vanishing point measurement model is added to the sliding window for joint optimization. Comparative tests on the EuRoC and TUM-VI public datasets validated the effectiveness of the proposed method.

A self-supervised monocular odometry with visual-inertial and depth representations

Background:Self-supervised monocular visual-odometry (VO) methods have demonstrated impressive results in relative pose estimation, but they encounter challenges when visual information is unreliable. To address this, we introduce a self-supervised visual-inertial odometry (VIO) system. Methods:Our method adopts a trinocular assumption, where three consecutive frames are treated as if captured simultaneously by three cameras. This approach expands the field of view and enhances robustness in visual constraints for 3D structures. Additionally, we incorporate Inertial Measurement Unit (IMU) data into the system. This integration, achieved by an attention-based fusion strategy, allows our visual-inertial odometry (VIO) system to adaptively select the most reliable representations for accurate relative pose estimation. Results:Experiments on public and self-collected datasets show that the proposed method is state-of-the-art in visual odometry. Conclusion:The encouraging results show that it is feasible to apply this learning-based VIO system in autonomous cars and robots.

RVIO: An Effective Localization Algorithm for Range-Aided Visual-Inertial Odometry System

RVIO: An Effective Localization Algorithm for Range-Aided Visual-Inertial Odometry System

Point feature correction based rolling shutter modeling for EKF-based visual-inertial odometry

The application of consumer-grade rolling-shutter (RS) cameras in visual inertial odometry (VIO) systems deserves attention. In previous works, VIO systems with RS cameras usually adopt different dimensions of RS modeling to represent the camera motion. Although the complex camera motion expression improves accuracy, adding more variables greatly increases the computation load. Therefore, considering the limitations of camera motion modeling, this paper introduces a point feature correction strategy to improve the performance of the system in terms of both speed and accuracy. The optimized camera poses and high-frequency inertial measurement unit (IMU) are employed to correct the point features in the RS image to new pixel coordinates in the global shutter view (in the middle of the RS image readout time) by epipolar transfer. At the same time, we handle the zero velocity and abnormal cases of RS modeling to improve the robustness of the system. Furthermore, the readout time of the RS camera is self-calibrated online during the state augmentation stage and can quickly converge and stabilize to near the true value. The readout time of the RS camera is locally observable, except for two degenerate motions. Experimental results demonstrate that the proposed method outperforms the state-of-the-art methods on a public dataset in terms of accuracy and computational cost. We port our algorithm to an Android-based mobile phone and our system outputs real-time trajectories on the mobile phone. Then, we use the motion capture system to obtain the ground truth phone pose to test the performance of our algorithm.

Research on Multi-Sensor Simultaneous Localization and Mapping Technology for Complex Environment of Construction Machinery

Simultaneous localization and mapping (SLAM), as a key task of unmanned vehicles for construction machinery, is of great significance for later path planning and control. Construction tasks in the engineering field are mostly carried out in bridges, tunnels, open fields, etc. The prominent features of these environments are high scene similarity, few geometric features, and large-scale repetitive texture information, which is prone to sensor detection degradation. This leads to positioning drift and map building failure. The traditional method of motion estimation and 3D reconstruction uses a single sensor, which lacks enough information, has poor adaptability to the environment, and cannot guarantee good positioning accuracy and robustness in complex environments. Currently, the strategy of multi-sensor fusion is proven to be an effective solution and is widely studied. This paper proposes a SLAM framework that integrates LiDAR, IMU, and camera. It tightly couples the texture information observed by camera, the geometric information scanned by LiDAR, and the measured value of IMU, allowing visual-inertial odometry (VIO) and LiDAR-inertial odometry (LIO) common implementation. The LIO subsystem extracts point cloud features and matches them with the global map. The obtained pose estimation can be used for the initialization of the VIO subsystem. The VIO system uses direct method to minimize the photometric error and IMU measurement error between images to estimate the pose of the robot and the geometric structure of the scene. The two subsystems assist each other to perform pose estimation, and can operate normally even when any subsystem fails. A factor graph is used to combine all constraints to achieve global pose optimization. Keyframe and sliding window strategies are used to ensure real-time performance. Through real-vehicle testing, the system can perform incremental and real-time state estimation and reconstruct a dense 3D point cloud map, which can effectively solve the problems of positioning drift and mapping failure in the lack of geometric features or challenging construction environments.

S-VIO: Exploiting Structural Constraints for RGB-D Visual Inertial Odometry

Vision-based localization is an essential problem for autonomous systems while the performance of vision-based odometry degrades in the challenging scenario. This letter presents S-VIO, an RGB-D visual inertial odometry (VIO) which fully uses multi-sensor measurements (i.e. depth, RGB and IMU), heterogeneous landmarks (i.e. points, lines and planes) and structural regularity of the environment to obtain a robust and accurate localization result. In order to detect the underlying structural regularity of the environment, a two-step Atlanta world inference method is proposed. Leveraging the gravity direction estimated by the VIO system, the proposed algorithm first generates horizontal Atlanta axis hypotheses from a set of recently optimized plane landmarks. Then the following plane landmarks and line clusters are used to filter out the occasionally observed axes based on the persistence of the observation. The remaining axes will survive and be saved in the Atlanta map for future re-observations. Particularly, an efficient mined-and-stabbed (MnS) method is applied to classify the structural line and extract the vanishing point from each line cluster. In addition, a closed-form initialization method for the structural line feature is proposed, which leverages the known direction to obtain a better initial estimation. Integrated with the above novelties, S-VIO is tested on two public real-world RGB-D inertial datasets. Experiments demonstrate that S-VIO has better accuracy and robustness compared to state-of-the-art VIO and RGB-D VIO algorithms. The code and the supplementary material will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/oranfire/SVIO</uri> .

RD-VIO: Relative-depth-aided visual-inertial odometry for autonomous underwater vehicles

Because of the autonomous underwater vehicles' (AUVs') slow motion and insufficient excitations, monocular visual-inertial odometry (VIO) systems for AUVs still have large-scale errors and a lengthy initialization period. In this paper, we present a relative-depth-aided monocular visual-inertial odometry (RD-VIO) system based on two innovative methods: Inertial Measurement Unit (IMU)-depth joint initialization and relative depth tightly coupled with visual-inertial. The RD-VIO performs reprojection bundle adjustment with dynamic weight factors for visual residuals. Depth measurements acquired by depth gauge or pressure sensor are compensated with reliable inter-image IMU pre-integration to account for different sensor frequencies. Also, we adopt an IMU-depth joint initialization using relative depth measurements and IMU excitations to estimate the scale and gravity direction together, which captures the complete AUV motion changes, improving scale estimation and decreasing initialization time. To reduce scale drift and improve global accuracy and consistency, a tight coupling of relative depth and visual-inertial in the RD-VIO is employed to optimize the AUV pose from depth measurements. Using two open-source datasets and our experiments from Sanya, we evaluated the performance of RD-VIO. The evaluation results show that the proposed system can achieve a more accurate trajectory than the state-of-the-art algorithms with less system initialization time and scale error.

Editorial Issue 34.2

Editorial Issue 34.2

An online temporal calibration method for nonlinear optimization-based stereo visual-inertial odometry

Mobile robots have been widely used in geological exploration, accident rescue, and other fields. As the core technology of these robots, simultaneous localization and mapping is responsible for estimating the robot’s position and building a scene map. With the rapid development of simultaneous localization and mapping technology in multi-sensor fusion, many visual-inertial odometry systems based on camera and inertial measurement unit fusion have been proposed. Although visual-inertial odometry has advantages over visual simultaneous localization and mapping in accuracy and robustness, they still cannot meet the requirements of robots to work stably for a long time. The time misalignment (time offset) between visual and inertial on the timeline is one of the main reasons for that problem. Therefore, this article proposes an online temporal calibration method based on nonlinear optimization, suitable for low-cost, self-assembled stereo visual-inertial odometry systems. We take full advantage of stereo camera, which constructs a new error factor concerning time offset using the epipolar constraint in the stereo camera. The proposed method could reduce the negative influence of features’ velocity errors during temporal calibration and improve the system’s robustness and accuracy. Experiments on the EuRoC data set show that our method’s temporal calibration results are closer to the actual value than the state-of-the-art calibration methods. Our method can significantly improve the system’s accuracy and robustness.

Event-based feature tracking in a visual inertial odometry framework.

Introduction: Event cameras report pixel-wise brightness changes at high temporal resolutions, allowing for high speed tracking of features in visual inertial odometry (VIO) estimation, but require a paradigm shift, as common practices from the past decades using conventional cameras, such as feature detection and tracking, do not translate directly. One method for feature detection and tracking is the Eventbased Kanade-Lucas-Tomasi tracker (EKLT), an hybrid approach that combines frames with events to provide a high speed tracking of features. Despite the high temporal resolution of the events, the local nature of the registration of features imposes conservative limits to the camera motion speed. Methods: Our proposed approach expands on EKLT by relying on the concurrent use of the event-based feature tracker with a visual inertial odometry system performing pose estimation, leveraging frames, events and Inertial Measurement Unit (IMU) information to improve tracking. The problem of temporally combining high-rate IMU information with asynchronous event cameras is solved by means of an asynchronous probabilistic filter, in particular an Unscented Kalman Filter (UKF). The proposed method of feature tracking based on EKLT takes into account the state estimation of the pose estimator running in parallel and provides this information to the feature tracker, resulting in a synergy that can improve not only the feature tracking, but also the pose estimation. This approach can be seen as a feedback, where the state estimation of the filter is fed back into the tracker, which then produces visual information for the filter, creating a "closed loop". Results: The method is tested on rotational motions only, and comparisons between a conventional (not event-based) approach and the proposed approach are made, using synthetic and real datasets. Results support that the use of events for the task improve performance. Discussion: To the best of our knowledge, this is the first work proposing the fusion of visual with inertial information using events cameras by means of an UKF, as well as the use of EKLT in the context of pose estimation. Furthermore, our closed loop approach proved to be an improvement over the base EKLT, resulting in better feature tracking and pose estimation. The inertial information, despite prone to drifting over time, allows keeping track of the features that would otherwise be lost. Then, feature tracking synergically helps estimating and minimizing the drift.

ESVIO: Event-Based Stereo Visual-Inertial Odometry.

The emerging event cameras are bio-inspired sensors that can output pixel-level brightness changes at extremely high rates, and event-based visual-inertial odometry (VIO) is widely studied and used in autonomous robots. In this paper, we propose an event-based stereo VIO system, namely ESVIO. Firstly, we present a novel direct event-based VIO method, which fuses events' depth, Time-Surface images, and pre-integrated inertial measurement to estimate the camera motion and inertial measurement unit (IMU) biases in a sliding window non-linear optimization framework, effectively improving the state estimation accuracy and robustness. Secondly, we design an event-inertia semi-joint initialization method, through two steps of event-only initialization and event-inertia initial optimization, to rapidly and accurately solve the initialization parameters of the VIO system, thereby further improving the state estimation accuracy. Based on these two methods, we implement the ESVIO system and evaluate the effectiveness and robustness of ESVIO on various public datasets. The experimental results show that ESVIO achieves good performance in both accuracy and robustness when compared with other state-of-the-art event-based VIO and stereo visual odometry (VO) systems, and, at the same time, with no compromise to real-time performance.

GNSS Reconstrainted Visual–Inertial Odometry System Using Factor Graphs

Monocular vision sensors are often affected by the rapid direction change in load platform and violent illumination change when the mobile device moves autonomously with high maneuverability. The images collected by the visual sensor will also have a lot of dynamic blur, which together with the weak texture environment reduces the continuity and accuracy of the visual autonomous navigation system. To enhance the stability of the system, in the letter, we propose a vision-led multisource data fusion navigation algorithm. The system combines the visual information for trajectory estimation, adds the inertial measurement unit (IMU) measurement information to the sliding window for optimization, and finally uses the global navigation satellite system (GNSS) data as a reconstraint condition through factor graph optimization to further optimize the trajectory accuracy. Experiments on the public datasets containing a variety of different scene categories show that the trajectory tracking results generated by our algorithm are more complete and stable and can better meet the system’s autonomous navigation requirements.

A Benchmark Comparison of Four Off-the-Shelf Proprietary Visual–Inertial Odometry Systems

Commercial visual-inertial odometry (VIO) systems have been gaining attention as cost-effective, off-the-shelf, six-degree-of-freedom (6-DoF) ego-motion-tracking sensors for estimating accurate and consistent camera pose data, in addition to their ability to operate without external localization from motion capture or global positioning systems. It is unclear from existing results, however, which commercial VIO platforms are the most stable, consistent, and accurate in terms of state estimation for indoor and outdoor robotic applications. We assessed four popular proprietary VIO systems (Apple ARKit, Google ARCore, Intel RealSense T265, and Stereolabs ZED 2) through a series of both indoor and outdoor experiments in which we showed their positioning stability, consistency, and accuracy. After evaluating four popular VIO sensors in challenging real-world indoor and outdoor scenarios, Apple ARKit showed the most stable and high accuracy/consistency, and the relative pose error was a drift error of about 0.02 m per second. We present our complete results as a benchmark comparison for the research community.

GNSS-denied UAV indoor navigation with UWB incorporated visual inertial odometry

The localization of unmanned aerial vehicles (UAVs) in GPS-denied areas is an essential issue for indoor navigation. This paper presents a technique to improve the accuracy of visual inertial odometry (VIO) by combining the ultra-wideband (UWB) positioning technology. The proposed architecture is divided into two stages. In the initial stage, the constraint on UWB short-term position change is adopted to improve the pose estimation results of the VIO system. It is also used to mitigate the translation error caused by the vibration and lack of features during the flight. In the second stage, a loose coupling approach based on nonlinear optimization is utilized to fuse the local pose estimator of the VIO system with the global constraints from the UWB positioning. At the beginning of each operation, the alignment between the VIO and UWB frames is estimated to avoid the influence of coordinate transformation due to the VIO cumulative error. It is shown that our optimization-based fusion method is able to achieve a smooth localization trajectory under the global coordinate frame. In the experiments, the performance evaluation carried out in the real-world scenes has demonstrated the effectiveness of the proposed technique.

Pedestrian Gait Information Aided Visual Inertial SLAM for Indoor Positioning Using Handheld Smartphones

Simultaneous localization and mapping (SLAM) is currently a widely used technology for indoor positioning. Many studies have focused on the smartphone-based localization and navigation for its portability and feature-rich sensors. But due to the low-quality sensors and complex environments, current SLAM-based positioning technology using smartphones still poses great challenges, such as inherent cumulative global drift and potential divergence facing texture-less indoor regions. For the regular gait characteristics of pedestrians when naturally walking, the gait motion model can certainly provide effective observation of the pedestrian motion state. We propose a visual-inertial odometry (VIO) assisted by pedestrian gait information for smartphone-based indoor positioning. This work mainly builds two additional state constraints, pedestrian velocity, and step displacement, obtained by the pedestrian dead reckoning (PDR) algorithm for the visual-inertial tracking system. For each step, the corresponding residual term of step length and velocity constraints is constructed and added to the cost function for nonlinear sliding-window optimization. Furthermore, the step displacement is applied again in the four-degree-of-freedom (4-DOF) graph-based optimization to refine the trajectory. VIO system also assists the PDR algorithm in mode switching, to improve the accuracy of gait information by applying the adaptive step length formula. Field experiments were conducted, and the results indicate that with the aiding of the pedestrian gait information, the accuracy and robustness of the visual-inertial pedestrian tracking system using smartphones have been significantly improved. Compared with the state-of-the-art algorithm monocular visual-inertial navigation system (VINS-MONO), our method improves the accuracy by 54.6% on average in our field tests in challenging environments.

Structural Regularity Aided Visual-Inertial Odometry With Novel Coordinate Alignment and Line Triangulation

Man-made buildings exhibit structural regularity, which can provide strongly geometrical constraints for Visual-Inertial Odometry (VIO) systems. To make full use of the structural information, we propose a new structural regularity aided VIO with novel coordinate alignment and line triangulation under Manhattan world assumption. The proposed VIO system is built upon OpenVINS [1] and partly based on our previous work [2]. The proposed coordinate alignment method makes the Jacobians and reprojection errors become concise but also makes the required computation number for transformations decrease. In addition, a novel structural line triangulation method is provided, in which the global orientation of a structural line is used to refine its 3D position. All the novelties result in a more accurate and fast VIO system. The system is tested on EuRoC MAV dataset and a self-collected dataset. Experimental results demonstrate that the proposed method obtains better accuracy compared with state-of-the-art (SOTA) point-based systems (VINS-Mono [3] and OpenVINS [1]), point-line-based systems (PL-VINS [4] and Wei et al. [2]), and structural line-based system (StructVIO [5]). Notably, the self-collected dataset is recorded on Manhattan world scenes, and is also full of challenging weak texture and motion blur situations. On the dataset, the accuracy of our method is increased by 40.7% compared with the SOTA point-line-based systems.

Fast visual inertial odometry with point–line features using adaptive EDLines algorithm

In mainstream visual inertial odometry (VIO) systems, the method of positional solution by feature point extraction and matching in the image is widely used. However, the tracking accuracy of point features is dependent on the texture richness in the environment. Although many existing algorithms introduce line features in the front end to improve the system’s environmental adaptability, most of them sacrifice system real-time in exchange for higher positioning accuracy. The extraction and matching of line features often require more time, thus failing to meet the real-time requirements of the system for localization. In this paper, we therefore propose a fast VIO fused with point and line features, which enables the system to maintain a high level of positioning robustness in dim and changing light environments with low time cost. The point–line features VIO algorithm is based on adaptive thresholding of EDLines. By adding an adaptive thresholding component to the EDLines algorithm, the robustness of line feature extraction is enhanced to better adapt to changes in ambient lighting. The time needed for line feature extraction is also significantly reduced. A line feature matching algorithm based on geometric information and structural similarity is proposed, which enables fast and accurate line feature matching. The algorithm is compared with point-line visual-inertial odometry and monocular visual-inertial state estimator algorithms on the European robotics challenge dataset and real-world scenes. Many experiments prove that the algorithm has improved in both real time and accuracy.