Visual-inertial Odometry Algorithm Research Articles

Implementation and observability analysis of visual-inertial-wheel odometry with robust initialization and online extrinsic calibration

Combining camera, IMU and wheel encoder is a wise choice for car positioning because of the low cost and complementarity of the sensors. We propose a novel extended visual-inertial odometry algorithm based on sliding window tightly fusing data from the above three sensors. Firstly we propose an IMU-odometer pre-integration approach utilizing complete IMU measurements and wheel encoder readings, to make scale estimation more accurate in subsequent 4-degrees of freedom (DoF) optimization. Secondly we develop an original initialization module where encoder readings are fully utilized to refine gravity direction and provide an initial value for camera pose in real scale. Thirdly, we design a computationally efficient online extrinsic calibration method by fixing the linearization point for the rotational component of IMU-odometer extrinsic parameters, which is deployed depending on the convergence of accelerometer bias. Fourthly, we give an observability analysis of our optimization based approach under more general assumption. Extensive experiments are performed on two sets of data in various scenes, bringing the state-of-the-art visual odometry and visual-inertial odometry algorithms into comparison. Experimental results prove the overwhelmingly better performance of our proposed approach on the above two sets of data, as well as the robustness of our initialization module and the improvement resulted from online extrinsic calibration.

LRPL-VIO: A Lightweight and Robust Visual-Inertial Odometry with Point and Line Features.

Visual-inertial odometry (VIO) algorithms, fusing various features such as points and lines, are able to improve their performance in challenging scenes while the running time severely increases. In this paper, we propose a novel lightweight point-line visual-inertial odometry algorithm to solve this problem, called LRPL-VIO. Firstly, a fast line matching method is proposed based on the assumption that the photometric values of endpoints and midpoints are invariant between consecutive frames, which greatly reduces the time consumption of the front end. Then, an efficient filter-based state estimation framework is designed to finish information fusion (point, line, and inertial). Fresh measurements of line features with good tracking quality are selected for state estimation using a unique feature selection scheme, which improves the efficiency of the proposed algorithm. Finally, validation experiments are conducted on public datasets and in real-world tests to evaluate the performance of LRPL-VIO and the results show that we outperform other state-of-the-art algorithms especially in terms of speed and robustness.

S-VIO: Exploiting Structural Constraints for RGB-D Visual Inertial Odometry

Vision-based localization is an essential problem for autonomous systems while the performance of vision-based odometry degrades in the challenging scenario. This letter presents S-VIO, an RGB-D visual inertial odometry (VIO) which fully uses multi-sensor measurements (i.e. depth, RGB and IMU), heterogeneous landmarks (i.e. points, lines and planes) and structural regularity of the environment to obtain a robust and accurate localization result. In order to detect the underlying structural regularity of the environment, a two-step Atlanta world inference method is proposed. Leveraging the gravity direction estimated by the VIO system, the proposed algorithm first generates horizontal Atlanta axis hypotheses from a set of recently optimized plane landmarks. Then the following plane landmarks and line clusters are used to filter out the occasionally observed axes based on the persistence of the observation. The remaining axes will survive and be saved in the Atlanta map for future re-observations. Particularly, an efficient mined-and-stabbed (MnS) method is applied to classify the structural line and extract the vanishing point from each line cluster. In addition, a closed-form initialization method for the structural line feature is proposed, which leverages the known direction to obtain a better initial estimation. Integrated with the above novelties, S-VIO is tested on two public real-world RGB-D inertial datasets. Experiments demonstrate that S-VIO has better accuracy and robustness compared to state-of-the-art VIO and RGB-D VIO algorithms. The code and the supplementary material will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/oranfire/SVIO</uri> .

M2C-GVIO: motion manifold constraint aided GNSS-visual-inertial odometry for ground vehicles

Visual-Inertial Odometry (VIO) has been developed from Simultaneous Localization and Mapping (SLAM) as a low-cost and versatile sensor fusion approach and attracted increasing attention in ground vehicle positioning. However, VIOs usually have the degraded performance in challenging environments and degenerated motion scenarios. In this paper, we propose a ground vehicle-based VIO algorithm based on the Multi-State Constraint Kalman Filter (MSCKF) framework. Based on a unified motion manifold assumption, we derive the measurement model of manifold constraints, including velocity, rotation, and translation constraints. Then we present a robust filter-based algorithm dedicated to ground vehicles, whose key is the real-time manifold noise estimation and adaptive measurement update. Besides, GNSS position measurements are loosely coupled into our approach, where the transformation between GNSS and VIO frame is optimized online. Finally, we theoretically analyze the system observability matrix and observability measures. Our algorithm is tested on both the simulation test and public datasets including Brno Urban dataset and Kaist Urban dataset. We compare the performance of our algorithm with classical VIO algorithms (MSCKF, VINS-Mono, R-VIO, ORB_SLAM3) and GVIO algorithms (GNSS-MSCKF, VINS-Fusion). The results demonstrate that our algorithm is more robust than other compared algorithms, showing a competitive position accuracy and computational efficiency.

Improving the Computational Efficiency of ROVIO

ROVIO is one of the state-of-the-art monocular visual inertial odometry algorithms. It uses an Iterative Extended Kalman Filter (IEKF) to align visual features and update the vehicle state simultaneously by including the feature locations in the state vector of the IEKF. This algorithm is single-core intensive, which allows the other cores to be used for other algorithms, such as object detection and path optimization. However, the computational cost of the algorithm grows rapidly with the maximum number of features to track. Each feature adds three new states (a 2D bearing vector and inverse depth), leading to bigger matrix multiplications that are computationally expensive. The main computational load of ROVIO is the iterative update step of the IEKF. In this work, we reduce the average computational cost of ROVIO by [Formula: see text] on an NVIDIA Jetson TX2, without affecting the accuracy of the algorithm. This computational gain is mainly achieved by exploiting the sparse matrices in ROVIO. Furthermore, we reduce the computational peaks by pre-selecting new features based on their already calculated FAST score. The combination of both modifications allows us to run ROVIO on the computationally restricted Raspberry Pi Zero 2[Formula: see text]W.

An improved semi-synthetic approach for creating visual-inertial odometry datasets

Capturing outdoor visual-inertial datasets is a challenging yet vital aspect of developing robust visual-inertial odometry (VIO) algorithms. A significant hurdle is that high-accuracy-ground-truth systems (e.g., motion capture) are not practical for outdoor use. One solution is to use a “semi-synthetic” approach that combines rendered images with real IMU data. This approach can produce sequences containing challenging imagery and accurate ground truth but with less simulated data than a fully synthetic sequence. Existing methods (used by popular tools/datasets) record IMU measurements from a visual-inertial system while measuring its trajectory using motion capture, then rendering images along that trajectory. This work identifies a major flaw in that approach, specifically that using motion capture alone to estimate the pose of the robot/system results in the generation of inconsistent visual-inertial data that is not suitable for evaluating VIO algorithms. However, we show that it is possible to generate high-quality semi-synthetic data for VIO algorithm evaluation. We do so using an open-source full-batch optimisation tool to incorporate both mocap and IMU measurements when estimating the IMU’s trajectory. We demonstrate that this improved trajectory results in better consistency between the IMU data and rendered images and that the resulting data improves VIO trajectory error by 79% compared to existing methods. Furthermore, we examine the effect of visual-inertial data inconsistency (as a result of trajectory noise) on VIO performance to provide a foundation for future work targeting real-time applications.

Visual Inertial Odometry-Based Gait Analysis Using Waist-Attached RGB-D Camera and Inertial Sensors

In this article, a visual–inertial odometry algorithm is proposed to estimate walking stride length and reconstruct walking trajectory. Depth and color image data from a downward-looking waist-mounted red green blue-depth (RGB-D) camera is fused with its internal inertial measurement unit (IMU) data in an estimation algorithm to perform foot detection and position estimation. Floor plane and foot positions in stance phases are calculated and used as landmarks to construct measurement equations for updating the filter. A smoothing problem is formulated as a linear optimization problem to improve filter results. Experiments are performed to evaluate the walking trajectory reconstruction and the overall root mean square errors (RMSEs) of walking stride length estimation is about 3.8 cm.

FDO-Calibr: visual-aided IMU calibration based on frequency-domain optimization

Nowadays, multi-sensor fusion technology is a fundamental prerequisite to achieve highly autonomous robots and robustness. Many studies have been conducted, such as visual–inertial odometry (VIO), integrated navigation, and LiDAR–inertial odometry. Typically, for VIO, gratifying results have been achieved, ascribing to the complementary sensing capabilities of inertial measurement units (IMUs) and cameras. However, this work mainly focuses on the fusion of visual and inertial data, while the IMU error is less considered, especially for low-cost or poorly calibrated microelectromechanical system (MEMS) IMU. Such errors may have a significant effect on the VIO performance. In this study, we compensated for the IMU using camera assistance. The key characteristic of the method is that we optimize the compensation parameters (scale factor) from coarse to fine by combining the time domain with the frequency domain. The proposed method is to use the time-domain and frequency-domain optimization to suppress large noise in the dynamic calibration process of the extremely low-cost sensor platform. The effectiveness of this method is validated through experiments and simulations. The minimal calibration error (0.46%) is commensurate with the advanced work. By feeding the compensated IMU into the VIO algorithm, the localization accuracy is improved by 9% to 15%. This method improves the performance in the VIO algorithm, which is equipped with the low-cost or poorly calibrated MEMS IMU and reduces the hardware and deployment costs of the system.

Visual-Inertial Odometry Using High Flying Altitude Drone Datasets

Positioning of unoccupied aerial systems (UAS, drones) is predominantly based on Global Navigation Satellite Systems (GNSS). Due to potential signal disruptions, redundant positioning systems are needed for reliable operation. The objective of this study was to implement and assess a redundant positioning system for high flying altitude drone operation based on visual-inertial odometry (VIO). A new sensor suite with stereo cameras and an inertial measurement unit (IMU) was developed, and a state-of-the-art VIO algorithm, VINS-Fusion, was used for localisation. Empirical testing of the system was carried out at flying altitudes of 40–100 m, which cover the common flight altitude range of outdoor drone operations. The performance of various implementations was studied, including stereo-visual-odometry (stereo-VO), monocular-visual-inertial-odometry (mono-VIO) and stereo-visual-inertial-odometry (stereo-VIO). The stereo-VIO provided the best results; the flight altitude of 40–60 m was the most optimal for the stereo baseline of 30 cm. The best positioning accuracy was 2.186 m for a 800 m-long trajectory. The performance of the stereo-VO degraded with the increasing flight altitude due to the degrading base-to-height ratio. The mono-VIO provided acceptable results, although it did not reach the performance level of the stereo-VIO. This work presented new hardware and research results on localisation algorithms for high flying altitude drones that are of great importance since the use of autonomous drones and beyond visual line-of-sight flying are increasing and will require redundant positioning solutions that compensate for potential disruptions in GNSS positioning. The data collected in this study are published for analysis and further studies.

Deep IMU Bias Inference for Robust Visual-Inertial Odometry With Factor Graphs

Visual Inertial Odometry (VIO) is one of the most established state estimation methods for mobile platforms. However, when visual tracking fails, VIO algorithms quickly diverge due to rapid error accumulation during inertial data integration. This error is typically modeled as a combination of additive Gaussian noise and a slowly changing bias which evolves as a random walk. In this work, we propose to train a neural network to learn the true bias evolution. We implement and compare two common sequential deep learning architectures: LSTMs and Transformers. Our approach follows from recent learning-based inertial estimators, but, instead of learning a motion model, we target IMU bias explicitly, which allows us to generalize to locomotion patterns unseen in training. We show that our proposed method improves state estimation in visually challenging situations across a wide range of motions by quadrupedal robots, walking humans, and drones. Our experiments show an average 15% reduction in drift rate, with much larger reductions when there is total vision failure. Importantly, we also demonstrate that models trained with one locomotion pattern (human walking) can be applied to another (quadruped robot trotting) without retraining.

Resolution and Frequency Effects on UAVs Semi-Direct Visual-Inertial Odometry (SVO) for Warehouse Logistics.

For the commercial sector, warehouses are becoming increasingly vital. Constant efforts are in progress to increase the efficiency of these facilities while reducing costs. The inventory part of the goods is a time-consuming task that impacts the company's revenue. This article presents an analysis of the performance of a state-of-the-art, visual-inertial odometry algorithm, SVO Pro Open, when varying the resolution and frequency of video streaming in an industrial environment. To perform efficiently this task, achieving an optimal system in terms of localization accuracy, robustness, and computational cost is necessary. Different resolutions are selected with a constant aspect ratio, and an accurate calibration for each resolution configuration is performed. A stable operating point in terms of robustness, accuracy of localization, and CPU utilization is found and the trends obtained are studied. To keep the system robust against sudden divergence, the feature loss factor extracted from optical sensors is analyzed. Innovative trends and translation errors on the order of a few tens of centimeters are achieved, allowing the system to navigate safely in the warehouse. The best result is obtained at a resolution of 636 × 600 px, where the localization errors (x, y, and z) are all under 0.25 m. In addition, the CPU (Central Processing Unit) usage of the onboard computer is kept below 60%, remaining usable for other relevant onboard processing tasks.

An improved semi-synthetic approach for creating visual-inertial odometry datasets

Capturing outdoor visual-inertial datasets is a challenging yet vital aspect of developing robust visual-inertial odometry (VIO) algorithms. A significant hurdle is that high-accuracy-ground-truth systems (e.g., motion capture) are not practical for outdoor use. One solution is to use a “semi-synthetic” approach that combines rendered images with real IMU data. This approach can produce sequences containing challenging imagery and accurate ground truth but with less simulated data than a fully synthetic sequence. Existing methods (used by popular tools/datasets) record IMU measurements from a visual-inertial system while measuring its trajectory using motion capture, then rendering images along that trajectory. This work identifies a major flaw in that approach, specifically that using motion capture alone to estimate the pose of the robot/system results in the generation of inconsistent visual-inertial data that is not suitable for evaluating VIO algorithms. However, we show that it is possible to generate high-quality semi-synthetic data for VIO algorithm evaluation. We do so using an open-source full-batch optimization tool to incorporate both mocap and IMU measurements when estimating the IMU’s trajectory. We demonstrate that this improved trajectory results in better consistency between the IMU data and rendered images and that the resulting data improves VIO trajectory error by 79% compared to existing methods. Furthermore, we examine the effect of visual-inertial data inconsistency (as a result of trajectory noise) on VIO performance to provide a foundation for future work targeting real-time applications.

EMA-VIO: Deep Visual–Inertial Odometry With External Memory Attention

Accurate and robust localization is a fundamental need for mobile agents. Visual–inertial odometry (VIO) algorithms exploit the information from the camera and inertial sensors to estimate position and translation. Recent deep-learning-based VIO models attract attention as they provide pose information in a data-driven way, without the need of designing hand-crafted algorithms. Existing learning-based VIO models rely on recurrent models to fuse multimodal data and process sensor signals, which are hard to train and not efficient enough. We propose a novel learning-based VIO framework with external memory attention that effectively and efficiently combines visual and inertial features for state estimation. Our proposed model is able to estimate pose accurately and robustly, even in challenging scenarios, for example, on overcast days and water-filled ground, which are difficult for traditional VIO algorithms to extract visual features. Experiments validate that it outperforms both traditional and learning-based VIO baselines in different scenes.

Exploring Event Camera-Based Odometry for Planetary Robots

Due to their resilience to motion blur and high robustness in low-light and high dynamic range conditions, event cameras are poised to become enabling sensors for vision-based exploration on future Mars helicopter missions. However, existing event-based visual-inertial odometry (VIO) algorithms either suffer from high tracking errors or are brittle, since they cannot cope with significant depth uncertainties caused by an unforeseen loss of tracking or other effects. In this work, we introduce EKLT-VIO, which addresses both limitations by combining a state-of-the-art event-based frontend with a filter-based backend. This makes it both accurate and robust to uncertainties, outperforming event- and frame-based VIO algorithms on challenging benchmarks by 32%. In addition, we demonstrate accurate performance in hover-like conditions (outperforming existing event-based methods) as well as high robustness in newly collected Mars-like and high-dynamic-range sequences, where existing frame-based methods fail. In doing so, we show that event-based VIO is the way forward for vision-based exploration on Mars.

Fast visual inertial odometry with point–line features using adaptive EDLines algorithm

In mainstream visual inertial odometry (VIO) systems, the method of positional solution by feature point extraction and matching in the image is widely used. However, the tracking accuracy of point features is dependent on the texture richness in the environment. Although many existing algorithms introduce line features in the front end to improve the system’s environmental adaptability, most of them sacrifice system real-time in exchange for higher positioning accuracy. The extraction and matching of line features often require more time, thus failing to meet the real-time requirements of the system for localization. In this paper, we therefore propose a fast VIO fused with point and line features, which enables the system to maintain a high level of positioning robustness in dim and changing light environments with low time cost. The point–line features VIO algorithm is based on adaptive thresholding of EDLines. By adding an adaptive thresholding component to the EDLines algorithm, the robustness of line feature extraction is enhanced to better adapt to changes in ambient lighting. The time needed for line feature extraction is also significantly reduced. A line feature matching algorithm based on geometric information and structural similarity is proposed, which enables fast and accurate line feature matching. The algorithm is compared with point-line visual-inertial odometry and monocular visual-inertial state estimator algorithms on the European robotics challenge dataset and real-world scenes. Many experiments prove that the algorithm has improved in both real time and accuracy.

Navigation of frameless fixation for gamma\xa0knife radiosurgery using fixed augmented reality

Augmented reality (AR) offers a new medical treatment approach. We aimed to evaluate frameless (mask) fixation navigation using a 3D-printed patient model with fixed-AR technology for gamma knife radiosurgery (GKRS). Fixed-AR navigation was developed using the inside-out method with visual inertial odometry algorithms, and the flexible Quick Response marker was created for object-feature recognition. Virtual 3D-patient models for AR-rendering were created via 3D-scanning utilizing TrueDepth and cone-beam computed tomography (CBCT) to generate a new GammaKnife Icon™ model. A 3D-printed patient model included fiducial markers, and virtual 3D-patient models were used to validate registration accuracy. Registration accuracy between initial frameless fixation and re-fixation navigated fixed-AR was validated through visualization and quantitative method. The quantitative method was validated through set-up errors, fiducial marker coordinates, and high-definition motion management (HDMM) values. A 3D-printed model and virtual models were correctly overlapped under frameless fixation. Virtual models from both 3D-scanning and CBCT were enough to tolerate the navigated frameless re-fixation. Although the CBCT virtual model consistently delivered more accurate results, 3D-scanning was sufficient. Frameless re-fixation accuracy navigated in virtual models had mean set-up errors within 1 mm and 1.5° in all axes. Mean fiducial marker differences from coordinates in virtual models were within 2.5 mm in all axes, and mean 3D errors were within 3 mm. Mean HDMM difference values in virtual models were within 1.5 mm of initial HDMM values. The variability from navigation fixed-AR is enough to consider repositioning frameless fixation without CBCT scanning for treating patients fractionated with large multiple metastases lesions (> 3 cm) who have difficulty enduring long beam-on time. This system could be applied to novel GKRS navigation for frameless fixation with reduced preparation time.

Perception-Aware Receding Horizon Trajectory Planning for Multicopters With Visual-Inertial Odometry

Visual inertial odometry (VIO) is widely used for the state estimation of multicopters, but it may function poorly in environments with few visual features or in overly aggressive flights. In this work, we propose a perception-aware collision avoidance trajectory planner for multicopters, that may be used with any feature-based VIO algorithm. Our approach is able to fly the vehicle to a goal position at high speed, avoiding obstacles in an unknown stationary environment while achieving good VIO state estimation accuracy. The proposed planner samples a group of minimum jerk trajectories and finds collision-free trajectories among them, which are then evaluated based on their speed to the goal and perception quality. Both the motion blur of features and their locations are considered for the perception quality. Our novel consideration of the motion blur of features enables automatic adaptation of the trajectory&#x2019;s aggressiveness under environments with different light levels. The best trajectory from the evaluation is tracked by the vehicle and is updated in a receding horizon manner when new images are received from the camera. Only generic assumptions about the VIO are made, so that the planner may be used with various existing systems. The proposed method can run in real time on a small embedded computer on board. We validated the effectiveness of our proposed approach through experiments in both indoor and outdoor environments. Compared to a perception-agnostic planner, the proposed planner kept more features in the camera&#x2019;s view and made the flight less aggressive, making the VIO more accurate. It also reduced VIO failures, which occurred for the perception-agnostic planner but not for the proposed planner. The ability of the proposed planner to fly through dense obstacles was also validated. The experiment video can be found at https://youtu.be/qO3LZIrpwtQ.

PC-SD-VIO: A constant intensity semi-direct monocular visual-inertial odometry with online photometric calibration

The brightness constancy assumption is the cornerstone of direct or semi-direct visual odometry (VO) and visual simultaneous localization and mapping (SLAM). However, due to the existence of automatic exposure time, nonlinear camera response function, and vignetting, this assumption is difficult to hold in practical applications. Therefore, the corresponding algorithm performs poorly on arbitrary video sequences and uncalibrated cameras. Hence, we propose a novel constant intensity semi-direct visual-inertial odometry (VIO) integrated with online photometric calibration, which combines the exactness of the feature-based method and the quickness of the direct method. We combine gain-adaptive direct image alignment and gain-adaptive Kanade–Lucas–Tomasi (KLT) optical flow tracking to complete the feature matching and use it as the input for the back-end optimization and online photometric calibration. Our photometric calibration module can complete the estimation of all photometric parameters without any prior knowledge and cooperate with the front-end to complete the real-time photometric calibration of the latest frame. Experiments on the TUM Mono VO dataset, EuRoC dataset, and real environments prove that the algorithm can reliably calibrate the photometric parameters of an arbitrary video sequence. The semi-direct VIO algorithm integrated with the photometric calibration algorithm achieves a good balance between speed, accuracy, and robustness.

Robust Mono Visual-Inertial Odometry Using Sparse Optical Flow With Edge Detection

In this paper, we present a fast and robust visual-inertial odometry (VIO) algorithm, which realizes the tight coupling of monocular visual odometry (VO) and low-cost inertial measurement unit (IMU). Aiming at the problem of tracking loss caused by the interference of moving objects or blurred images in the traditional VIO algorithm, a sparse optical flow method combining edge detection algorithms is proposed. In the image preprocessing, Laplace edge detection is performed on the original image first, and an area with the most texture is searched according to the sharpened image. When tracking feature points with the sparse optical flow method, the feature points within the searched area are used as the main tracking targets to reduce the impact because of some unclear part in the image on the visual front-end. The trust region dogleg method is used for non-linear optimization in the back-end of the VIO system. The effectiveness of the proposed VIO system is validated on the public data set named EuRoC MAV and compared with the advanced VIO algorithm in recent years. Experimental results show that the new VIO system proposed in this paper has good robustness and can be applied to complex scenes such as fast motion, lighting changes, lack of features, and image blur. Compared with the traditional VIO algorithm, the pose estimation speed of the proposed VIO algorithm can be improved by 10% or more.

VIODE: A Simulated Dataset to Address the Challenges of Visual-Inertial Odometry in Dynamic Environments

Dynamic environments such as urban areas are still challenging for popular visual-inertial odometry (VIO) algorithms. Existing datasets typically fail to capture the dynamic nature of these environments, therefore making it difficult to quantitatively evaluate the robustness of existing VIO methods. To address this issue, we propose three contributions: firstly, we provide the VIODE benchmark, a novel dataset recorded from a simulated UAV that navigates in challenging dynamic environments. The unique feature of the VIODE dataset is the systematic introduction of moving objects into the scenes. It includes three environments, each of which is available in four dynamic levels that progressively add moving objects. The dataset contains synchronized stereo images and IMU data, as well as ground-truth trajectories and instance segmentation masks. Secondly, we compare state-of-the-art VIO algorithms on the VIODE dataset and show that they display substantial performance degradation in highly dynamic scenes. Thirdly, we propose a simple extension for visual localization algorithms that relies on semantic information. Our results show that scene semantics are an effective way to mitigate the adverse effects of dynamic objects on VIO algorithms. Finally, we make the VIODE dataset publicly available at https://github.com/kminoda/VIODE.