Visual-inertial Odometry Research Articles

A REVIEW OF VISUAL ODOMETRY FOR UAV AUTONOMOUS NAVIGATION

Visual odometry (VO) has become a popular technique for autonomous naviga-tion of unmanned aerial vehicles (UAVs) in GPS-denied environments. Although, the concept of VO is not new, it has been successfully implemented for on-ground vehi-cles only. There are several product level solutions of autonomous navigation, which rely on VO or contain VO as an important subpart. Meanwhile, a proper system of VO for UAVs is not yet achieved and continues to be a subject of an active research. In this paper, we show the comparison of the latest VO techniques for UAV nav-igation. We begin by describing the basic principles of VO, then review the different types of VO algorithms commonly used in UAV navigation, such as monocular, stereo, and RGB-D approaches. For each type of algorithm, we discuss the advantages and limitations, and provide examples of recent research and applications. Finally, we dis-cuss some of the emerging trends and future directions in visual odometry for UAV navigation, such as Visual inertial odometry approaches and real-time implementa-tions. This review paper provides a comprehensive overview of the current state-of-the-art in visual odometry for UAV autonomous navigation, highlighting the ad-vantages and limitations of different types of algorithms and identifying key challeng-es and future research directions.

UAV Navigation With Monocular Visual Inertial Odometry Under GNSS-Denied Environment

UAV Navigation With Monocular Visual Inertial Odometry Under GNSS-Denied Environment

GNSS Reconstrainted Visual–Inertial Odometry System Using Factor Graphs

Monocular vision sensors are often affected by the rapid direction change in load platform and violent illumination change when the mobile device moves autonomously with high maneuverability. The images collected by the visual sensor will also have a lot of dynamic blur, which together with the weak texture environment reduces the continuity and accuracy of the visual autonomous navigation system. To enhance the stability of the system, in the letter, we propose a vision-led multisource data fusion navigation algorithm. The system combines the visual information for trajectory estimation, adds the inertial measurement unit (IMU) measurement information to the sliding window for optimization, and finally uses the global navigation satellite system (GNSS) data as a reconstraint condition through factor graph optimization to further optimize the trajectory accuracy. Experiments on the public datasets containing a variety of different scene categories show that the trajectory tracking results generated by our algorithm are more complete and stable and can better meet the system’s autonomous navigation requirements.

Resolution and Frequency Effects on UAVs Semi-Direct Visual-Inertial Odometry (SVO) for Warehouse Logistics.

For the commercial sector, warehouses are becoming increasingly vital. Constant efforts are in progress to increase the efficiency of these facilities while reducing costs. The inventory part of the goods is a time-consuming task that impacts the company's revenue. This article presents an analysis of the performance of a state-of-the-art, visual-inertial odometry algorithm, SVO Pro Open, when varying the resolution and frequency of video streaming in an industrial environment. To perform efficiently this task, achieving an optimal system in terms of localization accuracy, robustness, and computational cost is necessary. Different resolutions are selected with a constant aspect ratio, and an accurate calibration for each resolution configuration is performed. A stable operating point in terms of robustness, accuracy of localization, and CPU utilization is found and the trends obtained are studied. To keep the system robust against sudden divergence, the feature loss factor extracted from optical sensors is analyzed. Innovative trends and translation errors on the order of a few tens of centimeters are achieved, allowing the system to navigate safely in the warehouse. The best result is obtained at a resolution of 636 × 600 px, where the localization errors (x, y, and z) are all under 0.25 m. In addition, the CPU (Central Processing Unit) usage of the onboard computer is kept below 60%, remaining usable for other relevant onboard processing tasks.

A Benchmark Comparison of Four Off-the-Shelf Proprietary Visual–Inertial Odometry Systems

Commercial visual-inertial odometry (VIO) systems have been gaining attention as cost-effective, off-the-shelf, six-degree-of-freedom (6-DoF) ego-motion-tracking sensors for estimating accurate and consistent camera pose data, in addition to their ability to operate without external localization from motion capture or global positioning systems. It is unclear from existing results, however, which commercial VIO platforms are the most stable, consistent, and accurate in terms of state estimation for indoor and outdoor robotic applications. We assessed four popular proprietary VIO systems (Apple ARKit, Google ARCore, Intel RealSense T265, and Stereolabs ZED 2) through a series of both indoor and outdoor experiments in which we showed their positioning stability, consistency, and accuracy. After evaluating four popular VIO sensors in challenging real-world indoor and outdoor scenarios, Apple ARKit showed the most stable and high accuracy/consistency, and the relative pose error was a drift error of about 0.02 m per second. We present our complete results as a benchmark comparison for the research community.

Landmark-Based Scale Estimation and Correction of Visual Inertial Odometry for VTOL UAVs in a GPS-Denied Environment.

With the rapid development of technology, unmanned aerial vehicles (UAVs) have become more popular and are applied in many areas. However, there are some environments where the Global Positioning System (GPS) is unavailable or has the problem of GPS signal outages, such as indoor and bridge inspections. Visual inertial odometry (VIO) is a popular research solution for non-GPS navigation. However, VIO has problems of scale errors and long-term drift. This study proposes a method to correct the position errors of VIO without the help of GPS information for vertical takeoff and landing (VTOL) UAVs. In the initial process, artificial landmarks are utilized to improve the positioning results of VIO by the known landmark information. The position of the UAV is estimated by VIO. Then, the accurate position is estimated by the extended Kalman filter (EKF) with the known landmark, which is used to obtain the scale correction using the least squares method. The Inertial Measurement Unit (IMU) data are used for integration in the time-update process. The EKF can be updated with two measurements. One is the visual odometry (VO) estimated directly by a landmark. The other is the VIO with scale correction. When the landmark is detected during takeoff phase, or the UAV is returning to the takeoff location during landing phase, the trajectory estimated by the landmark is used to update the scale correction. At the beginning of the experiments, preliminary verification was conducted on the ground. A self-developed UAV equipped with a visual-inertial sensor to collect data and a high-precision real time kinematic (RTK) to verify trajectory are applied to flight tests. The experimental results show that the method proposed in this research effectively solves the problems of scale and the long-term drift of VIO.

An improved semi-synthetic approach for creating visual-inertial odometry datasets

Capturing outdoor visual-inertial datasets is a challenging yet vital aspect of developing robust visual-inertial odometry (VIO) algorithms. A significant hurdle is that high-accuracy-ground-truth systems (e.g., motion capture) are not practical for outdoor use. One solution is to use a “semi-synthetic” approach that combines rendered images with real IMU data. This approach can produce sequences containing challenging imagery and accurate ground truth but with less simulated data than a fully synthetic sequence. Existing methods (used by popular tools/datasets) record IMU measurements from a visual-inertial system while measuring its trajectory using motion capture, then rendering images along that trajectory. This work identifies a major flaw in that approach, specifically that using motion capture alone to estimate the pose of the robot/system results in the generation of inconsistent visual-inertial data that is not suitable for evaluating VIO algorithms. However, we show that it is possible to generate high-quality semi-synthetic data for VIO algorithm evaluation. We do so using an open-source full-batch optimization tool to incorporate both mocap and IMU measurements when estimating the IMU’s trajectory. We demonstrate that this improved trajectory results in better consistency between the IMU data and rendered images and that the resulting data improves VIO trajectory error by 79% compared to existing methods. Furthermore, we examine the effect of visual-inertial data inconsistency (as a result of trajectory noise) on VIO performance to provide a foundation for future work targeting real-time applications.

Omni-Swarm: A Decentralized Omnidirectional Visual–Inertial–UWB State Estimation System for Aerial Swarms

Decentralized state estimation is one of the most fundamental components of autonomous aerial swarm systems in GPS-denied areas yet it still remains a highly challenging research topic. Omni-swarm, a decentralized omnidirectional visual-inertial-UWB state estimation system for aerial swarms, is proposed in this paper to address this research niche. To solve the issues of observability, complicated initialization, insufficient accuracy, and lack of global consistency, we introduce an omnidirectional perception front-end in Omni-swarm. It consists of stereo wide-FoV cameras and ultra-wideband sensors, visual-inertial odometry, multi-drone map-based localization, and visual drone tracking algorithms. The measurements from the front-end are fused with graph-based optimization in the back-end. The proposed method achieves centimeter-level relative state estimation accuracy while guaranteeing global consistency in the aerial swarm, as evidenced by the experimental results. Moreover, supported by Omni-swarm, inter-drone collision avoidance can be accomplished without any external devices, demonstrating the potential of Omni-swarm as the foundation of autonomous aerial swarms.

GNSS-denied UAV indoor navigation with UWB incorporated visual inertial odometry

The localization of unmanned aerial vehicles (UAVs) in GPS-denied areas is an essential issue for indoor navigation. This paper presents a technique to improve the accuracy of visual inertial odometry (VIO) by combining the ultra-wideband (UWB) positioning technology. The proposed architecture is divided into two stages. In the initial stage, the constraint on UWB short-term position change is adopted to improve the pose estimation results of the VIO system. It is also used to mitigate the translation error caused by the vibration and lack of features during the flight. In the second stage, a loose coupling approach based on nonlinear optimization is utilized to fuse the local pose estimator of the VIO system with the global constraints from the UWB positioning. At the beginning of each operation, the alignment between the VIO and UWB frames is estimated to avoid the influence of coordinate transformation due to the VIO cumulative error. It is shown that our optimization-based fusion method is able to achieve a smooth localization trajectory under the global coordinate frame. In the experiments, the performance evaluation carried out in the real-world scenes has demonstrated the effectiveness of the proposed technique.

EMA-VIO: Deep Visual–Inertial Odometry With External Memory Attention

Accurate and robust localization is a fundamental need for mobile agents. Visual–inertial odometry (VIO) algorithms exploit the information from the camera and inertial sensors to estimate position and translation. Recent deep-learning-based VIO models attract attention as they provide pose information in a data-driven way, without the need of designing hand-crafted algorithms. Existing learning-based VIO models rely on recurrent models to fuse multimodal data and process sensor signals, which are hard to train and not efficient enough. We propose a novel learning-based VIO framework with external memory attention that effectively and efficiently combines visual and inertial features for state estimation. Our proposed model is able to estimate pose accurately and robustly, even in challenging scenarios, for example, on overcast days and water-filled ground, which are difficult for traditional VIO algorithms to extract visual features. Experiments validate that it outperforms both traditional and learning-based VIO baselines in different scenes.

C2VIR-SLAM: Centralized Collaborative Visual-Inertial-Range Simultaneous Localization and Mapping

Collaborative simultaneous localization and mapping have a great impact on various applications such as search-and-rescue and agriculture. For each agent, the key to performing collaboration is to measure the motion relative to other participants or external anchors; currently, this is mainly accompanied by (1) matching to the shared maps from other agents or (2) measuring the range to anchors with UWB devices. While requiring multiple agents to visit the same area can decrease the task efficiency and anchors demand a distribution process, this paper proposes to use a monocular camera, an inertial measurement unit (IMU), and a UWB device as the onboard sensors on each agent to build an accurate and efficient centralized collaborative SLAM system. For each participant, visual-inertial odometry is adopted to estimate the motion parameters and build a local map of the explored areas. The agent-to-agent range is measured by the onboard UWB and is published to the central server together with the estimated motion parameters and the reconstructed maps. We designed a global optimization algorithm to make use of the cross-agent map match information detected by a visual place technique, and the agent-to-agent range information to optimize the motion parameter of all the participants and merge the local maps into a global map. Compared with existing collaborative SLAM systems, the proposed system can perform collaboration with onboard UWB measurements only, vision only, and a combination of these; this greatly improves the adaptiveness and robustness of the collaborative system. We also present an in-depth analysis of C2VIR-SLAM in multiple UAV real-flight datasets.

REVIO: Range- and Event-Based Visual-Inertial Odometry for Bio-Inspired Sensors.

Visual-inertial odometry is critical for Unmanned Aerial Vehicles (UAVs) and robotics. However, there are problems of motion drift and motion blur in sharp brightness changes and fast-motion scenes. It may cause the degradation of image quality, which leads to poor location. Event cameras are bio-inspired vision sensors that offer significant advantages in high-dynamic scenes. Leveraging this property, this paper presents a new range and event-based visual-inertial odometry (REVIO). Firstly, we propose an event-based visual-inertial odometry (EVIO) using sliding window nonlinear optimization. Secondly, REVIO is developed on the basis of EVIO, which fuses events and distances to obtain clear event images and improves the accuracy of position estimation by constructing additional range constraints. Finally, the EVIO and REVIO are tested in three experiments-dataset, handheld and flight-to evaluate the localization performance. The error of REVIO can be reduced by nearly 29% compared with EVIO in the handheld experiment and almost 28% compared with VINS-Mono in the flight experiment, which demonstrates the higher accuracy of REVIO in some fast-motion and high-dynamic scenes.

Pedestrian Gait Information Aided Visual Inertial SLAM for Indoor Positioning Using Handheld Smartphones

Simultaneous localization and mapping (SLAM) is currently a widely used technology for indoor positioning. Many studies have focused on the smartphone-based localization and navigation for its portability and feature-rich sensors. But due to the low-quality sensors and complex environments, current SLAM-based positioning technology using smartphones still poses great challenges, such as inherent cumulative global drift and potential divergence facing texture-less indoor regions. For the regular gait characteristics of pedestrians when naturally walking, the gait motion model can certainly provide effective observation of the pedestrian motion state. We propose a visual-inertial odometry (VIO) assisted by pedestrian gait information for smartphone-based indoor positioning. This work mainly builds two additional state constraints, pedestrian velocity, and step displacement, obtained by the pedestrian dead reckoning (PDR) algorithm for the visual-inertial tracking system. For each step, the corresponding residual term of step length and velocity constraints is constructed and added to the cost function for nonlinear sliding-window optimization. Furthermore, the step displacement is applied again in the four-degree-of-freedom (4-DOF) graph-based optimization to refine the trajectory. VIO system also assists the PDR algorithm in mode switching, to improve the accuracy of gait information by applying the adaptive step length formula. Field experiments were conducted, and the results indicate that with the aiding of the pedestrian gait information, the accuracy and robustness of the visual-inertial pedestrian tracking system using smartphones have been significantly improved. Compared with the state-of-the-art algorithm monocular visual-inertial navigation system (VINS-MONO), our method improves the accuracy by 54.6% on average in our field tests in challenging environments.

A Reconfigurable Visual-Inertial Odometry Accelerated Core with High Area and Energy Efficiency for Autonomous Mobile Robots.

With the wide application of autonomous mobile robots (AMRs), the visual inertial odometer (VIO) system that realizes the positioning function through the integration of a camera and inertial measurement unit (IMU) has developed rapidly, but it is still limited by the high complexity of the algorithm, the long development cycle of the dedicated accelerator, and the low power supply capacity of AMRs. This work designs a reconfigurable accelerated core that supports different VIO algorithms and has high area and energy efficiency, precision, and speed processing characteristics. Experimental results show that the loss of accuracy of the proposed accelerator is negligible on the most authoritative dataset. The on-chip memory usage of 70 KB is at least 10× smaller than the state-of-the-art works. Thus, the FPGA implementation’s hardware-resource consumption, power dissipation, and synthesis in the 28 nm CMOS outperform the previous works with the same platform.

Square-Root Extended Information Filter for Visual-Inertial Odometry for Planetary Landing

A novel sequential information filter formulation for computationally efficient visual-inertial odometry and mapping is developed in this work and applied to a realistic moon landing scenario. Careful construction of the square-root information matrix, in contrast to the full information or covariance matrix, provides easy and exact mean and covariance recovery throughout operation. Compared to an equivalent extended Kalman filter implementation, which provides identical results, the proposed filter does not require explicit marginalization of past landmark states to maintain constant-time complexity. Whereas measurements to opportunistic visual features only provide relative state information, resulting in drift over time unless a priori mapped landmarks are identified and tracked, the tight coupling of the inertial measurement unit provides some inertial state information. The results are presented in a terrain-relative navigation simulation for both a purely orbital case (with no active propulsion) and a landing case with a constant thrust.

Ctrl-VIO: Continuous-Time Visual-Inertial Odometry for Rolling Shutter Cameras

In this letter, we propose a probabilistic continuous-time visual-inertial odometry (VIO) for rolling shutter cameras. The continuous-time trajectory formulation naturally facilitates the fusion of asynchronized high-frequency IMU data and motion-distorted rolling shutter images. To prevent intractable computation load, the proposed VIO is sliding-window and keyframe-based. We propose to probabilistically marginalize the control points to keep the constant number of keyframes in the sliding window. Furthermore, the line exposure time difference (line delay) of the rolling shutter camera can be online calibrated in our continuous-time VIO. To extensively examine the performance of our continuous-time VIO, experiments are conducted on publicly-available WHU-RSVI, TUM-RSVI, and SenseTime-RSVI rolling shutter datasets. The results demonstrate the proposed continuous-time VIO significantly outperforms the existing state-of-the-art VIO methods.

Structural Regularity Aided Visual-Inertial Odometry With Novel Coordinate Alignment and Line Triangulation

Man-made buildings exhibit structural regularity, which can provide strongly geometrical constraints for Visual-Inertial Odometry (VIO) systems. To make full use of the structural information, we propose a new structural regularity aided VIO with novel coordinate alignment and line triangulation under Manhattan world assumption. The proposed VIO system is built upon OpenVINS [1] and partly based on our previous work [2]. The proposed coordinate alignment method makes the Jacobians and reprojection errors become concise but also makes the required computation number for transformations decrease. In addition, a novel structural line triangulation method is provided, in which the global orientation of a structural line is used to refine its 3D position. All the novelties result in a more accurate and fast VIO system. The system is tested on EuRoC MAV dataset and a self-collected dataset. Experimental results demonstrate that the proposed method obtains better accuracy compared with state-of-the-art (SOTA) point-based systems (VINS-Mono [3] and OpenVINS [1]), point-line-based systems (PL-VINS [4] and Wei et al. [2]), and structural line-based system (StructVIO [5]). Notably, the self-collected dataset is recorded on Manhattan world scenes, and is also full of challenging weak texture and motion blur situations. On the dataset, the accuracy of our method is increased by 40.7% compared with the SOTA point-line-based systems.

Exploring Event Camera-Based Odometry for Planetary Robots

Due to their resilience to motion blur and high robustness in low-light and high dynamic range conditions, event cameras are poised to become enabling sensors for vision-based exploration on future Mars helicopter missions. However, existing event-based visual-inertial odometry (VIO) algorithms either suffer from high tracking errors or are brittle, since they cannot cope with significant depth uncertainties caused by an unforeseen loss of tracking or other effects. In this work, we introduce EKLT-VIO, which addresses both limitations by combining a state-of-the-art event-based frontend with a filter-based backend. This makes it both accurate and robust to uncertainties, outperforming event- and frame-based VIO algorithms on challenging benchmarks by 32%. In addition, we demonstrate accurate performance in hover-like conditions (outperforming existing event-based methods) as well as high robustness in newly collected Mars-like and high-dynamic-range sequences, where existing frame-based methods fail. In doing so, we show that event-based VIO is the way forward for vision-based exploration on Mars.

DynaVINS: A Visual-Inertial SLAM for Dynamic Environments

Visual inertial odometry and SLAM algorithms are widely used in various fields, such as service robots, drones, and autonomous vehicles. Most of the SLAM algorithms are based on assumption that landmarks are static. However, in the real-world, various dynamic objects exist, and they degrade the pose estimation accuracy. In addition, temporarily static objects, which are static during observation but move when they are out of sight, trigger false positive loop closings. To overcome these problems, we propose a novel visual-inertial SLAM framework, called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DynaVINS</i> , which is robust against both dynamic objects and temporarily static objects. In our framework, we first present a robust bundle adjustment that could reject the features from dynamic objects by leveraging pose priors estimated by the IMU preintegration. Then, a keyframe grouping and a multi-hypothesis-based constraints grouping methods are proposed to reduce the effect of temporarily static objects in the loop closing. Subsequently, we evaluated our method in a public dataset that contains numerous dynamic objects. Finally, the experimental results corroborate that our <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DynaVINS</i> has promising performance compared with other state-of-the-art methods by successfully rejecting the effect of dynamic and temporarily static objects.

UWB-VIO Fusion for Accurate and Robust Relative Localization of Round Robotic Teams

The relative pose estimation is one of the most fundamental components for multi-robot systems, while it still remains an open and challenging research topic in infrastructure-free environment. In this letter, we target improving the accuracy and robustness of relative pose estimation for ground robotic teams, and propose to fuse range and odometry measurements to estimate the relative pose using sliding window optimization. In the system, multiple UWB tags for ranging are equipped on each robot, and visual inertial odometry is applied for estimating the ego-motion pose for each robot. Aiming for simple and effective relative pose initialization, the triangulation uncertainty for multi-tag robots is analyzed, and an initialization method is designed. To cope with the complex environments such as continuous NLOS condition, a NLOS detection and range measurements filtering method is presented. We have conducted series of experiments to demonstrate the performance of the proposed approach.