Visual-inertial Odometry Research Articles

Fast and Robust Semidirect Monocular Visual-Inertial Odometry for UAV

Fast and Robust Semidirect Monocular Visual-Inertial Odometry for UAV

FILNet: Fast Image-Based Indoor Localization Using an Anchor Control Network.

This paper designs a fast image-based indoor localization method based on an anchor control network (FILNet) to improve localization accuracy and shorten the duration of feature matching. Particularly, two stages are developed for the proposed algorithm. The offline stage is to construct an anchor feature fingerprint database based on the concept of an anchor control network. This introduces detailed surveys to infer anchor features according to the information of control anchors using the visual-inertial odometry (VIO) based on Google ARcore. In addition, an affine invariance enhancement algorithm based on feature multi-angle screening and supplementation is developed to solve the image perspective transformation problem and complete the feature fingerprint database construction. In the online stage, a fast spatial indexing approach is adopted to improve the feature matching speed by searching for active anchors and matching only anchor features around the active anchors. Further, to improve the correct matching rate, a homography matrix filter model is used to verify the correctness of feature matching, and the correct matching points are selected. Extensive experiments in real-world scenarios are performed to evaluate the proposed FILNet. The experimental results show that in terms of affine invariance, compared with the initial local features, FILNet significantly improves the recall of feature matching from 26% to 57% when the angular deviation is less than 60 degrees. In the image feature matching stage, compared with the initial K-D tree algorithm, FILNet significantly improves the efficiency of feature matching, and the average time of the test image dataset is reduced from 30.3 ms to 12.7 ms. In terms of localization accuracy, compared with the benchmark method based on image localization, FILNet significantly improves the localization accuracy, and the percentage of images with a localization error of less than 0.1m increases from 31.61% to 55.89%.

Pose Graph Relocalization with Deep Object Detection and BIM-Supported Object Landmark Dictionary

Indoor localization is a prerequisite for autonomous robot applications in the construction industry. However, traditional localization techniques rely on low-level features and do not exploit construction-related semantics. They also are sensitive to environmental factors such as illumination and reflection rate, and therefore suffer from unexpected drifts and failures. This study proposes a pose graph relocalization framework that utilizes object-level landmarks to enhance a traditional visual localization system. The proposed framework builds an object landmark dictionary from Building Information Model (BIM) as prior knowledge. Then a multimodal deep neural network (DNN) is proposed to realize 3D object detection in real time, followed by instance-level object association with false-positive rejection, and relative pose estimation with outlier removal. Finally, a keyframe-based graph optimization is performed to rectify the drifts of traditional visual localization. The proposed framework was validated using a mobile platform with red-green-blue-depth (RGB-D) and inertial sensors, and the test scene was an indoor office environment with furnishing elements. The object detection model achieved 62.9% mean average precision (mAP). The relocalization technique reduced translational drifts by 64.67% and rotational drifts by 41.59% compared with traditional visual–inertial odometry.

RNGC-VIWO: Robust Neural Gyroscope Calibration Aided Visual-Inertial-Wheel Odometry for Autonomous Vehicle

Accurate and robust localization using multi-modal sensors is crucial for autonomous driving applications. Although wheel encoder measurements can provide additional velocity information for visual-inertial odometry (VIO), the existing visual-inertial-wheel odometry (VIWO) still cannot avoid long-term drift caused by the low-precision attitude acquired by the gyroscope of a low-cost inertial measurement unit (IMU), especially in visually restricted scenes where the visual information cannot accurately correct for the IMU bias. In this work, leveraging the powerful data processing capability of deep learning, we propose a novel tightly coupled monocular visual-inertial-wheel odometry with neural gyroscope calibration (NGC) to obtain accurate, robust, and long-term localization for autonomous vehicles. First, to cure the drift of the gyroscope, we design a robust neural gyroscope calibration network for low-cost IMU gyroscope measurements (called NGC-Net). Following a carefully deduced mathematical calibration model, NGC-Net leverages the temporal convolutional network to extract different scale features from raw IMU measurements in the past and regress the gyroscope corrections to output the de-noised gyroscope. A series of experiments on public datasets show that our NGC-Net has better performance on gyroscope de-noising than learning methods and competes with state-of-the-art VIO methods. Moreover, based on the more accurate de-noised gyroscope, an effective strategy for combining the advantages of VIWO and NGC-Net outputs is proposed in a tightly coupled framework, which significantly improves the accuracy of the state-of-the-art VIO/VIWO methods. In long-term and large-scale urban environments, our RNGC-VIWO tracking system performs robustly, and experimental results demonstrate the superiority of our method in terms of robustness and accuracy.

Solid-State-LiDAR-Inertial-Visual Odometry and Mapping via Quadratic Motion Model and Reflectivity Information

This paper proposes a solid-state-LiDAR-inertial-visual fusion framework containing two subsystems: the solid-state-LiDAR-inertial odometry (SSLIO) subsystem and the visual-inertial odometry (VIO) subsystem. Our SSLIO subsystem has two novelties that enable it to handle drastic acceleration and angular velocity changes: (1) the quadratic motion model is adopted in the in-frame motion compensation step of the LiDAR feature points, and (2) the system has a weight function for each residual term to ensure consistency in geometry and reflectivity. The VIO subsystem renders the global map in addition to further optimizing the state output by the SSLIO. To save computing resources, we calibrate our VIO subsystem’s extrinsic parameter indirectly in advance, instead of using real-time estimation. We test the SSLIO subsystem using publicly available datasets and a steep ramp experiment, and show that our SSLIO exhibits better performance than the state-of-the-art LiDAR-inertial SLAM algorithm Point-LIO in terms of coping with strong vibrations transmitted to the sensors due to the violent motion of the crawler robot. Furthermore, we present several outdoor field experiments evaluating our framework. The results show that our proposed multi-sensor fusion framework can achieve good robustness, localization and mapping accuracy, as well as strong real-time performance.

IQ-VIO: adaptive visual inertial odometry via interference quantization under dynamic environments

IQ-VIO: adaptive visual inertial odometry via interference quantization under dynamic environments

Range–Visual–Inertial Odometry with Coarse-to-Fine Image Registration Fusion for UAV Localization

In Global Navigation Satellite System (GNSS)-denied environments, image registration has emerged as a prominent approach to utilize visual information for estimating the position of Unmanned Aerial Vehicles (UAVs). However, traditional image-registration-based localization methods encounter limitations, such as strong dependence on the prior initial position information. In this paper, we propose a systematic method for UAV geo-localization. In particular, an efficient range–visual–inertial odometry (RVIO) is proposed to provide local tracking, which utilizes measurements from a 1D Laser Range Finder (LRF) to suppress scale drift in the odometry. To overcome the differences in seasons, lighting conditions, and other factors between satellite and UAV images, we propose an image-registration-based geo-localization method in a coarse-to-fine manner that utilizes the powerful representation ability of Convolutional Neural Networks (CNNs). Furthermore, to ensure the accuracy of global optimization, we propose an adaptive weight assignment method based on the evaluation of the quality of image-registration-based localization. The proposed method is extensively evaluated in both synthetic and real-world environments. The results demonstrate that the proposed method achieves global drift-free estimation, enabling UAVs to accurately localize themselves in GNSS-denied environments.

Real-Time Localization and Colorful Three-Dimensional Mapping of Orchards Based on Multi-Sensor Fusion Using Extended Kalman Filter

To realize autonomous navigation and intelligent management in orchards, vehicles require real-time positioning and globally consistent mapping of surroundings with sufficient information. However, the unstructured and unstable characteristics of orchards present challenges for accurate and stable localization and mapping. This study proposes a framework fusing LiDAR, visual, and inertial data by using the extended Kalman filter (EKF) to achieve real-time localization and colorful LiDAR point-cloud mapping in orchards. First, the multi-sensor data were integrated into a loosely-coupled framework based on the EKF to improve the pose estimation, with the pose estimation from LiDAR and gyroscope acting as the predictions, while that from visual-inertial odometry acting as the observations. Then, the Loam_Livox algorithm was enhanced by incorporating color from the image into the LiDAR point cloud, enabling the real-time construction of a three-dimensional colorful map of the orchard. The method demonstrates a high accuracy for localization in different motion trajectories (average RMSE: 0.3436) and different scenarios (average RMSE: 0.1230) and clear and efficient construction of three-dimensional colorful mapping, taking only 75.01 ms in localization and mapping for a frame of LiDAR point cloud. This indicates the proposed method has a great potential for the autonomous navigation of agricultural vehicles.

Robust Monocular Visual Inertial Odometry in γ Radioactive Environments using Edge-based Point Features

Abstract In the γ radioactive environment, high-energy photons induce degradation of the image sensor, which effects feature detection and tracking in the Visual Inertial Odometry (VIO) algorithm and deteriorates its localization performance. To address this issue, in this work, we propose a monocular VIO method using edge-based point features. To mitigate the effects of radiation noise, firstly, in the image preprocessing module, the median filter is used for real-time image denoising. Secondly, in the data association module, both Shi-Tomasi and edge-based point features are detected. The edge-based point feature is the endpoint or corner point in the salient edge map, which is more robust to radiation noise. Then, the bi-directional motion parallaxes and the RANdom SAmple Consensus (RANSAC) method are exploited to reject outliers. Finally, the point features measurements and Inertial Measurement Unit (IMU) pre-integration measurements are added into a tightly-coupled sliding window optimization VIO framework for localization estimation. The proposed method is verified by synthetic and real γ radioactive environment datasets. The experimental results show that the proposed method achieves more accurate and robust localization than the state-of-the-art VIO approaches in the γ radioactive environments.

PVI-DSO: Leveraging Planar Regularities for Direct Sparse Visual-Inertial Odometry

The monocular visual-inertial odometry (VIO) based on the direct method can leverage all available pixels in the image to simultaneously estimate the camera motion and reconstruct the denser map of the scene in real time. However, the direct method is sensitive to photometric changes, which can be compensated by introducing geometric information in the environment. In this paper, we propose a monocular direct sparse visual-inertial odometry, which exploits the planar regularities (PVI-DSO). Our system detects the planar regularities from the 3D mesh built on the estimated map points. To improve the pose estimation accuracy with the geometric information, a tightly coupled coplanar constraint expression is used to express photometric error in the direct method. Additionally, to improve the optimization efficiency, we elaborately derive the analytical Jacobian of the linearization form for the coplanar constraint. Finally, the inertial measurement error, coplanar point photometric error, non-coplanar photometric error, and prior error are added into the optimizer, which simultaneously improves the pose estimation accuracy and mesh itself. We verified the performance of the whole system on simulation and real-world datasets. Extensive experiments have demonstrated that our system outperforms the state-of-the-art counterparts.

MIXO: Mixture Of Experts-Based Visual Odometry for Multicamera Autonomous Systems

Visual-inertial odometry is a fundamental technology exploited by autonomous vehicles and mobile robots to determine their position in unknown environments. Indeed, many techniques have been proposed for monocular and stereo cameras. State-of-the-art autonomous vehicles, however, are equipped with multiple cameras covering the entire vehicle environment. We present here MIxture of eXperts Odometry (MIXO), a data-driven, machine learning-based technique that loosely combines odometry outputs from multiple cameras to obtain a more accurate and robust global estimate. It stems from the intuition that each camera provides an optimal vantage point in specific driving scenarios. In MIXO, each camera (or expert) is individually processed by a state-of-the-art visual odometry algorithm (e.g., ORB-SLAM2). Then, the odometry estimates are mixed by a gating network, which selects the locally optimal experts in the current operational conditions and weights their contributions accordingly. MIXO is a lightweight module that can be easily implemented on top of any visual odometry algorithm. Experimental results on real-life data from autonomous vehicles show that MIXO achieves more robust and accurate results than any single camera, reducing the absolute rotational and translation error by 38% and 15%, respectively.

SRVIO: Super Robust Visual Inertial Odometry for Dynamic Environments and Challenging Loop-Closure Conditions

There has been extensive research on visual localization and odometry for autonomous robots and virtual reality during the past decades. Traditionally, this problem has been solved with the help of expensive sensors, such as light detection and ranging (LiDAR). Nowadays, the focus of the leading research in this field is on robust localization using more economic sensors, such as cameras and inertial measurement units. Consequently, geometric visual localization methods have become more accurate over time. However, these methods still suffer from significant loss and divergence in challenging environments, such as a room full of moving people. Scientists started using deep neural networks (DNNs) to mitigate this problem. The main idea behind using DNNs is to better understand challenging aspects of the data and overcome complex conditions such as the movement of a dynamic object in front of the camera that covers the full view of the camera, extreme lighting conditions, and the high speed of the camera. Prior end-to-end DNN methods did overcome some of these challenges. However, no general and robust framework is available to overcome all challenges together. In this article, we have combined geometric and DNN-based methods to have the generality and speed of geometric SLAM frameworks and overcome most of these challenging conditions with the help of DNNs and deliver the most robust framework so far. To do so, we have designed a framework based on VINS-Mono and shown that it can achieve state-of-the-art results on TUM-Dynamic, TUM-VI, ADVIO, and EuRoC datasets compared to geometric and end-to-end DNN-based simultaneous localization and mappings. Our proposed framework can also achieve outstanding results on extreme simulated cases resembling the aforementioned challenges.

IR-VIO: Illumination-Robust Visual-Inertial Odometry Based on Adaptive Weighting Algorithm With Two-Layer Confidence Maximization

Illumination change, image blur, and fast motion dramatically decrease the performance of visual-inertial navigation systems (VINS). This article presents a new illumination-robust visual-inertial odometry (IR-VIO) based on adaptive weighting algorithm with two-layer confidence maximization. First, to prevent the VIO performance degradation caused by poor image quality in complex scenes and ignoring the confidence differences of feature points, we develop a novel adaptive weighting algorithm on the multisensor layer and visual feature layer to better fuse multisensor information and maximize the overall confidence of VIO. Second, to solve the problems of image feature tracking difficulty and excessive image noise in illumination-changing scenes, an image enhancement algorithm is introduced to enhance consecutive images to the same brightness level, while a block noise removal algorithm with constraint protection mechanism is proposed to dynamically remove noise points. Finally, experimental results in the public dataset and real-world environments demonstrate that IR-VIO has superior performance in terms of accuracy and robustness compared with the state-of-the-art methods.

A Novel Three-Stage Robust Adaptive Filtering Algorithm for Visual-Inertial Odometry in GNSS-Denied Environments

A Novel Three-Stage Robust Adaptive Filtering Algorithm for Visual-Inertial Odometry in GNSS-Denied Environments

Research on Multi-Sensor Simultaneous Localization and Mapping Technology for Complex Environment of Construction Machinery

Simultaneous localization and mapping (SLAM), as a key task of unmanned vehicles for construction machinery, is of great significance for later path planning and control. Construction tasks in the engineering field are mostly carried out in bridges, tunnels, open fields, etc. The prominent features of these environments are high scene similarity, few geometric features, and large-scale repetitive texture information, which is prone to sensor detection degradation. This leads to positioning drift and map building failure. The traditional method of motion estimation and 3D reconstruction uses a single sensor, which lacks enough information, has poor adaptability to the environment, and cannot guarantee good positioning accuracy and robustness in complex environments. Currently, the strategy of multi-sensor fusion is proven to be an effective solution and is widely studied. This paper proposes a SLAM framework that integrates LiDAR, IMU, and camera. It tightly couples the texture information observed by camera, the geometric information scanned by LiDAR, and the measured value of IMU, allowing visual-inertial odometry (VIO) and LiDAR-inertial odometry (LIO) common implementation. The LIO subsystem extracts point cloud features and matches them with the global map. The obtained pose estimation can be used for the initialization of the VIO subsystem. The VIO system uses direct method to minimize the photometric error and IMU measurement error between images to estimate the pose of the robot and the geometric structure of the scene. The two subsystems assist each other to perform pose estimation, and can operate normally even when any subsystem fails. A factor graph is used to combine all constraints to achieve global pose optimization. Keyframe and sliding window strategies are used to ensure real-time performance. Through real-vehicle testing, the system can perform incremental and real-time state estimation and reconstruct a dense 3D point cloud map, which can effectively solve the problems of positioning drift and mapping failure in the lack of geometric features or challenging construction environments.

Validation of real-time inside-out tracking and depth realization technologies for augmented reality-based neuronavigation.

Concomitant with the significant advances in computing technology, the utilization of augmented reality-based navigation in clinical applications is being actively researched. In this light, we developed novel object tracking and depth realization technologies to apply augmented reality-based neuronavigation to brain surgery. We developed real-time inside-out tracking based on visual inertial odometry and a visual inertial simultaneous localization and mapping algorithm. The cube quick response marker and depth data obtained from light detection and ranging sensors are used for continuous tracking. For depth realization, order-independent transparency, clipping, and annotation and measurement functions were developed. In this study, the augmented reality model of a brain tumor patient was applied to its life-size three-dimensional (3D) printed model. Using real-time inside-out tracking, we confirmed that the augmented reality model remained consistent with the 3D printed patient model without flutter, regardless of the movement of the visualization device. The coordination accuracy during real-time inside-out tracking was also validated. The average movement error of the X and Y axes was 0.34 ± 0.21 and 0.04 ± 0.08mm, respectively. Further, the application of order-independent transparency with multilayer alpha blending and filtered alpha compositing improved the perception of overlapping internal brain structures. Clipping, and annotation and measurement functions were also developed to aid depth perception and worked perfectly during real-time coordination. We named this system METAMEDIP navigation. The results validate the efficacy of the real-time inside-out tracking and depth realization technology. With these novel technologies developed for continuous tracking and depth perception in augmented reality environments, we are able to overcome the critical obstacles in the development of clinically applicable augmented reality neuronavigation.

FLAGDroneRacing: An Autonomous Drone Racing System

This paper presents a complete system for autonomous drone racing combining image recognition, depth mapping, visual-inertial odometry (VIO), and collision-free trajectory planning. The proposed system focuses on simple, robust, and computationally efficient techniques to enable onboard hardware applications. A loosely coupled visual-inertial localization system is devised, to ensure real-time and robust localization. A lightweight CPU-based detection module is designed, which consists of autonomous mapping and gate detection components. We also introduce a robust and efficient trajectory planner to generate smooth and collision-free trajectories in real-time. The proposed methods are tested extensively through benchmark comparisons and challenging indoor flights, while simulation results show the validness and effectiveness of our proposed system. We release our implementation as an open-source ROS-package.

Robust Visual-Inertial Odometry Based on a Kalman Filter and Factor Graph

Robust Visual-Inertial Odometry Based on a Kalman Filter and Factor Graph

GNSS PPP-RTK tightly coupled with low-cost visual-inertial odometry aiming at urban canyons

GNSS PPP-RTK tightly coupled with low-cost visual-inertial odometry aiming at urban canyons

S-VIO: Exploiting Structural Constraints for RGB-D Visual Inertial Odometry

Vision-based localization is an essential problem for autonomous systems while the performance of vision-based odometry degrades in the challenging scenario. This letter presents S-VIO, an RGB-D visual inertial odometry (VIO) which fully uses multi-sensor measurements (i.e. depth, RGB and IMU), heterogeneous landmarks (i.e. points, lines and planes) and structural regularity of the environment to obtain a robust and accurate localization result. In order to detect the underlying structural regularity of the environment, a two-step Atlanta world inference method is proposed. Leveraging the gravity direction estimated by the VIO system, the proposed algorithm first generates horizontal Atlanta axis hypotheses from a set of recently optimized plane landmarks. Then the following plane landmarks and line clusters are used to filter out the occasionally observed axes based on the persistence of the observation. The remaining axes will survive and be saved in the Atlanta map for future re-observations. Particularly, an efficient mined-and-stabbed (MnS) method is applied to classify the structural line and extract the vanishing point from each line cluster. In addition, a closed-form initialization method for the structural line feature is proposed, which leverages the known direction to obtain a better initial estimation. Integrated with the above novelties, S-VIO is tested on two public real-world RGB-D inertial datasets. Experiments demonstrate that S-VIO has better accuracy and robustness compared to state-of-the-art VIO and RGB-D VIO algorithms. The code and the supplementary material will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/oranfire/SVIO</uri> .