Visual Odometry System Research Articles

CR-LDSO: Direct Sparse LiDAR-Assisted Visual Odometry With Cloud Reusing

LiDAR-assisted visual odometry (VO) is a widely-used solution for pose estimation and mapping. However, most existing LiDAR-assisted VO systems could suffer from the problems of 1) lacking distinctive and evenly distributed pixels for tracking due to the sparsity of LiDAR points and limited FOV overlap between a camera and LiDAR, and 2) nontrivial errors when processing LiDAR point clouds. To address above problems, we present CR-LDSO, a direct sparse LiDAR-assisted VO with the core parts being: 1) a novel cloud reusing method with point extraction/re-extraction to increase both the camera-LiDAR FOV overlap and the number of high-quality tracking pixels and 2) an occlusion removal method to exclude mismatching pixels due to occluded 3D object from sliding-window optimization and a point extraction strategy without depth interpolation. Extensive experimental results on public datasets demonstrates the superiority of our method to the existing state-of-the-art methods.

Learning generalized visual odometry using position-aware optical flow and geometric bundle adjustment

Recent visual odometry (VO) methods incorporating geometric algorithm into deep-learning architecture have shown outstanding performance on the challenging monocular VO task. Despite encouraging results are shown, previous methods ignore the requirement of generalization capability under noisy environment and various scenes. To address this challenging issue, this work first proposes a novel optical flow network (PANet). Compared with previous methods that predict optical flow as a direct regression task, our PANet computes optical flow by predicting it into the discrete position space with optical flow probability volume, and then converting it to optical flow. Next, we improve the bundle adjustment module to fit the self-supervised training pipeline by introducing multiple sampling, ego-motion initialization, dynamic damping factor adjustment, and Jacobi matrix weighting. In addition, a novel normalized photometric loss function is advanced to improve the depth estimation accuracy. The experiments show that the proposed system not only achieves comparable performance with other state-of-the-art self-supervised learning-based methods on the KITTI dataset, but also significantly improves the generalization capability compared with geometry-based, learning-based and hybrid VO systems on the noisy KITTI and the challenging outdoor (KAIST) scenes.

Robust Visual Odometry Leveraging Mixture of Manhattan Frames in Indoor Environments.

We propose a robust RGB-Depth (RGB-D) Visual Odometry (VO) system to improve the localization performance of indoor scenes by using geometric features, including point and line features. Previous VO/Simultaneous Localization and Mapping (SLAM) algorithms estimate the low-drift camera poses with the Manhattan World (MW)/Atlanta World (AW) assumption, which limits the applications of such systems. In this paper, we divide the indoor environments into two different scenes: MW and non-MW scenes. The Manhattan scenes are modeled as a Mixture of Manhattan Frames, in which each Manhattan Frame in itself defines a Manhattan World of a specific orientation. Moreover, we provide a method to detect Manhattan Frames (MFs) using the dominant directions extracted from the parallel lines. Our approach is designed with lower computational complexity than existing techniques using planes to detect Manhattan Frame (MF). For MW scenes, we separately estimate rotational and translational motion. A novel method is proposed to estimate the drift-free rotation using MF observations, unit direction vectors of lines, and surface normal vectors. Then, the translation part is recovered from point-line tracking. In non-MW scenes, the tracked and matched dominant directions are combined with the point and line features to estimate the full 6 degree of freedom (DoF) camera poses. Additionally, we exploit the rotation constraints generated from the multi-view dominant directions observations. The constraints are combined with the reprojection errors of points and lines to refine the camera pose through local map bundle adjustment. Evaluations on both synthesized and real-world datasets demonstrate that our approach outperforms state-of-the-art methods. On synthesized datasets, average localization accuracy is 1.5 cm, which is equivalent to state-of-the-art methods. On real-world datasets, the average localization accuracy is 1.7 cm, which outperforms the state-of-the-art methods by 43%. Our time consumption is reduced by 36%.

Robust visual odometry using sparse optical flow network

Deep learning-based visual odometry (VO) has recently attracted much attention. In performing computer vision tasks, deep models are often more robust than manually extracted features. Deep models, however, have limited reliability and generalization ability which constrain their application in the VO systems. The existing approaches to address these issues are often based on creating more realistic datasets or employing strategies such as unsupervised learning and online fine-tuning. In contrast to the previous research, here we tackle these issues from the model structure standpoint. We present a simple yet robust VO (namely SF-VO) based on an especially designed sparse optical flow network. We then show that this network becomes suitable for VO applications by decomposing the global optimization problem into a single-point optimization. Combining the network with a consistency verification module and a Perspective-n-Point (PnP) solver, we then form a frame-to-frame VO system using the traditional pose estimation pipeline. Extensive experiments show that the proposed VO system effectively generalizes to real scenes while only synthetic datasets are used in the training process. It is also shown that the proposed model also outperforms other deep learning-based methods with a model size of only 1.69 M. Comparisons with the state-of-art optical flow models and performing expansion experiments further confirm that the designed network demonstrates a higher level of generalization ability and is capable of being trained based on limited datasets.

A Lightweight Stereo Visual Odometry System for Navigation of Autonomous Vehicles in Low-Light Conditions

Localization of vehicles in a 3D environment is a basic task in autonomous driving. In the low-light environments, it is difficult to navigate independently using a visual odometry for autonomous driving. The main reason for this challenge is the blurred images in the scenes with insufficient illumination. Although numerous works focused on this issue, it still has a number of inherent drawbacks. In this paper, we propose a lightweight stereo visual odometry system for navigation of autonomous vehicles in low-light situations. Contrary to the existing recovery methods, we aim to divide the captured image into the illumination image as well as the reflectance image and only estimate the illumination one, where the enhanced map of the low-light image is acquired by using the retinex theory. In addition, we further utilize a simplified and rapid feature detection scheme, which reduces the computation time by about 85% but maintaining the matching accuracy similar to that of ORB features. Finally, the experiments show that our average memory consumption of our proposed method is much less than the conventional algorithm.

Improving Monocular Visual Odometry Using Learned Depth

Monocular visual odometry (VO) is an important task in robotics and computer vision. Thus far, how to build accurate and robust monocular VO systems that can work well in diverse scenarios remains largely unsolved. In this article, we propose a framework to exploit monocular depth estimation for improving VO. The core of our framework is a monocular depth estimation module with a strong generalization capability for diverse scenes. It consists of two separate working modes to assist the localization and mapping. With a single monocular image input, the depth estimation module predicts a relative depth to help the localization module on improving the accuracy. With a sparse depth map and an RGB image input, the depth estimation module can generate accurate scale-consistent depth for dense mapping. Compared with current learning-based VO methods, our method demonstrates a stronger generalization ability to diverse scenes. More significantly, our framework is able to boost the performances of existing geometry-based VO methods by a large margin.

Coarse TRVO: A Robust Visual Odometry with Detector-Free Local Feature

The visual SLAM system requires precise localization. To obtain consistent feature matching results, visual features acquired by neural networks are being increasingly used to replace traditional manual features in situations with weak texture, motion blur, or repeated patterns. However, to improve the level of accuracy, most deep learning enhanced SLAM systems, which have a decreased efficiency. In this paper, we propose Coarse TRVO, a visual odometry system that uses deep learning for feature matching. The deep learning network uses a CNN and transformer structures to provide dense high-quality end-to-end matches for a pair of images, even under indistinctive settings with low-texture regions or repeating patterns occupying the majority of the field of view. Meanwhile, we made the proposed model compatible with NVIDIA TensorRT runtime to boost the performance of the algorithm. After obtaining the matching point pairs, the camera pose is solved in an optimized way by minimizing the re-projection error of the feature points. Experiments based on multiple data sets and real environments show that Coarse TRVO achieves a higher robustness and relative positioning accuracy in comparison with the current mainstream visual SLAM system.

Fast visual inertial odometry with point–line features using adaptive EDLines algorithm

In mainstream visual inertial odometry (VIO) systems, the method of positional solution by feature point extraction and matching in the image is widely used. However, the tracking accuracy of point features is dependent on the texture richness in the environment. Although many existing algorithms introduce line features in the front end to improve the system’s environmental adaptability, most of them sacrifice system real-time in exchange for higher positioning accuracy. The extraction and matching of line features often require more time, thus failing to meet the real-time requirements of the system for localization. In this paper, we therefore propose a fast VIO fused with point and line features, which enables the system to maintain a high level of positioning robustness in dim and changing light environments with low time cost. The point–line features VIO algorithm is based on adaptive thresholding of EDLines. By adding an adaptive thresholding component to the EDLines algorithm, the robustness of line feature extraction is enhanced to better adapt to changes in ambient lighting. The time needed for line feature extraction is also significantly reduced. A line feature matching algorithm based on geometric information and structural similarity is proposed, which enables fast and accurate line feature matching. The algorithm is compared with point-line visual-inertial odometry and monocular visual-inertial state estimator algorithms on the European robotics challenge dataset and real-world scenes. Many experiments prove that the algorithm has improved in both real time and accuracy.

Hybrid sparse monocular visual odometry with online photometric calibration

Most monocular visual Simultaneous Localization and Mapping (vSLAM) and visual odometry (VO) algorithms focus on either feature-based methods or direct methods. Hybrid (semi-direct) approach is less studied although it is equally important. In this paper, a hybrid sparse visual odometry (HSO) algorithm with online photometric calibration is proposed for monocular vision. HSO introduces two novel measures, that is, direct image alignment with adaptive mode selection and image photometric description using ratio factors, to enhance the robustness against dramatic image intensity changes and motion blur. Moreover, HSO is able to establish pose constraints between keyframes far apart in time and space by using KLT tracking enhanced with a local-global brightness consistency. The convergence speed of candidate map points is adopted as the basis for keyframe selection, which strengthens the coordination between the front end and the back end. Photometric calibration is elegantly integrated into the VO system working in tandem: (1) Photometric interference from the camera, such as vignetting and changes in exposure time, is accurately calibrated and compensated in HSO, thereby improving the accuracy and robustness of VO. (2) On the other hand, VO provides pre-calculated data for the photometric calibration algorithm, which reduces resource consumption and improves the estimation accuracy of photometric parameters. Extensive experiments are performed on various public datasets to evaluate the proposed HSO against the state-of-the-art monocular vSLAM/VO and online photometric calibration methods. The results show that the proposed HSO achieves superior performance on VO and photometric calibration in terms of accuracy, robustness, and efficiency, being comparable with the state-of-the-art VO/vSLAM systems. We open source HSO for the benefit of the community.

A point selection strategy with edge and line detection for Direct Sparse Visual Odometry

In most feature-based Visual Simultaneous Localization and Mapping systems, the pixels in a current image are compared with the correlative pixels in previous images, and the difference in the coordinates of pixels shows the movement of the camera. Different from the feature-based systems, direct methods operate on image intensity directly. Every pixel on the image or selected pixels with sufficient intensity gradient can be utilized. However, the noises in the images may affect the performance of those algorithms as the pixels are not adequately selected. In this work, we propose a new pixel selection method for a direct visual odometry system that focuses on the edge pixels. The edge pixels are usually more stable and repeatable than normal pixels. We apply the traditional edge detection method with adaptive parameters to get rough edge results. Then the edges are separated by gradient and shape. We use straightness, smoothness, length, and gradient magnitude to select the meaningful edges. We replace the pixel selection step of Direct Sparse Odometry and Direct Sparse Odometry with Loop Closure to present the evaluation on open datasets. The experimental results indicate that our method improves the performance of existing direct visual odometry systems in man-made scenes but is not suitable for pure natural scenes.

Adaptive compensation visual odometry in dynamic scenarios

In dynamic scenarios, dynamic participants break the static assumptions of the visual odometry (VO) algorithm. Hence, dynamic participants are typically removed and only static participants are used as motion references. When semantic information is used to eliminate participants with dynamic semantic labels in the scene, the dynamic degree of the scene affects system robustness and poses estimation accuracy. In this paper, we propose a VO system that employs adaptive compensation for semantic information and utilizes an adaptive compensation model for sparse static feature map construction to make the compensation process a flexible, independent thread. Moreover, a layered extraction fusion framework is proposed to uniformly sample equi-probability feature points both globally and locally. This framework combines multi-level weights of scene mapping and spatial-temporal priority information of self-organization. Finally, in the candidate pixel extraction stage, system efficiency is improved through region matching candidate pixel detection, extraction and fusion semantic constraints. Experiments are done by using the TUM RGBD datasets. Comparisons with many conventional visual mileage calculation methods reveal that the absolute and relative trajectory errors of camera motion are significantly reduced in most scenes with different dynamic degrees. Thus, compared to previous algorithms, the proposed algorithm is more robust and precise in dynamic scenes.

Indoor Localization of Hand-Held OCT Probe Using Visual Odometry and Real-Time Segmentation Using Deep Learning.

Optical coherence tomography (OCT) is an established medical imaging modality that has found widespread use due to its ability to visualize tissue structures at a high resolution. Currently, OCT hand-held imaging probes lack positional information, making it difficult or even impossible to link a specific image to the location it was originally obtained. In this study, we propose a camera-based localization method to track and record the scanner position in real-time, as well as providing a deep learning-based segmentation method. We used camera-based visual odometry (VO) and simultaneous mapping and localization (SLAM) to compute and visualize the location of a hand-held OCT imaging probe. A deep convolutional neural network (CNN) was used for kidney tubule lumens segmentation. The mean absolute error (MAE) and the standard deviation (STD) for 1D translation were found to be 0.15 mm and 0.26mm respectively. For 2D translation, the MAE and STD were found to be 0.85 mm and 0.50 mm, respectively. The dice coefficient of the segmentation method was 0.7. The t-statistic of the T-test between predicted and actual average densities and predicted and actual average diameters were 7.7547e-13 and 2.2288e-15 respectively. We also experimented on a preserved kidney utilizing our localization method with automatic segmentation. Comparisons of the average density maps and average diameter maps were made between the 3D comprehensive scan and VO system scan. Our results demonstrate that VO can track the probe location at high accuracy, and provides a user-friendly visualization tool to review OCT 2D images in 3D space. It also indicates that deep learning can provide high accuracy and high speed for segmentation. The proposed methods can be potentially used to predict delayed graft function (DGF) in kidney transplantation.

Direct Visual Odometry by Fusing Luminosity and Depth Information

In the traditional direct visual odometry, it is difficult to satisfy the photometric invariant assumption due to the influence of illumination changes in the real environment, which will lead to errors and drift. This paper proposes an improved direct visual odometry system, which combines luminosity and depth information. The algorithm proposed in this paper uses Kinect 2 to collect RGB images with the corresponding depth information, and selects points with large changes of gray gradient to construct a luminosity error function and uses the corresponding depth information to construct a depth error function. The two error functions are merged into one function and converted into the least squares function of the pose of camera, the Levenberg-Marquardt algorithm is used to solve the camera pose. Finally, the Graph optimization theory and the g2o library are used to optimize the initial pose. Experiments show that the algorithm can reduce the error to a certain extent and reduce the drift caused by illumination changes.

Calib-Net: Calibrating the Low-Cost IMU via Deep Convolutional Neural Network.

The low-cost Inertial Measurement Unit (IMU) can provide orientation information and is widely used in our daily life. However, IMUs with bad calibration will provide inaccurate angular velocity and lead to rapid drift of integral orientation in a short time. In this paper, we present the Calib-Net which can achieve the accurate calibration of low-cost IMU via a simple deep convolutional neural network. Following a carefully designed mathematical calibration model, Calib-Net can output compensation components for gyroscope measurements dynamically. Dilation convolution is adopted in Calib-Net for spatio-temporal feature extraction of IMU measurements. We evaluate our proposed system on public datasets quantitively and qualitatively. The experimental results demonstrate that our Calib-Net achieves better calibration performance than other methods, what is more, and the estimated orientation with our Calib-Net is even comparable with the results from visual inertial odometry (VIO) systems.

An Integrated Visual Odometry System With Stereo Camera for Unmanned Underwater Vehicles

Navigation is a challenging problem in the area of underwater unmanned vehicles, due to the significant electronmagnetic wave attenuation and the uncertainties in underwater environments. The conventional methods, mainly implemented by acoustic devices, suffer limitations such as high cost, terrain effects and low refresh rate. In this paper, a novel low-cost underwater visual navigation method, named Integrated Visual Odometry with a Stereo Camera (IVO-S), has been investigated. Unlike pure visual odometry, the proposed method fuses the information from inertial sensors and a sonar so that it is able to work in context-sparse environments. In practical experiments, the vehicle was operated to follow specific closed-loop shapes. The Integrated Visual Odoemtry with Monocular Camera (IVO-M) method and other popular open source Visual SLAMs (Simultaneous Localisation and Mappings), such as ORB-SLAM2 and VINS-Mono, have been used to provide comparative results. The cumulative error ratio is used as the quantitative evaluation method to analyse the practical test results. It is shown that the IVO-S method is able to work in underwater sparse-feature environments with high accuracy, whilst also being a low cost solution.

RGB-D DSO: Direct Sparse Odometry With RGB-D Cameras for Indoor Scenes

Visual odometry (VO) is a fundamental technique for many robotics and augmented reality (AR) applications. However, most existing RGB-D VO systems suffer from large performance degradation when large occlusions are present and/or a large portion of depth values are invalid due to the limited range of an RGB-D camera, prohibiting the usage of most systems in practical applications. To address above two problems, we present RGB-D DSO, an RGB-D direct sparse odometry with the core part being sliding-window optimization with occlusion removal and a depth refinement module. Occlusion removal excludes negative effects arising from occluded objects when minimizing the final energy function for camera pose tracking. Depth refinement ensures sufficient valid depth values uniformly distributed for the depth map of a keyframe. Experimental results on three public datasets demonstrate that our method achieves smaller tracking error than most existing state-of-the-art methods. Meanwhile, our system takes only 21.93 ms to track a frame, which is faster than most existing methods.

ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual–Inertial, and Multimap SLAM

This paper presents ORB-SLAM3, the first system able to perform visual, visual-inertial and multi-map SLAM with monocular, stereo and RGB-D cameras, using pin-hole and fisheye lens models. The first main novelty is a feature-based tightly-integrated visual-inertial SLAM system that fully relies on Maximum-a-Posteriori (MAP) estimation, even during the IMU initialization phase. The result is a system that operates robustly in real-time, in small and large, indoor and outdoor environments, and is 2 to 5 times more accurate than previous approaches. The second main novelty is a multiple map system that relies on a new place recognition method with improved recall. Thanks to it, ORB-SLAM3 is able to survive to long periods of poor visual information: when it gets lost, it starts a new map that will be seamlessly merged with previous maps when revisiting mapped areas. Compared with visual odometry systems that only use information from the last few seconds, ORB-SLAM3 is the first system able to reuse in all the algorithm stages all previous information. This allows to include in bundle adjustment co-visible keyframes, that provide high parallax observations boosting accuracy, even if they are widely separated in time or if they come from a previous mapping session. Our experiments show that, in all sensor configurations, ORB-SLAM3 is as robust as the best systems available in the literature, and significantly more accurate. Notably, our stereo-inertial SLAM achieves an average accuracy of 3.6 cm on the EuRoC drone and 9 mm under quick hand-held motions in the room of TUM-VI dataset, a setting representative of AR/VR scenarios. For the benefit of the community we make public the source code.

Robustness Improvement of Using Pre-Trained Network in Visual Odometry for On-Road Driving

Robustness in on-road driving Visual Odometry (VO) systems is critical, as it determines the reliable performance in various scenarios and environments. Especially with the development of data-driven technology, the combination of data-driven VO and model-based VO has achieved accurate tracking performance. However, the lack of generalization of pre-trained deep neural networks (DNN) limits the robustness of such a combination in unseen environments. In this study, we introduce a novel framework with appropriate usage of DNN prediction and improve the robustness in the self-driving application. Based on the characteristic of on-road self-driving motion and the DNN output, we propose a two-step optimization strategy with a variable degree of freedom (DoF), i.e., the use of two types of DoF representations during pose estimation. Specifically, our two-step optimization operates according to the residual of the optimization with the motion label classification from the pre-trained DNN, as well as our proposed Motion Evaluation by essential matrix construction. Experimental results show that our framework obtains better tracking accuracy than the existing methods.

Visual odometry in local underwater navigation problems

The work processes of the ORB-SLAM algorithm are presented. The results of experimental studies on temporal comparisons of the operation of the algorithm with different parameters and cameras are presented. The necessity of forming a visual odometry (VO) system as a local navigation of remote-controlled and autonomous underwater robots has been substantiated. The two most suitable odometry methods in the underwater environment are described, such as their advantages and disadvantages. The work processes of the ORB-SLAM algorithm are presented. The results of experimental studies on temporal comparisons of the operation of the algorithm with different parameters and cameras are presented. The procedure for preparing video data is described: processing a video stream, adjusting camera parameters for calibration. The experiments represent the testing of the ORB-SLAM3 algorithm on a sample of video filmed as part of the ecological monitoring of the Caspian shelf in 2020.

An end-to-End deep context gate convolutional visual odometry system based on lightweight attention mechanism

PurposeConventional learning-based visual odometry (VO) systems usually use convolutional neural networks (CNN) to extract features, where some important context-related and attention-holding global features might be ignored. Without essential global features, VO system will be sensitive to various environmental perturbations. The purpose of this paper is to design a novel learning-based framework that aims to improve accuracy of learning-based VO without decreasing the generalization ability.Design/methodology/approachInstead of CNN, a context-gated convolution is adopted to build an end-to-end learning framework, which enables convolutional layers that dynamically capture representative local patterns and composes local features of interest under the guidance of global context. In addition, an attention mechanism module is introduced to further improve learning ability and enhance robustness and generalization ability of the VO system.FindingsThe proposed system is evaluated on the public data set KITTI and the self-collected data sets of our college building, where it shows competitive performance compared with some classical and state-of-the-art learning-based methods. Quantitative experimental results on the public data set KITTI show that compared with CNN-based VO methods, the average translational error and rotational error of all the test sequences are reduced by 45.63% and 37.22%, respectively.Originality/valueThe main contribution of this paper is that an end-to-end deep context gate convolutional VO system based on lightweight attention mechanism is proposed, which effectively improves the accuracy compared with other learning-based methods.