Odometry System Research Articles

A point selection strategy with edge and line detection for Direct Sparse Visual Odometry

In most feature-based Visual Simultaneous Localization and Mapping systems, the pixels in a current image are compared with the correlative pixels in previous images, and the difference in the coordinates of pixels shows the movement of the camera. Different from the feature-based systems, direct methods operate on image intensity directly. Every pixel on the image or selected pixels with sufficient intensity gradient can be utilized. However, the noises in the images may affect the performance of those algorithms as the pixels are not adequately selected. In this work, we propose a new pixel selection method for a direct visual odometry system that focuses on the edge pixels. The edge pixels are usually more stable and repeatable than normal pixels. We apply the traditional edge detection method with adaptive parameters to get rough edge results. Then the edges are separated by gradient and shape. We use straightness, smoothness, length, and gradient magnitude to select the meaningful edges. We replace the pixel selection step of Direct Sparse Odometry and Direct Sparse Odometry with Loop Closure to present the evaluation on open datasets. The experimental results indicate that our method improves the performance of existing direct visual odometry systems in man-made scenes but is not suitable for pure natural scenes.

RLP‐VIO: Robust and lightweight plane‐based visual‐inertial odometry for augmented reality

AbstractWe propose RLP‐VIO—a robust and lightweight monocular visual‐inertial odometry system using multiplane priors. With planes extracted from the point cloud, visual‐inertial‐plane PnP uses the plane information for fast localization. Depth estimation is susceptible to degenerated motion, so the planes are expanded in a reprojection consensus‐based way robust to depth errors. For sensor fusion, our sliding‐window optimization uses a novel structureless plane‐distance error cost, which prevents the fill‐in effect that poisons the BA problem's sparsity and permits the use of a smaller sliding window while maintaining good accuracy. The total computational cost is further reduced with our modified marginalization strategy. To further improve the tracking robustness, the landmark depths are constrained using the planes during degenerated motion. The whole system is parallelized with a three‐stage pipeline. Under controlled environments, this parallelization runs deterministically and produces consistent results. The resulting VIO system is tested on widely used datasets and compared with several state‐of‐the‐art systems. Our system achieves competitive accuracy and works robustly even on long and challenging sequences. To demonstrate the effectiveness of the proposed system, we also show the AR application running on mobile devices in real‐time.

Robust Tightly-Coupled Visual-Inertial Odometry with Pre-built Maps in High Latency Situations.

In this paper, we present a novel monocular visual-inertial odometry system with pre-built maps deployed on the remote server, which can robustly run in real-time on a mobile device even in high latency situations. By tightly coupling VIO with geometric priors from pre-built maps, our system can tolerate the high latency and low frequency of global localization service, which is especially suitable for practical applications when the localization service is deployed on the remote server. Firstly, sparse point clouds are obtained from the dense mesh by the ray casting method according to the localization results. The dense mesh can be reconstructed from the point clouds generated by Structure-from-Motion. We directly use the sparse point clouds in feature tracking and state update to suppress drift. In the process of feature tracking, the high local accuracy of VIO is fully utilized to effectively remove outliers and make our system robust. The experiments on EurocMav datasets and simulation datasets show that compared with state-of-the-art methods, our method can achieve better results in terms of both precision and robustness. The effectiveness of the proposed method is further demonstrated through a real-time AR demo on a mobile phone with the aid of visual localization on the remote server.

An Enhanced Hybrid Visual-Inertial Odometry System for Indoor Mobile Robot.

As mobile robots are being widely used, accurate localization of the robot counts for the system. Compared with position systems with a single sensor, multi-sensor fusion systems provide better performance and increase the accuracy and robustness. At present, camera and IMU (Inertial Measurement Unit) fusion positioning is extensively studied and many representative Visual–Inertial Odometry (VIO) systems have been produced. Multi-State Constraint Kalman Filter (MSCKF), one of the tightly coupled filtering methods, is characterized by high accuracy and low computational load among typical VIO methods. In the general framework, IMU information is not used after predicting the state and covariance propagation. In this article, we proposed a framework which introduce IMU pre-integration result into MSCKF framework as observation information to improve the system positioning accuracy. Additionally, the system uses the Helmert variance component estimation (HVCE) method to adjust the weight between feature points and pre-integration to further improve the positioning accuracy. Similarly, this article uses the wheel odometer information of the mobile robot to perform zero speed detection, zero-speed update, and pre-integration update to enhance the positioning accuracy of the system. Finally, after experiments carried out in Gazebo simulation environment, public dataset and real scenarios, it is proved that the proposed algorithm has better accuracy results while ensuring real-time performance than existing mainstream algorithms.

Balancing the Budget: Feature Selection and Tracking for Multi-Camera Visual-Inertial Odometry

We present a multi-camera visual-inertial odometry system based on factor graph optimization which estimates motion by using all cameras simultaneously while retaining a fixed overall feature budget. We focus on motion tracking in challenging environments, such as narrow corridors, dark spaces with aggressive motions, and abrupt lighting changes. These scenarios cause traditional monocular or stereo odometry to fail. While tracking motion with extra cameras should theoretically prevent failures, it leads to additional complexity and computational burden. To overcome these challenges, we introduce two novel methods to improve multi-camera feature tracking. First, instead of tracking features separately in each camera, we track features continuously as they move from one camera to another. This increases accuracy and achieves a more compact factor graph representation. Second, we select a fixed budget of tracked features across the cameras to reduce back-end optimization time. We have found that using a smaller set of informative features can maintain the same tracking accuracy. Our proposed method was extensively tested using a hardware-synchronized device consisting of an IMU and four cameras (a front stereo pair and two lateral) in scenarios including: an underground mine, large open spaces, and building interiors with narrow stairs and corridors. Compared to stereo-only state-of-the-art visual-inertial odometry methods, our approach reduces the drift rate, relative pose error, by up to 80% in translation and 39% in rotation.

DM-VIO: Delayed Marginalization Visual-Inertial Odometry

We present DM-VIO, a monocular visual-inertial odometry system based on two novel techniques called delayed marginalization and pose graph bundle adjustment. DM-VIO performs photometric bundle adjustment with a dynamic weight for visual residuals. We adopt marginalization, which is a popular strategy to keep the update time constrained, but it cannot easily be reversed, and linearization points of connected variables have to be fixed. To overcome this we propose delayed marginalization: The idea is to maintain a second factor graph, where marginalization is delayed. This allows us to later readvance this delayed graph, yielding an updated marginalization prior with new and consistent linearization points. In addition, delayed marginalization enables us to inject IMU information into already marginalized states. This is the foundation of the proposed pose graph bundle adjustment, which we use for IMU initialization. In contrast to prior works on IMU initialization, it is able to capture the full photometric uncertainty, improving the scale estimation. In order to cope with initially unobservable scale, we continue to optimize scale and gravity direction in the main system after IMU initialization is complete. We evaluate our system on the EuRoC, TUM-VI, and 4Seasons datasets, which comprise flying drone, large-scale handheld, and automotive scenarios. Thanks to the proposed IMU initialization, our system exceeds the state of the art in visual-inertial odometry, even outperforming stereo-inertial methods while using only a single camera and IMU. The code will be published at http://vision.in.tum.de/dm-vio

A low-cost visual inertial odometry for mobile vehicle based on double stage Kalman filter

When the low-cost inertial measurement units are used in actual vehicle localization, the measurements suffer from large noise density and unstable bias. Therefore, the integral gyroscope error becomes larger, which results in visual-inertial odometry system degradation or even failure. To this end, this paper proposes a robust visual-inertial odometry method based on a double-stage Kalman filter for a low-cost visual-inertial system, which consists of two Kalman filters. The first filter is a complementary Kalman filter, which uses the accelerometer to correct the gyroscope bias, and then an accurate initial pose estimation is calculated. The second filter is a multi-state observation-constrained Kalman filter, in which the re-projection error of features is calculated based on the multi-state observation constraint strategy to update the system states. Additionally, a Schur complement model is used for the sliding window to marginalize the oldest camera pose of the system states, avoiding the loss of associated information between images and improving the accuracy of the camera pose. Finally, the EuRoC dataset and a homemade low-cost visual-inertial hardware system are used to evaluate the performance of the proposed algorithm. The results show that the accuracy of the low-cost gyroscope bias estimation will decrease when the visual observation is inaccurate in the classic VIO, the proposed algorithm corrects the gyroscope bias with accelerometer measurements, which significantly improves the accuracy and robustness of the vehicle pose estimation when low-cost inertial measurement units are used.

Direct Visual Odometry by Fusing Luminosity and Depth Information

In the traditional direct visual odometry, it is difficult to satisfy the photometric invariant assumption due to the influence of illumination changes in the real environment, which will lead to errors and drift. This paper proposes an improved direct visual odometry system, which combines luminosity and depth information. The algorithm proposed in this paper uses Kinect 2 to collect RGB images with the corresponding depth information, and selects points with large changes of gray gradient to construct a luminosity error function and uses the corresponding depth information to construct a depth error function. The two error functions are merged into one function and converted into the least squares function of the pose of camera, the Levenberg-Marquardt algorithm is used to solve the camera pose. Finally, the Graph optimization theory and the g2o library are used to optimize the initial pose. Experiments show that the algorithm can reduce the error to a certain extent and reduce the drift caused by illumination changes.

SLC-VIO: a stereo visual-inertial odometry based on structural lines and points belonging to lines

AbstractTo improve mobile robot positioning accuracy in building environments and construct structural three-dimensional (3D) maps, this paper proposes a stereo visual-inertial odometry (VIO) system based on structural lines and points belonging to lines. The 2-degree-of-freedom (DoF) spatial structural lines based on the Manhattan world assumption are used to establish visual measurement constraints. The property of point belonging to a line (PPBL) is used to initialize the structural lines and establish spatial distance-residual constraints between point and line landmarks in the reconstructed 3D map. Compared with the 4-DoF spatial straight line, the 2-DoF structural line reduces the variables to be estimated and introduces the orientation information of scenes to the VIO system. The utilization of PPBL makes the proposed system fully exploit the prior geometric information of environments and then achieves better performance. Tests on public data sets and real-world experiments show that the proposed system can achieve higher positioning accuracy and construct 3D maps that better reflect the structure of scenes than existing VIO approaches.

Calib-Net: Calibrating the Low-Cost IMU via Deep Convolutional Neural Network.

The low-cost Inertial Measurement Unit (IMU) can provide orientation information and is widely used in our daily life. However, IMUs with bad calibration will provide inaccurate angular velocity and lead to rapid drift of integral orientation in a short time. In this paper, we present the Calib-Net which can achieve the accurate calibration of low-cost IMU via a simple deep convolutional neural network. Following a carefully designed mathematical calibration model, Calib-Net can output compensation components for gyroscope measurements dynamically. Dilation convolution is adopted in Calib-Net for spatio-temporal feature extraction of IMU measurements. We evaluate our proposed system on public datasets quantitively and qualitatively. The experimental results demonstrate that our Calib-Net achieves better calibration performance than other methods, what is more, and the estimated orientation with our Calib-Net is even comparable with the results from visual inertial odometry (VIO) systems.

Monocular Visual-Inertial Odometry for Agricultural Environments

The accuracy of autonomous robot localization using a monocular visual-inertial odometry system (VIO) is significantly reduced in an agricultural environment compared to an urban and indoor environment due to the unstructured scenes with unstable features, variation of light conditions, and rugged terrain. To address those challenges, we propose a monocular VIO system with modifications to the existing state-of-the-art monocular VIO system VINS-mono. Three modules of VINS-mono have improved in this work: vision processing front end, pose optimization, failure detection and recovery. In the vision processing front-end module, we proposed a keyframe selection algorithm base on vertical movement smoothness verification to prevent the rapid loss of feature tracks. In the pose optimization module, some feature points are removed using a depth-limited approach to improve the efficiency and accuracy of trajectory tracking. In the failure detection and recovery module, we propose a failure recovery method using the old trajectory poses, which can recover the trajectory tracking process interrupted by the failure and alleviate the orientation drift. Experiments on the Rosario dataset have shown that our system outperforms VINS-mono, and the absolute trajectory error in the agricultural environment can be at least reduced by 69% compared with the VINS-mono, which can effectively improve the localization accuracy of agriculture robots in the agricultural environment.

An Integrated Visual Odometry System With Stereo Camera for Unmanned Underwater Vehicles

Navigation is a challenging problem in the area of underwater unmanned vehicles, due to the significant electronmagnetic wave attenuation and the uncertainties in underwater environments. The conventional methods, mainly implemented by acoustic devices, suffer limitations such as high cost, terrain effects and low refresh rate. In this paper, a novel low-cost underwater visual navigation method, named Integrated Visual Odometry with a Stereo Camera (IVO-S), has been investigated. Unlike pure visual odometry, the proposed method fuses the information from inertial sensors and a sonar so that it is able to work in context-sparse environments. In practical experiments, the vehicle was operated to follow specific closed-loop shapes. The Integrated Visual Odoemtry with Monocular Camera (IVO-M) method and other popular open source Visual SLAMs (Simultaneous Localisation and Mappings), such as ORB-SLAM2 and VINS-Mono, have been used to provide comparative results. The cumulative error ratio is used as the quantitative evaluation method to analyse the practical test results. It is shown that the IVO-S method is able to work in underwater sparse-feature environments with high accuracy, whilst also being a low cost solution.

Research on a real-time odometry system integrating vision, LiDAR and IMU for autonomous driving

Research on a real-time odometry system integrating vision, LiDAR and IMU for autonomous driving

Robustness Improvement of Using Pre-Trained Network in Visual Odometry for On-Road Driving

Robustness in on-road driving Visual Odometry (VO) systems is critical, as it determines the reliable performance in various scenarios and environments. Especially with the development of data-driven technology, the combination of data-driven VO and model-based VO has achieved accurate tracking performance. However, the lack of generalization of pre-trained deep neural networks (DNN) limits the robustness of such a combination in unseen environments. In this study, we introduce a novel framework with appropriate usage of DNN prediction and improve the robustness in the self-driving application. Based on the characteristic of on-road self-driving motion and the DNN output, we propose a two-step optimization strategy with a variable degree of freedom (DoF), i.e., the use of two types of DoF representations during pose estimation. Specifically, our two-step optimization operates according to the residual of the optimization with the motion label classification from the pre-trained DNN, as well as our proposed Motion Evaluation by essential matrix construction. Experimental results show that our framework obtains better tracking accuracy than the existing methods.

ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual–Inertial, and Multimap SLAM

This paper presents ORB-SLAM3, the first system able to perform visual, visual-inertial and multi-map SLAM with monocular, stereo and RGB-D cameras, using pin-hole and fisheye lens models. The first main novelty is a feature-based tightly-integrated visual-inertial SLAM system that fully relies on Maximum-a-Posteriori (MAP) estimation, even during the IMU initialization phase. The result is a system that operates robustly in real-time, in small and large, indoor and outdoor environments, and is 2 to 5 times more accurate than previous approaches. The second main novelty is a multiple map system that relies on a new place recognition method with improved recall. Thanks to it, ORB-SLAM3 is able to survive to long periods of poor visual information: when it gets lost, it starts a new map that will be seamlessly merged with previous maps when revisiting mapped areas. Compared with visual odometry systems that only use information from the last few seconds, ORB-SLAM3 is the first system able to reuse in all the algorithm stages all previous information. This allows to include in bundle adjustment co-visible keyframes, that provide high parallax observations boosting accuracy, even if they are widely separated in time or if they come from a previous mapping session. Our experiments show that, in all sensor configurations, ORB-SLAM3 is as robust as the best systems available in the literature, and significantly more accurate. Notably, our stereo-inertial SLAM achieves an average accuracy of 3.6 cm on the EuRoC drone and 9 mm under quick hand-held motions in the room of TUM-VI dataset, a setting representative of AR/VR scenarios. For the benefit of the community we make public the source code.

Visual odometry in local underwater navigation problems

The work processes of the ORB-SLAM algorithm are presented. The results of experimental studies on temporal comparisons of the operation of the algorithm with different parameters and cameras are presented. The necessity of forming a visual odometry (VO) system as a local navigation of remote-controlled and autonomous underwater robots has been substantiated. The two most suitable odometry methods in the underwater environment are described, such as their advantages and disadvantages. The work processes of the ORB-SLAM algorithm are presented. The results of experimental studies on temporal comparisons of the operation of the algorithm with different parameters and cameras are presented. The procedure for preparing video data is described: processing a video stream, adjusting camera parameters for calibration. The experiments represent the testing of the ORB-SLAM3 algorithm on a sample of video filmed as part of the ecological monitoring of the Caspian shelf in 2020.

A Point-Line VIO System With Novel Feature Hybrids and With Novel Line Predicting-Matching

Weak texture and motion blur are always challenging problems for visual-inertial odometry (VIO) systems. To improve accuracy of VIO systems in the challenging scenes, we propose a point-line-based VIO system with novel feature hybrids and with novel predicting-matching for long line track. Point-line features with shorter tracks are categorized into “MSCKF” features and with longer tracks into “SLAM” features. Especially, “SLAM” lines are added into the state vector to improve accuracy of the proposed system. Besides, to ensure the reliability and stability of detection and tracking of line features, we also propose a new “Predicting-Matching” line segment tracking method to increase the track lengths of line segments. Experimental results show that the proposed method outperforms the state-of-the-art methods of VINS-Mono [1], PL-VINS [2] and OpenVINS [3]) on both a public dataset and a collected dataset in terms of accuracy. The collected dataset is full of extremely weak textures and motion blurs. On this dataset, the proposed method also obtains better accuracy than ORB-SLAM3 [4].

An end-to-End deep context gate convolutional visual odometry system based on lightweight attention mechanism

PurposeConventional learning-based visual odometry (VO) systems usually use convolutional neural networks (CNN) to extract features, where some important context-related and attention-holding global features might be ignored. Without essential global features, VO system will be sensitive to various environmental perturbations. The purpose of this paper is to design a novel learning-based framework that aims to improve accuracy of learning-based VO without decreasing the generalization ability.Design/methodology/approachInstead of CNN, a context-gated convolution is adopted to build an end-to-end learning framework, which enables convolutional layers that dynamically capture representative local patterns and composes local features of interest under the guidance of global context. In addition, an attention mechanism module is introduced to further improve learning ability and enhance robustness and generalization ability of the VO system.FindingsThe proposed system is evaluated on the public data set KITTI and the self-collected data sets of our college building, where it shows competitive performance compared with some classical and state-of-the-art learning-based methods. Quantitative experimental results on the public data set KITTI show that compared with CNN-based VO methods, the average translational error and rotational error of all the test sequences are reduced by 45.63% and 37.22%, respectively.Originality/valueThe main contribution of this paper is that an end-to-end deep context gate convolutional VO system based on lightweight attention mechanism is proposed, which effectively improves the accuracy compared with other learning-based methods.

Visual Odometry for Self-Driving with Multihypothesis and Network Prediction

Robustness in visual odometry (VO) systems is critical, as it determines reliable performance in various scenarios and challenging environments. Especially with the development of data-driven technology, such as deep learning, the combination of data-driven VO and traditional model-based VO has achieved accurate tracking performance. However, the existence of local optimums in the model-based cost function still limits the robustness. In this study, we introduce a novel framework with a particle filter (PF) in the optimization process, where the PF is constructed by deep neural network (DNN) prediction. We propose constructing the PF by motion prediction classification and its uncertainty based on the characteristic of on-road driving motion. At the same time, an interval DNN prediction strategy is introduced to improve the real-time performance. Experimental results show that our framework obtains better tracking accuracy and robustness than the existing works, while the time consumption is maintained.

Design and Implementation of an Autonomous Vehicle to Collect Tennis Balls Using Artificial Vision

The objective of this work was to design and implement an autonomous vehicle (robot) to collect tennis balls using different digital image processing techniques. The robot was built from an Arduino Nano microcontroller. A radio frequency antenna NRF24L01 receives the data from the control stage and the locomotion system integrated by motors and an odometry system composed of MPU6050 gyroscope encoders; additionally, the system has an emitter module that consists of an Arduino Uno and an antenna with the same characteristics. The prototype consists of two separate subsystems, one for collecting and processing information and the other specific for the vehicle on the ground. It is equipped with a Kinect camera that captures information from a defined area for image processing through a visual control algorithm that detects the balls by color and shape segmentation, determining their location in rectangular coordinates and sending them to the robot through a data transmission system. The Ackerman configuration mobile robot equipped with the wireless communication system receives the coordinates to carry out the movements that are controlled by sensors located on the wheels, with a maximum capacity of 4 balls. The complete running of the system obtained an accuracy of 96.9% in the collection of balls; it should be noted that the tests were carried out with several distractors whose objective was to confuse the system; these tests were carried out at various times the day in a real scenario.