Visual-inertial Odometry Research Articles

Stereo visual-inertial odometry using structural lines for localizing indoor wheeled robots

This paper proposes optimization-based stereo visual-inertial odometry (VIO) to locate indoor wheeled robots. The multiple Manhattan worlds (MWs) assumption is adopted to model the interior environment. Instead of treating these worlds as isolated ones, we fuse the latest MW with the previous ones if they are in the same direction, reducing the calculated errors on the orientation of the latest MW. Then, the structural lines that encode the orientation information of these worlds are taken as additional landmarks to improve the positioning accuracy and reduce the accumulated drift of the system, especially when the system is in a challenging environment (i.e. scenes with continuous turning and low textures). In addition, the structural lines are parameterized by only two variables, which improves the computational efficiency and simplifies the initialization of lines. Experiments on public benchmark datasets and in real-world environments demonstrate that the proposed VIO system can accurately position the wheeled robot in a complex indoor environment.

CDSFusion: Dense Semantic SLAM for Indoor Environment Using CPU Computing

Unmanned Aerial Vehicles (UAVs) require the ability to robustly perceive surrounding scenes for autonomous navigation. The semantic reconstruction of the scene is a truly functional understanding of the environment. However, high-performance computing is generally not available on most UAVs, so a lightweight real-time semantic reconstruction method is necessary. Existing methods rely on GPU, and it is difficult to achieve real-time semantic reconstruction on CPU. To solve the problem, an indoor dense semantic Simultaneous Localization and Mapping (SLAM) method using CPU computing is proposed in this paper, named CDSFusion. The CDSFusion is the first system integrating RGBD-based Visual-Inertial Odometry (VIO), semantic segmentation and 3D reconstruction in real-time on a CPU. In our VIO method, the depth information is introduced to improve the accuracy of pose estimation, and FAST features are used for faster tracking. In our semantic reconstruction method, the PSPNet (Pyramid Scene Parsing Network) pre-trained model is optimized to provide the semantic information in real-time on the CPU, and the semantic point clouds are fused using Voxblox. The experimental results demonstrate that camera tracking is accelerated without loss of accuracy in our VIO, and a 3D semantic map is reconstructed in real-time, which is comparable to one generated by the GPU-dependent method.

Region of Interest Constrained Negative Obstacle Detection and Tracking With a Stereo Camera

Road object detection has been increasingly considered an essential task in automotive applications. Road objects are divided into two groups: i) above-ground objects such as vehicles and pedestrians and ii) road obstacles such as potholes and cracks. While above-ground object detection has been actively studied and achieved great advance, road obstacle detection has not been extensively studied. This paper proposes the road obstacle detection method based on 3D point cloud candidates extracted by height analysis. The 2D candidates of road obstacles are extracted from 3D candidates. A variant of RANSAC is applied to reduce point cloud matching errors in 3D reconstruction. The proposed system is implemented on the Jetson Xavier NX platform with a 20Hz camera, providing an accurate detection rate with feature tracking and visual inertial odometry in real-time. The proposed method is evaluated on a database consisting of concrete and tar-roads scenes. The experimental results verify the superiority of the proposed method in detecting road obstacles over the state-of-the-art methods.

Real Estate Advisory Drone (READ): system for autonomous indoor space appraisals, based on Deep Learning and Visual Inertial Odometry

The present paper describes the development of a mobile platform as a support of the real estate appraisal procedure. Currently, the estate evaluation is performed by an expert that manually collects data, performs measurements, and grabs pictures of the inspected unit to finally evaluate its commercial value. The READ project aims at automatizing this process by developing a solution based on a mobile unit (drone or tablet) able to navigate the indoor environment and record data, which will be later processed on the cloud. To accomplish all these tasks, the platform is equipped with cameras, a LiDAR sensor, and a data process unit, with the goal of 1) understanding its motion and localization; 2) reconstructing a 3D map of the inspected space; 3) performing image-based analyses applying AI algorithms enabling the identification of the indoor space (e.g. bedroom or kitchen), the counting and the classification of furniture objects, and the detection of building imperfections or frauds. Tests have been performed in different scenarios providing promising results, laying the foundations for bringing these technologies into a real operational context.

Positioning in Urban Scenarios Based on Smartphone Camera/IMU and Digital TV Signal

The built-in sensors of smartphone, such as cameras, accelerometers and gyroscopes, provide abundant environmental information for positioning. VIO (visual-inertial odometry) can be derived from the embedded camera and inertial sensors to provide the estimation of a continuous and smooth trajectory of the pedestrian. However, the trajectory will drift due to the accumulation of system errors. Currently, DTV (digital TV) signal has been verified to be promising for precise ranging in urban scenarios. Therefore, this paper investigates a method of fusing the VIO with the ranging information from DTV signal to improve the positioning accuracy. An Extended Kalman filter is presented to estimate pedestrian mobile state. Field tests have been carried out and the fusion results can effectively reduce trajectory drift errors and improves positioning performance.

Human localization in robotized warehouses based on stereo odometry and ground-marker fusion

Modern logistic solutions for large warehouses consist of a fleet of robots that transfer goods, move racks, and perform other physically difficult and repetitive tasks. The shopfloor is usually enclosed with a safety fence and if a human needs to enter the warehouse all the robots are stopped, as opposed to only the ones in the most immediate vicinity of the human, thus significantly limiting the warehouse efficiency. To tackle this challenge, an integrated safety system is needed with human localization as one of its essential components. In this paper, we propose a novel human localization method for robotized warehouses that is based on a suite of wearable visual sensors installed on a vest worn by humans. The proposed method does not require any modifications of the warehouse environment and relies on the already existing infrastructure. Specifically, we estimate the human location by fusing stereo visual-inertial odometry data and distances to the known absolute poses of the detected ground-markers which robots use for their localization. Fusion is performed by building a pose graph, where we treat estimated human poses relative to markers as graph nodes and odometry estimates as graph edges. We conducted extensive laboratory and warehouse facility experiments, where we tested the reliability and accuracy of the proposed method and compared its performance to a state-of-the-art visual SLAM solution, namely ORB-SLAM2. The results indicate that our method can track absolute position in real-time and has competitive accuracy with respect to ORB-SLAM2, while ensuring higher localization reliability when faced with structural changes in the environment. Furthermore, we provide publicly the experimental datasets to the research community.

Monocular Visual-Inertial-Wheel Odometry Using Low-Grade IMU in Urban Areas

In this article, we propose a new methodology to fuse visual-inertial measurements for land vehicles in a challenging urban environment in which a GNSS signal is not available nor reliable. Motivated by a degenerate case caused by a large bias of a MEMS IMU, we redesign a system model of visual-inertial odometry in a framework of extended Kalman filter. In particular, the system model is propagated through a reduced inertial sensor system composed of a 3-axis gyroscope, a 2-axis accelerometer, and a single-axis odometer. An analytical observability derivation reveals unobservable bases of our estimator, and these directions are resolved by using intermittent position measurements from a GNSS receiver. Furthermore, we inspect the uncertainties of the state vector in a Monte-Carlo simulation that agrees with our theoretical results. The proposed method is validated through the KITTI benchmark dataset and an extensive field testing showing a position drift as 1.25% in tunnels on average and a mean position error of 2.81m in the street canyon over a 6.7km driving.

Hierarchical forest based fast online loop closure for low-latency consistent visual-inertial SLAM

Visual-inertial odometry (VIO) shows high localization accuracy in many environments. However, the inevitable estimation drift is constantly accumulating, which even leads to failures. Hence, an efficient loop closing (LC) is necessary for correcting drift. Unlike most existing methods rely on pre-trained model, we propose an incremental hierarchical forest (HF) based LC method (HF-LC), which online encodes frames for loop closing without any prior information, and it is suitable for arbitrary images. We thoroughly exploit the duality of binary descriptors, and propose a hierarchical forest based descriptor search method for loop frame retrieval, which is extremely fast with efficient descriptor clustering, binary search, and grid based loop closure selection modules. Moreover, we integrate HF-LC into a state-of-the-art keyframe based VIO, and develop a comprehensive hierarchical forest based visual-inertial SLAM (HFVIS) system. Furthermore, a lightweight geometric verification is designed for correctness checking and pose computing of loop keyframes. Finally, all keyframe poses are further optimized in global pose-graph optimization, which achieves better consistency. The efficacy of our method is validated on ground robot and aerial vehicle datasets. Extensive results confirm that the proposed HF-LC achieves better precision–recall performance, and our HFVIS system has higher localization accuracy and superior real-time performance.

LiDAR-Visual-Inertial Odometry Based on Optimized Visual Point-Line Features

This study presents a LiDAR-Visual-Inertial Odometry (LVIO) based on optimized visual point-line features, which can effectively compensate for the limitations of a single sensor in real-time localization and mapping. Firstly, an improved line feature extraction in scale space and constraint matching strategy, using the least square method, is proposed to provide a richer visual feature for the front-end of LVIO. Secondly, multi-frame LiDAR point clouds were projected into the visual frame for feature depth correlation. Thirdly, the initial estimation results of Visual-Inertial Odometry (VIO) were carried out to optimize the scanning matching accuracy of LiDAR. Finally, a factor graph based on Bayesian network is proposed to build the LVIO fusion system, in which GNSS factor and loop factor are introduced to constrain LVIO globally. The evaluations on indoor and outdoor datasets show that the proposed algorithm is superior to other state-of-the-art algorithms in real-time efficiency, positioning accuracy, and mapping effect. Specifically, the average RMSE of absolute trajectory in the indoor environment is 0.075 m and that in the outdoor environment is 3.77 m. These experimental results can prove that the proposed algorithm can effectively solve the problem of line feature mismatching and the accumulated error of local sensors in mobile carrier positioning.

SLC-VIO: a stereo visual-inertial odometry based on structural lines and points belonging to lines

AbstractTo improve mobile robot positioning accuracy in building environments and construct structural three-dimensional (3D) maps, this paper proposes a stereo visual-inertial odometry (VIO) system based on structural lines and points belonging to lines. The 2-degree-of-freedom (DoF) spatial structural lines based on the Manhattan world assumption are used to establish visual measurement constraints. The property of point belonging to a line (PPBL) is used to initialize the structural lines and establish spatial distance-residual constraints between point and line landmarks in the reconstructed 3D map. Compared with the 4-DoF spatial straight line, the 2-DoF structural line reduces the variables to be estimated and introduces the orientation information of scenes to the VIO system. The utilization of PPBL makes the proposed system fully exploit the prior geometric information of environments and then achieves better performance. Tests on public data sets and real-world experiments show that the proposed system can achieve higher positioning accuracy and construct 3D maps that better reflect the structure of scenes than existing VIO approaches.

A New Visual Inertial Simultaneous Localization and Mapping (SLAM) Algorithm Based on Point and Line Features

In view of traditional point-line feature visual inertial simultaneous localization and mapping (SLAM) system, which has weak performance in accuracy so that it cannot be processed in real time under the condition of weak indoor texture and light and shade change, this paper proposes an inertial SLAM method based on point-line vision for indoor weak texture and illumination. Firstly, based on Bilateral Filtering, we apply the Speeded Up Robust Features (SURF) point feature extraction and Fast Nearest neighbor (FLANN) algorithms to improve the robustness of point feature extraction result. Secondly, we establish a minimum density threshold and length suppression parameter selection strategy of line feature, and take the geometric constraint line feature matching into consideration to improve the efficiency of processing line feature. And the parameters and biases of visual inertia are initialized based on maximum posterior estimation method. Finally, the simulation experiments are compared with the traditional tightly-coupled monocular visual–inertial odometry using point and line features (PL-VIO) algorithm. The simulation results demonstrate that the proposed an inertial SLAM method based on point-line vision for indoor weak texture and illumination can be effectively operated in real time, and its positioning accuracy is 22% higher on average and 40% higher in the scenario that illumination changes and blurred image.

Calib-Net: Calibrating the Low-Cost IMU via Deep Convolutional Neural Network.

The low-cost Inertial Measurement Unit (IMU) can provide orientation information and is widely used in our daily life. However, IMUs with bad calibration will provide inaccurate angular velocity and lead to rapid drift of integral orientation in a short time. In this paper, we present the Calib-Net which can achieve the accurate calibration of low-cost IMU via a simple deep convolutional neural network. Following a carefully designed mathematical calibration model, Calib-Net can output compensation components for gyroscope measurements dynamically. Dilation convolution is adopted in Calib-Net for spatio-temporal feature extraction of IMU measurements. We evaluate our proposed system on public datasets quantitively and qualitatively. The experimental results demonstrate that our Calib-Net achieves better calibration performance than other methods, what is more, and the estimated orientation with our Calib-Net is even comparable with the results from visual inertial odometry (VIO) systems.

Perception-Aware Receding Horizon Trajectory Planning for Multicopters With Visual-Inertial Odometry

Visual inertial odometry (VIO) is widely used for the state estimation of multicopters, but it may function poorly in environments with few visual features or in overly aggressive flights. In this work, we propose a perception-aware collision avoidance trajectory planner for multicopters, that may be used with any feature-based VIO algorithm. Our approach is able to fly the vehicle to a goal position at high speed, avoiding obstacles in an unknown stationary environment while achieving good VIO state estimation accuracy. The proposed planner samples a group of minimum jerk trajectories and finds collision-free trajectories among them, which are then evaluated based on their speed to the goal and perception quality. Both the motion blur of features and their locations are considered for the perception quality. Our novel consideration of the motion blur of features enables automatic adaptation of the trajectory’s aggressiveness under environments with different light levels. The best trajectory from the evaluation is tracked by the vehicle and is updated in a receding horizon manner when new images are received from the camera. Only generic assumptions about the VIO are made, so that the planner may be used with various existing systems. The proposed method can run in real time on a small embedded computer on board. We validated the effectiveness of our proposed approach through experiments in both indoor and outdoor environments. Compared to a perception-agnostic planner, the proposed planner kept more features in the camera’s view and made the flight less aggressive, making the VIO more accurate. It also reduced VIO failures, which occurred for the perception-agnostic planner but not for the proposed planner. The ability of the proposed planner to fly through dense obstacles was also validated. The experiment video can be found at https://youtu.be/qO3LZIrpwtQ.

Visual-Inertial Odometry and Global Navigation Satellite System Location Algorithm Based on Point-Line Feature in Outdoor Scenes

Visual-Inertial Odometry and Global Navigation Satellite System Location Algorithm Based on Point-Line Feature in Outdoor Scenes

UniVIO: Unified Direct and Feature-Based Underwater Stereo Visual-Inertial Odometry

Providing an accurate trajectory in an underwater scene is challenging for visual odometry due to weak texture and varying illumination. In this article, we propose a novel underwater visual-inertial odometry (VIO) with the unifying direct method and feature-based method. Our approach starts with a new rectification method for underwater images. The new underwater image rectification method separately eliminates water-air refraction distortion and lens distortion with an approximate single viewpoint (SVP) camera model. Meanwhile, a unified optimization method is used in our approach to obtain robust and highly accurate odometry results. This method jointly optimizes projection errors and photometric errors. Moreover, the robust data association processing, which combines <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">keyframe</i> / <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">keypoint</i> selection, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">keypoint</i> position prediction, and robust local optical flow, was designed in our approach. The robust data association processing greatly improves the success rate of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">keypoints</i> tracking. To evaluate the precision of underwater visual odometry, an underwater dataset with complete and highly accurate ground truth trajectories is provided in this work. The performance of our system is compared with other state-of-the-art algorithms on multiple datasets, including visual odometry datasets and our proposed underwater dataset. The evaluation results show that the proposed system can achieve better results than the state-of-the-art algorithms. Furthermore, the proposed system is validated on public natural underwater datasets to show the performance of the system in a real underwater environment. The qualitative results demonstrate that the proposed system can work well in the natural underwater environment.

CVIDS: A Collaborative Localization and Dense Mapping Framework for Multi-Agent Based Visual-Inertial SLAM

Nowadays, visual SLAM (Simultaneous Localization And Mapping) has become a hot research topic due to its low costs and wide application scopes. Traditional visual SLAM frameworks are usually designed for single-agent systems, completing both the localization and the mapping with sensors equipped on a single robot or a mobile device. However, the mobility and work capacity of the single agent are usually limited. In reality, robots or mobile devices sometimes may be deployed in the form of clusters, such as drone formations, wearable motion capture systems, and so on. As far as we know, existing SLAM systems designed for multi-agents are still sporadic, and most of them have non-negligible limitations in functions. Specifically, on one hand, most of the existing multi-agent SLAM systems can only extract some key features and build sparse maps. On the other hand, schemes that can reconstruct the environment densely cannot get rid of the dependence on depth sensors, such as RGBD cameras or LiDARs. Systems that can yield high-density maps just with monocular camera suites are temporarily lacking. As an attempt to fill in the research gap to some extent, we design a novel collaborative SLAM system, namely CVIDS (Collaborative Visual-Inertial Dense SLAM), which follows a centralized and loosely coupled framework and can be integrated with any existing Visual-Inertial Odometry (VIO) to accomplish the co-localization and the dense reconstruction. Integrating our proposed robust loop closure detection module and two-stage pose-graph optimization pipeline, the co-localization module of CVIDS can estimate the poses of different agents in a unified coordinate system efficiently from the packed images and local poses sent by the client-ends of different agents. Besides, our motion-based dense mapping module can effectively recover the 3D structures of selected keyframes and then fuse their depth information to the global map for reconstruction. The superior performance of CVIDS is corroborated by both quantitative and qualitative experimental results. To make our results reproducible, the source code has been released at https://cslinzhang.github.io/CVIDS.

Gnss-Denied Uav Indoor Navigation With Uwb Incorporated Visual Inertial Odometry

Gnss-Denied Uav Indoor Navigation With Uwb Incorporated Visual Inertial Odometry

Monocular Visual-Inertial Odometry for Agricultural Environments

The accuracy of autonomous robot localization using a monocular visual-inertial odometry system (VIO) is significantly reduced in an agricultural environment compared to an urban and indoor environment due to the unstructured scenes with unstable features, variation of light conditions, and rugged terrain. To address those challenges, we propose a monocular VIO system with modifications to the existing state-of-the-art monocular VIO system VINS-mono. Three modules of VINS-mono have improved in this work: vision processing front end, pose optimization, failure detection and recovery. In the vision processing front-end module, we proposed a keyframe selection algorithm base on vertical movement smoothness verification to prevent the rapid loss of feature tracks. In the pose optimization module, some feature points are removed using a depth-limited approach to improve the efficiency and accuracy of trajectory tracking. In the failure detection and recovery module, we propose a failure recovery method using the old trajectory poses, which can recover the trajectory tracking process interrupted by the failure and alleviate the orientation drift. Experiments on the Rosario dataset have shown that our system outperforms VINS-mono, and the absolute trajectory error in the agricultural environment can be at least reduced by 69% compared with the VINS-mono, which can effectively improve the localization accuracy of agriculture robots in the agricultural environment.

A Real-Time CPU-GPU Embedded Implementation of a Tightly-Coupled Visual-Inertial Navigation System

In autonomous navigation technologies, the Multi-State Constraint Kalman Filter (MSCKF) is one of the most accurate and robust tightly-coupled fusion frameworks for Visual-Inertial Navigation (VIN). However, the adoption of the MSCKF VIN system in real-time embedded applications depends heavily on an efficient implementation of its tangled pipeline. This work initially proposes a novel parallel multi-thread implementation of the MSCKF VIN pipeline on an embedded CPU-enabled hardware that has speeded up the per-epoch processing time of the pipeline by 41% compared to the conventional sequential implementation. The heart of the MSCKF pipeline&#x2019;s visual backend is an inertially-aided 3D localization of visual feature points that are reduced to a set of nonlinear optimization problems which were conventionally solved in a serial fashion using the single-objective Gauss-Newton optimization algorithm. This work leveraged the parallel architecture of an embedded GPU and further proposes an efficient parallel implementation of a multi-objective Gauss-Newton algorithm. Integration of the proposed GPU-accelerated feature localization technique in the MSCKF parallel pipeline has resulted in 33% faster per-epoch processing time and consequently, the satisfaction of strict real-time constraints. The proposed parallel MSCKF VIN pipelines have been developed using C++ and CUDA on the NVIDIA Jetson TX2 embedded board. Experimental evaluations on a real visual-inertial odometry dataset have been provided to validate the efficacy and real-time performance enhancement of the proposed parallel implementation.

Comparison of feature detection and outlier removal strategies in a mono visual odometry algorithm for underwater navigation

This work aims to develop and evaluate a navigation strategy based on optical payloads for Autonomous Underwater Vehicles (AUVs). The use of cameras for navigation purposes can make possible a correct vehicle localization in particular working conditions, where other sensors, such as Doppler Velocity Log (DVL), could not be used. In particular, feature detection and outliers removal algorithms have been chosen as possible critical step in the whole algorithm and have been carefully investigated. Underwater environment introduces challenging conditions for a feature based navigation system and, consequently, the images need to be firstly processed. The developed visual-inertial odometry (VIO) algorithm has been employed for the vehicle translation estimation and this information has been fused with the altimeter, Inertial Measurement Unit (IMU) and Fiber Optic Gyroscope (FOG) measurements. The developed algorithm was tested with an image set acquired by Zeno AUV in the Haifa Bay, Israel (September 2018) and with an image set acquired by FeelHippo AUV in Vulcano, Italy (June 2019) and the results were compared with the path estimated exploiting the other on-board sensors (e.g., the DVL, which has been considered as reference sensor for the benchmark path computation). The algorithm performances are evaluated in both cases, focusing either on the estimate quality and on the requested computational load.