Visual-inertial Simultaneous Localization Research Articles

High-Precision Extrinsic Calibration Method of a Time-of-Flight IMU RGB-Camera With Loop Closure Constraints

Accurate localization is key to realizing an autonomous mobile robot. The fusion of depth data measured by a time-of-flight (TOF) camera with visual-inertial simultaneous localization and mapping (SLAM) can greatly improve depth estimation and mapping. To achieve a robust sensor fusion system, precise calibration of one sensor to another is essential. In this paper, we propose a joint extrinsic calibration method for a TOF- IMU RGB-camera system. Further, we design a pattern that consists of white circles with high reflectivity, and a black background with high absorptivity. Using our calibration method, circular features can be accurately extracted from amplitude images as well as from RGB images. We correct the TOF depth measurement errors, which depend on the pixel location and distance of the pixel away from the object, by establishing an error distribution model with a B-spline function. All sensors are fixed on a platform, and the position of each sensor relative to the other sensors is unchanged, forming loop closure constraints. Based on this prerequisite, we utilize the loop closure constraints between multiple sensors to obtain more accurate extrinsic parameters. By taking the loop closure constraints into account, the stability and robustness of the calibration is increased. The experimental results validate the proposed method and prove its accuracy. Thus, we achieve high precision in multi-sensor extrinsic calibration, which is the necessary foundation of SLAM research.

PLD-VINS: RGBD visual-inertial SLAM with point and line features

It is a big challenge for autonomous robots to estimate state efficiently and generate high-precision 3D maps under low texture indoor scenes. This paper proposes a visual-inertial simultaneous localization and mapping (SLAM) system with point features, line features and depth information provided by RGBD camera, which is named as PLD-VINS. The main advantage of PLD-VINS is that it can improve the accuracy of state estimation and dense 3D mapping with an RGBD camera. Firstly, line features are added to local state estimation to improve the accuracy of relative state estimation between keyframes, which differs from most of existing solutions that only rely on point features. Secondly, an improved EDLines algorithm is introduced to improve the quality and efficiency of line segment detection, and compensate disadvantages of traditional line feature detection algorithms, which often detect short or similar line segments, and are also over-segmentation in complicated scenes. Thirdly, optical flow method is employed to track the detected line segment, which can effectively reduce the cost of calculation and improve the efficiency of the proposed system. Finally, 3D point cloud maps are built which can be input to different kinds of post-processes. The performance of PLD-VINS is validated on public OpenLORIS-Scene datasets and real-world experiments. Comparing with other state-of-the-art algorithms, such as ORB-SLAM2, PL-VINS, VINS-RGBD, and so on, the proposed PLD-VINS is more exact, robust and reliable.

Panoramic Visual-Inertial SLAM Tightly Coupled with a Wheel Encoder

This paper presents a panoramic visual-inertial simultaneous localization and mapping (SLAM) system that is tightly coupled with a wheel encoder, which can be used in a mobile mapping system (MMS), robot, or driverless car. The whole SLAM system is made up of four modules: 1) measurement preprocessing; 2) system initialization; 3) tracking; and 4) local mapping. The core of the system is the combined adjustment we propose for the observations of the panoramic camera, inertial measurement unit (IMU), and wheel encoder, to optimize the system state, which includes the rotation, translation, velocity, IMU bias, and local map point coordinates in the initialization, tracking, and local mapping modules. In the preprocessing, the most important contribution is that we derive the wheel preintegration based on a two-wheel differential model, which combines the many wheel measurements between two frames into a single relative motion constraint. A novel initialization algorithm is also introduced, which can be divided into two steps: 1) the first step is to initialize the visual scale and parameters of the IMU; and 2) the second step is to perform the combined adjustment for the camera, IMU, and preintegrated wheel encoder data, to refine the initial parameters. The proposed panoramic camera-IMU-wheel SLAM (PIW-SLAM) system can achieve high-precision and robust localization in challenging scenes through multi-sensor fusion. We validated the performance of the proposed system on seven different challenging data sequences from the University of Michigan North Campus Long-Term Vision and LiDAR Dataset (NCLT) dataset. The results showed that the PIW-SLAM system has a higher localization accuracy than the other state-of-the-art SLAM systems, and it also showed a superior robustness in various complex environments. We also verified the accuracy of the scale obtained by the initialization algorithm. Furthermore, we confirmed the performance of the proposed PIW-SLAM system in a wide-baseline scene of the ground motion measurement system.

PLF-VINS: Real-Time Monocular Visual-Inertial SLAM With Point-Line Fusion and Parallel-Line Fusion

This letter presents a real-time monocular visual-inertial simultaneous localization and mapping with point-line fusion and parallel-line fusion. The corner and line features provide plenty of information about object structures. In the 2D image plane, such corner and line features have a positional similarity. The corner feature represents an endpoint of an object's edge, and the line feature represents a straight edge. We propose two novel methods for fusing corner and line features to improve localization accuracy. The first method is for fusing corner and line features. Using the positional similarity of corner and line features, we search the relationship between two features by utilizing the proposed point-line coupled residual. The second method is for fusing parallel 3D lines. First, we search for line features clustered based on a vanishing point. Then, the outliers in the parallel 3D lines are removed using the proposed consistency check during the multi-view line clustering. Finally, the parallel 3D lines are used in the proposed parallel 3D line residual. Experimental results show that real-time localization accuracy is improved when two proposed residuals are integrated with the sliding-window optimization. The proposed PLF-VINS is compared with other state-of-the-art algorithms using the public EuRoC dataset.

Evaluation of the Intel RealSense T265 for tracking natural human head motion

Accurate and robust tracking of natural human head motion in natural environments is important for a number of applications including virtual and augmented reality, clinical diagnostics, as well as basic scientific research. IMU provide a versatile solution for recording inertial data including linear acceleration and angular velocity, but reconstructing head position is difficult or impossible. This problem can be solved by incorporating visual data using a technique known as visual-inertial simultaneous localization and mapping (VI-SLAM). A recently released commercial solution, the Intel RealSense T265, uses a proprietary VI-SLAM algorithm to estimate linear and angular position and velocity, but the performance of this device for tracking of natural human head motion in natural environments has not yet been comprehensively evaluated against gold-standard methods. In this study, we used a wide range of metrics to evaluate the performance of the T265 with different walking speeds in different environments, both indoor and outdoor, against two gold-standard methods, an optical tracking system and a so-called perambulator. Overall, we find that performance of the T265 relative to these gold-standard methods is most accurate for slow to normal walking speeds in small- to medium-sized environments. The suitability of this device for future scientific studies depends on the application; data presented here can be useful in making that determination.

Position Detection for Electric Vehicle Dwcs Using Vi-Slam Method

The dynamic wireless charging system (DWCS) is developed to solve the problems of large battery volume and mileage anxiety of electric vehicles. However, the accurate position detection for electric vehicle DWCS is facing challenge. The traditional communication, detection and estimation methods are difficult to accurately obtain the position. To tackle this problem, the visual inertial simultaneous localization and mapping (VI-SLAM) method is applied to the electric vehicles DWCS. Firstly, the graph optimization based tight coupling method is used to integrate the monocular visual and IMU measurements. Secondly, the NVIDIA TX2 and MYNT VI-sensor suite are assembled, which the MTi300-IMU is treated as the ground truth system. Finally, the mobile vehicle is controlled to race on the simulated DWCS pathway. The experimental result shows that the method achieves great performance with the accuracy of centimeter level. In particular, the root mean square error (RMSE) in X, Y, Z directions are 0.086,0.092,0.102, respectively.

Fast Monocular Visual‐Inertial Initialization with an Improved Iterative Strategy

The initialization process has a great effect on the performance of the monocular visual inertial simultaneous localization and mapping (VI‐SLAM) system. The initial estimation is usually solved by least squares such as the Gauss‐Newton (G‐N) algorithm, but the large iteration increment might lead to the slow convergence or even divergence. In order to solve this problem, an improved iterative strategy for initial estimation is proposed. The methodology of our initialization can be divided into four steps: Firstly, the pure visual ORB‐SLAM model is utilized to make all variables observable. Secondly, the IMU preintegration technology is adopted for IMU‐camera frequency alignment at the same time with key frame generation. Thirdly, an improved iterative strategy which is based on the trust region is introduced for the gyroscope bias estimation as well as the gravity direction is refined. Finally, the accelerometer bias and visual scale are estimated on the basis of previous estimations. Experimental results on the public datasets show that the estimation of initial values can be converged faster, as well as the velocity and pose of sensor suite can be estimated more accurately than the original method.

Renal Volume Estimation Using Freehand Ultrasound Scans: An Ex Vivo Demonstration

Renal volume has the potential to serve as a robust biomarker for tracking the onset and progression of renal diseases and also for quantifying renal function. We propose a method to estimate renal volumes using freehand ultrasound scans at the point of care. A conventional ultrasound probe was augmented with an Intel RealSense D435 i camera. Visual inertial simultaneous localization and mapping was used to localize the probe in free space. The acquired 2-D ultrasound images, segmented by trained clinicians, were combined with the estimated poses of the probe to yield accurate volumes. The method was tested on two ex vivo sheep kidneys embedded in gelatin phantoms. Four different scanning protocols were tested: transverse linear, transverse fan, longitudinal linear and longitudinal fan. The estimated renal volumes were compared with those obtained using the water displacement method, the ellipsoidal method and computed tomography imaging. The water displacement method yielded mean volumes of 66.00 and 66.20 mL for kidneys 1 and 2, respectively (ground truth). Freehand ultrasound scans produced mean volumes of 64.08 mL (2.90% error) and 65.25 mL (1.40% error); the ellipsoidal method yielded volumes of 57.49 mL (12.90% error) and 60.15 mL (9.13% error); and computed tomography yielded a volume of 63.00 mL (4.54% error).

SD-VIS: A Fast and Accurate Semi-Direct Monocular Visual-Inertial Simultaneous Localization and Mapping (SLAM).

In practical applications, how to achieve a perfect balance between high accuracy and computational efficiency can be the main challenge for simultaneous localization and mapping (SLAM). To solve this challenge, we propose SD-VIS, a novel fast and accurate semi-direct visual-inertial SLAM framework, which can estimate camera motion and structure of surrounding sparse scenes. In the initialization procedure, we align the pre-integrated IMU measurements and visual images and calibrate out the metric scale, initial velocity, gravity vector, and gyroscope bias by using multiple view geometry (MVG) theory based on the feature-based method. At the front-end, keyframes are tracked by feature-based method and used for back-end optimization and loop closure detection, while non-keyframes are utilized for fast-tracking by direct method. This strategy makes the system not only have the better real-time performance of direct method, but also have high accuracy and loop closing detection ability based on feature-based method. At the back-end, we propose a sliding window-based tightly-coupled optimization framework, which can get more accurate state estimation by minimizing the visual and IMU measurement errors. In order to limit the computational complexity, we adopt the marginalization strategy to fix the number of keyframes in the sliding window. Experimental evaluation on EuRoC dataset demonstrates the feasibility and superior real-time performance of SD-VIS. Compared with state-of-the-art SLAM systems, we can achieve a better balance between accuracy and speed.

Real-Time Semantic Plane Reconstruction on a Monocular Drone Using Sparse Fusion

A semantic map, which can provide important data to facilitate a drone's understanding to the environment, is critical for a fully autonomous drone. However, recent methods for producing such a map implemented on small drones that use low-/middle-grade processors and graphics processing units can hardly achieve real-time performance. In addition, few existing methods can reconstruct semantic planes based on a sparse depth map, which can greatly reduce computational load while encountering challenges in semantic reconstruction. To address this problem, this paper presents a novel on-board approach, called sparse fusion, which demonstrates real-time reconstruction of a semantic plane on a self-designed small drone. The approach combines the sparse depth map derived from a visual-inertial simultaneous localization and mapping with the semantic labels derived from a convolutional neural network for each frame using sparse fusion. Our proposed local plane optimization function greatly improves the accuracy of the semantic plane. Experimental results on various scenarios demonstrate that our sparse fusion module running on the drone platform can update the semantic plane within 1 ms, which is faster and achieves greater accuracy than the state-of-the-art real-time semantic reconstruction method. We also conducted experiments in real environments to demonstrate the performance of our method.

A Stereo Visual-Inertial SLAM Approach for Indoor Mobile Robots in Unknown Environments Without Occlusions

When mobile robots are working in indoor unknown environments, the surrounding scenes are mainly low texture or repeating texture. This means that image features are easily lost when tracking the robots, and poses are difficult to estimate as the robot moves back and forth in a narrow area. In order to improve such tracking problems, we propose a one-circle feature-matching method, which refers to a sequence of the circle matching for the time after space (STCM), and an STCM-based visual-inertial simultaneous localization and mapping (STCM-SLAM) technique. This strategy tightly couples the stereo camera and the inertial measurement unit (IMU) in order to better estimate poses of the mobile robot when working indoors. Forward backward optical flow is used to track image features. The absolute accuracy and relative accuracy of STCM increase by 37.869% and 129.167%, respectively, when compared with correlation flow. In addition, we compare our proposed method with other state-of-the-art methods. In terms of relative pose error, the accuracy of STCM-SLAM is an order of magnitude greater than ORB-SLAM2, and two orders of magnitude greater than OKVIS. Our experiments show that STCM-SLAM has obvious advantages over the OKVIS method, specifically in terms of scale error, running frequency, and CPU load. STCM-SLAM also performs the best under real-time conditions. In the indoor experiments, STCM-SLAM is able to accurately estimate the trajectory of the mobile robot. Based on the root mean square error, mean error, and standard deviation, the accuracy of STCM-SLAM is ultimately superior to that of either ORB-SLAM2 or OKVIS.

A Review of Visual-Inertial Simultaneous Localization and Mapping from Filtering-Based and Optimization-Based Perspectives

Visual-inertial simultaneous localization and mapping (VI-SLAM) is popular research topic in robotics. Because of its advantages in terms of robustness, VI-SLAM enjoys wide applications in the field of localization and mapping, including in mobile robotics, self-driving cars, unmanned aerial vehicles, and autonomous underwater vehicles. This study provides a comprehensive survey on VI-SLAM. Following a short introduction, this study is the first to review VI-SLAM techniques from filtering-based and optimization-based perspectives. It summarizes state-of-the-art studies over the last 10 years based on the back-end approach, camera type, and sensor fusion type. Key VI-SLAM technologies are also introduced such as feature extraction and tracking, core theory, and loop closure. The performance of representative VI-SLAM methods and famous VI-SLAM datasets are also surveyed. Finally, this study contributes to the comparison of filtering-based and optimization-based methods through experiments. A comparative study of VI-SLAM methods helps understand the differences in their operating principles. Optimization-based methods achieve excellent localization accuracy and lower memory utilization, while filtering-based methods have advantages in terms of computing resources. Furthermore, this study proposes future development trends and research directions for VI-SLAM. It provides a detailed survey of VI-SLAM techniques and can serve as a brief guide to newcomers in the field of SLAM and experienced researchers looking for possible directions for future work.

Visual-inertial SLAM method based on optical flow in a GPS-denied environment

PurposeThis study aims to present a visual-inertial simultaneous localization and mapping (SLAM) method for accurate positioning and navigation of mobile robots in the event of global positioning system (GPS) signal failure in buildings, trees and other obstacles.Design/methodology/approachIn this framework, a feature extraction method distributes features on the image under texture-less scenes. The assumption of constant luminosity is improved, and the features are tracked by the optical flow to enhance the stability of the system. The camera data and inertial measurement unit data are tightly coupled to estimate the pose by nonlinear optimization.FindingsThe method is successfully performed on the mobile robot and steadily extracts the features on low texture environments and tracks features. The end-to-end error is 1.375 m with respect to the total length of 762 m. The authors achieve better relative pose error, scale and CPU load than ORB-SLAM2 on EuRoC data sets.Originality/valueThe main contribution of this study is the theoretical derivation and experimental application of a new visual-inertial SLAM method that has excellent accuracy and stability on weak texture scenes.

Input Uncertainty Sensitivity Enhanced Nonsingleton Fuzzy Logic Controllers for Long-Term Navigation of Quadrotor UAVs

Input uncertainty, e.g., noise on the on-board camera and inertial measurement unit, in vision-based control of unmanned aerial vehicles (UAVs) is an inevitable problem. In order to handle input uncertainties as well as further analyze the interaction between the input and the antecedent fuzzy sets (FSs) of nonsingleton fuzzy logic controllers (NSFLCs), an input uncertainty sensitivity enhanced NSFLC has been developed in robot operating system using the C++ programming language. Based on recent advances in nonsingleton inference, the centroid of the intersection of the input and antecedent FSs (Cen-NSFLC) is utilized to calculate the firing strength of each rule instead of the maximum of the intersection used in traditional NSFLC (Tra-NSFLC). An 8-shaped trajectory, consisting of straight and curved lines, is used for the real-time validation of the proposed controllers for a trajectory following problem. An accurate monocular keyframe-based visual-inertial simultaneous localization and mapping (SLAM) approach is used to estimate the position of the quadrotor UAV in GPS-denied unknown environments. The performance of the Cen-NSFLC is compared with a conventional proportional-integral derivative (PID) controller, a singleton FLC and a Tra-NSFLC. All controllers are evaluated for different flight speeds, thus introducing different levels of uncertainty into the control problem. Visual-inertial SLAM-based real-time quadrotor UAV flight tests demonstrate that not only does the Cen-NSFLC achieve the best control performance among the four controllers, but it also shows better control performance when compared to their singleton counterparts. Considering the bias in the use of model-based controllers, e.g., PID, for the control of UAVs, this paper advocates an alternative method, namely Cen-NSFLCs, in uncertain working environments.

Accurate Monocular Visual-Inertial SLAM Using a Map-Assisted EKF Approach

This paper presents a novel tightly-coupled monocular visual-inertial Simultaneous Localization and Mapping algorithm, which provides accurate and robust localization within the globally consistent map in real time on a standard CPU. This is achieved by firstly performing the visual-inertial extended kalman filter(EKF) to provide motion estimate at a high rate. However the filter becomes inconsistent due to the well known linearization issues. So we perform a keyframe-based visual-inertial bundle adjustment to improve the consistency and accuracy of the system. In addition, a loop closure detection and correction module is also added to eliminate the accumulated drift when revisiting an area. Finally, the optimized motion estimates and map are fed back to the EKF-based visual-inertial odometry module, thus the inconsistency and estimation error of the EKF estimator are reduced. In this way, the system can continuously provide reliable motion estimates for the long-term operation. The performance of the algorithm is validated on public datasets and real-world experiments, which proves the superiority of the proposed algorithm.

Accurate Initial State Estimation in a Monocular Visual-Inertial SLAM System.

The fusion of monocular visual and inertial cues has become popular in robotics, unmanned vehicles and augmented reality fields. Recent results have shown that optimization-based fusion strategies outperform filtering strategies. Robust state estimation is the core capability for optimization-based visual–inertial Simultaneous Localization and Mapping (SLAM) systems. As a result of the nonlinearity of visual–inertial systems, the performance heavily relies on the accuracy of initial values (visual scale, gravity, velocity and Inertial Measurement Unit (IMU) biases). Therefore, this paper aims to propose a more accurate initial state estimation method. On the basis of the known gravity magnitude, we propose an approach to refine the estimated gravity vector by optimizing the two-dimensional (2D) error state on its tangent space, then estimate the accelerometer bias separately, which is difficult to be distinguished under small rotation. Additionally, we propose an automatic termination criterion to determine when the initialization is successful. Once the initial state estimation converges, the initial estimated values are used to launch the nonlinear tightly coupled visual–inertial SLAM system. We have tested our approaches with the public EuRoC dataset. Experimental results show that the proposed methods can achieve good initial state estimation, the gravity refinement approach is able to efficiently speed up the convergence process of the estimated gravity vector, and the termination criterion performs well.

Monocular Visual-Inertial SLAM:Continuous Preintegration and Reliable Initialization.

In this paper, we propose a new visual-inertial Simultaneous Localization and Mapping (SLAM) algorithm. With the tightly coupled sensor fusion of a global shutter monocular camera and a low-cost Inertial Measurement Unit (IMU), this algorithm is able to achieve robust and real-time estimates of the sensor poses in unknown environment. To address the real-time visual-inertial fusion problem, we present a parallel framework with a novel IMU initialization method. Our algorithm also benefits from the novel IMU factor, the continuous preintegration method, the vision factor of directional error, the separability trick and the robust initialization criterion which can efficiently output reliable estimates in real-time on modern Central Processing Unit (CPU). Tremendous experiments also validate the proposed algorithm and prove it is comparable to the state-of-art method.

Visual Inertial SLAM: Application to Unmanned Aerial Vehicles

Unmanned Aerial Vehicles (UAVs) require an accurate estimate of their state. Computer vision provides a number of benefits over conventionally used sensors, such as the Global Positioning System (GPS) or a Motion Capture System (MCS), in order to achieve state estimation and localization relative to a scene. Our work uses the output of an existing Visual Simultaneous Localization and Mapping (VSLAM) system which provides a scaled position measurement. We propose an observer design to estimate vehicle position and linear velocity. The observer fuses an accelerometer measurement from an Inertial Measurement Unit (IMU) and VSLAM system output. The observer depends on an attitude estimate from an Attitude and Heading Reference System (AHRS). A change of coordinates is used to transform the system into a Linear Time-Varying (LTV) form. Using these coordinates we consider the observability of the Visual Inertial Simultaneous Localization and Mapping (VISLAM) problem. Two observer designs are proposed and their performance is validated in simulation and experiment.

Visual-Inertial Monocular SLAM With Map Reuse

In recent years there have been excellent results in Visual-Inertial Odometry techniques, which aim to compute the incremental motion of the sensor with high accuracy and robustness. However these approaches lack the capability to close loops, and trajectory estimation accumulates drift even if the sensor is continually revisiting the same place. In this work we present a novel tightly-coupled Visual-Inertial Simultaneous Localization and Mapping system that is able to close loops and reuse its map to achieve zero-drift localization in already mapped areas. While our approach can be applied to any camera configuration, we address here the most general problem of a monocular camera, with its well-known scale ambiguity. We also propose a novel IMU initialization method, which computes the scale, the gravity direction, the velocity, and gyroscope and accelerometer biases, in a few seconds with high accuracy. We test our system in the 11 sequences of a recent micro-aerial vehicle public dataset achieving a typical scale factor error of 1% and centimeter precision. We compare to the state-of-the-art in visual-inertial odometry in sequences with revisiting, proving the better accuracy of our method due to map reuse and no drift accumulation.

Asynchronous adaptive conditioning for visual–inertial SLAM

This paper is concerned with real-time monocular visual–inertial simultaneous localization and mapping (SLAM). In particular a tightly coupled nonlinear-optimization-based solution that can match the global optimal result in real time is proposed. The methodology is motivated by the requirement to produce a scale-correct visual map, in an optimization framework that is able to incorporate relocalization and loop closure constraints. Special attention is paid to achieve robustness to many real world difficulties, including degenerate motions and unobservablity. A variety of helpful techniques are used, including: a relative manifold representation, a minimal-state inverse depth parameterization, and robust non-metric initialization and tracking. Importantly, to enable real-time operation and robustness, a novel numerical dog-leg solver is presented that employs multi-threaded, asynchronous, adaptive conditioning. In this approach, the conditioning edges of the SLAM graph are adaptively identified and solved for both synchronously and asynchronously. In this way one thread focuses on a small number of temporally immediate parameters and hence constitute a natural “front-end”; the other thread adaptively focuses on larger portions of the SLAM problem, and hence is able to re-estimate past parameters in the presence of new information: an ability that is useful for self-calibration, during degenerate motions, or when bias and the direction of gravity are poorly observed. Experiments with real and simulated data for both indoor and outdoor scenarios demonstrate that asynchronous adaptive conditioning is accurate, and able to closely track the batch SLAM maximum likelihood solution in real time.