Vision-based Simultaneous Localization Research Articles

Virtual Vectors for Vision-Based Simultaneous Localization and Mapping System

Virtual Vectors for Vision-Based Simultaneous Localization and Mapping System

Analysis of feature point matching technology in SLAM based on binocular vision

With rapidly developing computing technology, intelligent robots have been used extensively in many areas of life, improving work efficiency and lowering labor costs. At the same time, with the increasing number of robot application scenarios, the demand for robot intelligence has also increased. Computer vision-based simultaneous localization and mapping (SLAM) technology is a key technology that helps robots achieve real-time positioning and navigation in order to improve their intelligence. Feature point matching is an important component of this technology. This paper mainly analyzes the current development status of feature point matching technology in visual SLAM. Firstly, a brief introduction was given to the three widely used SLAMs, and the advantages and disadvantages of different SLAM technologies were analyzed. Second, a brief presentation and explanation of the working principle of binocular SLAM technology were provided. Then, the basic principles and key points of feature point recognition and matching techniques in several different algorithms for visual SLAM were summarized. Analyzed the advantages and disadvantages of various algorithms, as well as improvements based on this algorithm. Finally, summarize and provide suggestions for the future development of visual SLAM technology.

Dynam-SLAM: An Accurate, Robust Stereo Visual-Inertial SLAM Method in Dynamic Environments

Most existing vision-based simultaneous localization and mapping (SLAM) systems and their variants still assume that the observation is absolutely static and cannot work well in dynamic environments. Here, we present the Dynam-SLAM (Dynam), a stereo visual-inertial SLAM system capable of robust, accurate, and continuous work in high dynamic environments. Our approach is devoted to loosely coupling the stereo scene flow with an inertial measurement unit (IMU) for dynamic feature detection and tightly coupling the dynamic and static features with the IMU measurements for nonlinear optimization. First, the scene flow uncertainty caused by measurement noise is modeled to derive the accurate motion likelihood of landmarks. Meanwhile, to cope with highly dynamic environments, we additionally construct the virtual landmarks based on the detected dynamic features. Then, we build a tightly coupled, nonlinear optimization-based SLAM system to estimate the camera state by fusing IMU measurements and feature observations. Finally, we evaluate the proposed dynamic feature detection module (DFM) and the overall SLAM system in various benchmark datasets. Experimental results show that the Dynam is almost unaffected by DFM and performs well in static EuRoC datasets. Dynam outperforms the current state-of-the-art visual and visual-inertial SLAM implementations in terms of accuracy and robustness in self-collected dynamic datasets. The average absolute trajectory error of Dynam in the dynamic benchmark datasets is <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\sim$</tex-math></inline-formula> 90% lower than that of VINS-Fusion, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\sim$</tex-math></inline-formula> 84% lower than that of ORB-SLAM3, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\sim$</tex-math></inline-formula> 88% lower than that of Kimera.

LRD-SLAM: A Lightweight Robust Dynamic SLAM Method by Semantic Segmentation Network

With the development of intelligent concepts in various fields, research on driverless and intelligent industrial robots has increased. Vision-based simultaneous localization and mapping (SLAM) is a widely used technique. Most conventional visual SLAM algorithms are assumed to work in ideal static environments; however, such environments rarely exist in real life. Thus, it is important to develop visual SLAM algorithms that can determine their own positions and perceive the environment in real dynamic environments. This paper proposes a lightweight robust dynamic SLAM system based on a novel semantic segmentation network (LRD-SLAM). In the proposed system, a fast deep convolutional neural network (FNet) is implemented into ORB-SLAM2 as a semantic segmentation thread. In addition, a multiview geometry method is introduced, in which the accuracy of detecting dynamic points is further improved through the difference in parallax angle and depth, and the information of the keyframes is used to repair the static background information absent from the removal of dynamic objects, to facilitate the subsequent reconstruction of the point cloud map. Experimental results obtained using the TUM RGB-D dataset demonstrate that the proposed system improves the positioning accuracy and robustness of visual SLAM in indoor pedestrian dynamic environments.

Visual-Inertial RGB-D SLAM With Encoders for a Differential Wheeled Robot

Recent years have seen multiple impressive results in visual-inertial odometry (VIO) techniques, by which accurate state estimation can be achieved via the extended Kalman filter (EKF) or nonlinear optimization. However, these approaches are rarely open source, and they tend to fail in real experiments due to the temporary lack of feature points and fast motion. Therefore, in this study, we used encoders to overcome the temporary failure of purely vision-based simultaneous localization and mapping (SLAM). Here, we propose a generative measurement model for encoders and derive an expression for the maximum a posteriori (MAP) estimate and necessary Jacobians for optimization. We use our theory to present a novel tightly coupled visual-inertial encoder RGB-Depth (RGB-D) SLAM system. Tests on our system on an in-house dataset confirmed that our modeling effort led to accurate (with a root mean squared error (RMSE) of approximately 2–7 cm) and robust state estimation in real time. The source code and our dataset containing the encoder information have been published for verification.

Hierarchical Multi-Level Information Fusion for Robust and Consistent Visual SLAM

Vision-based simultaneous localization and mapping (SLAM) has been indispensable for autonomous vehicles. However, in challenging environments, such as sparsely textured parking lots, the state-of-the-art methods often fail to obtain sufficiently reliable features for robust localization and consistent mapping. This paper proposes a multi-level visual information hierarchical fusion SLAM framework based on a multi-camera system. High-level features like object edges are extracted by different segmentation methods. With the guidance of high-level information, low-level key points are selected. The hierarchical fusion structure consists of two parts. High-level visual information is loosely fused with the wheel encoder to estimate initial data association. Then, low-level feature points of multiple views are tightly fused to facilitate robust localization. With such a structure, a consistent map can be built even under the condition that one view is lost. In addition, the lost view can be reinitialized in the single map with other information instead of waiting for re-localization or creating additional maps. Real-time experiments in autonomous parking scenario validate the robustness and consistency of the proposed method.

LiDARTag: A Real-Time Fiducial Tag System for Point Clouds

Image-based fiducial markers are useful in problems such as object tracking in cluttered or textureless environments, camera (and multi-sensor) calibration tasks, and vision-based simultaneous localization and mapping (SLAM). The state-of-the-art fiducial marker detection algorithms rely on the consistency of the ambient lighting. This paper introduces LiDARTag, a novel fiducial tag design and detection algorithm suitable for light detection and ranging (LiDAR) point clouds. The proposed method runs in real-time and can process data at 100 Hz, which is faster than the currently available LiDAR sensor frequencies. Because of the LiDAR sensors' nature, rapidly changing ambient lighting will not affect the detection of a LiDARTag; hence, the proposed fiducial marker can operate in a completely dark environment. In addition, the LiDARTag nicely complements and is compatible with existing visual fiducial markers, such as AprilTags, allowing for efficient multi-sensor fusion and calibration tasks. We further propose a concept of minimizing a fitting error between a point cloud and the marker's template to estimate the marker's pose. The proposed method achieves millimeter error in translation and a few degrees in rotation. Due to LiDAR returns' sparsity, the point cloud is lifted to a continuous function in a reproducing kernel Hilbert space where the inner product can be used to determine a marker's ID. The experimental results, verified by a motion capture system, confirm that the proposed method can reliably provide a tag's pose and unique ID code. The rejection of false positives is validated on the Google Cartographer indoor dataset and the Honda H3D outdoor dataset. All implementations are coded in C++ and are available at: https://github.com/UMich-BipedLab/LiDARTag.

Fast Reconstruction of 3D Point Cloud Model Using Visual SLAM on Embedded UAV Development Platform

In recent years, the rapid development of unmanned aerial vehicle (UAV) technologies has made data acquisition increasingly convenient, and three-dimensional (3D) reconstruction has emerged as a popular subject of research in this context. These 3D models have many advantages, such as the ability to represent realistic scenes and a large amount of information. However, traditional 3D reconstruction methods are expensive, and require long and complex processing. As a result, they cannot rapidly respond when used in time-sensitive applications, e.g., those for such natural disasters as earthquakes, debris flow, etc. Computer vision-based simultaneous localization and mapping (SLAM) along with hardware development based on embedded systems, can provide a solution to this problem. Based on an analysis of the principle and implementation of the visual SLAM algorithm, this study proposes a fast method to quickly reconstruct a dense 3D point cloud model on a UAV platform combined with an embedded graphics processing unit (GPU). The main contributions are as follows: (1) to resolve the contradiction between the resource limitations and the computational complexity of visual SLAM on UAV platforms, the technologies needed to compute resource allocation, communication between nodes, and data transmission and visualization in an embedded environment were investigated to achieve real-time data acquisition and processing. Visual monitoring to this end is also designed and implemented. (2) To solve the problem of time-consuming algorithmic processing, a corresponding parallel algorithm was designed and implemented based on the parallel programming framework of the compute unified device architecture (CUDA). (3) The visual odometer and methods of 3D “map” reconstruction were designed using under a monocular vision sensor to implement the prototype of the fast 3D reconstruction system. Based on preliminary results of the 3D modeling, the following was noted: (1) the proposed method was feasible. By combining UAV, SLAM, and parallel computing, a simple and efficient 3D reconstruction model of an unknown area was obtained for specific applications. (2) The parallel SLAM algorithm used in this method improved the efficiency of the SLAM algorithm. On the one hand, the SLAM algorithm required 1/6 of the time taken by the structure-from-motion algorithm. On the other hand, the speedup obtained using the parallel SLAM algorithm based on the embedded GPU on our test platform was 7.55 × that of the serial algorithm. (3) The depth map results show that the effective pixel with an error less than 15cm is close to 60%.

Multi-camera visual SLAM for off-road navigation

With the rapid development of computer vision, vision-based simultaneous localization and mapping (vSLAM) plays an increasingly important role in the field of unmanned driving. However, traditional SLAM methods based on a monocular camera only perform well in simple indoor environments or urban environments with obvious structural features. In off-road environments, the situation that SLAM encounters could be complicated by problems such as direct sunlight, leaf occlusion, rough roads, sensor failure, sparsity of stably trackable texture. Traditional methods are highly susceptible to these factors, which lead to compromised stability and reliability. To counter such problems, we propose a panoramic vision SLAM method based on multi-camera collaboration, aiming at utilizing the characters of panoramic vision and stereo perception to improve the localization precision in off-road environments. At the same time, the independence and information sharing of each camera in multi-camera system can improve its ability to resist bumps, illumination, occlusion and sparse texture in an off-road environment, and enable our method to recover the metric scale. These characters ensure unmanned ground vehicles (UGVs) to locate and navigate safely and reliably in complex off-road environments.

Vision-Based Race Track SLAM Based Only on Lane Curvature

This paper presents a novel approach for vision-based Simultaneous Localization and Mapping (SLAM) on a high curved track without pose-based landmarks. The proposed approach combines an Iterative Closest Points (ICP) method and Stochastic Gradient Descent (SGD) optimization and comprises four main steps. First, a Kalman filter with a simple circular lane model is used to estimate the road curvature using images from a front camera. Then, the vehicle position and orientation are reconstructed by using the yaw rate and longitudinal speed from inertial sensors. Drift and misalignment of the constructed map are corrected using ICP under the assumption that the vehicle continuously travels the same track. The final map is obtained using SGD optimization, which enforces curvature matching. We evaluate the performance of the proposed algorithm with an environment of the winding track of Hyundai-Kia California Proving Ground (CPG) located in Southern California and the customized ellipsoidal track. The experimental results show the effectiveness of the proposed approach.

Tightly Coupled Integration of GNSS and Vision SLAM Using 10-DoF Optimization on Manifold

Vision navigation technique, especially the vision-based simultaneous localization and mapping (V-SLAM), plays a critical role in robotic navigation. As a relative positioning technique, V-SLAM often suffers from drift and scale uncertainty problems which incur bias increasing over time. To overcome these drawbacks and to improve the robustness and accuracy of localization, an effective way is to fuse global navigation satellite system (GNSS) with V-SLAM. In this paper, we propose a novel GNSS and SLAM fusion algorithm, which provides ego-motion estimation through tightly coupling GNSS pseudo-range measurements and camera feature points. It first decomposes the pose state into basic motion vectors, based on which an asynchronous tracking is performed. Then, a 10-DoF joint-optimization formulation on manifold is proposed to achieve tight fusion of the raw measurement from camera and GNSS. Finally, this formulation is solved to calculate the ego-motion state. The proposed algorithm is verified on an autonomous ground vehicle in two typical environments. The results demonstrated that, the new algorithm can amend the bias in vision SLAM and constrain the GNSS solution, which achieves a better localization result than the traditional methods.

RGB-D visual odometry with point and line features in dynamic environment

Vision-based simultaneous localization and mapping (SLAM) technology is the key to realize autonomous navigation of mobile robots. When the robot is in an unfamiliar environment, it usually uses the point features of the surrounding environment to estimate its pose. However, if the feature information in the environment is not rich and there are many dynamic objects, the camera trajectory cannot be accurately estimated. To this end, this paper proposed an RGB-D visual odometry that combines point features and line features simultaneously. The dynamic line features are eliminated by calculating the static weight of the line features, and the camera pose is estimated based on the point features and the remaining line features. Compared with other feature-based SLAM systems, the performance and accuracy of systematic pose estimation can be improved in the absence of feature points or dynamic environments.

Co-localized augmented human and X-ray observers in collaborative surgical ecosystem.

Image-guided percutaneous interventions are safer alternatives to conventional orthopedic and trauma surgeries. To advance surgical tools in complex bony structures during these procedures with confidence, a large number of images is acquired. While image-guidance is the de facto standard to guarantee acceptable outcome, when these images are presented on monitors far from the surgical site the information content cannot be associated easily with the 3D patient anatomy. In this article, we propose a collaborative augmented reality (AR) surgical ecosystem to jointly co-localize the C-arm X-ray and surgeon viewer. The technical contributions of this work include (1) joint calibration of a visual tracker on a C-arm scanner and its X-ray source via a hand-eye calibration strategy, and (2) inside-out co-localization of human and X-ray observers in shared tracking and augmentation environments using vision-based simultaneous localization and mapping. We present a thorough evaluation of the hand-eye calibration procedure. Results suggest convergence when using 50 pose pairs or more. The mean translation and rotation errors at convergence are 5.7mm and [Formula: see text], respectively. Further, user-in-the-loop studies were conducted to estimate the end-to-end target augmentation error. The mean distance between landmarks in real and virtual environment was 10.8mm. The proposed AR solution provides a shared augmented experience between the human and X-ray viewer. The collaborative surgical AR system has the potential to simplify hand-eye coordination for surgeons or intuitively inform C-arm technologists for prospective X-ray view-point planning.

Dynamic objects elimination in SLAM based on image fusion

In the process of vision-based simultaneous localization and mapping (SLAM), if there is a moving object entering the view field of the camera, a trajectory of the object will be preserved in the constructed point cloud map. However, the other types of maps converted from the point cloud maps cannot be directly used for navigation. This paper studies the dynamic objects elimination in SLAM based on image fusion algorithm. First, we construct a camera motion model for the moving platform. Then, the camera motion is decomposed into two parts: translation and rotation, two constrains were proposed to locate the dynamic regions. Finally, these dynamic regions in the image sequence are set to blank, while we choose one of the image planes as the projection plane, and map the information of other images to the plane to get a fusion image. The results contain all environment information, and the fusion image can be used to replace the image sequence in SLAM. Experiment results demonstrate that our method can eliminate the influence of the dynamic objects effectively.

Computationally efficient algorithm for vision-based simultaneous localization and mapping of mobile robots

PurposeFastSLAM is a popular method to solve the problem of simultaneous localization and mapping (SLAM). However, when the number of landmarks present in real environments increases, there are excessive comparisons of the measurement with all the existing landmarks in each particle. As a result, the execution speed will be too slow to achieve the objective of real-time navigation. Thus, this paper aims to improve the computational efficiency and estimation accuracy of conventional SLAM algorithms.Design/methodology/approachAs an attempt to solve this problem, this paper presents a computationally efficient SLAM (CESLAM) algorithm, where odometer information is considered for updating the robot’s pose in particles. When a measurement has a maximum likelihood with the known landmark in the particle, the particle state is updated before updating the landmark estimates.FindingsSimulation results show that the proposed CESLAM can overcome the problem of heavy computational burden while improving the accuracy of localization and mapping building. To practically evaluate the performance of the proposed method, a Pioneer 3-DX robot with a Kinect sensor is used to develop an RGB-D-based computationally efficient visual SLAM (CEVSLAM) based on Speeded-Up Robust Features (SURF). Experimental results confirm that the proposed CEVSLAM system is capable of successfully estimating the robot pose and building the map with satisfactory accuracy.Originality/valueThe proposed CESLAM algorithm overcomes the problem of the time-consuming process because of unnecessary comparisons in existing FastSLAM algorithms. Simulations show that accuracy of robot pose and landmark estimation is greatly improved by the CESLAM. Combining CESLAM and SURF, the authors establish a CEVSLAM to significantly improve the estimation accuracy and computational efficiency. Practical experiments by using a Kinect visual sensor show that the variance and average error by using the proposed CEVSLAM are smaller than those by using the other visual SLAM algorithms.

Heterogeneous Multisensor Fusion for Mobile Platform Three-Dimensional Pose Estimation

Precise, robust, and consistent localization is an important subject in many areas of science such as vision-based control, path planning, and simultaneous localization and mapping (SLAM). To estimate the pose of a platform, sensors such as inertial measurement units (IMUs), global positioning system (GPS), and cameras are commonly employed. Each of these sensors has their strengths and weaknesses. Sensor fusion is a known approach that combines the data measured by different sensors to achieve a more accurate or complete pose estimation and to cope with sensor outages. In this paper, a three-dimensional (3D) pose estimation algorithm is presented for a unmanned aerial vehicle (UAV) in an unknown GPS-denied environment. A UAV can be fully localized by three position coordinates and three orientation angles. The proposed algorithm fuses the data from an IMU, a camera, and a two-dimensional (2D) light detection and ranging (LiDAR) using extended Kalman filter (EKF) to achieve accurate localization. Among the employed sensors, LiDAR has not received proper attention in the past; mostly because a two-dimensional (2D) LiDAR can only provide pose estimation in its scanning plane, and thus, it cannot obtain a full pose estimation in a 3D environment. A novel method is introduced in this paper that employs a 2D LiDAR to improve the full 3D pose estimation accuracy acquired from an IMU and a camera, and it is shown that this method can significantly improve the precision of the localization algorithm. The proposed approach is evaluated and justified by simulation and real world experiments.

Human Cooperative Wheelchair With Brain–Machine Interaction Based on Shared Control Strategy

In this paper, a human–machine shared control strategy is proposed for the navigation control of a wheelchair, employing both brain–machine control mode and autonomous control mode. In the brain–machine control mode, contrary to the traditional four-direction control signals, a novel brain–machine interface using steady-state visual evoked potentials is presented, which utilizes two brain signals to produce a polar polynomial trajectory. The produced trajectory is continuous in curvature without violating dynamic constraints of the wheelchair. In the autonomous control mode, the synthesis of angle-based potential field and vision-based simultaneous localization and mapping technique is proposed to guide the robot navigating among the obstacles. Extensive experiments have been conducted to test the developed shared control wheelchair in several scenarios with a number of volunteers, and the results have verified the effectiveness of the proposed shared control scheme.

Delayed Monocular SLAM Approach Applied to Unmanned Aerial Vehicles.

In recent years, many researchers have addressed the issue of making Unmanned Aerial Vehicles (UAVs) more and more autonomous. In this context, the state estimation of the vehicle position is a fundamental necessity for any application involving autonomy. However, the problem of position estimation could not be solved in some scenarios, even when a GPS signal is available, for instance, an application requiring performing precision manoeuvres in a complex environment. Therefore, some additional sensory information should be integrated into the system in order to improve accuracy and robustness. In this work, a novel vision-based simultaneous localization and mapping (SLAM) method with application to unmanned aerial vehicles is proposed. One of the contributions of this work is to design and develop a novel technique for estimating features depth which is based on a stochastic technique of triangulation. In the proposed method the camera is mounted over a servo-controlled gimbal that counteracts the changes in attitude of the quadcopter. Due to the above assumption, the overall problem is simplified and it is focused on the position estimation of the aerial vehicle. Also, the tracking process of visual features is made easier due to the stabilized video. Another contribution of this work is to demonstrate that the integration of very noisy GPS measurements into the system for an initial short period of time is enough to initialize the metric scale. The performance of this proposed method is validated by means of experiments with real data carried out in unstructured outdoor environments. A comparative study shows that, when compared with related methods, the proposed approach performs better in terms of accuracy and computational time.

Comparison of Local Feature Extraction Paradigms Applied to Visual SLAM

The detection and description of locally salient regions is one of the most widely used low-level processes in modern computer vision systems. The general approach relies on the detection of stable and invariant image features that can be uniquely characterized using compact descriptors. Many detection and description algorithms have been proposed, most of them derived using different assumptions or problem models. This work presents a comparison of different approaches towards the feature extraction problem, namely: (1) standard computer vision techniques; (2) automatic synthesis techniques based on genetic programming (GP); and (3) a new local descriptor based on composite correlation filtering, proposed for the first time in this paper. The considered methods are evaluated on a difficult real-world problem, vision-based simultaneous localization and mapping (SLAM). Using three experimental scenarios, results indicate that the GP-based methods and the correlation filtering techniques outperform widely used computer vision algorithms such as the Harris and Shi-Tomasi detectors and the Speeded Up Robust Features descriptor.

Vision-Based SLAM System for Unmanned Aerial Vehicles.

The present paper describes a vision-based simultaneous localization and mapping system to be applied to Unmanned Aerial Vehicles (UAVs). The main contribution of this work is to propose a novel estimator relying on an Extended Kalman Filter. The estimator is designed in order to fuse the measurements obtained from: (i) an orientation sensor (AHRS); (ii) a position sensor (GPS); and (iii) a monocular camera. The estimated state consists of the full state of the vehicle: position and orientation and their first derivatives, as well as the location of the landmarks observed by the camera. The position sensor will be used only during the initialization period in order to recover the metric scale of the world. Afterwards, the estimated map of landmarks will be used to perform a fully vision-based navigation when the position sensor is not available. Experimental results obtained with simulations and real data show the benefits of the inclusion of camera measurements into the system. In this sense the estimation of the trajectory of the vehicle is considerably improved, compared with the estimates obtained using only the measurements from the position sensor, which are commonly low-rated and highly noisy.