Stereo Simultaneous Localization And Mapping Research Articles

BDIS-SLAM: a lightweight CPU-based dense stereo SLAM for surgery.

Common dense stereo simultaneous localization and mapping (SLAM) approaches in minimally invasive surgery (MIS) require high-end parallel computational resources for real-time implementation. Yet, it is not always feasible since the computational resources should be allocated to other tasks like segmentation, detection, and tracking. To solve the problem of limited parallel computational power, this research aims at a lightweight dense stereo SLAM system that works on a single-core CPU and achieves real-time performance (more than 30Hz in typical scenarios). A new dense stereo mapping module is integrated with the ORB-SLAM2 system and named BDIS-SLAM. Our new dense stereo mapping module includes stereo matching and 3D dense depth mosaic methods. Stereo matching is achieved with the recently proposed CPU-level real-time matching algorithm Bayesian Dense Inverse Searching (BDIS). A BDIS-based shape recovery and a depth mosaic strategy are integrated as a new thread and coupled with the backbone ORB-SLAM2 system for real-time stereo shape recovery. Experiments on in vivo data sets show that BDIS-SLAM runs at over 30Hz speed on modern single-core CPU in typical endoscopy/colonoscopy scenarios. BDIS-SLAM only consumes around an additional time compared with the backbone ORB-SLAM2. Although our lightweight BDIS-SLAM simplifies the process by ignoring deformation and fusion procedures, it can provide a usable dense mapping for modern MIS on computationally constrained devices. The proposed BDIS-SLAM is a lightweight stereo dense SLAM system for MIS. It achieves 30Hz on a modern single-core CPU in typical endoscopy/colonoscopy scenarios (image size around ). BDIS-SLAM provides a low-cost solution for dense mapping in MIS and has the potential to be applied in surgical robots and AR systems. Code is available at https://github.com/JingweiSong/BDIS-SLAM .

INS/LIDAR/Stereo SLAM Integration for Precision Navigation in GNSS-Denied Environments.

Traditionally, navigation systems have relied solely on global navigation satellite system (GNSS)/inertial navigation system (INS) integration. When a temporal loss of GNSS signal lock is encountered, these systems would rely on INS, which can sustain short bursts of outages, albeit drift significantly in prolonged outages. In this study, an extended Kalman filter (EKF) is proposed to develop an integrated INS/LiDAR/Stereo simultaneous localization and mapping (SLAM) navigation system. The first update stage of the filter integrates the INS with the LiDAR, after which the resultant navigation solution is integrated with the stereo SLAM solution, which yields the final integrated navigation solution. The system was tested for different driving scenarios in urban and rural environments using the raw Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) dataset in the complete absence of the GNSS signal. In addition, the selected KITTI drives covered low and high driving speeds in feature-rich and feature-poor environments. It is shown that the proposed INS/LiDAR/Stereo SLAM navigation system yielded better position estimations in comparison to using the INS without any assistance from onboard sensors. The accuracy improvement was expressed as a reduction of the root-mean-square error (RMSE) by 83% and 82% in the horizontal and up directions, respectively. In addition, the proposed system outperformed the positioning accuracy of some of the state-of-the-art algorithms.

An effective stereo SLAM with high-level primitives in underwater environment

Visual simultaneous localization and mapping (SLAM) algorithms face challenges in complex underwater scenarios, such as turbidity, dynamism, and low texture, where point features are unreliable and can lead to weakened or even failed systems. To overcome these issues, high-level object features are considered due to their accuracy and robustness. In this paper, we introduce an effective object-level SLAM method that employs a stereo camera to enhance the navigation robustness of autonomous underwater vehicles and generates a detailed semantic map. Point features and object features are integrated to serve the proposed approach. We begin by detecting 2D objects in images using a state-of-the-art neural network, followed by obtaining 3D objects described by the general model through the principle of multi-view geometry and eventually constructing semantic landmarks. To account for object data association, we present an object match method that takes into consideration the stereo camera characteristics in a single stereo frame and a filter-based approach to track the landmarks in odometry. Experiments are also conducted using the KITTI dataset and our sequences collected from the pool and coast. The evaluation results indicate that the proposed method can improve the performance of ORBSLAM2 in terms of both navigation robustness and mapping information in underwater scenarios.

Improved Real-Time Monocular SLAM Using Semantic Segmentation on Selective Frames

Monocular simultaneous localization and mapping (SLAM) is emerging in advanced driver assistance systems and autonomous driving, because a single camera is cheap and easy to install. Conventional monocular SLAM has two major challenges leading inaccurate localization and mapping. First, it is challenging to estimate scales in localization and mapping. Second, conventional monocular SLAM uses inappropriate mapping factors such as dynamic objects and low-parallax areas in mapping. This paper proposes an improved real-time monocular SLAM that resolves the aforementioned challenges by efficiently using deep learning-based semantic segmentation. To achieve the real-time execution of the proposed method, we apply semantic segmentation only to downsampled keyframes in parallel with mapping processes. In addition, the proposed method corrects scales of camera poses and three-dimensional (3D) points, using estimated ground plane from road-labeled 3D points and the real camera height. The proposed method also removes inappropriate corner features labeled as moving objects and low parallax areas. Experiments with eight video sequences demonstrate that the proposed monocular SLAM system achieves significantly improved and comparable trajectory tracking accuracy, compared to existing state-of-the-art monocular and stereo SLAM systems, respectively. The proposed system can achieve real-time tracking on a standard CPU potentially with a standard GPU support, whereas existing segmentation-aided monocular SLAM does not.

TwistSLAM: Constrained SLAM in Dynamic Environment

Classical visual simultaneous localization and mapping (SLAM) algorithms usually assume the environment to be rigid. This assumption limits the applicability of those algorithms as they are unable to accurately estimate the camera poses and world structure in real life scenes containing moving objects (e.g. cars, bikes, pedestrians, etc.). To tackle this issue, we propose TwistSLAM: a semantic, dynamic and stereo SLAM system that can track dynamic objects in the environment. Our algorithm creates clusters of points according to their semantic class. Thanks to the definition of inter-cluster constraints modeled by mechanical joints (function of the semantic class), a novel constrained bundle adjustment is then able to jointly estimate both poses and velocities of moving objects along with the classical world structure and camera trajectory. We evaluate our approach on several sequences from the public KITTI dataset and demonstrate quantitatively that it improves camera and object tracking compared to state-of-the-art approaches.

GPGM-SLAM: a Robust SLAM System for Unstructured Planetary Environments with Gaussian Process Gradient Maps

Simultaneous Localization and Mapping (SLAM) techniques play a key role towards long-term autonomy of mobile robots due to the ability to correct localization errors and produce consistent maps of an environment over time. Contrarily to urban or man-made environments, where the presence of unique objects and structures offer unique cues for localization, the apperance of unstructured natural environments is often ambiguous and self-similar, hindering the performances of loop closure detection. In this paper, we present an approach to improve the robustness of place recognition in the context of a submap-based stereo SLAM based on Gaussian Process Gradient Maps (GPGMaps). GPGMaps embed a continuous representation of the gradients of the local terrain elevation by means of Gaussian Process regression and Structured Kernel Interpolation, given solely noisy elevation measurements. We leverage the imagelike structure of GPGMaps to detect loop closures using traditional visual features and Bag of Words. GPGMap matching is performed as an SE(2) alignment to establish loop closure constraints within a pose graph. We evaluate the proposed pipeline on a variety of datasets recorded on Mt. Etna, Sicily and in the Morocco desert, respectively Moon- and Mars-like environments, and we compare the localization performances with state-of-the-art approaches for visual SLAM and visual loop closure detection.

Point-Line Visual Stereo SLAM Using EDlines and PL-BoW

Visual Simultaneous Localization and Mapping (SLAM) technologies based on point features achieve high positioning accuracy and complete map construction. However, despite their time efficiency and accuracy, such SLAM systems are prone to instability and even failure in poor texture environments. In this paper, line features are integrated with point features to enhance the robustness and reliability of stereo SLAM systems in poor texture environments. Firstly, method Edge Drawing lines (EDlines) is applied to reduce the line feature detection time. Meanwhile, the proposed method improves the reliability of features by eliminating outliers of line features based on the entropy scale and geometric constraints. Furthermore, this paper proposes a novel Bags of Word (BoW) model combining the point and line features to improve the accuracy and robustness of loop detection used in SLAM. The proposed PL-BoW technique achieves this by taking into account the co-occurrence information and spatial proximity of visual words. Experiments using the KITTI and EuRoC datasets demonstrate that the proposed stereo Point and EDlines SLAM (PEL-SLAM) achieves high accuracy consistently, including in challenging environments difficult to sense accurately. The processing time of the proposed method is reduced by 9.9% and 4.5% when compared to the Point and Line SLAM (PL-SLAM) and Point and stereo Point and Line based Visual Odometry (sPLVO) methods, respectively.

3D global mapping of large-scale unstructured orchard integrating eye-in-hand stereo vision and SLAM

Large-scale, high-accuracy, and adaptive three-dimensional (3D) perception are the basic technical requirements for constructing a practical and stable fruit-picking robot. The latest vision-based fruit-picking robots have been able to adapt to the complex background, uneven lighting and low color contrast of the orchard environment. However, most of them have, until now, been limited to a small field of view or rigid sampling manners. Although the simultaneous localization and mapping (SLAM) methods have the potential to realize large scale sensing, it was critically revealed in this study that the classic SLAM pipeline is not completely adapted to orchard picking tasks. In this study, the eye-in-hand stereo vision and SLAM system were integrated to provide detailed global map supporting long-term, flexible and large-scale orchard picking. To be specific, a mobile robot based on eye-in-hand vision was built and an effective hand-eye calibration method was proposed; a state-of-the-art object detection network was trained and used to establish a dynamic stereo matching method adapted well to complex orchard environments; a SLAM system was deployed and combined with the above eye-in-hand stereo vision system to obtain a detailed, wide 3D orchard map. The main contribution of this work is to build a new global mapping framework compatible to the nature of orchard picking tasks. Compared with the existing studies, this work pays more attention to the structural details of the orchard. Experimental results indicated that the constructed global map achieved both large-scale and high-resolution. This is an exploratory work providing theoretical and technical references for the future research on more stable, accurate and practical mobile fruit picking robots.

MS-SLAM: Motion State Decision of Keyframes for UAV-Based Vision Localization

To improve the robustness of UAVs in rotating and motion scenes, we propose a stereo simultaneous localization and mapping (SLAM) for UAVs with a new keyframe strategy that differs from current SLAM systems. Moreover, it has two strategies. 1) In the dominant strategy based on image quality, we filter and retain the strong feature points in each image frame by our own defined rules; this leads to longer survival time and attribute invariance in image tracking, and we save the image frames containing more strong feature points as keyframes. 2) In the secondary strategy based on motion state, we quantify the motion state during the UAV’s motion (called the compound rotation amount), and characterize the intensity of the motion. Since we want the UAV to have better robustness in the rotating scene, this strategy generates keyframes when the compound rotation amount meets the threshold. These two strategies are used to cope with gentle motion scenes and rotational motion scenes, respectively. Thus, the insertion of our keyframes is determined by the motion state; our SLAM system is proposed based on the motion state (MS-SLAM). In the back-end part of the system we construct a new weighted cost function to optimize the pose. Finally, through comparison experiments on the public dataset EuRoc, we demonstrate that our algorithm is more advantageous than some current mainstream algorithms. In difficult sequences, our algorithm compares the absolute trajectory error with ORB-SLAM2, SVO+gtsam, and VINS-Mono, the absolute trajectory error of our algorithm can be reduced by 87%.

Robust Stereo Visual SLAM for Dynamic Environments With Moving Object

The accuracy of localization and mapping of automated guided vehicles (AGVs) using visual simultaneous localization and mapping (SLAM) is significantly reduced in a dynamic environment compared to a static environment due to incorrect data association caused by dynamic objects. To solve this problem, a robust stereo SLAM algorithm based on dynamic region rejection is proposed. The algorithm first detects dynamic feature points from the fundamental matrix of consecutive frames and then divides the current frame into superpixels and labels its boundaries with disparity. Finally, dynamic regions are obtained from dynamic feature points and superpixel boundaries types; only the static area is used to estimate the pose to improve the localization accuracy and robustness of the algorithm. Experiments show that the proposed algorithm outperforms ORB-SLAM2 in the KITTI dataset, and the absolute trajectory error in the actual dynamic environment can be reduced by 84% compared with the conventional ORB-SLAM2, which can effectively improve the localization and mapping accuracy of AGVs in dynamic environments.

PL-SLAM: A Stereo SLAM System Through the Combination of Points and Line Segments

Traditional approaches to stereo visual simultaneous localization and mapping (SLAM) rely on point features to estimate the camera trajectory and build a map of the environment. In low-textured environments, though, it is often difficult to find a sufficient number of reliable point features and, as a consequence, the performance of such algorithms degrades. This paper proposes PL-SLAM, a stereo visual SLAM system that combines both points and line segments to work robustly in a wider variety of scenarios, particularly in those where point features are scarce or not well-distributed in the image. PL-SLAM leverages both points and line segments at all the instances of the process: visual odometry, keyframe selection, bundle adjustment, etc. We contribute also with a loop-closure procedure through a novel bag-of-words approach that exploits the combined descriptive power of the two kinds of features. Additionally, the resulting map is richer and more diverse in three-dimensional elements, which can be exploited to infer valuable, high-level scene structures, such as planes, empty spaces, ground plane, etc. (not addressed in this paper). Our proposal has been tested with several popular datasets (such as EuRoC or KITTI), and is compared with state-of-the-art methods such as ORB-SLAM2, revealing a more robust performance in most of the experiments while still running in real time. An open-source version of the PL-SLAM C++ code has been released for the benefit of the community.

Line-Based Stereo SLAM by Junction Matching and Vanishing Point Alignment

In this paper, we propose a stereo simultaneous localization and mapping (SLAM) method based on line segments. For the front-end module of SLAM, we designed a novel method based on the coplanar junction detection, description, and matching. Then the junctions along with their multi-scale rotated BRIEF descriptors are used in other SLAM modules, including line tracking, mapping, and loop closure. The line extraction and matching thread runs at 20 ~ 40Hz for stereo image sequences on a laptop, making it a practical front-end for line-based SLAM system. For the back-end module, a cost function is designed to minimize both of the reprojection error of line segments and alignment error of the vanishing points. The experimental results demonstrate that the proposed method exhibits more accurate localization and reconstruction than state-of-the-art line-based SLAM systems in line-rich environments.

From the Calibration of a Light-Field Camera to Direct Plenoptic Odometry

This paper presents a complete framework from the calibration of a plenoptic camera toward plenoptic camera based visual odometry. This is achieved by establishing the multiple view geometry for plenoptic cameras. Based on this novel multiple view geometry, a calibration approach is developed. The approach optimizes all intrinsic parameters of the plenoptic camera model, the 3D coordinates of the calibration points, and all camera poses in a single bundle adjustment. Our plenoptic camera based visual odometry algorithm, called direct plenoptic odometry (DPO), is a direct and semi-dense approach, which takes advantage of the full sensor resolution. DPO also relies on our multiple view geometry for plenoptic cameras. Tracking and mapping works directly on the micro images formed by the micro lens array and, therefore, has not to deal with aliasing effects in the spatial domain. The algorithm generates a semi-dense depth map based on correspondences between subsequent light-field frames, while taking differently focused micro images into account. Up to our knowledge, it is the first method that performs tracking and mapping for plenoptic cameras directly on the micro images. DPO outperforms state-of-the-art direct monocular simultaneous localization and mapping (SLAM) algorithms and can compete in accuracy with latest stereo SLAM approaches, while supplying much more detailed point clouds.

Incremental Loop Closure Verification by Guided Sampling

Loop closure detection, which is the task of identifying locations revisited by a robot in a sequence of odometry and perceptual observations, is typically formulated as a combination of two subtasks: (1) bag-of-words image retrieval and (2) post-verification using random sample consensus (RANSAC) geometric verification. The main contribution of this study is the proposal of a novel post-verification framework that achieves good precision recall trade-off in loop closure detection. This study is motivated by the fact that not all loop closure hypotheses are equally plausible (e.g., owing to mutual consistency between loop closure constraints) and that if we have evidence that one hypothesis is more plausible than the others, then it should be verified more frequently. We demonstrate that the loop closure detection problem can be viewed as an instance of a multi-model hypothesize-and-verify framework. Thus, we can build guided sampling strategies on this framework where loop closures proposed using image retrieval are verified in a planned order (rather than in a conventional uniform order) to operate in a constant time. Experimental results using a stereo simultaneous localization and mapping (SLAM) system confirm that the proposed strategy, the use of loop closure constraints and robot trajectory hypotheses as a guide, achieves promising results despite the fact that there exists a significant number of false positive constraints and hypotheses.

A constant-time SLAM back-end in the continuum between global mapping and submapping: application to visual stereo SLAM

This work addresses the development and application of a novel approach, called sparser relative bundle adjustment (SRBA), which exploits the inherent flexibility of the relative bundle adjustment (RBA) framework to devise a continuum of strategies, ranging from RBA with linear graphs to classic bundle adjustment (BA) in global coordinates, where submapping with local maps emerges as a natural intermediate solution. This method leads to graphs that can be optimized in bounded time even at loop closures, regardless of the loop length. Furthermore, it is shown that the pattern in which relative coordinate variables are defined among keyframes has a significant impact on the graph optimization problem. By using the proposed scheme, optimization can be done more efficiently than in standard RBA, allowing the optimization of larger local maps for any given maximum computational cost. The main algorithms involved in the graph management, along with their complexity analyses, are presented to prove their bounded-time nature. One key advance of the present work is the demonstration that, under mild assumptions, the spanning trees for every single keyframe in the map can be incrementally built by a constant-time algorithm, even for arbitrary graph topologies. We validate our proposal within the scope of visual stereo simultaneous localization and mapping (SLAM) by developing a complete system that includes a front-end that seamlessly integrates several state-of-the-art computer vision techniques such as ORB features and bag-of-words, along with a decision scheme for keyframe insertion and a SRBA-based back-end that operates as graph optimizer. Finally, a set of experiments in both indoor and outdoor conditions is presented to test the capabilities of this approach. Open-source implementations of the SRBA back-end and the stereo front-end have been released online.

Building a 3-D Line-Based Map Using Stereo SLAM

This paper presents a graph-based visual simultaneous localization and mapping (SLAM) system using straight lines as features. Compared with point features, lines provide far richer information about the structure of the environment and make it possible to infer spatial semantics from the map. Using a stereo rig as the sole sensor, our proposed system utilizes many advanced techniques, such as motion estimation, pose optimization, and bundle adjustment. We use two different representations to parameterize 3-D lines in this paper: Plucker line coordinates for efficient initialization of newly observed line features and projection of 3-D lines, and orthonormal representation for graph optimization. The proposed system is tested with indoor and outdoor sequences, and it exhibits better reconstruction performance against a point-based SLAM system in line-rich environments.

Stereo Vision Specific Observation Model for EKF-Based SLAM

This work addresses a new probabilistic observation model for a stereo simultaneous localization and mapping (SLAM) system within the standard Extended-Kalman filter (EKF) framework. The observation modal was derived by using the inverse depth parameterization as the landmark modal, and contributes to both bearing and range information into the EKF estimation. In this way the inherently non-linear problem cause by the projection equations is resolved and real depth uncertainty distribution of landmarks features can be accurately estimated. The system was demonstrated with real-world outdoor data. Analysis results show landmark feature depth estimation is more stable and the uncertainty noise converges faster than the traditional approach.

Single and Multi Camera Simultaneous Localization and Mapping Using the Extended Kalman Filter

Simultaneous Localization and Mapping (SLAM) has received quite a lot of attention in the last decades because of its relevance for many applications centered on a mobile observer, such as service robotics and intelligent transportation systems. This paper focuses on the use of recursive Bayesian filtering, as implemented by the Extendend Kalman Filter (EKF), to face the Visual SLAM problem, i.e., when using data from visual sources. In Monocular SLAM, which uses a single camera as unique source of information, it is not possible to directly estimate the depth of a feature from a single image. To handle the severely non-normal distribution representing such uncertainty, inverse parametrizations were developed, capable to deal with such uncertainty and still relying on Gaussian variables. In the paper, after an introduction to EKF-SLAM, we provide a review of different inverse parametrizations, and we introduce a novel proposal, the Framed Inverse Depth (FID) parametrization, which, in terms of consistency, performs similarly to state of the art Monocular SLAM parametrizations, but at a reduced computational cost. All these parametrizations can be used in a stereo and multi camera setting too. An extensive analysis is presented for both Monocular and stereo SLAM, for a simulated environment widely used in the literature as well as on a widely used real dataset.