Monocular Visual Simultaneous Localization And Mapping Research Articles

SVR-Net: A Sparse Voxelized Recurrent Network for Robust Monocular SLAM with Direct TSDF Mapping.

Simultaneous localization and mapping (SLAM) plays a fundamental role in downstream tasks including navigation and planning. However, monocular visual SLAM faces challenges in robust pose estimation and map construction. This study proposes a monocular SLAM system based on a sparse voxelized recurrent network, SVR-Net. It extracts voxel features from a pair of frames for correlation and recursively matches them to estimate pose and dense map. The sparse voxelized structure is designed to reduce memory occupation of voxel features. Meanwhile, gated recurrent units are incorporated to iteratively search for optimal matches on correlation maps, thereby enhancing the robustness of the system. Additionally, Gauss-Newton updates are embedded in iterations to impose geometrical constraints, which ensure accurate pose estimation. After end-to-end training on ScanNet, SVR-Net is evaluated on TUM-RGBD and successfully estimates poses on all nine scenes, while traditional ORB-SLAM fails on most of them. Furthermore, absolute trajectory error (ATE) results demonstrate that the tracking accuracy is comparable to that of DeepV2D. Unlike most previous monocular SLAM systems, SVR-Net directly estimates dense TSDF maps suitable for downstream tasks with high efficiency of data exploitation. This study contributes to the development of robust monocular visual SLAM systems and direct TSDF mapping.

Monocular Visual SLAM for Markerless Tracking Algorithm to Augmented Reality

Augmented Reality (AR) tries to seamlessly integrate virtual content into the real world of the user. Ideally, the virtual content would behave exactly like real objects. This necessitates a correct and precise estimation of the user’s viewpoint (or that of a camera) with regard to the virtual content’s coordinate system. Therefore, the real-time establishment of 3-dimension (3D) maps in real scenes is particularly important for augmented reality technology. So in this paper, we integrate Simultaneous Localization and Mapping (SLAM) technology into augmented reality. Our research is to implement an augmented reality system without markers using the ORB-SLAM2 framework algorithm. In this paper we propose an improved method for Oriented FAST and Rotated BRIEF (ORB) feature extraction and optimized key frame selection, as well as the use of the Progressive Sample Consensus (PROSAC) algorithm for planar estimation of augmented reality implementations, thus solving the problem of increased system runtime because of the loss of large amounts of texture information in images. In this paper, we get better results by comparing experiments and data analysis. However, there are some improved methods of PROSAC algorithm which are more suitable for the detection of plane feature points.

A monocular visual SLAM system augmented by lightweight deep local feature extractor using in-house and low-cost LIDAR-camera integrated device

ABSTRACT Simultaneous Localization and Mapping (SLAM) has been widely used in emergency response, self-driving and city-scale 3D mapping and navigation. Recent deep-learning based feature point extractors have demonstrated superior performance in dealing with the complex environmental challenges (e.g. extreme lighting) while the traditional extractors are struggling. In this paper, we have successfully improved the robustness and accuracy of a monocular visual SLAM system under various complex scenes by adding a deep learning based visual localization thread as an augmentation to the visual SLAM framework. In this thread, our feature extractor with an efficient lightweight deep neural network is used for absolute pose and scale estimation in real time using the highly accurate georeferenced prior map database at 20cm geometric accuracy created by our in-house and low-cost LiDAR and camera integrated device. The closed-loop error provided by our SLAM system with and without this enhancement is 1.03m and 18.28m respectively. The scale estimation of the monocular visual SLAM is also significantly improved (0.01 versus 0.98). In addition, a novel camera-LiDAR calibration workflow is also provided for large-scale 3D mapping. This paper demonstrates the application and research potential of deep-learning based vision SLAM with image and LiDAR sensors.

PLJ-SLAM: Monocular Visual SLAM With Points, Lines, and Junctions of Coplanar Lines

Existing monocular visual simultaneous localization and mapping (SLAM) mainly focuses on point and line features, and the constraints among line features are not fully explored. In this paper, a multi-feature monocular SLAM with ORB points, lines, and junctions of coplanar lines is proposed for indoor environments. To create 3D junctions of coplanar lines, an adaptive coordinate confidence is designed to describe the coordinate stability of 2D junctions on the image plane. Based on the matched 2D junctions between the current keyframe and other two associated keyframes, a multi-view coplanarity verification is performed for 3D junction creation. Moreover, reprojection verification is conducted to update the observation of 3D junction, which is used to dynamically update the confidence of coplanarity for confidence-based local bundle adjustment. The camera pose is optimized through multiple constraints of ORB points, lines, and junctions of the coplanar lines. As a result, the performance of monocular SLAM is improved. The experiment verification shows the effectiveness of the proposed method.

OG-SLAM: A real-time and high-accurate monocular visual SLAM framework

The challenge of improving the accuracy of monocular Simultaneous Localization and Mapping (SLAM) is considered, which widely appears in computer vision, autonomous robotics, and remote sensing. A new framework (ORB-GMS-SLAM (or OG-SLAM)) is proposed, which introduces the region-based motion smoothness into a typical Visual SLAM (V-SLAM) system. The region-based motion smoothness is implemented by integrating the Oriented Fast and Rotated Brief (ORB) features and the Grid-based Motion Statistics (GMS) algorithm into the feature matching process. The OG-SLAM significantly reduces the absolute trajectory error (ATE) on the key-frame trajectory estimation without compromising the real-time performance. This study compares the proposed OG-SLAM to an advanced V-SLAM system (ORB-SLAM2). The results indicate the highest accuracy improvement of almost 75% on a typical RGB-D SLAM benchmark. Compared with other ORB-SLAM2 settings (1800 key points), the OG-SLAM improves the accuracy by around 20% without losing performance in real-time. The OG-SLAM framework has a significant advantage over the ORB-SLAM2 system in that it is more robust for rotation, loop-free, and long ground-truth length scenarios. Furthermore, as far as the authors are aware, this framework is the first attempt to integrate the GMS algorithm into the V-SLAM.

Robust Tightly Coupled Pose Measurement Based on Multi-Sensor Fusion in Mobile Robot System.

Currently, simultaneous localization and mapping (SLAM) is one of the main research topics in the robotics field. Visual-inertia SLAM, which consists of a camera and an inertial measurement unit (IMU), can significantly improve robustness and enable scale weak-visibility, whereas monocular visual SLAM is scale-invisible. For ground mobile robots, the introduction of a wheel speed sensor can solve the scale weak-visibility problem and improve robustness under abnormal conditions. In this paper, a multi-sensor fusion SLAM algorithm using monocular vision, inertia, and wheel speed measurements is proposed. The sensor measurements are combined in a tightly coupled manner, and a nonlinear optimization method is used to maximize the posterior probability to solve the optimal state estimation. Loop detection and back-end optimization are added to help reduce or even eliminate the cumulative error of the estimated poses, thus ensuring global consistency of the trajectory and map. The outstanding contribution of this paper is that the wheel odometer pre-integration algorithm, which combines the chassis speed and IMU angular speed, can avoid the repeated integration caused by linearization point changes during iterative optimization; state initialization based on the wheel odometer and IMU enables a quick and reliable calculation of the initial state values required by the state estimator in both stationary and moving states. Comparative experiments were conducted in room-scale scenes, building scale scenes, and visual loss scenarios. The results showed that the proposed algorithm is highly accurate—2.2 m of cumulative error after moving 812 m (0.28%, loopback optimization disabled)—robust, and has an effective localization capability even in the event of sensor loss, including visual loss. The accuracy and robustness of the proposed method are superior to those of monocular visual inertia SLAM and traditional wheel odometers.

Incorporating learnt local and global embeddings into monocular visual SLAM

Traditional approaches for Visual Simultaneous Localization and Mapping (VSLAM) rely on low-level vision information for state estimation, such as handcrafted local features or the image gradient. While significant progress has been made through this track, under more challenging configuration for monocular VSLAM, e.g., varying illumination, the performance of state-of-the-art systems generally degrades. As a consequence, robustness and accuracy for monocular VSLAM are still widely concerned. This paper presents a monocular VSLAM system that fully exploits learnt features for better state estimation. The proposed system leverages both learnt local features and global embeddings at different modules of the system: direct camera pose estimation, inter-frame feature association, and loop closure detection. With a probabilistic explanation of keypoint prediction, we formulate the camera pose tracking in a direct manner and parameterize local features with uncertainty taken into account. To alleviate the quantization effect, we adapt the mapping module to generate 3D landmarks better to guarantee the system’s robustness. Detecting temporal loop closure via deep global embeddings further improves the robustness and accuracy of the proposed system. The proposed system is extensively evaluated on public datasets (Tsukuba, EuRoC, and KITTI), and compared against the state-of-the-art methods. The competitive performance of camera pose estimation confirms the effectiveness of our method.

Improved Point-Line Feature Based Visual SLAM Method for Complex Environments.

Traditional visual simultaneous localization and mapping (SLAM) systems rely on point features to estimate camera trajectories. However, feature-based systems are usually not robust in complex environments such as weak textures or obvious brightness changes. To solve this problem, we used more environmental structure information by introducing line segments features and designed a monocular visual SLAM system. This system combines points and line segments to effectively make up for the shortcomings of traditional positioning based only on point features. First, ORB algorithm based on local adaptive threshold was proposed. Subsequently, we not only optimized the extracted line features, but also added a screening step before the traditional descriptor matching to combine the point features matching results with the line features matching. Finally, the weighting idea was introduced. When constructing the optimized cost function, we allocated weights reasonably according to the richness and dispersion of features. Our evaluation on publicly available datasets demonstrated that the improved point-line feature method is competitive with the state-of-the-art methods. In addition, the trajectory graph significantly reduced drift and loss, which proves that our system increases the robustness of SLAM.

LIFT-SLAM: A deep-learning feature-based monocular visual SLAM method

The Simultaneous Localization and Mapping (SLAM) problem addresses the possibility of a robot to localize itself in an unknown environment and simultaneously build a consistent map of this environment. Recently, cameras have been successfully used to get the environment’s features to perform SLAM, which is referred to as visual SLAM (VSLAM). However, classical VSLAM algorithms can be easily induced to fail when either the motion of the robot or the environment is too challenging. Although new approaches based on Deep Neural Networks (DNNs) have achieved promising results in VSLAM, they still are unable to outperform traditional methods. To leverage the robustness of deep learning to enhance traditional VSLAM systems, we propose to combine the potential of deep learning-based feature descriptors with the traditional geometry-based VSLAM, building a new VSLAM system called LIFT-SLAM. Experiments conducted on KITTI and Euroc datasets show that deep learning can be used to improve the performance of traditional VSLAM systems, as the proposed approach was able to achieve results comparable to the state-of-the-art while being robust to sensorial noise. We enhance the proposed VSLAM pipeline by avoiding parameter tuning for specific datasets with an adaptive approach while evaluating how transfer learning can affect the quality of the features extracted.

Monocular Visual SLAM with Points and Lines for Ground Robots in Particular Scenes: Parameterization for Lines on Ground

Visual simultaneous localization and mapping (V-SLAM) has attracted a lot of attention lately from the robotics communities due to its vast applications and importance. This paper addresses the problem of V-SLAM with points and lines in particular scenes where there are many lines on an approximately planar ground. All the lines in these particular scenes are treated as 3D lines with four degree-of-freedom (DoF) in most V-SLAM systems with lines. However, lines on ground only have two DoF. The redundant parameters will increase the estimation uncertainty of lines on ground. In order to restrict the lines on ground to the correct solution space, we propose two parameterization methods for it. The first method still treats lines on ground as 3D lines, and then we propose a planar constraint for the representation of 3D lines to loosely constrain the lines to the ground plane. Further, to strictly constrain the lines on ground to the ground plane, the second method treats these lines as 2D lines in a plane, and then we propose the corresponding parameterization method and geometric computation method from initialization to bundle adjustment. After that, to better exploit lines on ground during localization and mapping by using the proposed parameterization methods, we propose the graph optimization-based monocular V-SLAM system with points and lines to deal with lines on ground differently from general 3D lines. Assisted by wheel encoders, the proposed system generates a structural map. We perform experiments on both simulated data and real-world data to demonstrate that the proposed two parameterization methods can better exploit lines on ground than 3D line parameterization method that is used to represent the lines on ground in the state-of-the-art V-SLAM works with lines.

A Semi-Direct Monocular Visual SLAM Algorithm in Complex Environments

A novel monocular visual simultaneous localization and mapping (SLAM) algorithm built on the semi-direct method is proposed to deal with some problems in complex environments, such as low-texture, moving objects and perceptual aliasing. The proposed algorithm takes advantage of direct and feature-based methods. On one hand, a direct method is used to track the camera poses and solve the feature alignment. On the other hand, ORB features in keyframes are extracted and matched for optimization and loop closure. To improve the localization accuracy in dynamic environments, a motion detection module that is robust to illumination change is adopted. In addition, for the sake of resolving the loop closure detection problem in perceptual aliasing scenes, this paper fuses the spatial information between two visual words into the bag of visual words (BoVW) model and employs an improved pyramid term frequency-inverse document frequency (TF-IDF) scoring match scheme. Experimental results prove that the proposed algorithm behaves better performance than ORB-SLAM with regard to overall accuracy and speed in complex environments.

Attention-SLAM: A Visual Monocular SLAM Learning From Human Gaze

This paper proposes a novel simultaneous localization and mapping (SLAM) approach, namely Attention-SLAM, which simulates human navigation mode by combining a visual saliency model (SalNavNet) with traditional monocular visual SLAM. Firstly a visual saliency model namely SalNavNet is proposed. In SalNavNet, we introduce a correlation module and propose an adaptive Exponential Moving Average (EMA) module. These modules mitigate the center bias, which most current saliency models have. This novel idea enables the saliency maps generated by SalNavNet to pay more attention to the same salient object. An open-source saliency SLAM dataset namely Salient-Euroc is published, it consists of Euroc dataset and corresponding saliency maps. Moreover, we propose a new optimization method called Weighted Bundle Adjustment (Weighted BA) in Attention-SLAM. Most SLAM methods treat all the features extracted from the images as equal importance during the optimization process. In weighted BA, the feature points extracted from the salient regions have greater importance. Comprehensive test results prove that Attention-SLAM outperforms benchmarks such as Direct Sparse Odometry (DSO), ORB-SLAM, and Salient DSO in the 7 of 11 test cases. The test cases are all indoor scenes, with varying brightness, speed, and image distortion. Compared with ORB-SLAM, our method improves the accuracy by 4% and efficiency by 6.5% on average.

Study on the Influence of Image Noise on Monocular Feature-Based Visual SLAM Based on FFDNet.

Noise appears in images captured by real cameras. This paper studies the influence of noise on monocular feature-based visual Simultaneous Localization and Mapping (SLAM). First, an open-source synthetic dataset with different noise levels is introduced in this paper. Then the images in the dataset are denoised using the Fast and Flexible Denoising convolutional neural Network (FFDNet); the matching performances of Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF) and Oriented FAST and Rotated BRIEF (ORB) which are commonly used in feature-based SLAM are analyzed in comparison and the results show that ORB has a higher correct matching rate than that of SIFT and SURF, the denoised images have a higher correct matching rate than noisy images. Next, the Absolute Trajectory Error (ATE) of noisy and denoised sequences are evaluated on ORB-SLAM2 and the results show that the denoised sequences perform better than the noisy sequences at any noise level. Finally, the completely clean sequence in the dataset and the sequences in the KITTI dataset are denoised and compared with the original sequence through comprehensive experiments. For the clean sequence, the Root-Mean-Square Error (RMSE) of ATE after denoising has decreased by 16.75%; for KITTI sequences, 7 out of 10 sequences have lower RMSE than the original sequences. The results show that the denoised image can achieve higher accuracy in the monocular feature-based visual SLAM under certain conditions.

A brain-inspired compact cognitive mapping system.

In many simultaneous localization and mapping (SLAM) systems, the map of the environment grows over time as the robot explores the environment. The ever-growing map prevents long-term mapping, especially in large-scale environments. In this paper, we develop a compact cognitive mapping approach inspired by neurobiological experiments. Mimicking the firing activities of neighborhood cells, neighborhood fields determined by movement information, i.e. translation and rotation, are modeled to describe one of the distinct segments of the explored environment. The vertices with low neighborhood field activities are avoided to be added into the cognitive map. The optimization of the cognitive map is formulated as a robust non-linear least squares problem constrained by the transitions between vertices, and is numerically solved efficiently. According to the cognitive decision-making of place familiarity, loop closure edges are clustered depending on time intervals, and then batch global optimization of the cognitive map is performed to satisfy the combined constraint of the whole cluster. After the loop closure process, scene integration is performed, in which revisited vertices are removed subsequently to further reduce the size of the cognitive map. The compact cognitive mapping approach is tested on a monocular visual SLAM system in a naturalistic maze for a biomimetic animated robot. Our results demonstrate that the proposed method largely restricts the growth of the size of the cognitive map over time, and meanwhile, the compact cognitive map correctly represents the overall layout of the environment. The compact cognitive mapping method is well suitable for the representation of large-scale environments to achieve long-term robot navigation.

Monocular Visual SLAM Based on a Cooperative UAV-Target System.

To obtain autonomy in applications that involve Unmanned Aerial Vehicles (UAVs), the capacity of self-location and perception of the operational environment is a fundamental requirement. To this effect, GPS represents the typical solution for determining the position of a UAV operating in outdoor and open environments. On the other hand, GPS cannot be a reliable solution for a different kind of environments like cluttered and indoor ones. In this scenario, a good alternative is represented by the monocular SLAM (Simultaneous Localization and Mapping) methods. A monocular SLAM system allows a UAV to operate in a priori unknown environment using an onboard camera to simultaneously build a map of its surroundings while at the same time locates itself respect to this map. So, given the problem of an aerial robot that must follow a free-moving cooperative target in a GPS denied environment, this work presents a monocular-based SLAM approach for cooperative UAV–Target systems that addresses the state estimation problem of (i) the UAV position and velocity, (ii) the target position and velocity, (iii) the landmarks positions (map). The proposed monocular SLAM system incorporates altitude measurements obtained from an altimeter. In this case, an observability analysis is carried out to show that the observability properties of the system are improved by incorporating altitude measurements. Furthermore, a novel technique to estimate the approximate depth of the new visual landmarks is proposed, which takes advantage of the cooperative target. Additionally, a control system is proposed for maintaining a stable flight formation of the UAV with respect to the target. In this case, the stability of control laws is proved using the Lyapunov theory. The experimental results obtained from real data as well as the results obtained from computer simulations show that the proposed scheme can provide good performance.

Simple But Effective Scale Estimation for Monocular Visual Odometry in Road Driving Scenarios

In large-scale environments, scale drift is a crucial problem of monocular visual simultaneous localization and mapping (SLAM). A common solution is to utilize the camera height, which can be obtained using the reconstructed 3D ground points (3DGPs) from two successive frames, as prior knowledge. Increasing the number of 3DGPs by using more proceeding frames can be a natural extension of this solution to estimate a more precise camera height. However, merely employing multiple frames based on conventional methods is hard to be directly applicable in a real-world scenario because the vehicle motion and inaccurate feature matching inevitably cause large uncertainty and noisy 3DGPs. In this study, we propose an elaborate method to collect confident 3DGPs from multiple frames for robust scale estimation. First, we gather 3DGP candidates that can be seen in more than a predefined number of frames. To verify the 3DGP candidates, we filter out the 3D points at the exterior of the road region obtained by the deep-learning-based road segmentation model. In addition, we formulate an optimization problem constrained by a simple but effective geometric assumption that the normal vector of the ground plane lies in the null space of a movement vector of the camera center, and provide a closed-form solution. ORB-SLAM with the proposed scale estimation method achieves the average translation error with 1.19% on the KITTI dataset, which outperforms the state-of-the-art conventional monocular visual SLAM methods in road driving scenarios.

Accurate and Robust Monocular SLAM with Omnidirectional Cameras.

Simultaneous localization and mapping (SLAM) are fundamental elements for many emerging technologies, such as autonomous driving and augmented reality. For this paper, to get more information, we developed an improved monocular visual SLAM system by using omnidirectional cameras. Our method extends the ORB-SLAM framework with the enhanced unified camera model as a projection function, which can be applied to catadioptric systems and wide-angle fisheye cameras with 195 degrees field-of-view. The proposed system can use the full area of the images even with strong distortion. For omnidirectional cameras, a map initialization method is proposed. We analytically derive the Jacobian matrices of the reprojection errors with respect to the camera pose and 3D position of points. The proposed SLAM has been extensively tested in real-world datasets. The results show positioning error is less than 0.1% in a small indoor environment and is less than 1.5% in a large environment. The results demonstrate that our method is real-time, and increases its accuracy and robustness over the normal systems based on the pinhole model.

Semi-direct monocular visual and visual-inertial SLAM with loop closure detection

A novel semi-direct monocular visual simultaneous localization and mapping (SLAM) system is proposed to maintain the fast performance of a direct method and the high precision and loop closure capability of a feature-based method. This system extracts and matches Oriented FAST and Rotated BRIEF features in a keyframe and tracks a non-keyframe via a direct method without the requirement of extracting and matching features. A keyframe is used for global or local optimization and loop closure, whereas a non-keyframe is used for fast tracking and localization, thereby combining the advantages of direct and feature-based methods. A monocular visual-inertial SLAM system that fuses inertial measurement data with visual SLAM is also proposed. This system successfully recovers the metric scale successfully. The evaluation on datasets shows that the proposed approach accomplishes loop closure detection successfully and requires less time to achieve accuracy comparable with that of feature-based method. The physical experiment demonstrates the feasibility and robustness of the proposed SLAM. The approach achieves good balance between speed and accuracy and provides valuable references for design and improvement of other SLAM methods.

Distributed monocular visual SLAM as a basis for a collaborative augmented reality framework

Visual Simultaneous Localization and Mapping (SLAM) has been used for markerless tracking in augmented reality applications. Distributed SLAM helps multiple agents to collaboratively explore and build a global map of the environment while estimating their locations in it. One of the main challenges in distributed SLAM is to identify local map overlaps of these agents, especially when their initial relative positions are not known. We developed a collaborative AR framework with freely moving agents having no knowledge of their initial relative positions. Each agent in our framework uses a camera as the only input device for its SLAM process. Furthermore, the framework identifies map overlaps of agents using an appearance-based method. We also proposed a quality measure to determine the best keypoint detector/descriptor combination for our framework.

Multi-camera visual SLAM for autonomous navigation of micro aerial vehicles

In this paper, we present a visual simultaneous localization and mapping (SLAM) system which integrates measurements from multiple cameras to achieve robust pose tracking for autonomous navigation of micro aerial vehicles (MAVs) in unknown complex environments. We analyze the iterative optimizations for pose tracking and map refinement of visual SLAM in multi-camera cases. The analysis ensures the soundness and accuracy of each optimization update. A well-known monocular visual SLAM system is extended to utilize two cameras with non-overlapping fields of view (FOVs) in the final implementation. The resulting visual SLAM system enables autonomous navigation of an MAV in complex scenarios. The theory behind this system can easily be extended to multi-camera configurations, when the onboard computational capability allows this. For operations in large-scale environments, we modify the resulting visual SLAM system to be a constant-time robust visual odometry. To form a full visual SLAM system, we further implement an efficient back-end for loop closing. The back-end maintains a keyframe-based global map, which is also used for loop-closure detection. An adaptive-window pose-graph optimization method is proposed to refine keyframe poses of the global map and thus correct pose drift that is inherent in the visual odometry. We demonstrate the efficiency of the proposed visual SLAM system for applications onboard of MAVs in experiments with both autonomous and manual flights. The pose tracking results are compared with ground truth data provided by an external tracking system.