Real-time Visual SLAM Research Articles

Strong-SLAM: real-time RGB-D visual SLAM in dynamic environments based on StrongSORT

Abstract The assumptions of a static environment and scene rigidity are important theoretical underpinnings of traditional visual simultaneous localization and mapping (SLAM) algorithms. However, these assumptions are difficult to work in dynamic environments containing non-rigid objects, and cannot effectively handle the characteristics of local areas of non-rigid moving objects, seriously affecting the robustness and accuracy of the SLAM system in localization and mapping. To address these problems, we improved ORB-SLAM3 and proposed a real-time RGB-D visual SLAM framework for dynamic environments based on StrongSORT—Strong-SLAM. First, we combine YOLOv7-tiny with StrongSORT to match the semantic information of dynamic targets. Optical flow and epipolar constraints are then used to initially extract geometric and motion information between adjacent frames. Subsequently, based on an improved adaptive threshold segmentation algorithm and geometric residuals, a background model and a Gaussian residual model are constructed to further extract the geometric information of dynamic targets. Finally, semantic and geometric information are integrated to perform global feature motion level classification, and motion probabilities and optimization weights are defined to participate in global pose estimation and optimization. Experimental results on the publicly available TUM RGB-D dataset show that Strong-SLAM reduces the absolute trajectory error and relative pose error by at least 90% compared to ORB-SLAM3, achieving performance comparable to the most advanced dynamic SLAM solutions.

A real-time visual SLAM based on semantic information and geometric information in dynamic environment

A real-time visual SLAM based on semantic information and geometric information in dynamic environment

Parallel implementation for real-time visual SLAM systems based on heterogeneous computing

Simultaneous localization and mapping has become rapidly developed and plays an indispensable role in intelligent vehicles. However, many state-of-the-art visual simultaneous localization and mapping (VSLAM) frameworks are very time-consuming both in front-end and back-end, especially for large-scale scenes. Nowadays, the increasingly popular use of graphics processors for general-purpose computing, and the progressively mature high-performance programming theory based on compute unified device architecture (CUDA) have given the possibility for large-scale VSLAM to solve the conflict between limited computing power and excessive computing tasks. The paper proposes a full-flow optimal parallelization scheme based on heterogeneous computing to speed up the time-consuming modules in VSLAM. Firstly, a parallel strategy for feature extraction and matching is designed to reduce the time consumption arising from multiple data transfers between devices. Secondly, a bundle adjustment method based solely on CUDA is developed. By fully optimizing memory scheduling and task allocation, a large increase in speed is achieved while maintaining accuracy. Besides, CUDA heterogeneous acceleration is fully utilized for tasks such as error computation and linear system construction in the VSLAM back-end to enhance the operation speed. Our proposed method is tested on numerous public datasets on both computer and embedded sides, respectively. A number of qualitative and quantitative experiments are performed to verify its superiority in terms of speed compared to other states-of-the-art.

Real-Time Dense Visual SLAM with Neural Factor Representation

Developing a high-quality, real-time, dense visual SLAM system poses a significant challenge in the field of computer vision. NeRF introduces neural implicit representation, marking a notable advancement in visual SLAM research. However, existing neural implicit SLAM methods suffer from long runtimes and face challenges when modeling complex structures in scenes. In this paper, we propose a neural implicit dense visual SLAM method that enables high-quality real-time reconstruction even on a desktop PC. Firstly, we propose a novel neural scene representation, encoding the geometry and appearance information of the scene as a combination of the basis and coefficient factors. This representation allows for efficient memory usage and the accurate modeling of high-frequency detail regions. Secondly, we introduce feature integration rendering to significantly improve rendering speed while maintaining the quality of color rendering. Extensive experiments on synthetic and real-world datasets demonstrate that our method achieves an average improvement of more than 60% for Depth L1 and ATE RMSE compared to existing state-of-the-art methods when running at 9.8 Hz on a desktop PC with a 3.20 GHz Intel Core i9-12900K CPU and a single NVIDIA RTX 3090 GPU. This remarkable advancement highlights the crucial importance of our approach in the field of dense visual SLAM.

RVD-SLAM: A Real-Time Visual SLAM Toward Dynamic Environments Based on Sparsely Semantic Segmentation and Outlier Prior

RVD-SLAM: A Real-Time Visual SLAM Toward Dynamic Environments Based on Sparsely Semantic Segmentation and Outlier Prior

FPGA-Based Feature Extraction and Tracking Accelerator for Real-Time Visual SLAM.

Due to its advantages of low latency, low power consumption, and high flexibility, FPGA-based acceleration technology has been more and more widely studied and applied in the field of computer vision in recent years. An FPGA-based feature extraction and tracking accelerator for real-time visual odometry (VO) and visual simultaneous localization and mapping (V-SLAM) is proposed, which can realize the complete acceleration processing capability of the image front-end. For the first time, we implement a hardware solution that combines features from accelerated segment test (FAST) feature points with Gunnar Farneback (GF) dense optical flow to achieve better feature tracking performance and provide more flexible technical route selection. In order to solve the scale invariance and rotation invariance lacking problems of FAST features, an efficient pyramid module with a five-layer thumbnail structure was designed and implemented. The accelerator was implemented on a modern Xilinx Zynq FPGA. The evaluation results showed that the accelerator could achieve stable tracking of features of violently shaking images and were consistent with the results from MATLAB code running on PCs. Compared to PC CPUs, which require seconds of processing time, the processing latency was greatly reduced to the order of milliseconds, making GF dense optical flow an efficient and practical technical solution on the edge side.

A Real-Time Monocular Visual SLAM Based on the Bundle Adjustment with Adaptive Robust Kernel

A Real-Time Monocular Visual SLAM Based on the Bundle Adjustment with Adaptive Robust Kernel

Edge Assisted Mobile Semantic Visual SLAM

Localization and navigation play a key role in many location-based services and have attracted numerous research efforts. In recent years, visual SLAM has been prevailing for autonomous driving. However, the ever-growing computation resources demanded by SLAM impede its applications to resource-constrained mobile devices. In this paper, we present the design, implementation, and evaluation of <i>edgeSLAM</i>, an edge-assisted real-time semantic visual SLAM service running on mobile devices. <i>edgeSLAM</i> leverages the state-of-the-art semantic segmentation algorithm to enhance localization and mapping accuracy, and speeds up the computation-intensive SLAM and semantic segmentation algorithms by computation offloading. The key innovations of <i>edgeSLAM</i> include an efficient computation offloading strategy, an opportunistic data sharing method, an adaptive task scheduling algorithm, and a multi-user support mechanism. We fully implement <i>edgeSLAM</i> and plan to open-source it. Extensive experiments are conducted under 3 datasets. The results show that <i>edgeSLAM</i> can run on mobile devices at 35fps and achieve 5cm localization accuracy from real-world experiments, outperforming existing solutions by more than 15&#x0025;. We also demonstrate the usability of <i>edgeSLAM</i> through 2 case studies of pedestrian localization and robot navigation. To the best of our knowledge, <i>edgeSLAM</i> is the first edge-assisted real-time semantic visual SLAM for mobile devices.

A Real-time and Robust Monocular Visual Inertial SLAM System Based on Point and Line Features for Mobile Robots of Smart Cities Toward 6G

Autonomous navigation of mobile robots in complex environments is challenging. Solving the problems of inaccuracy localization and frequent tracking losses of mobile robots in challenging scenes is beyond the power of point-based visual simultaneous localization and mapping (vSLAM). This paper proposes a real-time and robust point-line based monocular visual inertial SLAM (VINS) system for mobile robots of smart cities towards 6G. To extract robust line features for tracking in challenging scenes, EDLines with adaptive gamma correction is adopted to fast extract a larger ratio of long line features among all extracted line features. A real-time line feature matching approach is proposed to track the extracted line features between adjacent frames without the need of computing descriptors. Compared with LSD and KNN matching method based on LBD descriptors, the proposed method runs three times faster. Furthermore, a tightly coupled sensor fusion optimization framework is constructed for accurate state estimation, which contains point-line feature reprojection errors and IMU residuals. By evaluating on public benchmark datasets, our VINS system has high localization accuracy, real-time performance and robustness compared with other advanced SLAM systems. Our VINS system enables mobile robots to locate accurately in smart cities with complex environments.

Rdts-Slam: Real-Time Visual Slam for Dynamic Environments Using Simultaneous Target Detection and Semantic Segmentation

Rdts-Slam: Real-Time Visual Slam for Dynamic Environments Using Simultaneous Target Detection and Semantic Segmentation

RDS-SLAM: Real-Time Dynamic SLAM Using Semantic Segmentation Methods

The scene rigidity is a strong assumption in typical visual Simultaneous Localization and Mapping (vSLAM) algorithms. Such strong assumption limits the usage of most vSLAM in dynamic real-world environments, which are the target of several relevant applications such as augmented reality, semantic mapping, unmanned autonomous vehicles, and service robotics. Many solutions are proposed that use different kinds of semantic segmentation methods (e.g., Mask R-CNN, SegNet) to detect dynamic objects and remove outliers. However, as far as we know, such kind of methods wait for the semantic results in the tracking thread in their architecture, and the processing time depends on the segmentation methods used. In this paper, we present RDS-SLAM, a real-time visual dynamic SLAM algorithm that is built on ORB-SLAM3 and adds a semantic thread and a semantic-based optimization thread for robust tracking and mapping in dynamic environments in real-time. These novel threads run in parallel with the others, and therefore the tracking thread does not need to wait for the semantic information anymore. Besides, we propose an algorithm to obtain as the latest semantic information as possible, thereby making it possible to use segmentation methods with different speeds in a uniform way. We update and propagate semantic information using the moving probability, which is saved in the map and used to remove outliers from tracking using a data association algorithm. Finally, we evaluate the tracking accuracy and real-time performance using the public TUM RGB-D datasets and Kinect camera in dynamic indoor scenarios. Source code and demo: https://github.com/yubaoliu/RDS-SLAM.git.

Accurate real-time visual SLAM combining building models and GPS for mobile robot

This paper presents a novel 7 DOF (i.e., orientation, translation, and scale) visual simultaneous localization and mapping (vSLAM) system for mobile robots in outdoor environments. In the front end of this vSLAM system, a fast initialization method is designed for different vSLAM backbones, which upgrades the accuracy of trajectory and reconstruction of vSLAM with an absolute scale computed from depth maps generated by building blocks. In the back end of this vSLAM, we propose a nonlinear optimization mechanism throughout which multimodal data are combined for more robust optimization. The modality of building blocks in optimization can improve the tracking accuracy and the scale estimation. By integrating the pose estimated from visual information and the position received through GPS, the optimization further alleviates the drift. The experimental results prove that the proposed method is extremely suitable for outer AR application for outdoor environments, because our method has superior initialization performance, runs in real time, and achieves real scale, higher accuracy, and robustness.

VPS-SLAM: Visual Planar Semantic SLAM for Aerial Robotic Systems

Indoor environments have abundant presence of high-level semantic information which can provide a better understanding of the environment for robots to improve the uncertainty in their pose estimate. Although semantic information has proved to be useful, there are several challenges faced by the research community to accurately perceive, extract and utilize such semantic information from the environment. In order to address these challenges, in this paper we present a lightweight and real-time visual semantic SLAM framework running on board aerial robotic platforms. This novel method combines low-level visual/visual-inertial odometry (VO/VIO) along with geometrical information corresponding to planar surfaces extracted from detected semantic objects. Extracting the planar surfaces from selected semantic objects provides enhanced robustness and makes it possible to precisely improve the metric estimates rapidly, simultaneously generalizing to several object instances irrespective of their shape and size. Our graph-based approach can integrate several state of the art VO/VIO algorithms along with the state of the art object detectors in order to estimate the complete 6DoF pose of the robot while simultaneously creating a sparse semantic map of the environment. No prior knowledge of the objects is required, which is a significant advantage over other works. We test our approach on a standard RGB-D dataset comparing its performance with the state of the art SLAM algorithms. We also perform several challenging indoor experiments validating our approach in presence of distinct environmental conditions and furthermore test it on board an aerial robot. Video:https://vimeo.com/368217703Released Code:https://bitbucket.org/hridaybavle/semantic_slam.git.

A synthetic dataset for Visual SLAM evaluation

Vision-based self-localization methods are key functionalities for various research topics. Recent research results on related fields have catalyzed several accurate, versatile and reliable real-time Visual SLAM systems suitable for self-localization under a wide variety of environmental preconditions. These methods extend their functionalities from being only a good camera tracker to being able to recursively build up camera’s surroundings. The fast development of Visual SLAM research has proposed demands on innovating evaluation methods for Visual SLAM systems. However, retrieving images and ground truth from various kinds of environments, estimating calibration parameters between several sensors and annotating useful labels all require cumbersome human labor and will introduce inevitable errors. In this paper, we propose a method that uses virtually established models to automatically generate photorealistic images with accurate ground truth and several kinds of pixel-level annotations useful for Visual SLAM development and evaluation. We build and render a challenging dataset in low-texture environments with large scale camera movement, multiple moving objects and varying luminance status. We also propose several new evaluation criteria that can fully take advantage of ground truth and annotations from synthetic datasets. Experiments are conducted using the proposed datasets and criteria with several state-of-the-art Visual SLAM methods to demonstrate the functionality of our datasets.

Online Photometric Calibration of Auto Exposure Video for Realtime Visual Odometry and SLAM

Recent direct visual odometry and SLAM algorithms have demonstrated impressive levels of precision. However, they require a photometric camera calibration in order to achieve competitive results. Hence, the respective algorithm cannot be directly applied to an off-the-shelf-camera or to a video sequence acquired with an unknown camera. In this letter, we propose a method for online photometric calibration that enables to process auto exposure videos with visual odometry precisions that are on par with those of photometrically calibrated videos. Our algorithm recovers the exposure times of consecutive frames, the camera response function, and the attenuation factors of the sensor irradiance due to vignetting. Gain robust Kanade-Lucas-Tomasi (KLT) feature tracks are used to obtain scene point correspondences as input to a nonlinear optimization framework. We show that our approach can reliably calibrate arbitrary video sequences by evaluating it on datasets for which full photometric ground truth is available. We further show that our calibration can improve the performance of a state-of-the-art direct visual odometry method that works solely on pixel intensities, calibrating for photometric parameters in an online fashion in realtime.

A topometric system for wide area augmented reality

We describe a system designed to facilitate efficient communication of information relating to the physical world using augmented reality (AR). We bring together a range of technologies to create a system capable of operating in real-time, over wide areas and for both indoor and outdoor operations. The central concept is to integrate localised mapping and tracking based on real-time visual SLAM with global positioning from both GPS and indoor ultra-wide band (UWB) technology. The former allows accurate and repeatable creation and visualisation of AR annotations within local metric maps, whilst the latter provides a coarse global representation of the topology of the maps. We call this a ‘Topometric System’. The key elements are: robust and efficient vision based tracking and mapping using a Kalman filter framework; rapid and reliable vision based relocalisation of users within local maps; user interaction mechanisms for effective annotation insertion; and an integrated framework for managing and fusing mapping and positioning data. We present the results of experiments conducted over a wide area, with indoor and outdoor operations, which demonstrates successful creation and visualisation of large numbers of AR annotations over a range of different locations.