Visual SLAM Research Articles

FADM-SLAM: a fast and accurate dynamic intelligent motion SLAM for autonomous robot exploration involving movable objects

PurposeMany popular simultaneous localization and mapping (SLAM) techniques have low accuracy, especially when localizing environments containing dynamically moving objects since their presence can potentially cause inaccurate data associations. To address this issue, the proposed FADM-SLAM system aims to improve the accuracy of SLAM techniques in environments containing dynamically moving objects. It uses a pipeline of feature-based approaches accompanied by sparse optical flow and multi-view geometry as constraints to achieve this goal.Design/methodology/approachFADM-SLAM, which works with monocular, stereo and RGB-D sensors, combines an instance segmentation network incorporating an intelligent motion detection strategy (iM) with an optical flow technique to improve location accuracy. The proposed AS-SLAM system comprises four principal modules, which are the optical flow mask and iM, the ego motion estimation, dynamic point detection and the feature-based extraction framework.FindingsExperiment results using the publicly available RGBD-Bonn data set indicate that FADM-SLAM outperforms established visual SLAM systems in highly dynamic conditions.Originality/valueIn summary, the first module generates the indication of dynamic objects by using the optical flow and iM with geometric-wise segmentation, which is then used by the second module to compute the starting point of a posture. The third module, meanwhile, first searches for the dynamic feature points in the environment, and second, eliminates them from further processing. An algorithm based on epipolar constraints is implemented to do this. In this way, only the static feature points are retained, which are then fed to the fourth module for extracting important features.

Continual learning for topological geo-localisation

Geo-localisation from a single aerial image for Uncrewed Aerial Vehicles (UAVs) is an alternative to other vision-based methods, such as visual Simultaneous Localisation and Mapping (SLAM), seeking robustness under GPS failure. Due to the success of deep learning and the fact that UAVs can carry a low-cost camera, we can train a Convolutional Neural Network (CNN) to predict position from a single aerial image. However, conventional CNN-based methods adapted to this problem require off-board training that involves high computational processing time and where the model can not be used in the same flight mission. In this work, we explore the use of continual learning via latent replay to achieve online training with a CNN model that learns during the flight mission GPS coordinates associated with single aerial images. Thus, the learning process repeats the old data with the new ones using fewer images. Furthermore, inspired by the sub-mapping concept in visual SLAM, we propose a multi-model approach to assess the advantages of using compact models learned continuously with promising results. On average, our method achieved a processing speed of 150 fps with an accuracy of 0.71 to 0.85, demonstrating the effectiveness of our methodology for geo-localisation applications.

UW Deep SLAM-CNN Assisted Underwater SLAM

Abstract Underwater simultaneous localization and mapping (SLAM) poses significant challenges for modern visual SLAM systems. The integration of deep learning networks within computer vision offers promising potential for addressing these difficulties. Our research draws inspiration from deep learning approaches applied to interest point detection and matching, single image depth prediction and underwater image enhancement. In response, we propose 3D-Net, a deep learning-assisted network designed to tackle these three tasks simultaneously. The network consists of three branches, each serving a distinct purpose: interest point detection, descriptor generation, and depth prediction. The interest point detector and descriptor generator can effectively serve as a front end for a classical SLAM system. The predicted depth information is akin to a virtual depth camera, opening up possibilities for various applications. We provide quantitative and qualitative evaluations to illustrate some of these potential uses. The network was trained in in several steps, using in-air datasets and followed by generated underwater datasets. Further, the network is integrated into feature-based SALM systems ORBSLAM2 and ORBSSLAM3, providing a comprehensive assessment of its effectiveness for underwater navigation.

Visual Simultaneous Localization and Mapping for Sewer Pipe Networks Leveraging Cylindrical Regularity

This work proposes a novel visual Simultaneous Localisation and Mapping (vSLAM) approach for robots in sewer pipe networks. One problem of vSLAM in pipes is that the scale drifts and accuracy degrades. We propose the use of structural information to mitigate this problem via cylindrical regularity. The main novelty consists of an approach for cylinder detection that is more robust than previous methods in non-smooth sewer pipe environments. Cylindrical regularity is then incorporated into both local bundle adjustment and pose graph optimisation. The approach adopts a minimal cylinder representation with only five parameters, avoiding constraints during the optimisation in vSLAM. A further novelty is that the estimated cylinder is part of the scale drift estimation, which enables a correction to the translation estimate and this further improves the accuracy. The approach, termed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Cylindrical Regularity ORB-SLAM</i> (CRORB), is benchmarked and compared to leading visual SLAM algorithms ORB-SLAM2 and direct sparse odometry (DSO), as well as a vSLAM algorithm with cylindrical regularity developed for gas pipes, using real sewer pipe data and synthetic data generated with the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Gazebo</i> modelling software. The results demonstrate that CRORB improves substantially over the competitors, with a reduction of approximately 70% in error on real data.

Improving RGB-D SLAM accuracy in dynamic environments based on semantic and geometric constraints

Most current visual SLAM systems rely on static environment assumptions. However, in dynamic environments, the presence of dynamic objects can severely impair the performance of visual SLAM systems. To address this issue, we propose an algorithm that combines semantic segmentation and geometric constraints to eliminate the impact of dynamic objects on visual SLAM trajectory accuracy in dynamic indoor environments. First, a missed detection compensation algorithm based on the constant velocity model and region growing algorithm is proposed, which improves the accuracy of semantic or instance segmentation network and realizes accurate segmentation of dynamic objects at the semantic level in the new thread. Second, for the dynamic objects that cannot be recognized by the semantic segmentation network, a geometric constraint model based on the position correlation between feature points is presented to further eliminate the dynamic feature points that do not meet the motion consistency. Thereafter, the proposed method is integrated into the front-end of the ORB-SLAM2 system and evaluated using the RGB-D dataset. Experimental results demonstrate that the proposed method can improve the absolute trajectory accuracy of the ORB-SLAM2 system by an average of 96.90% in high dynamic scenes.

Review of Visual Simultaneous Localization and Mapping Based on Deep Learning

Due to the limitations of LiDAR, such as its high cost, short service life and massive volume, visual sensors with their lightweight and low cost are attracting more and more attention and becoming a research hotspot. As the hardware computation power and deep learning develop by leaps and bounds, new methods and ideas for dealing with visual simultaneous localization and mapping (VSLAM) problems have emerged. This paper systematically reviews the VSLAM methods based on deep learning. We briefly review the development process of VSLAM and introduce its fundamental principles and framework. Then, we focus on the integration of deep learning and VSLAM from three aspects: visual odometry (VO), loop closure detection, and mapping. We summarize and analyze the contribution and weakness of each algorithm in detail. In addition, we also provide a summary of widely used datasets and evaluation metrics. Finally, we discuss the open problems and future directions of combining VSLAM with deep learning.

DEEP LEARNING TO SUPPORT 3D MAPPING CAPABILITIES OF A PORTABLE VSLAM-BASED SYSTEM

Abstract. The use of vision-based localization and mapping techniques, such as visual odometry and SLAM, has become increasingly prevalent in the field of Geomatics, particularly in mobile mapping systems. These methods provide real-time estimation of the 3D scene as well as sensor's position and orientation using images or LiDAR sensors mounted on a moving platform. While visual odometry primarily focuses on the camera's position, SLAM also creates a 3D reconstruction of the environment. Conventional (geometric) and learning-based approaches are used in visual SLAM, with deep learning networks being integrated to perform semantic segmentation, object detection and depth prediction. The goal of this work is to report ongoing developments to extend the GuPho stereo-vision SLAM-based system with deep learning networks for tasks such as crack detection, obstacle detection and depth estimation. Our findings show how a neural network can be coupled to SLAM sequences in order to support 3D mapping application with semantic information.

SIMULATION OF LOW-COST MEMS-LIDAR AND ANALYSIS OF ITS EFFECT ON THE PERFORMANCES OF STATE-OF-THE-ART SLAMS

Abstract. Indoor Unmanned Aerial Vehicles have often been tasked with performing SLAM, and the sensors most used in literature and industry have been cameras. Spanning from stereo to event cameras, visual algorithms have often been the de facto choice for localization. While visual SLAM has reached a high level of accuracy in localization, accurate map reconstruction still proves to be challenging. Meanwhile, LiDAR sensors have been used for years to obtain accurate maps. First in surveying applications and in the past ten years in the automotive sector. The weight, power, and size constraints of most traditional LiDARs have prevented their installation on UAVs for indoor use. MEMS-based LiDARs have already been used in UAVs but had to rely on algorithms designed to deal with their small FOV. Recently, a MEMS-based LiDAR with a wide field of view (360°*59°) and weighing 265 g has sparked interest in its potential for indoor UAV SLAM. We performed an extensive battery of tests in simulation environments to provide a first look into its effect on state-of-the-art SLAM algorithms, highlight which ones can provide the best results, and what improvements may be most beneficial. This paper aims to provide assistance in further research in the field by releasing the tool used for this work.

FROG: A PORTABLE UNDERWATER MOBILE MAPPING SYSTEM

Abstract. Browsing the scientific and professional literature it appears that the concept of mobile mapping underwater is not as common as in ‘terrestrial’ applications. Nevertheless, exploring and mapping the ocean’s depths is a priority challenge for humankind, today more than ever. Radio waves, such as the GNSS or UWB signal, have a very limited transmission underwater, resulting in the absence of an underwater global positioning system. Consequently, the main sounding methods (i.e., depth measuring systems) are based on the fusion of inertial and acoustic sensors, which allow for systematic mapping of vast seafloor areas. However, photogrammetric surveying methods are preferred when high resolution and reliable colour information are essential aspects in the project economy. This class of approaches include visual odometry and visual SLAM (vSLAM), which represent a valid tool for navigation and positioning in GNSS-denied environments, such as underwater. In this paper, we present a portable underwater mobile mapping system, named FROG, which implements a vSLAM based solution to guide the survey according to photogrammetric principles. FROG is built upon the Guided Photogrammetry - GuPho concept and, thanks to its modular design, can be used by a diver or installed on a micro ROV and controlled remotely from a support vessel. In the paper, FROG characteristics will be detailed, and its potentialities demonstrated in real case applications at sea and in lakes.

ASSESSING TERRESTRIAL MMS 3D DATA FOR OUTDOOR MULTI-SCALE MODELLING

Abstract. Mobile Mapping Systems (MMS) have recently benefited from the development of many fusion-based technologies with countless systems development based on cars, drones, trolley, wearable or portable mapping system. The scale of applicating range from the urban to the architectural scale. Recent solution are also based on visual or Lidar SLAM (Simultaneous Localization and Mapping), which substantially takes advantage of environmental features and techniques of continuous co-registration of the clouds, also in case of absence on GNSS positioning measurement. FARO Technology has recently developed the Swift, a fusion-based hybrid solution that integrates the sensors for 3D mapping in a trolley system configuration and recently, an external camera Panocam Theta Z1 is equipping the system enabling the possibility to associate radiometry to the acquired data. The working principle is the exploitation of a system of static and mobile configuration, using the so-called “anchor scans” co-registration as an hybrid intermediate solution between a typical static scan and a profilometers-based MMS point cloud. The co-registered clouds therefore yield a trajectory mode such as SLAM but benefit from the comparable range and density characteristics, according to user-customized settings, of static scans, with a duration of a few seconds per scan and a few minutes overall. In the present research the Swift System is tested in two different context and the assessment are conducted aimed at satisfying both the urban and the architectural scale instances in the direction of improving further evaluations.

PERFORMANCE OF LOCALIZATION BASED ON VISUAL SLAM AND LIDAR POINT CLOUD REGISTRATION

Abstract. Self-driving car technology has become increasingly popular in recent years. Traditionally, these cars rely on GNSS and INS, but limitations in urban environments can affect their effectiveness. GNSS technology has developed to address this issue, but it still has limitations in certain environments. LiDAR technology has emerged as a solution to obtain high-precision point cloud maps and SLAM technology is now used to integrate these maps with sensor data, such as cameras, to achieve precise positioning. This enables visual SLAM to be performed in environments where GNSS signals are blocked. In this research, we aim to use SLAM technology to obtain posterior environmental information and match it with prior high-precision point cloud maps. The research will start with hardware configuration, actual measurement and analysis of SLAM algorithms, and 3D point cloud matching methods. Results and benefits will be analysed to compare the advantages and disadvantages of point cloud matching algorithms and applicable hardware.

Research on Environment Perception System of Quadruped Robots Based on LiDAR and Vision

Due to the high stability and adaptability, quadruped robots are currently highly discussed in the robotics field. To overcome the complicated environment indoor or outdoor, the quadruped robots should be configured with an environment perception system, which mostly contain LiDAR or a vision sensor, and SLAM (Simultaneous Localization and Mapping) is deployed. In this paper, the comparative experimental platforms, including a quadruped robot and a vehicle, with LiDAR and a vision sensor are established firstly. Secondly, a single sensor SLAM, including LiDAR SLAM and Visual SLAM, are investigated separately to highlight their advantages and disadvantages. Then, multi-sensor SLAM based on LiDAR and vision are addressed to improve the environmental perception performance. Thirdly, the improved YOLOv5 (You Only Look Once) by adding ASFF (adaptive spatial feature fusion) is employed to do the image processing of gesture recognition and achieve the human–machine interaction. Finally, the challenge of environment perception system for mobile robot based on comparison between wheeled and legged robots is discussed. This research provides an insight for the environment perception of legged robots.

VILO SLAM: Tightly Coupled Binocular Vision–Inertia SLAM Combined with LiDAR

For the existing visual–inertial SLAM algorithm, when the robot is moving at a constant speed or purely rotating and encounters scenes with insufficient visual features, problems of low accuracy and poor robustness arise. Aiming to solve the problems of low accuracy and robustness of the visual inertial SLAM algorithm, a tightly coupled vision-IMU-2D lidar odometry (VILO) algorithm is proposed. Firstly, low-cost 2D lidar observations and visual–inertial observations are fused in a tightly coupled manner. Secondly, the low-cost 2D lidar odometry model is used to derive the Jacobian matrix of the lidar residual with respect to the state variable to be estimated, and the residual constraint equation of the vision-IMU-2D lidar is constructed. Thirdly, the nonlinear solution method is used to obtain the optimal robot pose, which solves the problem of how to fuse 2D lidar observations with visual–inertial information in a tightly coupled manner. The results show that the algorithm still has reliable pose-estimation accuracy and robustness in many special environments, and the position error and yaw angle error are greatly reduced. Our research improves the accuracy and robustness of the multi-sensor fusion SLAM algorithm.

ULL-SLAM: underwater low-light enhancement for the front-end of visual SLAM

Underwater visual simultaneous localization and mapping (VSLAM), which can provide robot navigation and localization for underwater vehicles, is crucial in underwater exploration. Underwater SLAM is a challenging research topic due to the limitations of underwater vision and error accumulation over long-term operations. When an underwater vehicle goes down, it may inevitably enter a low-light environment. Although artificial light sources could help to some extent, they might also cause non-uniform illumination, which may have an adverse effect on feature point matching. Consequently, the capability of feature point extraction-based visual SLAM systems could only sometimes work. This paper proposes an end-to-end network for SLAM preprocessing in an underwater low-light environment to address this issue. Our model includes a low-light enhancement branch specific with a non-reference loss function, which can achieve low-light image enhancement without requiring paired low-light data. In addition, we design a self-supervised feature point detector and descriptor extraction branch to take advantage of self-supervised learning for feature points and descriptors matching to reduce the re-projection error. Unlike other works, our model does not require pseudo-ground truth. Finally, we design a unique matrix transformation method to improve the feature similarity between two adjacent video frames. Comparative experiments and ablation experiments confirm that the proposed method in this paper could effectively enhance the performance of VSLAM based on feature point extraction in an underwater low-light environment.

Actively-exploring thermography-enabled autonomous robotic system for detecting and registering HVAC thermal leaks

The inefficiency, labor-intensity, and possibility of automation of the current practice of HVAC system inspections, which heavily relies on human inspectors, are being questioned. This paper describes a lightweight and reproducible robotic system dubbed AcTEA-bot, that automatically detects and locates thermal leaks in ceiling environments while self-navigating. AcTEA-bot is designed and built upon three modules, including an active visual SLAM module, a thermal leak detection module, and a thermal leak registration module. To evaluate the effectiveness of AcTEA-bot, a true-to-scale ceiling environment containing five thermal leaks is built, and 30 experiments are conducted where AcTEA-bot starts with different poses and conducts the inspection task. Results demonstrate that thermal leaks of multiple types in ceiling environments can be accurately detected and located.

Increasing the localization accuracy of visual SLAM with semantic segmentation and motion consistency detection in dynamic scenes1

Visual Simultaneous Localization and Mapping (SLAM) is one of the key technologies for intelligent mobile robots. However, most of the existing SLAM algorithms have low localization accuracy in dynamic scenes. Therefore, a visual SLAM algorithm combining semantic segmentation and motion consistency detection is proposed. Firstly, the RGB images are segmented by SegNet network, the prior semantic information is established and the feature points of high-dynamic objects are removed; Secondly, motion consistency detection is carried out, the fundamental matrix is calculated by the improved Random Sample Consistency (RANSAC) algorithm, the abnormal feature points are output by the epipolar geometry method, and the feature points of low-dynamic objects are eliminated by combining the prior semantic information. Thirdly, the static feature points are used for pose estimation. Finally, the proposed algorithm is tested on the TUM dataset, the algorithm in this paper reduces the average RMSE of ORB-SLAM2 by 93.99% in highly dynamic scenes, which show that the algorithm can effectively improve the localization accuracy of the visual SLAM system in dynamic scenes.

Robot positioning and navigation technology is based on Integration of the Global Navigation Satellite System and real-time kinematics

This paper provides a new robot positioning and navigation technique that combines GNSS-RTK and visual SLAM to address the shortcomings of existing single robot positioning and navigation techniques, such as limited adaptation to scenarios and susceptibility to environmental influences. Real-time kinematics (RTK) is a real-time dynamic observation technique based on carrier phase observations, which can provide real-time 3D positioning results of the measurement site in a specified coordinate system, and can be positioned with centimetre-level accuracy, which is suitable for It is suitable for outdoor positioning of mobile robots. In this paper, a dataset is studied to compare and analyse the advantages and disadvantages of different SLAMs in order to determine the most suitable robot positioning and navigation technique for autonomous indoor and outdoor mobility. These sensors were well calibrated and the composition was applied to the RK3588. The results show good performance in indoor-outdoor switching scenarios, obtaining a mapping technique that cannot be achieved with a single visual SLAM, effectively improving the ability to adapt to multiple scenarios and enhancing the operational range of the transportable robot.

BoW3D: Bag of Words for Real-Time Loop Closing in 3D LiDAR SLAM

Loop closing is a fundamental part of simultaneous localization and mapping (SLAM) for autonomous mobile systems. In the field of visual SLAM, bag of words (BoW) has achieved great success in loop closure. The BoW features for loop searching can also be used in the subsequent 6-DoF loop correction. However, for 3D LiDAR SLAM, the state-of-the-art methods may fail to effectively recognize the loop in real time, and usually cannot correct the full 6-DoF loop pose. To address this limitation, we present a novel Bag of Words for real-time loop closing in 3D LiDAR SLAM, called BoW3D. Our method not only efficiently recognizes the revisited loop places, but also corrects the full 6-DoF loop pose in real time. BoW3D builds the bag of words based on the 3D LiDAR feature LinK3D, which is efficient, pose-invariant and can be used for accurate point-to-point matching. We furthermore embed our proposed method into 3D LiDAR odometry system to evaluate loop closing performance. We test our method on public dataset, and compare it against other state-of-the-art algorithms. BoW3D shows better performance in terms of F1 max and extended precision scores on most scenarios. It is noticeable that BoW3D takes an average of 48 ms to recognize and correct the loops on KITTI 00 (includes 4K+ 64-ray LiDAR scans), when executed on a notebook with an Intel Core i7 @2.2 GHz processor. We release the implementation of our method here: https://github.com/YungeCui/BoW3D.

Deep Learning for Visual SLAM: The State-of-the-Art and Future Trends

Visual Simultaneous Localization and Mapping (VSLAM) has been a hot topic of research since the 1990s, first based on traditional computer vision and recognition techniques and later on deep learning models. Although the implementation of VSLAM methods is far from perfect and complete, recent research in deep learning has yielded promising results for applications such as autonomous driving and navigation, service robots, virtual and augmented reality, and pose estimation. The pipeline of traditional VSLAM methods based on classical image processing algorithms consists of six main steps, including initialization (data acquisition), feature extraction, feature matching, pose estimation, map construction, and loop closure. Since 2017, deep learning has changed this approach from individual steps to implementation as a whole. Currently, three ways are developing with varying degrees of integration of deep learning into traditional VSLAM systems: (1) adding auxiliary modules based on deep learning, (2) replacing the original modules of traditional VSLAM with deep learning modules, and (3) replacing the traditional VSLAM system with end-to-end deep neural networks. The first way is the most elaborate and includes multiple algorithms. The other two are in the early stages of development due to complex requirements and criteria. The available datasets with multi-modal data are also of interest. The discussed challenges, advantages, and disadvantages underlie future VSLAM trends, guiding subsequent directions of research.

AHY-SLAM: Toward Faster and More Accurate Visual SLAM in Dynamic Scenes Using Homogenized Feature Extraction and Object Detection Method.

At present, SLAM is widely used in all kinds of dynamic scenes. It is difficult to distinguish dynamic targets in scenes using traditional visual SLAM. In the matching process, dynamic points are incorrectly added to the pose calculation with the camera, resulting in low precision and poor robustness in the pose estimation. This paper proposes a new dynamic scene visual SLAM algorithm based on adaptive threshold homogenized feature extraction and YOLOv5 object detection, named AHY-SLAM. This new method adds three new modules based on ORB-SLAM2: a keyframe selection module, a threshold calculation module, and an object detection module. The optical flow method is used to screen keyframes for each frame input in AHY-SLAM. An adaptive threshold is used to extract feature points for keyframes, and dynamic points are eliminated with YOLOv5. Compared with ORB-SLAM2, AHY-SLAM has significantly improved pose estimation accuracy over multiple dynamic scene sequences in the TUM open dataset, and the absolute pose estimation accuracy can be increased by up to 97%. Compared with other dynamic scene SLAM algorithms, the speed of AHY-SLAM is also significantly improved under a guarantee of acceptable accuracy.