Appearance-based Mapping Research Articles

Semisupervised Vector Quantization in Visual SLAM Using HGCN

We present a novel vector quantization (VQ) module for the two state-of-the-art long-range simultaneous localization and mapping (SLAM) algorithms. The VQ task in SLAM is generally performed using unsupervised methods. We provide an alternative approach trough embedding a semisupervised hyperbolic graph convolutional neural network (HGCN) in the VQ step of the SLAM processes. The SLAM platforms we have utilized for this purpose are fast appearance-based mapping (FABMAP) and oriented fast and rotated short (ORB), both of which rely on extracting the features of the captured images in their loop closure detection (LCD) module. For the first time, we have considered the space formed by these SURF features, robust image descriptors, as a graph, enabling us to apply an HGCN in the VQ section which results in an improved LCD performance. The HGCN vector quantizes the SURF feature space, leading to a bag-of-word (BoW) representation construction of the images. This representation is subsequently used to determine LCD accuracy and recall. Our approaches in this study are referred to as HGCN-FABMAP and HGCN-ORB. The main advantage of using HGCN in the LCD section is that it scales linearly when the features are accumulated. The benchmarking experiments show the superiority of our methods in terms of both trajectory generation accuracy in small-scale paths and LCD accuracy and recall for large-scale problems.

Real-Time 3D Reconstruction of UAV Acquisition System for the Urban Pipe Based on RTAB-Map

In urban underground projects, such as urban drainage systems, the real-time acquisition and generation of 3D models of pipes can provide an important foundation for pipe safety inspection and maintenance. The simultaneous localization and mapping (SLAM) technique, compared to the traditional structure from motion (SfM) reconstruction technique, offers high real-time performance and improves the efficiency of 3D object reconstruction. Underground pipes are situated in complex environments with unattended individuals and often lack natural lighting. To address this, this paper presents a real-time and cost-effective 3D perception and reconstruction system that utilizes an unmanned aerial vehicle (UAV) equipped with Intel RealSense D435 depth cameras and an artificial light-supplementation device. This system carries out real-time 3D reconstruction of underground pipes using the RTAB-Map (real-time appearance-based mapping) method. RTAB-Map is a graph-based visual SLAM method that combines closed-loop detection and graph optimization algorithms. The unique memory management mechanism of RTAB-Map enables synchronous mapping for multiple sessions during UAV flight. Experimental results demonstrate that the proposed system, based on RTAB-Map, exhibits the robustness, textures, and feasibility for 3D reconstruction of underground pipes.

DOPESLAM: High-Precision ROS-Based Semantic 3D SLAM in a Dynamic Environment.

Recent advancements in deep learning techniques have accelerated the growth of robotic vision systems. One way this technology can be applied is to use a mobile robot to automatically generate a 3D map and identify objects within it. This paper addresses the important challenge of labeling objects and generating 3D maps in a dynamic environment. It explores a solution to this problem by combining Deep Object Pose Estimation (DOPE) with Real-Time Appearance-Based Mapping (RTAB-Map) through means of loose-coupled parallel fusion. DOPE's abilities are enhanced by leveraging its belief map system to filter uncertain key points, which increases precision to ensure that only the best object labels end up on the map. Additionally, DOPE's pipeline is modified to enable shape-based object recognition using depth maps, allowing it to identify objects in complete darkness. Three experiments are performed to find the ideal training dataset, quantify the increased precision, and evaluate the overall performance of the system. The results show that the proposed solution outperforms existing methods in most intended scenarios, such as in unilluminated scenes. The proposed key point filtering technique has demonstrated an improvement in the average inference speed, achieving a speedup of 2.6× and improving the average distance to the ground truth compared to the original DOPE algorithm.

Voice Interaction Recognition Design in Real-Life Scenario Mobile Robot Applications

This paper designed a voice interactive robot system that can conveniently execute assigned service tasks in real-life scenarios. It is equipped without a microphone where users can control the robot with spoken commands; the voice commands are then recognized by a well-trained deep neural network model of automatic speech recognition (ASR), which enables the robot to execute and complete the command based on the navigation of a real-time simultaneous localization and mapping (SLAM) algorithm. The voice interaction recognition model is divided into two parts: (1) speaker separation and (2) ASR. The speaker separation is applied by a deep-learning system consisting of eight convolution layers, one LSTM layer, and two fully connected (FC) layers to separate the speaker’s voice. This model recognizes the speaker’s voice as a referrer that separates and holds the required voiceprint and removes noises from other people’s voiceprints. Its automatic speech recognition uses the novel sandwich-type conformer model with a stack of three layers, and combines convolution and self-attention to capture short-term and long-term interactions. Specifically, it contains a multi-head self-attention module to directly convert the voice data into text for command realization. The RGB-D vision-based camera uses a real-time appearance-based mapping algorithm to create the environment map and replace the localization with a visional odometer to allow the robot to navigate itself. Finally, the proposed ASR model was tested to check if the desired results will be obtained. Performance analysis was applied to determine the robot’s environment isolation and voice recognition abilities. The results showed that the practical robot system successfully completed the interactive service tasks in a real environment. This experiment demonstrates the outstanding performance with other ASR methods and voice control mobile robot systems. It also verified that the designed voice interaction recognition system enables the mobile robot to execute tasks in real-time, showing that it is a convenient way to complete the assigned service applications.

Research on Multi-Sensor Fusion SLAM Algorithm Based on Improved Gmapping

Simultaneous Localization and Mapping (SLAM) is the core technology of the intelligent robot system, and it is also the basis for its autonomous movement. In recent years, it has been found that SLAM using a single sensor has certain limitations, such as Inertial Measurement Unit (IMU) noise and serious drift, and 2D radar can only detect environmental information on the same horizontal plane. In this regard, this paper constructs a multi-sensor back-end fusion SLAM algorithm that combines vision, laser, encoder and IMU information. Experiments have proved that compared with using a single sensor, the application of a multi-sensor fusion system makes the edges of the constructed map clearer and the noise reduced. Aiming at the problem of increased calculation caused by particle degradation and too many particles, this paper improves the Gmappping algorithm, and uses the combination of selective resampling and Kullback-Leibler Distance (KLD) sampling to complete resampling. It has been proved by experiments that compared with the original algorithm of Gmapping, the application of the improved algorithm increases the particle convergence speed by 39.85% in the process of indoor mapping. Aiming at the problems that the traditional loop detection algorithm is easily affected by environmental factors, resulting in low detection accuracy, and the loop detection algorithm based on deep convolutional neural network has a large amount of calculation and takes a long time to detect. The main research of this paper is to apply a deep learning-based loop detection algorithm on the multi-sensor fusion framework, and use the combination of high-dimensional and low-dimensional features of the image for loop detection. This paper uses different algorithms to conduct comparative experiments on the dataset CityCentre. The experimental results show that compared with the traditional algorithms Bag of Words (BoW), AlexNet algorithm, VGG19 algorithm, and ResNet32 algorithm, the accuracy of the algorithm proposed in this paper has increased by 31.26%, 14.21%, 3.05%, and 1.56%, respectively. In addition, the comparison experiment results of SLAM mapping with the original Real-Time Appearance-Based Mapping (RTAB-MAP) algorithm prove that the loop closure detection algorithm based on deep learning proposed in this paper can enable the system to better build a globally consistent map, including more environmental information.

Autonomous vehicle navigation using high-definition maps through CARLA-ROS simulator bridge

Autonomous vehicles (AV) have garnered significant interest in recent years due to its potential for controlling thousands of accidents that happen yearly due to human error. However, AV bring with it very complex and sophisticated requirements and challenges related to extensive testing of the algorithms and hardware in the physical world. The evolution of automotive simulation tools provides an opportunity to fully test and validate AV architectures without the risk of creating hazardous situations in real world. This research demonstrates the application of HD (High Definition) maps in autonomous vehicle navigation using ROS interface integrated with the CARLA (CAR Learning to Act) simulator. The sensor data includes Light Detection and Ranging (LiDAR), RGB-Depth (RGB-D), and vehicle odometry. HD maps play an important role in robustness of the autonomous systems where due to sensor obstruction or weather conditions the vehicle is unable to perceive the information ahead. It also aids the vehicle to sense its environment even outside from the sensor’s field of view. The research is divided into the three fundamental concepts of Simultaneous Localization and Mapping (SLAM) approach that is, (i) mapping, (ii) localization, and (iii) navigation. Two ROS tools are used for mapping, (a) OctoMap mapping, and (ii) Real-Time Appearance-Based Mapping (RTAB-Map). We demonstrate the effectiveness of localization using RTAB-Map and compare actual path, position and orientation to their estimated equivalents. Our results show acceptable error in XY axes and exemplifies the error accumulated in Z axis.

Painting Path Planning for a Painting Robot with a RealSense Depth Sensor

The utilization of stereo cameras in robotic applications is presented in this paper. The use of a stereo depth sensor is a principal step in robotics applications, since it is the first step in sequences of robotic actions where the intent is to detect and extract windows and obstacles that are not meant to be painted from the surrounding wall. A RealSense D435 stereo camera was used for surface recording via a real-time, appearance-based (RTAB) mapping procedure, as well as to navigate the painting robot. Later, wall detection and the obstacle avoidance processes were performed using statistical filtering and a random sample consensus model (RANSAC) algorithm.

Subsurface robotic exploration for geomorphology, astrobiology and mining during MINAR6 campaign, Boulby Mine, UK: part II (Results and Discussion)

AbstractGeomorphological studies of the hidden and protected subsurface environments are crucial to obtain a greater insight into the evolution of planetary landforms, hydrology, climate, geology and mineralogy. From an astrobiological point of view subsurface environments are of interest for their potential habitability as they are local environments that are partially or fully shielded from the high levels of space and solar radiation. Furthermore, in the case of Mars, there is an increasing interest in searching for the presence of past or extant life in its subsurface. These applications make it mandatory to investigate equipment and instrumentation that allow for the study of subsurface geomorphology, as well as organic chemical biomarkers, such as biomolecules, carbon, nitrogen and sulphur isotopes, and other biologically significant minerals and gases. Mines on Earth can be used as analogues to investigate the geomorphology of Martian subsurface environments and perform astrobiology studies. With that goal, we have developed a low-cost, robust, remotely operable subsurface rover called KORE (KOmpact Rover for Exploration). This work illustrates the studies of a terrestrial analogue for the exploration of Mars using KORE during the Mine Analogue Research 6 (MINAR 6) campaign with the low-cost 3D mapping technology InXSpace 3D (In situ3D mapping tool eXploration of space 3D). InXSpace 3D utilizes an RGB-D camera that captures depth information in addition to the RGB data of an image, operating based on the structured light principle capable of providing depth information in mm scale resolution at sub 3 m mapping range. InXSpace 3D is used to capture point clouds of natural and artificial features, thereby obtaining information about geologically relevant structures and also to incorporate them in earth mining safety. We tested two of the dense simultaneous localization and mapping (SLAM) algorithms: Kintinuous and Real-Time Appearance-Based Mapping (RTAB-Map) to check the performance of InXSpace 3D in a dark mine environment. Also, the air accumulation of volatiles such as methane and formaldehyde due to thermogenic and mining process was measured with the environmental station payload on the rover platform, which caters to both astrobiological significance and mine safety. The main conclusions of this work are: (1) a comparison made between the RTAB-Map algorithm and Kintinuous algorithm showed the superiority of Kintinuous algorithm in providing better 3D reconstruction; although RTAB-Map algorithm captured more points than the Kintinuous algorithm in the dark mine environment; (2) a comparison of point cloud images captured with and without lighting conditions had a negligible effect on the surface density of the point clouds; (3) close-range imaging of the polygonal features occurring on the halite walls using InXSpace 3D provided mm-scale resolution to enable further characterization; (4) heuristic algorithms to quickly post-process the 3D point cloud data provided encouraging results for preliminary analyses; (5) we successfully demonstrated the application of KORE to mine safety; and (6) the multi-sensors platform on KORE successfully monitored the accumulated volatiles in the mine atmosphere during its operation. The findings obtained during this KORE campaign could be incorporated in designing and planning future subsurface rover explorations to potential planetary bodies such as Mars with synergistic applications to subsurface environments in mines on Earth.

A Benchmark of Popular Indoor 3D Reconstruction Technologies: Comparison of ARCore and RTAB-Map

The fast evolution in computational and sensor technologies brings previously niche solutions to a wider userbase. As such, 3D reconstruction technologies are reaching new use-cases in scientific and everyday areas where they were not present before. Cost-effective and easy-to-use solutions include camera-based 3D scanning techniques, such as photogrammetry. This paper provides an overview of the available solutions and discusses in detail the depth-image based Real-time Appearance-based Mapping (RTAB-Map) technique as well as a smartphone-based solution that utilises ARCore, the Augmented Reality (AR) framework of Google. To qualitatively compare the two 3D reconstruction technologies, a simple length measurement-based method was applied with a purpose-designed reference object. The captured data were then analysed by a processing algorithm. In addition to the experimental results, specific case studies are briefly discussed, evaluating the applicability based on the capabilities of the technologies. As such, the paper presents the use-case of interior surveying in an automated laboratory as well as an example for using the discussed techniques for landmark surveying. The major findings are that point clouds created with these technologies provide a direction- and shape-accurate model, but those contain mesh continuity errors, and the estimated scale factor has a large standard deviation.

Visual Odometry Implementation and Accuracy Evaluation Based on Real-time Appearance-based Mapping

Visual Odometry Implementation and Accuracy Evaluation Based on Real-time Appearance-based Mapping

Research of visual simultaneous localization and mapping-based navigation system for mobile robots

Subject of Research. The paper considers navigation system for mobile robots by building a map of the environment using Simultaneous Localization and Mapping algorithm and converting the 3D map collected by INTEL Realsense Depth camera into a 2D-cost map. Method. Real-Time Appearance-Based Mapping was used for building a virtual 3D map of the environment. A binary map was obtained by projecting the 3D map on a plane. The D* algorithm was applied on the binary map for planning a global path to the goal. The Dynamic-Window Approach was used as a local planner. Main Results. A point cloud of the environment was created and converted to a 2D map. A robot was safely navigated to the desired location. Practical Relevance. The proposed approach is fast and reliable and can be used for indoor navigation (factories and companies). Since the map needs to be designed only once, the calculation can be handled by CPU without any need for graphics processing unit.

Detecting the Expectancy of a Place Using Nearby Context for Appearance-Based Mapping

In recent years, place recognition techniques have been extensively studied in the domain of robotic mapping, referred to as appearance-based mapping. Nonetheless, the majority of these methods focus the challenges of place recognition in offline or supervised scenarios, which in certain conditions, e.g., unknown environments, is infeasible. In this paper, we address the challenges of online place recognition and demonstrate the general applicability of our approach in versatile environments. To this end, a modified growing self-organizing network of neurons is proposed, which incrementally adapts itself to learn the topology of the perceptual space formed by gist features. Given a query image and the network state at any time instant, the expected activity of the network is estimated using a proposed Bayesian framework, and the current place is categorized as familiar or novel. Exhaustive experiments on 11 challenging sequences signify the strength of our algorithm for a reliable and real-time place recognition on routes as large as 18 km. Compared to many state-of-the-art approaches, our method does not need offline training or environment-specific parameter tuning.

A novel incremental topological mapping using global visual features

Mapping is fundamental in the navigation task of autonomous mobile robots. In appearance-based mapping, the process of detecting visual loop closing determines whether the current observation comes from a previously visited location or a new one. The purpose of this paper is to present a new method of exploring indoor environments by an autonomous mobile robot, as well as building topological maps based on global visual attributes. This method takes advantage of the small size of the GIST descriptors, and the ease of their calculation. We also make use of omnidirectional images to build a single global visual descriptor showing an entire room. Furthermore, in order to handle the problem of a visual loop closing, we have employed a formula that correctly assigns each global descriptor to its location.

Visual Place Recognition for Autonomous Mobile Robots

Place recognition is an essential component of autonomous mobile robot navigation. It is used for loop-closure detection to maintain consistent maps, or to localize the robot along a route, or in kidnapped-robot situations. Camera sensors provide rich visual information for this task. We compare different approaches for visual place recognition: holistic methods (visual compass and warping), signature-based methods (using Fourier coefficients or feature descriptors (able for binary-appearance loop-closure evaluation, ABLE)), and feature-based methods (fast appearance-based mapping, FabMap). As new contributions we investigate whether warping, a successful visual homing method, is suitable for place recognition. In addition, we extend the well-known visual compass to use multiple scale planes, a concept also employed by warping. To achieve tolerance against changing illumination conditions, we examine the NSAD distance measure (normalized sum of absolute differences) on edge-filtered images. To reduce the impact of illumination changes on the distance values, we suggest to compute ratios of image distances to normalize these values to a common range. We test all methods on multiple indoor databases, as well as a small outdoor database, using images with constant or changing illumination conditions. ROC analysis (receiver-operator characteristics) and the metric distance between best-matching image pairs are used as evaluation measures. Most methods perform well under constant illumination conditions, but fail under changing illumination. The visual compass using the NSAD measure on edge-filtered images with multiple scale planes, while being slower than signature methods, performs best in the latter case.

Building Location Models for Visual Place Recognition

This paper deals with the task of appearance-based mapping and place recognition. Previously, the scope of a location generally varied between either using discrete poses or loosely defined sequences of poses, facing problems related to perceptual aliasing and path invariance respectively. Here, we present a unified framework for defining, modelling and recognizing places in a way which is directly related to the underlying structure of features in the environment. A covisibility map of the environment is incrementally maintained over time, where visual landmarks represent nodes in a graph which are connected if seen together. When queried, relevant places are retrieved as clusters from this map, and a novel probabilistic observation model is used to evaluate place recognition. Place retrieval is able to adapt to a given query and also inherently cope with trajectory variations, due to the use of the landmark covisibility structure. In addition, the chosen generative model is developed in a way which is robust to observation errors, mapping errors, perceptual aliasing, and parameter sensitivity. Validation is provided through a variety of tests using real-world datasets, which compare the behaviour of the proposed approach to other representative state-of-the-art methods (namely FAB-MAP and SeqSLAM).

Pre-Driving Needless System for Autonomous Mobile Robots Navigation in Real World Robot Challenge 2013

This paper describes navigation systems for autonomous mobile robots taking part in the real-world Tsukuba Challenge 2013 robot competition. Tsukuba Challenge 2013 enables any information on the route to be collected beforehand and used on the day of the challenge. At the same time, however, autonomous mobile robots should function appropriately in daily human life even in areas where they have never been before. System thus need not capture pre-driving details. We analyzed traverses in complex urban areas without prior environmental information using light detection and ranging (LIDAR). We also determined robot status, such as its position and orientation using the gauss maps derived from LIDAR without gyro sensors. Dead reckoning was combined with wheel odometry and orientation from above. We corrected 2D robot poses by matching electronics maps from the Web. Because drift inevitably causes errors, slippage and failure, etc., our robot also traced waypoints derived beforehand from the same electronics map, so localization is consistent even if we do not drive through an area ahead of time. Trajectory candidates are generated along global planning routes based on these waypoints and an optimal trajectory is selected. Tsukuba Challenge 2013 required that robots find specified human targets indicated by features released on the Web. To find the target correctly without driving in Tsukuba beforehand, we searched for point cloud clusters similar to specified human targets based on predefined features. These point clouds were then projected on the camera image at the time, and we extracted points of interest such as SURF to apply fast appearance-based mapping (FAB-MAP). This enabled us to find specified targets highly accurately. To demonstrate the feasibility of our system, experiments were conducted over our university route and over that in the Tsukuba Challenge.

Semantic labeling for indoor topological mapping using a wearable catadioptric system

An important part of current research on appearance based mapping works towards semantic representations of the environment, which may allow autonomous systems to perform higher level tasks and provide better human–robot interaction. This work presents a new omnidirectional vision based scene labeling approach for augmented indoor topological mapping. Omnidirectional vision systems are of particular interest because they allow us to have more compact and efficient representation of the environment. Our approach includes novel ideas in order to augment the semantic information of a typical indoor topological map: we pay special attention to the semantic labels of the different types of transitions between places, and propose a simple way to include this semantic information as part of the criteria to segment the environment into topological regions. This work is built on efficient catadioptric image representation based on the Gist descriptor, which is used to classify the acquired views into types of indoor regions. The basic types of indoor regions considered are Place and Transition, further divided into more specific subclasses, e.g., Transition into door, stairs and elevator. Besides using the result of this labeling, the proposed mapping approach includes a probabilistic model to account for spatio-temporal consistency. All the proposed ideas have been evaluated in a new indoor dataset also presented in this paper. This dataset has been acquired with our wearable catadioptric vision system,11http://robots.unizar.es/omnicam/. showing promising results in a realistic prototype.

CAT-SLAM: probabilistic localisation and mapping using a continuous appearance-based trajectory

This paper describes a new system, dubbed Continuous Appearance-based Trajectory Simultaneous Localisation and Mapping (CAT-SLAM), which augments sequential appearance-based place recognition with local metric pose filtering to improve the frequency and reliability of appearance-based loop closure. As in other approaches to appearance-based mapping, loop closure is performed without calculating global feature geometry or performing 3D map construction. Loop-closure filtering uses a probabilistic distribution of possible loop closures along the robot’s previous trajectory, which is represented by a linked list of previously visited locations linked by odometric information. Sequential appearance-based place recognition and local metric pose filtering are evaluated simultaneously using a Rao–Blackwellised particle filter, which weights particles based on appearance matching over sequential frames and the similarity of robot motion along the trajectory. The particle filter explicitly models both the likelihood of revisiting previous locations and exploring new locations. A modified resampling scheme counters particle deprivation and allows loop-closure updates to be performed in constant time for a given environment. We compare the performance of CAT-SLAM with FAB-MAP (a state-of-the-art appearance-only SLAM algorithm) using multiple real-world datasets, demonstrating an increase in the number of correct loop closures detected by CAT-SLAM.

Appearance-based mapping and localization for mobile robots using a feature stability histogram

The strength of appearance-based mapping models for mobile robots lies in their ability to represent the environment through high-level image features and to provide human-readable information. However, developing a mapping and a localization method using these kinds of models is very challenging, especially if robots must deal with long-term mapping, localization, navigation, occlusions, and dynamic environments. In other words, the mobile robot has to deal with environmental appearance change, which modifies its representation of the environment. This paper proposes an indoor appearance-based mapping and a localization method for mobile robots based on the human memory model, which was used to build a Feature Stability Histogram (FSH) at each node in the robot topological map. This FSH registers local feature stability over time through a voting scheme, and the most stable features were considered for mapping, for Bayesian localization and for incrementally updating the current appearance reference view in the topological map. The experimental results are presented using an omnidirectional images dataset acquired over the long-term and considering: illumination changes (time of day, different seasons), occlusions, random removal of features, and perceptual aliasing. The results include a comparison with the approach proposed by Dayoub and Duckett (2008) [19] and the popular Bag-of-Words (Bazeille and Filliat, 2010) [35] approach. The obtained results confirm the viability of our method and indicate that it can adapt the internal map representation over time to localize the robot both globally and locally.

Accelerating FAB-MAP With Concentration Inequalities

We outline an approach for using concentration inequalities to perform rapid approximate multi-hypothesis testing. In a scenario where multiple hypotheses are ranked according to a large set of features, our scheme improves the efficiency of selecting the best hypothesis by providing a “bail-out threshold” at which unpromising hypotheses can be excluded from further evaluation. We show how concentration inequalities can be used to derive principled bail-out thresholds, subject to a user-specified error tolerance. The technique is similar to the sequential probability ratio test, but is applicable in more general conditions. We apply the technique to improve the speed of the fast-appearance-based mapping system for appearance-based place recognition and mapping. The speed increase provided by the new approach is data dependent, but we demonstrate speed improvements of between 25x - 50x on real data, with only a slight degradation in accuracy.