RGB-D Simultaneous Localization And Mapping Research Articles

Real-Time Global Registration for Globally Consistent RGB-D SLAM

Real-time globally consistent camera localization is critical for visual simultaneous localization and mapping (SLAM) applications. Regardless the popularity of high efficient pose graph optimization as a backend in SLAM, its deficiency in accuracy can hardly benefit the reconstruction application. An alternative solution for the sake of high accuracy would be global registration, which minimizes the alignment error of all the corresponding observations, yet suffers from high complexity due to the tremendous observations that need to be considered. In this paper, we start by analyzing the complexity bottleneck of global point cloud registration problem, i.e., each observation (three-dimensional point feature) has to be linearized based on its local coordinate (camera poses), which however is nonlinear and dynamically changing, resulting in extensive computation during optimization. We further prove that such nonlinearity can be decoupled into linear component (feature position) and nonlinear components (camera poses), where the former linear one can be effectively represented by its compact second-order statistics, while the latter nonlinear one merely requires six degrees of freedom for each camera pose. Benefiting from the decoupled representation, the complexity can be significantly reduced without sacrifice in accuracy. Experiments show that the proposed algorithm achieves globally consistent pose estimation in real-time via CPU computing, and owns comparable accuracy as state-of-the-art that use GPU computing, enabling the practical usage of globally consistent RGB-D SLAM on highly computationally constrained devices.

ObjectFusion: An object detection and segmentation framework with RGB-D SLAM and convolutional neural networks

Given the driving advances on CNNs (Convolutional Neural Networks) [1], deep neural networks being deployed for accurate detection and semantic reconstruction in SLAM (Simultaneous Localization and Mapping) has become a trend. However, as far as we know, almost all existing methods focus on design a specific CNN architecture for single task. In this paper, we propose a novel framework which employs a general object detection CNN to fuse with a SLAM system towards obtaining better performances on both detection and semantic segmentation in 3D space. Our approach first use CNN-based detection network to obtain the 2D object proposals which can be used to establish the local target map. We then use the results estimated from SLAM to update the dynamic global target map based on the local target map obtained by CNNs. Finally, we are able to obtain the detection result for the current frame by projecting the global target map into 2D space. On the other hand, we send the estimation results back to SLAM and update the semantic surfel model in SLAM system. Therefore, we can acquire the segmentation result by projecting the updated 3D surfel model into 2D. Our fusion scheme privileges in object detection and segmentation by integrating with SLAM system to preserve the spatial continuity and temporal consistency. Evaluation performances on four datasets demonstrate the effectiveness and robustness of our method.

Incremental 3D Cuboid Modeling with Drift Compensation.

This paper presents a framework of incremental 3D cuboid modeling by using the mapping results of an RGB-D camera based simultaneous localization and mapping (SLAM) system. This framework is useful in accurately creating cuboid CAD models from a point cloud in an online manner. While performing the RGB-D SLAM, planes are incrementally reconstructed from a point cloud in each frame to create a plane map. Then, cuboids are detected in the plane map by analyzing the positional relationships between the planes, such as orthogonality, convexity, and proximity. Finally, the position, pose, and size of a cuboid are determined by computing the intersection of three perpendicular planes. To suppress the false detection of the cuboids, the cuboid shapes are incrementally updated with sequential measurements to check the uncertainty of the cuboids. In addition, the drift error of the SLAM is compensated by the registration of the cuboids. As an application of our framework, an augmented reality-based interactive cuboid modeling system was developed. In the evaluation at cluttered environments, the precision and recall of the cuboid detection were investigated, compared with a batch-based cuboid detection method, so that the advantages of our proposed method were clarified.

A Compatible Framework for RGB-D SLAM in Dynamic Scenes

Localization and mapping in a dynamic scene is a crucial problem for the indoor visual simultaneous localization and mapping (SLAM) system. Most existed visual odometry (VO) or SLAM systems are based on the assumption that the environment is static. The performance of a SLAM system may degenerate when it is operated in a severely dynamic environment. The assumption limits the applications of RGB-D SLAM in the dynamic environment. In this paper, we propose a workflow to segment the objects accurately, which will be marked as the potentially dynamic-object area based on the semantic information. A novel approach for motion detection and removal from the moving camera is introduced. We integrate the semantics-based motion detection and the segmentation approach with an RGB-D SLAM system. To evaluate the effectiveness of the proposed approach, we conduct the experiments on the challenging dynamic sequences of TUM-RGBD datasets. The experimental results suggest that our approach improves the accuracy of localization and outperforms the state-of-the-art dynamic-removal-based SLAM system in both severely dynamic and slightly dynamic scenes.

Real-Time Direct Monocular SLAM With Learning-Based Confidence Estimation

Direct monocular simultaneous localization and mapping (SLAM) methods, for which the image intensity is used for tracking and mapping instead of sparse feature points, have gained in popularity in recent years. However, feature-based methods usually have more accurate camera localization results than most direct methods, though direct methods can work better in a textureless environment. To tackle the localization issue, we develop a novel real-time large-scale direct SLAM model, namely, GCP-SLAM, by integrating the learning-based confidence estimation into the depth fusion and motion tracking optimization. In GCP-SLAM, a random regression forest is trained off-line with pre-defined confidence measures for learning confidence and detecting the ground control points (GCPs). Then, the confidence value along with the selected GCPs is utilized for depth refinement and camera localization. Our proposed method is shown experimentally more reliable in tracking and relocalization than the previous state-of-the-art direct method when compared with feature-based and RGBD SLAMs.

Application of multisensor fusion to develop a personal location and 3D mapping system

With the popularization of indoor pedestrian positioning systems in recent years, many 3D indoor map-building methods and stereoscopic maps for visualization purposes have been developed. This study presents a human-portable 3D simultaneous localization and mapping (SLAM) system that can adapt to various environmental scenarios. Based on RGB-D SLAM and IMU/laser SLAM, we propose a sensor fusion SLAM algorithm that combines the advantages of these two algorithms and fuses Microsoft Kinect, a Hokuyo laser range finder (LRF), and inertial measurement unit (IMU) sensors to implement 3D positioning and mapping. In terms of positioning, the data from the IMU is used to estimate the user’s velocity and attitude. To correct drift from the inertial sensor, the update procedure of the extended Kalman filter in this system mainly depends on displacement, which is estimated from the Kinect sensor. When the displacement estimated from the Kinect sensor is judged as a failure, the walking velocity estimated from the LRF is used to update the extended Kalman filter. Finally, the colored point cloud extracted from the Kinect is used to create dense 3D environmental maps. Experimental results for three different scenarios prove that our proposed algorithm surpasses the other two.

Dense RGB-D SLAM with Multiple Cameras.

A multi-camera dense RGB-D SLAM (simultaneous localization and mapping) system has the potential both to speed up scene reconstruction and to improve localization accuracy, thanks to multiple mounted sensors and an enlarged effective field of view. To effectively tap the potential of the system, two issues must be understood: first, how to calibrate the system where sensors usually shares small or no common field of view to maximally increase the effective field of view; second, how to fuse the location information from different sensors. In this work, a three-Kinect system is reported. For system calibration, two kinds of calibration methods are proposed, one is suitable for system with inertial measurement unit (IMU) using an improved hand–eye calibration method, the other for pure visual SLAM without any other auxiliary sensors. In the RGB-D SLAM stage, we extend and improve a state-of-art single RGB-D SLAM method to multi-camera system. We track the multiple cameras’ poses independently and select the one with the pose minimal-error as the reference pose at each moment to correct other cameras’ poses. To optimize the initial estimated pose, we improve the deformation graph by adding an attribute of device number to distinguish surfels built by different cameras and do deformations according to the device number. We verify the accuracy of our extrinsic calibration methods in the experiment section and show the satisfactory reconstructed models by our multi-camera dense RGB-D SLAM. The RMSE (root-mean-square error) of the lengths measured in our reconstructed mode is 1.55 cm (similar to the state-of-art single camera RGB-D SLAM systems).

RGB-D SLAM in Indoor Environments With STING-Based Plane Feature Extraction

In this paper, the RGB-D camera-based simultaneous localization and mapping (SLAM) of indoor environments is addressed using plane features. The plane parameter space (PPS) is defined for a compact representation of planes in the Cartesian space. The statistical information grid (STING) structure is constructed in the PPS to extract plane features. The plane association graph is developed to determine the correspondences between the plane features from two successive scans. The RGB-D camera pose is directly calculated using the matched plane features if they can provide sufficient constraints for the pose estimation. Otherwise, a novel STING-based scan matching method is developed in the PPS to achieve a full six degrees of freedom camera pose estimation. The proposed method uses only the plane features independent of any other features to estimate the RGB-D camera poses and can thus be suitable for some challenging scenes. The experimental results demonstrate that the proposed plane feature-based RGB-D SLAM can achieve high accuracy and robustness in both on-board and hand-held applications.

Combining Local and Global Optical Flow for RGB-D Point Cloud Alignment

3D scene reconstruction using RGB-D camera-based Simultaneous Localization and Mapping (SLAM) is constantly studied today. KinectFusion, GPU-based real-time 3D scene reconstruction framework, is mainly used for many other algorithms of RGB-D SLAM. One of the main limitation of KinectFusion depends only on geometric information in the camera pose estimation process. In this paper, we utilize both geometric and photometric information for point cloud alignment. To extract photometric information in color image, we combine local and global optical flow method, such as Lucas-Kanade and Horn-Schunck, respectively, and make not only dense but also robust to noise flow field. In experimental results, we show that our method can use dense and accurate photometric information.

RGB-D SLAM in Dynamic Environments Using Static Point Weighting

We propose a real-time depth edge based RGB-D SLAM system for dynamic environment. Our visual odometry method is based on frame-to-keyframe registration, where only depth edge points are used. To reduce the influence of dynamic objects, we propose a static weighting method for edge points in the keyframe. Static weight indicates the likelihood of one point being part of the static environment. This static weight is added into the intensity assisted iterative closest point (IAICP) method to perform the registration task. Furthermore, our method is integrated into a SLAM (Simultaneous Localization and Mapping) system, where an efficient loop closure detection strategy is used. Both our visual odometry method and SLAM system are evaluated with challenging dynamic sequences from the TUM RGB-D dataset. Compared to state-of-the-art methods for dynamic environment, our method reduces the tracking error significantly.

Tele-Operated Mobile Robot for 3D Visual Inspection Utilizing Distributed Operating System Platform

This research examines how humans work with tele-operated unmanned mobile robots to perform interaction to do the inspection in industrial plant area to produce a 3D map for further critical evaluation. This experiment focuses on two parts, the way human-robot doing the remote interactions using robust method and the way robot perceives its surrounding environment as a 3D perspective map. Robot operating system (ROS) as a tool was utilized in the development and implementation of this research which comes up with robust data communication method in the form of messages and topics. RGBD simultaneous localization and mapping (SLAM) performs the visual odometry and mapping function simultaneously to construct 3D map using Kinect sensor. The results showed that the mobile robot-based tele-operated system utilizing RGBD SLAM for 3D inspection task under ROS environment are successful in extending human perspective in term of remote surveillance in the large area of the industrial plant.

Improving RGB-D SLAM in dynamic environments: A motion removal approach

Visual Simultaneous Localization and Mapping (SLAM) based on RGB-D data has developed as a fundamental approach for robot perception over the past decades. There is an extensive literature regarding RGB-D SLAM and its applications. However, most of existing RGB-D SLAM methods assume that the traversed environments are static during the SLAM process. This is because moving objects in dynamic environments can severely degrade the SLAM performance. The static world assumption limits the applications of RGB-D SLAM in dynamic environments. In order to address this problem, we proposed a novel RGB-D data-based motion removal approach and integrated it into the front end of RGB-D SLAM. The motion removal approach acted as a pre-processing stage to filter out data that were associated with moving objects. We conducted experiments using a public RGB-D dataset. The results demonstrated that the proposed motion removal approach was able to effectively improve RGB-D SLAM in various challenging dynamic environments.

A multi-frame graph matching algorithm for low-bandwidth RGB-D SLAM

This paper presents a novel multi-frame graph matching algorithm for reliable partial alignments among point clouds. We use this algorithm to stitch frames for 3D environment reconstruction. The idea is to utilize both descriptor similarity and mutual spatial coherency of features existed in multiple frames to match these frames. The proposed multi-frame matching algorithm can extract coarse correspondence among multiple point clouds more reliably than pairwise matching algorithms, especially when the data are noisy and the overlap is relatively small. When there are insufficient consistent features that appeared in all these frames, our algorithm reduces the number of frames to match to deal with it adaptively. Hence, it is particularly suitable for cost-efficient robotic Simultaneous Localization and Mapping (SLAM). We design a prototype system integrating our matching and reconstruction algorithm on a remotely controlled navigation iRobot, equipped with a Kinect and a Raspberry Pi. Our reconstruction experiments demonstrate the effectiveness of our algorithm and design.

An Overview to Visual Odometry and Visual SLAM: Applications to Mobile Robotics

This paper is intended to pave the way for new researchers in the field of robotics and autonomous systems, particularly those who are interested in robot localization and mapping. We discuss the fundamentals of robot navigation requirements and provide a review of the state of the art techniques that form the bases of established solutions for mobile robots localization and mapping. The topics we discuss range from basic localization techniques such as wheel odometry and dead reckoning, to the more advance Visual Odometry (VO) and Simultaneous Localization and Mapping (SLAM) techniques. We discuss VO in both monocular and stereo vision systems using feature matching/tracking and optical flow techniques. We discuss and compare the basics of most common SLAM methods such as the Extended Kalman Filter SLAM (EKF-SLAM), Particle Filter and the most recent RGB-D SLAM. We also provide techniques that form the building blocks to those methods such as feature extraction (i.e. SIFT, SURF, FAST), feature matching, outlier removal and data association techniques.

Local optimized and scalable frame-to-model SLAM

In recent years, dense visual SLAM (Simultaneous Localization And Mapping) has been proved that it has lots of advantages on the accuracy of pose estimation compared to sparse features due to exploiting the more available information in image data. In this paper, we propose a scalable dense frame-to-model SLAM system based on KinectFusion algorithm. We design an effective active volume shift strategy to extend the tracking range without increment on storage resource. Furthermore, we propose three local optimization methods including motion predicting, weighted ICP (Iterative Closest Point) and multi-modelframe combined estimation to improve the tracking stability and decrease the accumulative error. Note that, as the basis of on-line augmented reality application, our work focus on local optimization rather than global closing loop optimization which is a classical problem in SLAM. At last, we evaluate our optimized approaches on publicly available benchmark datasets, and compare it with the original KinectFusion, a similar method and another two frame-to-frame RGBD SLAM methods. The results indicate our approaches yield a certain degree improvement on the performance of tracking accuracy.

Loop Closing Detection in RGB-D SLAM Combining Appearance and Geometric Constraints.

A kind of multi feature points matching algorithm fusing local geometric constraints is proposed for the purpose of quickly loop closing detection in RGB-D Simultaneous Localization and Mapping (SLAM). The visual feature is encoded with BRAND (binary robust appearance and normals descriptor), which efficiently combines appearance and geometric shape information from RGB-D images. Furthermore, the feature descriptors are stored using the Locality-Sensitive-Hashing (LSH) technique and hierarchical clustering trees are used to search for these binary features. Finally, the algorithm for matching of multi feature points using local geometric constraints is provided, which can effectively reject the possible false closure hypotheses. We demonstrate the efficiency of our algorithms by real-time RGB-D SLAM with loop closing detection in indoor image sequences taken with a handheld Kinect camera and comparative experiments using other algorithms in RTAB-Map dealing with a benchmark dataset.

RGB-D SLAM using vanishing point and door plate information in corridor environment

This paper proposes a novel approach to an RGB-D simultaneous localization and mapping (SLAM) system that uses the vanishing point and door plates in a corridor environment simultaneously for navigation. To increase the stability of the SLAM process in such an environment, the vanishing point and door plates are utilized as landmarks for extended Kalman filter-based SLAM (i.e., EKF SLAM). The vanishing point is a semi-global unique feature usually observed in the corridor frontage, and a door plate has strong signature information (i.e., the room number) for the data association process. Thus, using these types of reliable features as landmarks maintains the stability of the SLAM process. A dense 3D map is represented by an octree structure that includes room number information, which can be useful semantic information. The experimental results showed that the proposed scheme yields a better performance than previous SLAM systems.

Real-time large-scale dense RGB-D SLAM with volumetric fusion

We present a new simultaneous localization and mapping (SLAM) system capable of producing high-quality globally consistent surface reconstructions over hundreds of meters in real time with only a low-cost commodity RGB-D sensor. By using a fused volumetric surface reconstruction we achieve a much higher quality map over what would be achieved using raw RGB-D point clouds. In this paper we highlight three key techniques associated with applying a volumetric fusion-based mapping system to the SLAM problem in real time. First, the use of a GPU-based 3D cyclical buffer trick to efficiently extend dense every-frame volumetric fusion of depth maps to function over an unbounded spatial region. Second, overcoming camera pose estimation limitations in a wide variety of environments by combining both dense geometric and photometric camera pose constraints. Third, efficiently updating the dense map according to place recognition and subsequent loop closure constraints by the use of an ‘as-rigid-as-possible’ space deformation. We present results on a wide variety of aspects of the system and show through evaluation on de facto standard RGB-D benchmarks that our system performs strongly in terms of trajectory estimation, map quality and computational performance in comparison to other state-of-the-art systems.