Sparse 3D Map Research Articles

NMC3D: Non-Overlapping Multi-Camera Calibration Based on Sparse 3D Map.

With the advancement of computer vision and sensor technologies, many multi-camera systems are being developed for the control, planning, and other functionalities of unmanned systems or robots. The calibration of multi-camera systems determines the accuracy of their operation. However, calibration of multi-camera systems without overlapping parts is inaccurate. Furthermore, the potential of feature matching points and their spatial extent in calculating the extrinsic parameters of multi-camera systems has not yet been fully realized. To this end, we propose a multi-camera calibration algorithm to solve the problem of the high-precision calibration of multi-camera systems without overlapping parts. The calibration of multi-camera systems is simplified to the problem of solving the transformation relationship of extrinsic parameters using a map constructed by multiple cameras. Firstly, the calibration environment map is constructed by running the SLAM algorithm separately for each camera in the multi-camera system in closed-loop motion. Secondly, uniformly distributed matching points are selected among the similar feature points between the maps. Then, these matching points are used to solve the transformation relationship between the multi-camera external parameters. Finally, the reprojection error is minimized to optimize the extrinsic parameter transformation relationship. We conduct comprehensive experiments in multiple scenarios and provide results of the extrinsic parameters for multiple cameras. The results demonstrate that the proposed method accurately calibrates the extrinsic parameters for multiple cameras, even under conditions where the main camera and auxiliary cameras rotate 180°.

Application and Challenges of Virtual Reality (VR) in Art Exhibition Planning

Abstract The combination of VR technology and art exhibition promotes a new form of art exhibition and enriches the multi-sensory experience of visitors. After analyzing the development form of VR art exhibition and the design process of VR art exhibition, this paper puts forward the solution countermeasures of using the sparse 3D map to optimize the presentation of virtual reality visualization scene, refining the 3D scene model and improving the presentation of the surface scene based on the Possion reconstruction for the lack of development of interactive experience in VR art exhibition. As well as based on the dimensions of interactive experiences, Kinect somatosensory interaction technology is used to update the interactive experience design in VR art exhibition planning. Explore the optimization effect of VR art exhibition planning, analyze the visitors’ interactive experience task completion time, participation, and interactive experience degree, and put forward effective countermeasures for the future development of VR art exhibitions. The mean value of scores of VR exhibitions and offline exhibitions are 5.887 and 5.657, respectively, with a difference of 0.23 and a total score difference of 2.07, which indicates that the art exhibition constructed on the basis of VR has a better sense of experience.

Toward real-time and accurate dense 3D mapping of crop fields for combine harvesters using a stereo camera.

Fast and accurate 3D scene perception is a crucial prerequisite for the autonomous navigation and harvesting of combine harvesters. However, crop field scenarios pose severe challenges for vision-based perception systems due to repetitive scenes, illumination changes and real-time constraints on embedded computing platforms. In this paper, we propose a feature-based, two-stage approach for real-time dense 3D mapping for combine harvesters. In the first stage, our approach constructs a sparse 3D map using reliable feature matching, which provides prior knowledge about the environment. In the second stage, our method formulates per-pixel disparity calculation as probabilistic inference. The key to our approach is the ability to compute dense 3D maps by combining Bayesian estimation with efficient and discriminative point cues from images, exhibiting tolerance against visual measurement uncertainties due to repetitive textures and uneven lighting in crop fields. We validate the performance of the proposed method using real crop field data, and the results demonstrate that our dense 3D maps provide detailed spatial metric information while maintaining a balance between accuracy and efficiency. This makes our approach highly valuable for online perception in combine harvesters operating with resource-limited systems.

Semi-Direct Multimap SLAM System for Real-Time Sparse 3-D Map Reconstruction

The lack of a sufficient number of reliable corners in low-textured environments is a big challenge for classical visual simultaneous localization and mapping (SLAM), especially for point feature-based methods. Many other features (i.e., line and plane segments) are often combined with points to restore an environmental structure. However, using such features requires much computational time. This work focuses on the reliable and high-performance real-time operation of a SLAM system in low-textured scenarios. It proposes a semi-direct multi-map monocular SLAM system (SM-SLAM) that combines direct tracking and feature-based map maintenance with point features and line segments. The proposed system tracks non-keyframes based on a sparse image alignment method for fast tracking, extracts and matches point features and line segments in keyframes for high-quality environment structure and motion optimization. We present an extensive evaluation on two widely-used public datasets and some challenging real-world scenarios. Experimental results show that SM-SLAM can well reconstruct a sparse 3D map with geometrical structure information in 30-40 Hz. Notably, SM-SLAM shows an accuracy improvement of more than 20% than the Oriented FAST and Rotated BRIEF feature-based SLAM (ORB-SLAM) on some low-speed, small-range camera motion datasets and performs well in low-texture scenarios.

Keyframe-Based Camera Relocalization Method Using Landmark and Keypoint Matching

Camera relocalization is a challenging task, especially based on the sparse 3D map or keyframes. In this paper, we present an accurate method for RGB camera relocalization in the case of a very sparse 3D map built by limited keyframes. The core of our approach is a top-to-down feature matching strategy to provide a set of accurate 2D-to-3D matches. Specifically, we first use the landmark-based place recognition method to generate from keyframes the images similar to the current view along with the set of pairwise matched landmarks. This step constrains the 3D model points that can be matched with the current view. Then, the points are matched within the landmark pairs and combined afterward. This is in contrast to the conventional methods of feature matching that typically match points between the entire images and the whole 3D map, which, as a result, may not be robust to large viewpoint changes, the main challenge of the relocalization based on the sparse map. After feature matching, the camera pose is calculated by an efficient novel Perspective-n-Points (PnP) algorithm. We conduct experiments on challenging datasets to demonstrate that the camera poses estimated by our method based on the sparse 3D point cloud are more accurate than the classical methods using the dense map or a large number of training images.

Real-time monocular 3D perception with ORB-Features

PurposeThis paper aims to propose a new solution for real-time 3D perception with monocular camera. Most of the industrial robots’ solutions use active sensors to acquire 3D structure information, which limit their applications to indoor scenarios. By only using monocular camera, some state of art method provides up-to-scale 3D structure information, but scale information of corresponding objects is uncertain.Design/methodology/approachFirst, high-accuracy and scale-informed camera pose and sparse 3D map are provided by leveraging ORB-SLAM and marker. Second, for each frame captured by a camera, a specially designed depth estimation pipeline is used to compute corresponding 3D structure called depth map in real-time. Finally, depth map is integrated into volumetric scene model. A feedback module has been designed for users to visualize intermediate scene surface in real-time.FindingsThe system provides more robust tracking performance and compelling results. The implementation runs near 25 Hz on mainstream laptop based on parallel computation technique.Originality/valueA new solution for 3D perception is using monocular camera by leveraging ORB-SLAM systems. Results in our system are visually comparable to active sensor systems such as elastic fusion in small scenes. The system is also both efficient and easy to implement, and algorithms and specific configurations involved are introduced in detail.

Real-time enhancement of sparse 3D maps using a parallel segmentation scheme based on superpixels

In this work, we focus on the problem of feature-based 3D mapping of environments with large textureless regions, which generates sparse 3D maps that may not represent well the mapped scene. To deal with this problem, based on our previous work (Cruz Martinez et al. in 2016 IEEE international symposium on mixed and augmented reality (ISMAR) adjunct proceedings, IEEE, 2016), we propose to enhance sparse 3D maps by using a superpixel-based segmentation with the aim of generating denser 3D maps of the scene in real time. Superpixels are middle-level features, which represent similar regions in an image, which can be connected in order to segment textureless areas. We propose a graphics processor unit architecture for (1) superpixel extraction considering chromatic and depth information, (2) superpixel-based segmentation, generation of connectivity matrix to compute the connected components algorithm and (3) mapping of segmented regions to 3D points. We use the ORB-SLAM system (Mur-Artal et al. in IEEE Trans Robot 31(5):1147–1163, 2015) to generate a sparse 3D map and to project the textureless segments onto it at 27 frames per second. We assessed our approach in terms of segmentation and map quality. Regarding the latter, covered area by the generated map, depth accuracy, and computational performance are reported.

A Real-Time Augmented Reality System to See-Through Cars.

One of the most hazardous driving scenario is the overtaking of a slower vehicle, indeed, in this case the front vehicle (being overtaken) can occlude an important part of the field of view of the rear vehicle's driver. This lack of visibility is the most probable cause of accidents in this context. Recent research works tend to prove that augmented reality applied to assisted driving can significantly reduce the risk of accidents. In this paper, we present a real-time marker-less system to see through cars. For this purpose, two cars are equipped with cameras and an appropriate wireless communication system. The stereo vision system mounted on the front car allows to create a sparse 3D map of the environment where the rear car can be localized. Using this inter-car pose estimation, a synthetic image is generated to overcome the occlusion and to create a seamless see-through effect which preserves the structure of the scene.

MAPPING THE STELLAR STRUCTURE OF THE MILKY WAY THICK DISK AND HALO USING SEGUE PHOTOMETRY

We map the stellar structure of the Galactic thick disk and halo by applying color-magnitude diagram (CMD) fitting to photometric data from the SEGUE survey, allowing, for the first time, a comprehensive analysis of their structure at both high and low latitudes using uniform SDSS photometry. Incorporating photometry of all relevant stars simultaneously, CMD fitting bypasses the need to choose single tracer populations. Using old stellar populations of differing metallicities as templates we obtain a sparse 3D map of the stellar mass distribution at |Z|>1 kpc. Fitting a smooth Milky Way model comprising exponential thin and thick disks and an axisymmetric power-law halo allows us to constrain the structural parameters of the thick disk and halo. The thick-disk scale height and length are well constrained at 0.75+-0.07 kpc and 4.1+-0.4 kpc, respectively. We find a stellar halo flattening within ~25 kpc of c/a=0.88+-0.03 and a power-law index of 2.75+-0.07 (for 7<R_{GC}<~30 kpc). The model fits yield thick-disk and stellar halo densities at the solar location of rho_{thick,sun}=10^{-2.3+-0.1} M_sun pc^{-3} and rho_{halo,sun}=10^{-4.20+-0.05} M_sun pc^{-3}, averaging over any substructures. Our analysis provides the first clear in situ evidence for a radial metallicity gradient in the Milky Way's stellar halo: within R<~15 kpc the stellar halo has a mean metallicity of [Fe/H]=-1.6, which shifts to [Fe/H]=-2.2 at larger radii. Subtraction of the best-fit smooth and symmetric model from the overall density maps reveals a wealth of substructures at all latitudes, some attributable to known streams and overdensities, and some new. A simple warp cannot account for the low latitude substructure, as overdensities occur simultaneously above and below the Galactic plane. (abridged)