3D Range Data Research Articles

Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments

Environment models serve as important resources for an autonomous robot by providing it with the necessary task-relevant information about its habitat. Their use enables robots to perform their tasks more reliably, flexibly, and efficiently. As autonomous robotic platforms get more sophisticated manipulation capabilities, they also need more expressive and comprehensive environment models: for manipulation purposes their models have to include the objects present in the world, together with their position, form, and other aspects, as well as an interpretation of these objects with respect to the robot tasks. The dissertation presented in this article (Rusu, PhD thesis, 2009) proposes Semantic 3D Object Models as a novel representation of the robot’s operating environment that satisfies these requirements and shows how these models can be automatically acquired from dense 3D range data.

A Layered Approach to People Detection in 3D Range Data

People tracking is a key technology for autonomous systems, intelligent cars and social robots operating in populated environments. What makes the task difficult is that the appearance of humans in range data can change drastically as a function of body pose, distance to the sensor, self-occlusion and occlusion by other objects. In this paper we propose a novel approach to pedestrian detection in 3D range data based on supervised learning techniques to create a bank of classifiers for different height levels of the human body. In particular, our approach applies AdaBoost to train a strong classifier from geometrical and statistical features of groups of neighboring points at the same height. In a second step, the AdaBoost classifiers mutually enforce their evidence across different heights by voting into a continuous space. Pedestrians are finally found efficiently by mean-shift search for local maxima in the voting space. Experimental results carried out with 3D laser range data illustrate the robustness and efficiency of our approach even in cluttered urban environments. The learned people detector reaches a classification rate up to 96% from a single 3D scan.

Edge Detecting for Range Data Using Laplacian Operators

Feature extraction in image data has been investigated for many years, and more recently the problem of processing images containing irregularly distributed data has become prominent. Range data are now commonly used in the areas of image processing and computer vision. However, due to the data irregularity found in range images that occurs with a variety of image sensors, direct image processing, in particular edge detection, is a non-trivial problem. Typically, irregular range data would require to be interpolated to a regular grid prior to processing. One example of an edge detection technique than can be directly applied to range images is the scan-line approximation, but this does not employ exact data locations. Therefore, we present novel Laplacian operators that can be applied directly to irregularly distributed data, and in particular we focus on application to irregularly distributed 3D range data for the purpose of edge detection. Within the data distribution framework commonly occurring in range data acquisition devices, our results illustrate that the approach works well over a range of levels of irregularity of data distribution. The use of Laplacian operators on range data is also found to be much less susceptible to noise than the traditional use of Laplacian operators on intensity images.

Robust 3D Plane Estimation for Autonomous Vehicle Applications

A novel plane estimation algorithm from 3D range data is presented. The proposed solution is based on the minimization of a nonlinear prediction error cost function inspired by the mathematical definition of Gibbs' entropy. The method has been experimentally tested and compared with a standard implementation of the RANSAC algorithm. Results suggest that the proposed approach has the potential of performing better in terms of precision and reliability while requiring a lower computational effort.

Multi-Part People Detection Using 2D Range Data

People detection is a key capacity for robotics systems that have to interact with humans. This paper addresses the problem of detecting people using multiple layers of 2D laser range scans. Each layer contains a classifier able to detect a particular body part such as a head, an upper body or a leg. These classifiers are learned using a supervised approach based on AdaBoost. The final person detector is composed of a probabilistic combination of the outputs from the different classifiers. Experimental results with real data demonstrate the effectiveness of our approach to detect persons in indoor environments and its ability to deal with occlusions.

3D mapping with multi-resolution occupied voxel lists

Most current navigation algorithms in mobile robotics produce 2D maps from data provided by 2D sensors. In large part this is due to the availability of suitable 3D sensors and difficulties of managing the large amount of data supplied by 3D sensors. This paper presents a novel, multi-resolution algorithm that aligns 3D range data stored in occupied voxel lists so as to facilitate the construction of 3D maps. Multi-resolution occupied voxel lists (MROL) are voxel based data structures that efficiently represent 3D scan and map information. The process described in this research can align a sequence of scans to produce maps and localise a range sensor within a prior global map. An office environment (200 square metres) is mapped in 3D at 0.02 m resolution, resulting in a 200,000 voxel occupied voxel list. Global localisation within this map, with no prior pose estimate, is completed in 5 seconds on a 2 GHz processor. The MROL based sequential scan matching is compared to a standard iterative closest point (ICP) implementation with an error in the initial pose estimate of plus or minus 1 m and 90 degrees. MROL correctly scan matches 94% of scans to within 0.1 m as opposed to ICP with 30% within 0.1 m.

Robust palmprint verification using 2D and 3D features

This paper presents a new personal authentication system that simultaneously exploits 2D and 3D palmprint features. The objective of our work is to improve accuracy and robustness of existing palmprint authentication systems using 3D palmprint features. The proposed multilevel framework for personal authentication efficiently utilizes the robustness (against spoof attacks) of the 3D features and the high discriminating power of the 2D features. The developed system uses an active stereo technique, structured light, to simultaneously capture 3D image or range data and a registered intensity image of the palm. The surface curvature feature based method is investigated for 3D palmprint feature extraction while Gabor feature based competitive coding scheme is used for 2D representation. We comparatively analyze these representations for their individual performance and attempt to achieve performance improvement using the proposed multilevel matcher that utilizes fixed score level combination scheme to integrate information. Our experiments on a database of 108 subjects achieved significant improvement in performance with the integration of 3D features as compared to the case when 2D palmprint features alone are employed. We also present experimental results to demonstrate that the proposed biometric system is extremely difficult to circumvent, as compared to the currently proposed palmprint authentication approaches in the literature.

Model based vehicle detection and tracking for autonomous urban driving

Situational awareness is crucial for autonomous driving in urban environments. This paper describes the moving vehicle detection and tracking module that we developed for our autonomous driving robot Junior. The robot won second place in the Urban Grand Challenge, an autonomous driving race organized by the U.S. Government in 2007. The module provides reliable detection and tracking of moving vehicles from a high-speed moving platform using laser range finders. Our approach models both dynamic and geometric properties of the tracked vehicles and estimates them using a single Bayes filter per vehicle. We present the notion of motion evidence, which allows us to overcome the low signal-to-noise ratio that arises during rapid detection of moving vehicles in noisy urban environments. Furthermore, we show how to build consistent and efficient 2D representations out of 3D range data and how to detect poorly visible black vehicles. Experimental validation includes the most challenging conditions presented at the Urban Grand Challenge as well as other urban settings.

Automated Online Measurement of Particle Size Distribution using 3D Range Data

Abstract Fully automated online measurement of the size distribution of limestone fragments on conveyor belt is presented based on 3D range data collected every minute during 13 hours of production. The research establishes the necessary measurement technology to facilitate automatic control of particle breaking or aggregating processes to improve both energy efficiency and product quality. Techniques are presented covering; sizing of fragments, determination of non-overlapped and overlapped fragments, and mapping of sizing results to distributions comparable to sieving. Detailed variations in the product sieve size are shown with an abrupt change when the size range of the limestone fragments was changed.

Simultaneous Nonrigid Registration of Multiple Point Sets and Atlas Construction

Groupwise registration of a set of shapes represented by unlabeled point sets is a challenging problem since, usually, this involves solving for point correspondence in a nonrigid motion setting. In this paper, we propose a novel and robust algorithm that is capable of simultaneously computing the mean shape, represented by a probability density function, from multiple unlabeled point sets(represented by finite-mixture models), and registering them nonrigidly to this emerging mean shape. This algorithm avoids the correspondence problem by minimizing the Jensen-Shannon (JS) divergence between the point sets represented as finite mixtures of Gaussian densities. We motivate the use of the JS divergence by pointing out its close relationship to hypothesis testing. Essentially,minimizing the JS divergence is asymptotically equivalent to maximizing the likelihood ratio formed from a probability density of the pooled point sets and the product of the probability densities of the individual point sets. We derive the analytic gradient of the cost function, namely, the JS-divergence, in order to efficiently achieve the optimal solution. The cost function is fully symmetric, with no bias toward any of the given shapes to be registered and whose mean is being sought. A by-product of the registration process is a probabilistic atlas, which is defined as the convex combination of the probability densities of the input point sets being aligned. Our algorithm can be especially useful for creating atlases of various shapes present in images and for simultaneously (rigidly or nonrigidly)registering 3D range data sets (in vision and graphics applications), without having to establish any correspondence. We present experimental results on nonrigidly registering 2D and 3D real and synthetic data (point sets).

Rectification of 3D data obtained from moving range sensor by using extended projective multiple view geometry

To measure the 3D shape of large objects, scanning by a moving range sensor is one of the most efficient methods. However, if we use moving range sensors, the obtained data have some distortions due to the movement of the sensor during the scanning process. In this paper, we propose a method for recovering correct 3D range data from a moving range sensor by using the multiple view geometry under projective projections in space-time. We assume that range sensor radiates laser beams in a raster scan order, and they are observed from two cameras. We first show that we can deal with range data as 2D images, and show that the extended multiple view geometry can be used for representing the relationship between the 2D image of range data and the 2D image of cameras. We next show that the extended multiple view geometry can be used for rectifying 3D data obtained by the moving range sensor. The method is implemented and tested in synthetic images and range data. The stability of the recovered 3D shape is also evaluated.

Three-dimensional data merging using Holoimage

Three-dimensional data merging is vital for full-field three-dimensional (3D) shape measurement. All 3D range data patches, acquired from either different sensors or the same sensor in different viewing angles, have to be merged into a single piece to facilitate future data analysis. A novel method for 3D data merging using Holoimage is proposed. Similar to the 3D shape measurement system using a phase-shifting method, Holoimage is a phase-shifting–based computer synthesized fringe image. The 3D information is retrieved from Holoimage using a phase-shifting method. If two patches of 3D data with overlapping areas are rendered by OpenGL, the overlapping areas are resolved by the graphics pipeline, that is, only the front geometry can be visualized. Therefore, the merging is performed if the front geometry information can be obtained. Holoimage is to obtain the front geometry by projecting the fringe patterns onto the rendered scene. We also demonstrated that each point of the geometry in the overlapping area can be obtained by averaging the corresponding point of the geometries reconstructed from Holoimage for each patch. Moreover, using Holoimage, the texture can also be obtained. Both simulation and experiments demonstrated the success of the proposed method.

Local 3D structure recognition in range images

A feature detector and a feature descriptor are presented, which are applicable to 3D range data. The feature detector is used to identify locations in the range data at which the feature descriptor is applied. The feature descriptor, or feature transform, calculates a signature for each identified location on the basis of local shape information. The approach used in both the feature detector and the descriptor is motivated by the success of the scale invariant feature transform and speeded up robust features approaches in the 2D case. Using synthetic data, the authors evaluate the repeatability of the detector and robustness of the descriptor to global transformations and image noise. The complete system is then applied to the problem of automatic detection of repeated structure in real range images.

Application of the structured illumination method for automated visual inspection of the loudspeaker cones

The purpose of this study is to develop an automated visual inspection of the loudspeaker cones, for determining the shape and concentricity of the cone. The study integrates a visual module, a control module, and a mechanism module into one non-contact 3D automatic measurement system for solving the problems concerning the automated inspection of the cone. The measurement techniques include (1) measuring local feature of the cone using the multi-image to improve repeatability accuracy instead of measuring the whole cone by using only one image for implementation, (2) applying triangulation-based to obtain 3D range data of the cone can be converted into the shape of the cone for obtaining height measurement. The experimental results show that the inspecting system has excellent repeatable accuracy and reliability. The machine vision system is thus able to keep up with the loudspeakers cones production process, can inspect every cone, and reject the defective ones.

Using backpropagation neural network for face recognition with 2D+3D hybrid information

Biometric measurements received an increasing interest for security applications in the last two decades. After the 911 terrorist attacks, face recognition has been an active research in this area. However, very few research group focus on face recognition from both 2D and 3D facial images. Almost all existing recognition systems rely on a single type of face information: 2D intensity (or color) image or 3D range data set [Wang, Y., Chua, C., & Ho, Y. (2002). Facial feature detection and face recognition from 3D and 3D images. Pattern Recognition Letters, 23, 1191-1202]. The objective of this study is to develop an effective face recognition system that extracts and combines 2D and 3D face features to improve the recognition performance. The proposed method derived the information of 3D face (disparity face) using a designed synchronous Hopfield neural network. Then, we retrieved 2D and 3D face features with principle component analysis (PCA) and local autocorrelation coefficient (LAC) respectively. Eventually, the information of features was learned and classified using backpropagation neural networks. An experiment was conducted with 100 subjects, and for each subject thirteen stereo face images were taken with different expressions. Among them, seven faces with expressions were used for training, and the rest of the expressions were used for testing. The experimental results show that the proposed method effectively improved the recognition rate by combining the 2D with 3D face information.

A comparison of line extraction algorithms using 2D range data for indoor mobile robotics

This paper presents an experimental evaluation of different line extraction algorithms applied to 2D laser scans for indoor environments. Six popular algorithms in mobile robotics and computer vision are selected and tested. Real scan data collected from two office environments by using different platforms are used in the experiments in order to evaluate the algorithms. Several comparison criteria are proposed and discussed to highlight the advantages and drawbacks of each algorithm, including speed, complexity, correctness and precision. The results of the algorithms are compared with ground truth using standard statistical methods. An extended case study is performed to further evaluate the algorithms in a SLAM application.

Smart particle filtering for high-dimensional tracking

Tracking articulated structures like a hand or body within a reasonable time is challenging because of the high-dimensionality of the state space. Recently, a new optimization method, called ‘ Stochastic Meta-Descent’ (SMD) has been introduced in computer vision. This is a gradient descent scheme with adaptive and parameter-specific step sizes able to operate in a constrained space. However, while the local optimization works very well, reaching the global optimum is not guaranteed. We therefore propose an enhanced algorithm that wraps a particle filter around multiple SMD-based trackers, which play the role of many particles, i.e. that act as ‘smart particles’. After the standard particle propagation on the basis of a simple motion model, SMD is performed and the resulting new particle set is included such that the original Bayesian distribution is not altered. The resulting ‘ Smart Particle Filter’ (SPF) tracks high-dimensional articulated structures with far fewer samples than previous methods. Additionally, it can handle multiple hypotheses and clutter, where pure optimization approaches have problems. Good performance is demonstrated for the case of hand tracking from 3D range data.

2P1-G03 自律移動ロボットを用いた屋内三次元環境モデリング(Localization and Mapping)

This paper describes a method of generating 3D indoor environment model by a mobile robot with laser range finder. Generally, there are more complicated objects in indoor environment than those in outdoor environment, and it is an issue how we model them. First, we obtain 3D range data by moving the laser range finder which is put on an autonomous mobile robot verticaly. Next, we rearrange the data in order to make it easier to generate planes from the rearranged data. Then we generate planes from data. Finally, we make textures which are suitable for planes from the images of cameras, and map them on the planes.

3D Pose estimation of symmetrical objects of unknown shape

The estimation of the pose of an object is a frequent requirement in computer vision tasks. This paper outlines a method for estimating the pose of an object having a plane of bilateral symmetry but of otherwise unknown shape. The method can operate with a set of 3D range data suffering from self-occlusion, as is obtained from a single viewpoint.

Robust face recognition from 2D and 3D images using structural Hausdorff distance

This paper presents a recognition system that is invariant to both viewing directions and facial expressions. This system is based on both 3D range data as well as 2D grey-level facial images. An irregular 2D mesh labeled by 12 landmarks and a 3D region labeled by four landmarks are defined in each face for feature extraction. Nodes of the 2D mesh are described by Gabor filter responses and 3D points are represented by Point signatures. A subset of mesh nodes (feature nodes), from which discriminating and expression-invariant 2D features can be extracted, is automatically selected for each subject. In the face library, each subject is represented, using both 2D and 3D features, by a frontal face with a neutral facial expression. To classify test faces under varying views or varying facial expressions, a robust Structural Hausdorff Distance is proposed to handle the possible case of matching incomplete data under structural constraints. The best matched model is determined based on the linear integration of matching results in 2D and 3D domains. Good experimental results have been obtained based on our database (involving 80 persons with different facial expressions and viewpoints).