Time-of-flight Camera Research Articles

SR-4000 and CamCube3.0 Time of Flight (ToF) Cameras: Tests and Comparison

In this paper experimental comparisons between two Time-of-Flight (ToF) cameras are reported in order to test their performance and to give some procedures for testing data delivered by this kind of technology. In particular, the SR-4000 camera by Mesa Imaging AG and the CamCube3.0 by PMD Technologies have been evaluated since they have good performances and are well known to researchers dealing with Time-of-Flight (ToF) cameras. After a brief overview of commercial ToF cameras available on the market and the main specifications of the tested devices, two topics are presented in this paper. First, the influence of camera warm-up on distance measurement is analyzed: a warm-up of 40 minutes is suggested to obtain the measurement stability, especially in the case of the CamCube3.0 camera, that exhibits distance measurement variations of several centimeters. Secondly, the variation of distance measurement precision variation over integration time is presented: distance measurement precisions of some millimeters are obtained in both cases. Finally, a comparison between the two cameras based on the experiments and some information about future work on evaluation of sunlight influence on distance measurements are reported.

Using GNG to improve 3D feature extraction—Application to 6DoF egomotion

Several recent works deal with 3D data in mobile robotic problems, e.g. mapping or egomotion. Data comes from any kind of sensor such as stereo vision systems, time of flight cameras or 3D lasers, providing a huge amount of unorganized 3D data. In this paper, we describe an efficient method to build complete 3D models from a Growing Neural Gas (GNG). The GNG is applied to the 3D raw data and it reduces both the subjacent error and the number of points, keeping the topology of the 3D data. The GNG output is then used in a 3D feature extraction method. We have performed a deep study in which we quantitatively show that the use of GNG improves the 3D feature extraction method. We also show that our method can be applied to any kind of 3D data. The 3D features obtained are used as input in an Iterative Closest Point (ICP)-like method to compute the 6DoF movement performed by a mobile robot. A comparison with standard ICP is performed, showing that the use of GNG improves the results. Final results of 3D mapping from the egomotion calculated are also shown.

Non-visual 2D representation of obstacles

Mobility of blind and visually impaired people always suffers from various hindrances while walking indoor and outdoor, due to unaware distributions and properties of obstacles. In our research, we propose to use a 3D TOF camera to detect the surroundings, which not only obtains precise distance and orientation of obstacles, but also analyzes more properties (e.g. size, hanging height) through a 3D segmentation algorithm. Furthermore, a novel touch-sensitive tactile display is employed to present the 3D spatial scene into a 2D perspective. The non-visual representation let visually impaired users sense the distribution of obstacles precisely and explicitly, as well as the surroundings environments. An evaluation plan has been addressed to compare the effectiveness of audio and tactile representation, and investigate the feasibility of extending the while cane.

Range images of time-of-flight cameras enhancement

Range-images of the Time-of-Flight (TOF) camera are affected by errors owing to the scattered light inside the camera body (flare light) and by the indirect (diffuse) light. The 3D images are distorted because the errors of the range-image depend on the surfaces’ reflectivity. The presented algorithm, based on a modified TOF camera, computes a correction vector that corresponds to the average effect of the diffuse and flare light. The enhanced range-image is obtained by subtracting this vector from the recorded vector image. In this manner, the range-image distortions in the dark regions are reduced.

Real-Time Distance-Dependent Mapping for a Hybrid ToF Multi-Camera Rig

We propose a real-time mapping procedure for data matching to deal with hybrid time-of-flight (ToF) multi-camera rig data fusion. Our approach takes advantage of the depth information provided by the ToF camera to calculate the distance-dependent disparity between the two cameras that constitute the system. As a consequence, the not co-centric binocular system behaves as a co-centric system with co-linear optical axes between their sensors. The association between mapped and non-mapped image coordinates can be described by a set of look-up tables. This, in turn, reduces the complexity of the whole process to a simple indexing step, and thus, performs in real-time. The experimental results show that in addition to being straightforward and easy to compute, our proposed data matching approach is highly accurate which facilitates further fusion operations.

제스처 인식을 활용한 모바일 게임 제어

Mobile game have an advantage of mobility, portability, and simple interface. These advantages are useful for gesture recognition based game which should not have much content quantity and complex interface. This paper suggests gesture recognition based mobile game content with user movement could be applied directly to the mobile game wherever recognition system is equipped. Gesture is recognized by obtaining user area in image from the depth image of TOF camera and going through SVM(Support Vectorn Machine) using EOH(Edge Of Histogram) features of user area. And we confirmed that gesture recognition can be utilized to user input of mobile game content. Proposed technique can be applied to a variety of content, but this paper shows a simple way of game contents which is consisted of moving and jumping newly.

New Sensors for Cultural Heritage Metric Survey: The ToF Cameras

ToF cameras are new instruments based on CCD/CMOS sensors which measure distances instead of radiometry. The resulting point clouds show the same properties (both in terms of accuracy and resolution) of the point clouds acquired by means of traditional LiDAR devices. ToF cameras are cheap instruments (less than 10.000 €) based on video real time distance measurements and can represent an interesting alternative to the more expensive LiDAR instruments. In addition, the limited weight and dimensions of ToF cameras allow a reduction of some practical problems such as transportation and on-site management. Most of the commercial ToF cameras use the phase-shift method to measure distances. Due to the use of only one wavelength, most of them have limited range of application (usually about 5 or 10 m). After a brief description of the main characteristics of these instruments, this paper explains and comments the results of the first experimental applications of ToF cameras in Cultural Heritage 3D metric survey. The possibility to acquire more than 30 frames/s and future developments of these devices in terms of use of more than one wavelength to overcome the ambiguity problem allow to foresee new interesting applications.

Exploiting sparsity in time-of-flight range acquisition using a single time-resolved sensor

Range acquisition systems such as light detection and ranging (LIDAR) and time-of-flight (TOF) cameras operate by measuring the time difference of arrival between a transmitted pulse and the scene reflection. We introduce the design of a range acquisition system for acquiring depth maps of piecewise-planar scenes with high spatial resolution using a single, omnidirectional, time-resolved photodetector and no scanning components. In our experiment, we reconstructed 64 × 64-pixel depth maps of scenes comprising two to four planar shapes using only 205 spatially-patterned, femtosecond illuminations of the scene. The reconstruction uses parametric signal modeling to recover a set of depths present in the scene. Then, a convex optimization that exploits sparsity of the Laplacian of the depth map of a typical scene determines correspondences between spatial positions and depths. In contrast with 2D laser scanning used in LIDAR systems and low-resolution 2D sensor arrays used in TOF cameras, our experiment demonstrates that it is possible to build a non-scanning range acquisition system with high spatial resolution using only a standard, low-cost photodetector and a spatial light modulator.

Range Camera Self-Calibration Based on Integrated Bundle Adjustment via Joint Setup with a 2D Digital Camera

Time-of-flight cameras, based on Photonic Mixer Device (PMD) technology, are capable of measuring distances to objects at high frame rates, however, the measured ranges and the intensity data contain systematic errors that need to be corrected. In this paper, a new integrated range camera self-calibration method via joint setup with a digital (RGB) camera is presented. This method can simultaneously estimate the systematic range error parameters as well as the interior and external orientation parameters of the camera. The calibration approach is based on photogrammetric bundle adjustment of observation equations originating from collinearity condition and a range errors model. Addition of a digital camera to the calibration process overcomes the limitations of small field of view and low pixel resolution of the range camera. The tests are performed on a dataset captured by a PMD[vision]-O3 camera from a multi-resolution test field of high contrast targets. An average improvement of 83% in RMS of range error and 72% in RMS of coordinate residual, over that achieved with basic calibration, was realized in an independent accuracy assessment. Our proposed calibration method also achieved 25% and 36% improvement on RMS of range error and coordinate residual, respectively, over that obtained by integrated calibration of the single PMD camera.

Lock-in Time-of-Flight (ToF) Cameras: A Survey

This paper reviews the state-of-the art in the field of lock-in time-of-flight (ToF) cameras, their advantages, their limitations, the existing calibration methods, and the way they are being used, sometimes in combination with other sensors. Even though lock-in ToF cameras provide neither higher resolution nor larger ambiguity-free range compared to other range map estimation systems, advantages such as registered depth and intensity data at a high frame rate, compact design, low weight, and reduced power consumption have motivated their increasing usage in several research areas, such as computer graphics, machine vision, and robotics.

Autonomous grasp and manipulation planning using a ToF camera

A time-of-flight camera can help a service robot to sense its 3D environment. In this paper, we introduce our methods for sensor calibration and 3D data segmentation to use it to automatically plan grasps and manipulation actions for a service robot. Impedance control is intensively used to further compensate the modeling error and to apply the computed forces. The methods are further demonstrated in three service robotic applications. Sensor-based motion planning allows the robot to move within dynamic and cluttered environment without collision. Unknown objects can be detected and grasped. In the autonomous ice cream serving scenario, the robot captures the surface of ice cream and plans a manipulation trajectory to scoop a portion of ice cream.

3D scene capturing using stereoscopic cameras and a time-of-flight camera

Since advanced 3D video systems employ multiview generation techniques using depth data to provide more realistic and comfortable depth impression, many research works are related to depth generation exploiting a time-offlight (TOF) camera. Most of them use time-consuming depth enhancement methods since the TOF camera generates noisy and unstable depth maps. In this paper, a 3D scene capturing method using stereoscopic cameras and one TOF camera is presented. The proposed method generates two viewpoints' depth videos corresponding to those of color cameras using depth up-converting and 3D warping techniques instead of using complex and time-consuming depth enhancement processes. In addition, the proposed method is implemented with the multi-thread technique to develop a fast system. Experimental results show that the proposed method generates two viewpoints' depth videos at 15 fps and those are used for generating multi-view images.

Statistical analysis of signal measurement in time-of-flight cameras

Three-dimensional imaging systems have evolved significantly in the last two decades due to increase in demand for tasks in the field of close range photogrammetry. The fast and growing need of 3D imaging devices has given rise to range image technology, especially time-of-flight (TOF) cameras, that provide direct measurement of distance between the camera and the targeted surface. A significant advantage of TOF devices over traditional range data sensors is their capability to provide frame rate range data over a full image array. In phase shift TOF cameras, phase shift sampling of the received signal is used to measure amplitude, phase and the offset (intensity) of the received signal. As a result, the quality of the measurement of these sensors depends heavily on signal-to-noise (SNR) of the incoming signal and the subsequent processing algorithms. A detailed understanding of the statistical distributions of the measurement parameters is crucial for accurate distance measurement analysis especially in low SNR scenarios. In this paper, we provide explicit noise models for the three parameters of amplitude, phase and intensity. The proposed stochastic model helps in investigating the effect of noise on signal and classifying range data reliability in TOF cameras. The model is used for prediction of errors in a TOF camera under various SNR conditions. Experimental verification confirms the validity of the model using real data for range error classification under different noise conditions.

MITK-ToF—Range data within MITK

The time-of-flight (ToF) technique is an emerging technique for rapidly acquiring distance information and is becoming increasingly popular for intra-operative surface acquisition. Using the ToF technique as an intra-operative imaging modality requires seamless integration into the clinical workflow. We thus aim to integrate ToF support in an existing framework for medical image processing. MITK-ToF was implemented as an extension of the open-source C++ Medical Imaging Interaction Toolkit (MITK) and provides the basic functionality needed for rapid prototyping and development of image-guided therapy (IGT) applications that utilize range data for intra-operative surface acquisition. This framework was designed with a module-based architecture separating the hardware-dependent image acquisition task from the processing of the range data. The first version of MITK-ToF has been released as an open-source toolkit and supports several ToF cameras and basic processing algorithms. The toolkit, a sample application, and a tutorial are available from http://mitk.org. With the increased popularity of time-of-flight cameras for intra-operative surface acquisition, integration of range data supports into medical image processing toolkits such as MITK is a necessary step. Handling acquisition of range data from different cameras and processing of the data requires the establishment and use of software design principles that emphasize flexibility, extendibility, robustness, performance, and portability. The open-source toolkit MITK-ToF satisfies these requirements for the image-guided therapy community and was already used in several research projects.

Learning Sparse Representations of Depth

This paper introduces a new method for learning and inferring sparse representations of depth (disparity) maps. The proposed algorithm relaxes the usual assumption of the stationary noise model in sparse coding. This enables learning from data corrupted with spatially varying noise or uncertainty, typically obtained by laser range scanners or structured light depth cameras. Sparse representations are learned from the Middlebury database disparity maps and then exploited in a two-layer graphical model for inferring depth from stereo, by including a sparsity prior on the learned features. Since they capture higher-order dependencies in the depth structure, these priors can complement smoothness priors commonly used in depth inference based on Markov Random Field (MRF) models. Inference on the proposed graph is achieved using an alternating iterative optimization technique, where the first layer is solved using an existing MRF-based stereo matching algorithm, then held fixed as the second layer is solved using the proposed non-stationary sparse coding algorithm. This leads to a general method for improving solutions of state of the art MRF-based depth estimation algorithms. Our experimental results first show that depth inference using learned representations leads to state of the art denoising of depth maps obtained from laser range scanners and a time of flight camera. Furthermore, we show that adding sparse priors improves the results of two depth estimation methods: the classical graph cut algorithm by Boykov et al. and the more recent algorithm of Woodford et al.

3D registration of point clouds from time-of-flight cameras

A novel approach for aligning two 3D point clouds acquired by time-of-flight (TOF) cameras is presented. Given that the density of 3D data of these devices is not high, the alignment will be carried out using descriptors based on a direct encoding of the environment of a set of previously selected points-of-interest. This will allow the design of a similarity measure that behaves correctly when both the density of the point clouds and the overlap between them are low. Furthermore, this combination of descriptor and similarity measure facilitates the alignment of point clouds from devices with different accuracy and resolution (such as a laser scanner and a TOF camera).

The Attentive Co-Pilot: Towards a Proactive Biologically-Inspired Advanced Driver Assistance System

Driver assistance functionalities on the market are getting more and more sophisticated, which will lead to integrated systems that fuse the data of multiple sensors (e.g., camera, Photonic Mixer Device, Radar) and internal system percepts (e.g., detected objects and their states, detected road). One important future challenge will be to find smart solutions in system design that allow an efficient control of said integrated systems. A promising way for achieving this is to get inspiration from known signal-processing principles in the human brain. The contribution presents a biologically motivated Advanced Driver Assistance System (ADAS) that uses the generic principle of attention as common front-end of all visual processes. Based on the attention principle an early task-dependent pre-selection of interesting image regions is done, which decreases the scene complexity. Furthermore, internal information fusion increases the system performance (e.g., the attention is used to improve the object tracking, road-detection results can improve the attention). Based on streams of a challenging traffic scenario it is shown, how the system builds up and verifies its environment-related expectations relying on the attention principle. The ADAS is controlled by a central behavior control module that tunes sub-modules and parameters. The behavior control module has a simple structure, but still allows for robustly performing various tasks, since the complexity is distributed over the system in form of local control loops mimicking human cognition aspects.

Get Out of the Way – Obstacle Avoidance and Learning by Demonstration for Manipulation

AbstractHumans acquire manipulation skills by trial and error within a few trials, whereas programming a robot to perform the same task requires robotic expertise and effort. This paper presents a robot which learns a movement from demonstrations with the ability to generalize the movement to new goal poses and avoid the collision with obstacles in the workspace. The general movement is represented by dynamic movement primitives (DMP) augmented by potential fields in order to modulate the motion in the presence of obstacles. The approach is validated in experiments with a robotic arm in which dynamic obstacles partially blocking the movement are detected by a Photonic-Mixer-Devices (PMD) camera.

Fusing Time-of-Flight Cameras and Inertial Measurement Units for Ego-Motion Estimation

This paper presents an approach for ego-motion estimation based on a Time-of-Flight (ToF) camera. It is applied in real-time on a mobile robot platform during mission.The proposed method utilizes the coherence of depth and reflectance data of ToF cameras by detecting image features on reflectance data and estimating the motion on related depth data. This motion estimate is fused with data from an inertial measurement unit in order to gain higher accuracy and robustness, especially in situations when image registration fails.Results are benchmarked against reference poses from an accurate laser ranger finder-based localization.

Model-based spacecraft pose estimation and motion prediction using a photonic mixer device camera

Determination of relative distances and orientations, as well as motion identification is essential in rendezvous and docking and formation flying tasks. A 3D-image generated by a PMD (photonic mixer device) camera, which employs a phase shift measurement of emitted modulated light to derive distance and color information on each pixel, provides in this contribution the basis for near range motion detection and prediction. A novel algorithm based on rotation- and scale-invariant features commonly used for scan matching in high-resolution images is presented. The performance of the PMD camera and of the proposed data processing is characterized in scenarios, considering factors like relative velocities, rotation dynamics, illumination and optical properties of target surface materials, which are major effects for disturbances in this context.