Time-of-flight Camera Research Articles

Study of the depth accuracy and entropy characteristics of a ToF camera with coupled noise

By illuminating targets with a modulated light signal, a time-of-flight (ToF) camera can calculate depth maps based on the phase shifts generated by the round-trip travel of the light from the targets back to the sensor. The accuracy of such depth imaging depends on the quality of the returned light. However, the calculation can be disturbed by various factors, including the external environment and the internal structure of the camera. Multiple coupled interferences can introduce noise into the depth data collected by a ToF camera in the spatial domain. It is difficult to express the relationship between the noise in depth maps and these mixed disturbances in a mathematical model. Based on the theory of differential entropy and a large amount of depth data from a ToF camera, this paper analyzes the characteristics of depth imaging entropy, proposes an evaluation method for depth image quality, and presents a multilayer perceptron model with information entropy (E-MLP) trained to optimize the accuracy of depth imaging. Experimental results show that this method can significantly improve the depth accuracy in the case of mixed noise.

Multipath Interference Suppression in Time-of-Flight Sensors by Exploiting the Amplitude Envelope of the Transmission Signal

Time-of-flight (ToF) cameras provide depth information and they have been applied in many areas. However, multipath interference (MpI) affects the depth measurements. This article proposes an algorithm to suppress MpI during ToF camera depth measurements by exploiting the amplitude envelope of the camera's transmission wave. More precisely, it uses the fact that the amplitude envelope of the ToF camera's transmission wave is constant. The envelope of the reflected signal in the case of MpI deviates from that of the MpI-free envelope. By minimizing the deviation using an adaptive filter, the depth measurement error can be corrected. The method is applied at the sensor side of the ToF camera and lays the theoretical foundation for MpI suppression in ToF cameras. The simulation and experimental results demonstrate that the proposed algorithm is able to significantly improve the depth measurement accuracy.

Stereovision Tracking System for Monitoring Loader Crane Tip Position

This paper presents the idea and experimental tests of a non-stationary stereovision system used for supervision of a loading crane. The system based on two independent gimbals, equipped with cameras, allows simultaneously to measure and track a group of individual spatial points. The four steps calibration procedure is required for each gimbal axis, camera optics, gimbals positions, and loading crane position. Modern camera systems have been widely applied in several fields of the industry allowing for accurate measurements and real-time quality control. The systems based on stereo camera models and appropriate image analysis algorithms can measure a group of spatial points, complex surfaces, small displacements, or vibrations. There are also several vision systems such as laser tracker, camera mounted on fully controlled rotating head (called gimbal), or TOF camera, which can be used to follow and measure real-time object displacements. Unfortunately, they all have significant individual limitations explained in this paper. Accuracy tests of 3D measurements including active tracking of length pattern and crane tip have been performed within a selected working area. Based on reconstructed 3D data points, different types of geometric errors have been presented. The results revealed effects of accuracy degradation on the workspace boundaries and for higher observation angles, as well as 3D data bias suggesting mechanical issues. Several improvements in calibration and gimbal construction might result in far enough accuracy for normal operations of the crane.

A Parallel-Phase Demodulation-Based Distance-Measurement Method Using Dual-Frequency Modulation

Time-of-flight (ToF) measurement technology based on the amplitude-modulated continuous-wave (AMCW) model has emerged as a state-of-the-art distance-measurement method for various engineering applications. However, many of the ToF cameras employing the AMCW process phase demodulation sequentially, which requires time latency for a single distance measurement. This can result in significant distance errors, especially in non-static environments (e.g., robots and vehicles) such as those containing objects moving relatively to the sensors. To reduce the measurement time required for a distance measurement, this paper proposes a novel, parallel-phase demodulation method. The proposed method processes phase demodulation of signal in parallel rather than sequentially. Based on the parallel phase demodulation, 2π ambiguity problem is also solved in this work by adopting dual frequency modulation to increase the maximum range while maintaining the accuracy. The performance of proposed method was verified through distance measurements under various conditions. The improved distance measurement accuracy was demonstrated throughout an extended measurement range (1–10 m).

ASSESSMENT OF A PORTABLE TOF CAMERA AND COMPARISON WITH SMARTPHONE STEREO VISION

Abstract. Nowadays time-of-flight (ToF) cameras and multiple RGB cameras are being embedded in an increasing number of high-end smartphones: despite their integration in mobile devices is mostly motivated by photographic applications, their availability can be exploited to enable 3D reconstructions directly on smartphones. Furthermore, even when a ToF camera is not embedded in a smartphone, low cost solutions are available on the market in order to easily provide standard mobile devices with a lightweight and extremely portable ToF camera. This work deals with the assessment of a low cost ToF camera, namely Pico Zense DCAM710, which perfectly fits with the above description. According to the results obtained in the considered tests, the ranging error (precision) of the DCAM710 camera increases linearly approximately up to the nominal maximum range in the considered working mode, up to approximately 1 cm. Despite the device allows to acquire measurements also at larger ranges, the measurement quality significantly worsen. After assessing the main characteristics of such ToF camera, this paper aims at comparing its 3D reconstruction ability with that of a smartphone stereo vision system. In particular, the comparison of a 3D reconstruction obtained with stereo vision from images acquired with an LG G6 shows that the stereo reconstruction leads to a much larger point cloud. However, points generated by the ToF camera are more homogeneously distributed, and they seem to slightly better describe the real geometry of the reconstructed object. The combination of such two technologies, which will be investigated in our future work, can potentially lead to a denser cloud with respect to the ToF camera, while preserving a reasonable accuracy.

Design of multipath error correction algorithm of coarse and fine sparse decomposition based on compressed sensing in time-of-flight cameras

ABSTRACTA single pixel in time of flight cameras receives multiple reflected light from different scene points, resulting in erroneous depth information. In this paper, based on the proposed sparse decomposition, a coarse and fine sparse decomposition based on compressed sensing is applied to multipath separation. The applied method uses a linear combination of multiple frequency signals to modulate the source. The measured vector obtained through finite random measurements is subjected to two sparse decompositions – rough separation and detailed positioning, and finally the minimum direct path depth is accurately recovered. Under the premise of the same number of measurements, calculation amount, and storage space, the accuracy of the coarse and fine sparse decomposition based on compressed sensing is improved by nearly an order of magnitude compared to the sparse decomposition without compressed sensing. Moreover, our method can basically achieve multi-path separation accuracy to the sub-millimeter level.

Real Time Static Gesture Recognition using Time of Flight Camera

Hand gesture recognition is challenging task in machine vision due to similarity between inter class samples and high amount of variation in intra class samples. The gesture recognition independent of light intensity, independent of color has drawn some attention due to its requirement where system should perform during night time also. This paper provides an insight into dynamic hand gesture recognition using depth data and images collected from time of flight camera. It provides user interface to track down natural gestures. The area of interest and hand area is first segmented out using adaptive thresholding and region labeling. It is assumed that hand is the closet object to camera. A novel algorithm is proposed to segment the hand region only. The noise due to ToF camera measurement is eliminated by preprocessing algorithms. There are two algorithms which we have proposed for extracting the hand gestures features. The first algorithm is based on computing the region distance between the fingers and second one is about computing the shape descriptor of gesture boundary in radial fashion from the centroid of hand gestures. For matching the gesture the distance between two independent regions is computed for every row and column. Same process is repeated across the columns. The number of total region transitions are computed for every row and column. These number of transitions across rows and columns forms the feature vector. The proposed solution is easily able to deal with static and dynamic gestures. In case of second approach we compute the distance between the gesture centroid and shape boundaries at various angles from 0 to 360 degrees. These distances forms the feature vector. Comparison of result shows that this method is very effective in extracting the shape features and competent enough in terms of accuracy and speed. The gesture recognition algorithm mentioned in this paper can be used in automotive infotainment systems, consumer electronics where hardware needs to be cost effective and the response of the system should be fast enough.

Registration and fusion of 3D surface data from CT and ToF camera for position verification in radiotherapy

The purpose of this study is to introduce a low-cost method for verification of patient positioning before and during irradiation in radiotherapy. The experiments strategy is implemented by using a phantom like a human head in two modes: with and without thermoplastic masks. Radiotherapy is an effective clinical option for topical tumors treatment. Ability to the monitoring and control the patient’s position during treatment is among the important items in radiotherapy. We registered to modality of 3D surface image in point cloud format, dynamic 3D image of time-of-flight camera (includes outer shape of head) and static 3D image of CT slice sectional by coherent point drift algorithm. Accuracy of 3D surface registration was calculated by average symmetrical surface distances (ASD) that it is a new method based on surface distance. The mean ASD calculated for rotation around X and Y axes of phantom is 2.62 ± 0.97 mm with immobilized thermoplastic mask and accuracy be increased to 1.69 ± 0.88 mm in non-thermoplastic mode. By performing paired T tests for ASDs, it was found that the use of a thermoplastic mask does not make a significant difference in the data acquisition. By developing this research to image fusion of that’s 3D data sets, we will design to augmented reality in radiotherapy for verification of treatment planning and compare it with the electronic portal imaging device.

Robustness of ToF and stereo fusion for high‐accuracy depth map

A depth map can be used in many applications such as robotic navigation, driverless, video production and 3D reconstruction. Both passive stereo and time‐of‐flight (ToF) cameras can provide the depth map for the captured real scenes, but they both have innate limitations. Since ToF cameras and passive stereo are intrinsically complementary for certain scenes, it is desirable to appropriately leverage all the available information by ToF cameras and passive stereo. As a result, this study proposes an approach to integrate ToF cameras and passive stereo to obtain high‐accuracy depth maps. The main contributions are: the first step is to design an energy cost function to utilise the depth map from ToF cameras to guide the stereo matching of passive stereo and the second step is to design their weight function for depth maps pixel‐level fusion. The experiments show that the proposed approach achieves the improved results with high accuracy and robustness.

A High Reliability 3D Object Tracking Method for Robot Teaching Application

3D Object tracking is the task of capturing the 3D position and pose of an object from each time-series image frame. As we known, 3D sensing technique can be realized by stereo vision, structured light, and ToF (time-of-flight) camera. All of them can capture the point cloud data for describing the depth information in a workspace. In past research, the reliability in 3D object tracking was a big problem for real industrial application, therefore, we address a different way to enhance the tracking accuracy and stabilize the tracing path for raising the reliability. In order to build a 3D tracking model and the workspace environment, we adopted an RGB-D camera which is the Intel® RealSense™ D400 Series depth modules to collect the cloud point data and RGB values. The built 3D tracking model should contains the information which includes points, normal and texture for producing many 2D object images with different perspectives. Then the produced images were fed to a SSD (single-shot detector) neural network to learn the object’s features for 2D tracking. In dynamic tracking process, the image frames were through the sematic image segmentation by DeepLabV3+ for only extracting the object information without hands and background. Thus, the reserved data only included object’s cloud point data and texture information in workspace. Then we use the iterative closest point (ICP) algorithm and the RGB intensity correlation method to confirm the object’s position and posture in workspace. The result shows that our method has a better performance than SSD method for tracking a self-predefined object.

Precise drone location and tracking by adaptive matched filtering from a top-view ToF camera

Nowadays the use of drones is rapidly growing, and the autonomous navigation capability depends on knowing their position at any time. The precision and robustness of a local positioning system for these devices is of particular importance in takeoff and landing maneuvers, especially in GPS-denied environments. The main contribution of this work lies in the development of a precise local positioning and tracking system for drones. For this purpose, a ToF camera has been installed on the ceiling in order to make use of its depth maps. Taking as a reference the disturbance caused by a quadrotor in one of these maps, a novel 2D matched filter has been designed based on a Gaussian wavelet. This filter allows the system to quickly detect all drones flying in the scene. Moreover, it is dynamically adapted to the image portion that they occupy, taking into account the variation of this parameter with their flying altitude, which has also been theoretically determined. The whole algorithm leads to a precise 3D drone positioning.In addition, the proposed system is also robust against short-time occlusions, since the measurements history can be used to predict future positions, thus avoiding the complete loss of tracking.

Evaluation of Lung Volume Under Nasal High Flow With a Time of Flight Camera

Evaluation of Lung Volume Under Nasal High Flow With a Time of Flight Camera

Cyan emitting Ca3Sc2Si1.5Ge1.5O12:Ce3+ phosphor with 10.4 ns lifetime

Synthesis and photoluminescence properties of Ca3Sc2Si1.5Ge1.5O12:Ce3+ phosphor are described and compared with that of Ca3Sc2Si3O12:Ce3+. Partial replacement of Si by Ge, produces interesting changes in luminescence properties of Ca3Sc2Si3O12:Ce3+. Emission maximum shifts from 510 to 485 nm and that of excitation from 450 to 428 nm. The blue shifts are attributable to the splitting of Ce3+ 5d levels by crystal field. Centroid shift also contributes to these shifts. Cyan emitting phosphor, in combination with other phosphors, can be useful for generating true white spectrum. Moreover, lifetime of Ca3Sc2Si1.5Ge1.5O12:Ce3+ phosphor is very short, 10.4 ns, as compared to 62.7 for Ca3Sc2Si3O12:Ce3+. Short lifetime makes Ca3Sc2Si1.5Ge1.5O12:Ce3+ suitable for applications like scintillator, time of flight camera, etc.

Computer Vision Based 3D Reconstruction : A Review

3D reconstruction are used in many fields starts from the object reconstruction such as site, and cultural artifacts in both ground and under the sea levels. The scientist are beneficial for these task in order to learn and keep the environment into 3D data due to the extinction. In this paper explained vision setup that is commonly used such as single camera, stereo camera, Kinect / Structured Light/ Time of Flight camera and fusion approach. The prior works also explained how the 3D reconstruction perform in many fields and using various algorithms.

Low Power Depth Estimation of Rigid Objects for Time-of-Flight Imaging

Depth sensing is useful in a variety of applications that range from augmented reality to robotics. Time-of-flight (TOF) cameras are appealing because they obtain dense depth measurements with minimal latency. However, for many battery-powered devices, the illumination source of a TOF camera is power hungry and can limit the battery life of the device. To address this issue, we present an algorithm that lowers the power for depth sensing by reducing the usage of the TOF camera and estimating depth maps using concurrently collected images. Our technique also adaptively controls the TOF camera and enables it when an accurate depth map cannot be estimated. To ensure that the overall system power for depth sensing is reduced, we design our algorithm to run on a low power embedded platform, where it outputs $640\times 480$ depth maps at 30 frames per second. We evaluate our approach on several RGB-D datasets, where it produces depth maps with an overall mean relative error of 0.96% and reduces the usage of the TOF camera by 85%. When used with commercial TOF cameras, we estimate that our algorithm can lower the total power for depth sensing by up to 73%.

Depth from phasor distortions in fog

This paper presents a time-of-flight (ToF) measurement method for use in foggy weather. The depth measured by a ToF camera is greatly distorted in fog because the light scattered in the fog reaches the camera much faster than the target reflection. We reveal that the multi-frequency measurements contain a cue whether two arbitrary pixels have the same depth. After clustering the same depth pixels using this cue, the original depth can be recovered for each cluster by line fitting in the Cartesian coordinate frame. The effectiveness of this method is evaluated numerically via real-world and road-scale experiments.

Measurement of respiratory effort in sleep by 3D camera and respiratory inductance plethysmography

Polysomnography systems used for the diagnosis of sleep respiratory disorders are comprised of multiple sensors, including abdomen and thorax respiratory inductance plethysmography (RIP) belts to record respiratory effort. However, RIP belts are known to be susceptible to signal loss. To resolve this, we utilized a contactless three-dimensional (3D) time-of-flight (TOF) camera to monitor respiratory effort. We aimed to show that respiratory effort monitoring can be achieved by 3D TOF camera recording instead of RIP belts. The use of RIP belt signals is twofold. Firstly, the signals are used to classify the apnea events into obstructive, central, and mixed. Additionally, the American Academy of Sleep Medicine (AASM) Scoring Manual recommends the scoring of apneas and hypopneas using RIPSum (the sum of the abdomen and thorax RIP signals) when the airflow sensors are unavailable. We therefore used the 3D effort signal to classify apneas and compared it to the RIP signal classification. Reduced effort events from RIP and 3D effort signals were compared to the apnea and hypopnea events. Furthermore, the changes in effort during the events were compared between the two effort signals. Classification by 3D effort signal performed well, with 80% accuracy. It worked best for central apneas, with an accuracy of 99%. There was a high correlation of r = 0.88 (r ≠ 0, p = 0.0001) between the 3D effort signal events and RIPSum events. There was also a significant correlation of 0.62 (r ≠ 0, p = 0.0001) between 3D effort signal and RIPSum in the decrease of effort during apnea and hypopnea events. We conclude that respiratory effort derived from a 3D TOF camera can be used as an alternative to RIP belts for scoring of apneas and hypopneas and classification of apneas.

Anisotropic non-rigid Iterative Closest Point Algorithm for respiratory motion abdominal surface matching

Surface registration is a one of the crucial and actual problems of computer aided surgery. This paper presents the modification of the non-rigid Iterative Closest Point Algorithm which takes into account an anisotropic noise model and landmarks as guided correspondence at the transformation step in every iteration. The presented approach was validated on human abdominal briefing surface data from a time-of-flight camera. We took the median of the resulting measures and the outcome is presented: the median of means of surfaces distance was at the same level for both variants of the ICP algorithm and is comparable with the isotropic variant, the median of mean landmark position errors decreased by 0.93 units (over 20% improvement) and the median of percentage of single correspondences in target point cloud increased by 11.96%. The results showed that the introduction of the anisotropic model of noise for the ToF camera allows for the improvement the percentage of target cloud points which had only one correspondent over 10% impartment and additional weighting of markers also improves the measure of the quality of finding real correspondents over 20% improvement. In the examined dataset, where the average initial distance between the clouds of points in the inspiratory and expiration is equal to approx. 7.5 mm, a more than 10% improvement in the quality of the correspondence improves the accuracy of matching the surface within 1 mm which is a significant value in application of minimally invasive image guided interventions.

Compressive time-of-flight 3D imaging using block-structured sensing matrices

Spatially and temporally highly resolved depth information enables numerous applications including human–machine interaction in gaming or safety functions in the automotive industry. In this paper, we address this issue using time-of-flight (ToF) 3D cameras which are compact devices providing highly resolved depth information. Practical restrictions often require one to reduce the amount of data to be read-out and transmitted. Using standard ToF cameras, this can only be achieved by lowering the spatial or temporal resolution. To overcome such a limitation, we propose a compressive ToF camera design using block-structured sensing matrices that allows one to reduce the amount of data while keeping high spatial and temporal resolution. We propose the use of efficient reconstruction algorithms based on -minimization and TV-regularization. The reconstruction methods are applied to data captured by a real ToF camera system and evaluated in terms of reconstruction quality and computational effort. For both -minimization and TV-regularization, we use a local as well as a global reconstruction strategy. For all considered instances, global TV-regularization turns out to clearly perform best in terms of evaluation metrics including the PSNR.

Principle and Robotic Applications of Conical Scanning Method

We have proposed a surrounding environmental recognition method using conical scanning distance measurement for mobile robots. The conical scanning method has a high robustness against ambient light noise and its calculation cost is low. This report describes the principle of this method and the two implementations. The first application is the quadrupedal wheeled robot. The robot can recognize a stair in sight, approach to it and climb. A second application is the indoor-outdoor wheeled mobile robot. The robot can recognize its position and obstacles on floor from information of a laser range finder and a 3D ToF camera. The validity of this method was confirmed by these robotic applications.