Time-of-flight Camera Research Articles

Mozart: A Mobile ToF System for Sensing in the Dark Through Phase Manipulation

Sensing in low-light and dark environments has a wide range of applications. However, existing sensing technologies suffer several major challenges, such as excessive noise and low resolution. In this work, we propose Mozart - a new mobile sensing system that leverages off-the-shelf Time-of-Flight (ToF) depth cameras to generate high-resolution and rich-in-texture maps for applications in dark scenarios. The design of Mozart is based on our key observation that the phase components of ToF measurements can be manipulated to expose texture information. Through in-depth analysis of the physical reflection model, we show that the textures can be exposed and enhanced using highly compute-efficient phase manipulation functions. Moreover, by exploiting the physics texture models, we propose an autoencoderbased unsupervised learning approach that can automatically learn efficient representations from phase components to generate high-resolution maps. We implemented Mozart on several Android smartphone models and an edge testbed with standalone ToF camera platforms for various applications in the dark. The results show that Mozart can work in real time and delivers significant improvement over existing sensing technologies. The demo video at https:// www.youtube.com/watch?v=L_sxyTZxIdU shows that Mozart offers a low-cost, highperformance sensing technology for nextgeneration applications in the dark.

Deep Learning for Generating Time-of-Flight Camera Artifacts.

Time-of-Flight (ToF) cameras are subject to high levels of noise and errors due to Multi-Path-Interference (MPI). To correct these errors, algorithms and neuronal networks require training data. However, the limited availability of real data has led to the use of physically simulated data, which often involves simplifications and computational constraints. The simulation of such sensors is an essential building block for hardware design and application development. Therefore, the simulation data must capture the major sensor characteristics. This work presents a learning-based approach that leverages high-quality laser scan data to generate realistic ToF camera data. The proposed method employs MCW-Net (Multi-Level Connection and Wide Regional Non-Local Block Network) for domain transfer, transforming laser scan data into the ToF camera domain. Different training variations are explored using a real-world dataset. Additionally, a noise model is introduced to compensate for the lack of noise in the initial step. The effectiveness of the method is evaluated on reference scenes to quantitatively compare to physically simulated data.

Non-systematic noise reduction framework for ToF camera

Time-of-flight (ToF) cameras enable a diverse range of applications due to their high frame rate, high resolution, and low cost. However, these cameras suffer from non-systematic noise during the acquisition of high-quality depth images, severely affecting their range accuracy. In this paper, we propose a non-systematic noise reduction framework named “DCS2Noise” to address this issue. This framework comprises a three-stage denoising strategy, involving noise standardization, deep learning-based differential correlation sampling (DCS) denoising and further enhancement with pairwise noise suppression. This framework directly captures and denoises DCS images during the ToF imaging process, making it more suitable for non-systematic noise reduction in ToF cameras. Compared to traditional methods, our approach significantly reduces the root mean squared error (RMSE) and improves the noise reduction ratio, peak signal to noise ratio (PSNR), and structural similarity index measure (SSIM). We believe that this study provides new insights into understanding noise in ToF cameras and offers effective references for reducing non-systematic noise in three-dimensional measuring instruments.

Localization of asparagus spears using time-of-flight imaging for robotic harvesting

PurposeTime of flight (ToF) imaging is a promising emerging technology for the purposes of crop identification. This paper aim to presents localization system for identifying and localizing asparagus in the field based on point clouds from ToF imaging. Since the semantics are not included in the point cloud, it contains the geometric information of other objects such as stones and weeds other than asparagus spears. An approach is required for extracting the spear information so that a robotic system can be used for harvesting.Design/methodology/approachA real-time convolutional neural network (CNN)-based method is used for filtering the point cloud generated by a ToF camera, allowing subsequent processing methods to operate over smaller and more information-dense data sets, resulting in reduced processing time. The segmented point cloud can then be split into clusters of points representing each individual spear. Geometric filters are developed to eliminate the non-asparagus points in each cluster so that each spear can be modelled and localized. The spear information can then be used for harvesting decisions.FindingsThe localization system is integrated into a robotic harvesting prototype system. Several field trials have been conducted with satisfactory performance. The identification of a spear from the point cloud is the key to successful localization. Segmentation and clustering points into individual spears are two major failures for future improvements.Originality/valueMost crop localizations in agricultural robotic applications using ToF imaging technology are implemented in a very controlled environment, such as a greenhouse. The target crop and the robotic system are stationary during the localization process. The novel proposed method for asparagus localization has been tested in outdoor farms and integrated with a robotic harvesting platform. Asparagus detection and localization are achieved in real time on a continuously moving robotic platform in a cluttered and unstructured environment.

Single-shot synthetic wavelength imaging: Sub-mm precision ToF sensing with conventional CMOS sensors

We present a novel single-shot interferometric ToF camera targeted for precise 3D measurements of dynamic objects. The camera concept is based on Synthetic Wavelength Interferometry, a technique that allows retrieval of depth maps of objects with optically rough surfaces at submillimeter depth precision. In contrast to conventional ToF cameras, our device uses only off-the-shelf CCD/CMOS detectors and works at their native chip resolution (as of today, theoretically up to 20 Mp and beyond). Moreover, we can obtain a full 3D model of the object in single-shot, meaning that no temporal sequence of exposures or temporal illumination modulation (such as amplitude or frequency modulation) is necessary, which makes our camera robust against object motion.In this paper, we introduce the novel camera concept and show the first measurements that demonstrate the capabilities of our system. We present 3D measurements of small (cm-sized) objects with > 2 Mp point cloud resolution (the native pixel resolution of our used detector) and up to sub-mm depth precision. We also report a “single-shot 3D video” acquisition and a first single-shot “Non-Line-of-Sight” measurement. Our technique has great potential for high-precision applications with dynamic object movement, e.g., in AR/VR, industrial inspection, medical imaging, and imaging through scattering media like fog or human tissue.

Coal flow volume detection method for conveyor belt based on TOF vision

Coal flow volume is the essential basic data support for intelligent speed regulation and energy-saving control of coal mine transportation systems. To accurately measure the coal flow volume of conveyor belts, an innovative coal flow volume detection method for conveyor belts based on TOF vision was proposed in the paper. Both depth and grayscale images of the coal flow were collected by a TOF camera. Then an improved fast marching method based on the grayscale image was used to achieve depth image restoration. The coal flow volume of conveyor belts could be calculated by using the surface fitting method. Experimental results demonstrate that the coal flow detection accuracy of the proposed method can reach 97.35%, and the single-frame image processing time is less than 70.72ms. The proposed method is verified to meet the accuracy and real-time requirements of coal mines.

Motion prediction of tumbling uncooperative spacecraft during proximity operations.

The relative attitude estimation between chasers and uncooperative targets is an important prerequisite for executing in orbit service (OOS) tasks. Only by efficiently obtaining relative pose parameters can chasers design close-range rendezvous trajectories close to uncooperative targets. The focus of this article is on active systems, such as TOF cameras or LIDAR. This paper proposes an attitude estimation scheme to obtain relative attitude parameters between uncooperative targets. This scheme utilizes LIDAR to obtain three-dimensional point clouds of non-cooperative targets, extracts key points and simplifies the number of point clouds through joint farthest point sampling and point cloud feature analysis, and then uses point fast feature histograms (FPFHs) and robust iterative closest point algorithms to achieve point cloud registration between every two frames. Finally, a filtering framework was designed, whose scheme is an extended Kalman filter designed for updating measurements of relative position, velocity, attitude, and angular velocity estimation. The experimental results show that this method can effectively achieve point cloud registration for close range rotation and translation motion, and can estimate the motion state of the target.

Model-Based 3D Gaze Estimation Using a TOF Camera.

Among the numerous gaze-estimation methods currently available, appearance-based methods predominantly use RGB images as input and employ convolutional neural networks (CNNs) to detect facial images to regressively obtain gaze angles or gaze points. Model-based methods require high-resolution images to obtain a clear eyeball geometric model. These methods face significant challenges in outdoor environments and practical application scenarios. This paper proposes a model-based gaze-estimation algorithm using a low-resolution 3D TOF camera. This study uses infrared images instead of RGB images as input to overcome the impact of varying illumination intensity in the environment on gaze estimation. We utilized a trained YOLOv8 neural network model to detect eye landmarks in captured facial images. Combined with the depth map from a time-of-flight (TOF) camera, we calculated the 3D coordinates of the canthus points of a single eye of the subject. Based on this, we fitted a 3D geometric model of the eyeball to determine the subject's gaze angle. Experimental validation showed that our method achieved a root mean square error of 6.03° and 4.83° in the horizontal and vertical directions, respectively, for the detection of the subject's gaze angle. We also tested the proposed method in a real car driving environment, achieving stable driver gaze detection at various locations inside the car, such as the dashboard, driver mirror, and the in-vehicle screen.

Obstacle detection using a two-dimensional ToF camera for the landing a fixed-wing UAV

Obstacle detection using a two-dimensional ToF camera for the landing a fixed-wing UAV

EVALUATION AND COMPARISON OF DIFFERENT TIME OF FLIGHT CAMERAS FOR OUTDOOR APPLICATIONS

Abstract. Time-of-Flight (ToF) cameras have gained prominence in robotics, augmented reality, and gesture recognition due to their costeffective direct measurement of 3D environments. However, their outdoor applications remain limited, mainly due to challenges like sunlight interference. Through systematic testing under challenging outdoor conditions, we aim to assess the suitability of ToF cameras, specifically Azure Kinect and Pmd Monstar, in smart city contexts and contribute to the state-of-the-art and future trends of 3D sensing technology. Our experiments focus on three high-reflectivity cases: license plates, reflective road marking paint on cement and asphalt boards, and traffic cones. Results indicated that Azure Kinect offered a longer measurement range but was more susceptible to flying pixels. Pmd Monstar provided more stable depth measurements and was less sensitive to flying pixels. Differences in performance were attributed to their modulation frequencies and the distinct approaches to handling low-confidence points. By addressing the identified limitations and challenges, researchers and engineers can enhance ToF camera capabilities, ultimately improving their performance and expanding their applicability in outdoor transportation, autonomous driving, and other related smart city fields.

Reconstructing Depth Images for Time-of-Flight Cameras Based on Second-Order Correlation Functions

Depth cameras are closely related to our daily lives and have been widely used in fields such as machine vision, autonomous driving, and virtual reality. Despite their diverse applications, depth cameras still encounter challenges like multi-path interference and mixed pixels. Compared to traditional sensors, depth cameras have lower resolution and a lower signal-to-noise ratio. Moreover, when used in environments with scattering media, object information scatters multiple times, making it difficult for time-of-flight (ToF) cameras to obtain effective object data. To tackle these issues, we propose a solution that combines ToF cameras with second-order correlation transform theory. In this article, we explore the utilization of ToF camera depth information within a computational correlated imaging system under ambient light conditions. We integrate compressed sensing and non-training neural networks with ToF technology to reconstruct depth images from a series of measurements at a low sampling rate. The research indicates that by leveraging the depth data collected by the camera, we can recover negative depth images. We analyzed and addressed the reasons behind the generation of negative depth images. Additionally, under undersampling conditions, the use of reconstruction algorithms results in a higher peak signal-to-noise ratio compared to images obtained from the original camera. The results demonstrate that the introduced second-order correlation transformation can effectively reduce noise originating from the ToF camera itself and direct ambient light, thereby enabling the use of ToF cameras in complex environments such as scattering media.

GPU Accelerated Simulation of Scene Generation of 3D Photonic Mixer Device Camera

Simulation of Photonic Mixer Device (PMD) sensors have the capability to create virtual environment to test 3D camera design. This simulation comprises of multiple steps like scene generation using ray tracing, power calculation, raw data generation and raw data processing. However, each step-in situation process takes longer time to implement and they are simulation process, simulators need to be faster. In this paper, we propose parallel implementation method for scene generation using GPGPUs. The feasibility of the method is confirmed using Amdahl’s law before implementation. The method is implemented and tested on GeForce 820M, GeForce 750Ti and Volta V100.Tthe highest speed up obtained is 219.913 using Volta (GV100) GPU for block size 1024. Thus, parallel method optimizes the scene generation time as compared to serial processing and the implemented results are better than the state of the art in the literature.

Fast Radius Outlier Filter Variant for Large Point Clouds

Currently, several devices (such as laser scanners, Kinect, time of flight cameras, medical imaging equipment (CT, MRI, intraoral scanners)), and technologies (e.g., photogrammetry) are capable of generating 3D point clouds. Each point cloud type has its unique structure or characteristics, but they have a common point: they may be loaded with errors. Before further data processing, these unwanted portions of the data must be removed with filtering and outlier detection. There are several algorithms for detecting outliers, but their performances decrease when the size of the point cloud increases. The industry has a high demand for efficient algorithms to deal with large point clouds. The most commonly used algorithm is the radius outlier filter (ROL or ROR), which has several improvements (e.g., statistical outlier removal, SOR). Unfortunately, this algorithm is also limited since it is slow on a large number of points. This paper introduces a novel algorithm, based on the idea of the ROL filter, that finds outliers in huge point clouds while its time complexity is not exponential. As a result of the linear complexity, the algorithm can handle extra large point clouds, and the effectiveness of this is demonstrated in several tests.

Technology of Automatic Evaluation of Dairy Herd Fatness

The global recent development trend in dairy farming emphasizes the automation and robotization of milk production. The rapid development rate of dairy farming requires new technologies to increase the economic efficiency and improve production. The research goal was to increase the milk production efficiency by introducing the technology to automatically assess the fatness of a dairy herd in 0.25-point step on a 5-point scale. Experimental data were collected on the 3D ToF camera O3D 303 installed in a walk-through machine on robotic free-stall farms in the period from August 2020 to November 2022. The authors collected data on 182 animals and processed 546 images. All animals were between 450 and 700 kg in weight. Based on the regression analysis, they developed software to find and identify the main five regions of interest: the spinous processes of the lumbar spine and back; the transverse processes of the lumbar spine and the gluteal fossa area; the malar and sciatic tuberosities; the tail base; and the vulva and anus region. The adequacy of the proposed technology was verified by means of a parallel expert survey. The developed technology was tested on 3 farms with a total of 1810 cows and is helpful for the non-contact evaluation of the fatness of a dairy herd within the herd’s life cycle. The developed method can be used to evaluate the tail base area with 100% accuracy. The hungry hole can be determined with a 98.9% probability; the vulva and anus area—with a 95.10% probability. Protruding vertebrae—namely, spinous processes and transverse processes—were evaluated with a 52.20% and 51.10% probability. The system’s overall accuracy was assessed as 93.4%, which was a positive result. Animals in the condition of 2.5 to 3.5 at 5–6 months were considered healthy. The developed system makes it possible to divide the animals into three groups, confirming their physiological status: normal range body condition, exhaustion, and obesity. By means of a correlation dependence equal to R = 0.849 (Pearson method), the authors revealed that animals of the same breed and in the same lactation range have a linear dependence of weight-to-fatness score. They have developed an algorithm for automated assessment of the fatness of animals with further staging of their physiological state. The economic effect of implementing the proposed system has been demonstrated. The effect of increasing the production efficiency of a dairy farm by introducing the technology of automatic evaluation of the fatness of a dairy herd with a 0.25-point step on a 5-point scale had been achieved. The overall accuracy of the system was estimated at 93.4%.

Determining Body Condition of Dairy Cows for Early Diagnosis of Physiological Diseases

The paper points out the problem with automated diagnostics of body condition in dairy cattle, including ketosis. The conducted research is aimed at determining the possibility for non-contact automated diagnostics of the cattle physiological state on a daily basis. (Research purpose) To develop an algorithm for complex operational diagnostics of the physiological state of dairy cows by their live weight and body condition. (Materials and methods) Field data were collected in 2021-2022 on the FSUE Grigoryevskoye (Yaroslavl Region), Istra Cheese Factory and Lenin Dairy State Farm (Moscow Region). A commercial 3D ToF (Time-of-Flight) camera O3D303 was used. The 3D camera is capable of calculating and displaying the Point Cloud space as a multidimensional array. The program received 144 images, 136 images passed the filtering stage, 6 images did not detect the areas of interest, because of the high level of image noise, and the sacrum was not detected. 62 cows were subject to research. (Results and discussion) The sample and the dependence are proved to be representative as the Pearson correlation coefficient equals R=0.849, which shows a strong linear relationship between the body condition score and live weight. It was determined that in 24 percent of cases the body condition score is less than the least normal one. An algorithm was developed to help veterinarians to detect the animals that need additional examination. (Conclusions) It was found that the developed algorithm helps to quickly detect ketosis in dairy cows and automatically diagnose physiological diseases at an early stage, without additional labor and monetary costs.

A Hybrid ToF Image Sensor for Long-Range 3D Depth Measurement Under High Ambient Light Conditions

A new indirect time of flight (iToF) sensor realizing long-range measurement of 30 m has been demonstrated by a hybrid ToF (hToF) operation, which uses multiple time windows (TWs) prepared by multi-tap pixels and range-shifted subframes. The VGA-resolution hToF image sensor with 4-tap and 1-drain pixels, fabricated by the BSI process, can measure a depth of up to 30 m for indoor operation and 20 m for outdoor operation under high ambient light of 100 klux. The new hToF operation with overlapped TWs between subframes for mitigating an issue on the motion artifact is implemented. The sensor works at 120 frames/s for a single subframe operation. Interference between multiple ToF cameras in IoT systems is suppressed by a technique of emission cycle-time changing.

Accuracy of a time-of-flight (ToF) imaging system for monitoring deep-inspiration breath-hold radiotherapy (DIBH-RT) for left breast cancer patients.

Deep inspiration breath-hold radiotherapy (DIBH-RT) reduces cardiac dose by over 50%. However, poor breath-hold reproducibility could result in target miss which compromises the treatment success. This study aimed to benchmark the accuracy of a Time-of-Flight (ToF) imaging system for monitoring breath-hold during DIBH-RT. The accuracy of an Argos P330 3D ToF camera (Bluetechnix, Austria) was evaluated for patient setup verification and intra-fraction monitoring among 13 DIBH-RT left breast cancer patients. The ToF imaging was performed simultaneously with in-room cone beam computed tomography (CBCT) and electronic portal imaging device (EPID) imaging systems during patient setup and treatment delivery, respectively. Patient surface depths (PSD) during setup were extracted from the ToF and the CBCT images during free breathing and DIBH using MATLAB (MathWorks, Natick, MA) and the chest surface displacement were compared. The mean difference ± standard deviation, correlation coefficient, and limit of agreement between the CBCT and ToF were 2.88 ± 5.89mm, 0.92, and -7.36, 1.60mm, respectively. The breath-hold stability and reproducibility were estimated using the central lung depth extracted from the EPID images during treatment and compared with the PSD from the ToF. The average correlation between ToF and EPID was -0.84. The average intra-field reproducibility for all the fields was within 2.70mm. The average intra-fraction reproducibility and stability were 3.74mm, and 0.80mm, respectively. The study demonstrated the feasibility of using ToF camera for monitoring breath-hold during DIBH-RT and shows good breath-hold reproducibility and stability during the treatment delivery.

Range selective phase-shifting frequency-modulated digital holography with temporal-heterodyning.

We demonstrate that a time of flight (TOF) camera can be used to perform range selective temporal-heterodyne frequency-modulated continuous wave digital holography (TH FMCW DH). The modulated arrayed detection of a TOF camera allows efficient integration of holograms at a selected range with range resolutions significantly less than the optical system's depth of field. TH FMCW DH also allows for on-axis geometries to be achieved, where background light not at the internal modulation frequency of the camera is filtered out. Range selective TH FMCW DH imaging was achieved for both image holograms and Fresnel holograms using on-axis DH geometries. A 2.39GHz FMCW chirp bandwidth resulted in a DH range resolution of 6.3cm.

Experimental investigation to determine necking of commercially pure titanium sheets using a time-of-flight camera and Heaviside-digital image correlation

The plastic instability evolution in polycrystalline metallic materials is a key to understanding the deformation processes during mechanical loading. Plasticity assessments of commercially pure titanium (CP-Ti) can be performed to enhance its mechanical properties. The aim of the present investigation is to determine the plastic deformation during the metal forming processes of CP-Ti sheets at room temperature. This localized deformation might affect the ductility of the sheets’ materials during the forming processes due to their complex strain paths. Therefore, this study is focused on the determination of Forming Limit Diagrams (FLDs) of CP-Ti sheets in different strain paths based on experimental Nakazima test. The latter is employed to determine the complete FLD using several specimens with various geometries to achieve different strain paths in tensile tests. For this purpose, the strain field measurements before cracking are carried out by the Digital Image Correlation (DIC) procedure. The detection of macroscopic cracks is determined by the Heaviside-DIC (H-DIC). During experimental tests, several images are taken with an integrated camera. The captured images are analyzed by DIC and H-DIC. In this work, a second depth camera is accurately calibrated. Thus, depth and standard cameras address most error sources inherent in the time-of-flight (ToF) technology within a unified calibration framework. Therefore, the automatic calibration method of the consumer ToF camera is used and developed for the first time with H-DIC technique, in the framework of material formability, to better determine the ductility limit of CP-Ti.

Sparsity-Aware 3-D ToF Sensing

In this work, we propose several time-of-flight (ToF) sensing schemes which tackle the challenge of covering very-wide areas and long ranges in nearly real time, with relatively simple implementation and low associated computational load. We thoroughly describe two methodologies for the resolution of the inverse problem. First, we extend a greedy algorithm, such as orthogonal matching pursuit (OMP), by considering an initial refinement of the spatial domain in which the signal recovery is performed. Then, we propose various nonadaptive techniques for the construction of the sensing matrices, relying on the optimization of coherence and density. We further develop them by including an additional verification step which accounts for the noninstantaneous transitions from one element to another of the code and avoids any possible coincidence between rising and falling edges which may degrade the coherence. We also investigate the upper super-resolution limit, the over-sampling rate which yields unitary coherence, by considering the instrument response function (IRF) of the ToF sensor. Second, we expand an adaptive sensing scheme in which the rows of the sensing matrix are generated accounting for the information from previous measurements, such as adaptive progressive edge growth algorithm (APEG), by considering several groups of signals during the adaptation of the sensing matrices. The signals are then individually recovered for each pixel via OMP over the identified joint signal support. We validate the proposed methodologies by numerical simulations over datasets from stereo and ToF cameras.