Time-of-flight Camera Research Articles

Face Mask Identification Using Spatial and Frequency Features in Depth Image from Time-of-Flight Camera.

Face masks can effectively prevent the spread of viruses. It is necessary to determine the wearing condition of masks in various locations, such as traffic stations, hospitals, and other places with a risk of infection. Therefore, achieving fast and accurate identification in different application scenarios is an urgent problem to be solved. Contactless mask recognition can avoid the waste of human resources and the risk of exposure. We propose a novel method for face mask recognition, which is demonstrated using the spatial and frequency features from the 3D information. A ToF camera with a simple system and robust data are used to capture the depth images. The facial contour of the depth image is extracted accurately by the designed method, which can reduce the dimension of the depth data to improve the recognition speed. Additionally, the classification process is further divided into two parts. The wearing condition of the mask is first identified by features extracted from the facial contour. The types of masks are then classified by new features extracted from the spatial and frequency curves. With appropriate thresholds and a voting method, the total recall accuracy of the proposed algorithm can achieve 96.21%. Especially, the recall accuracy for images without mask can reach 99.21%.

Non-Cooperative Spacecraft Pose Measurement with Binocular Camera and TOF Camera Collaboration

Non-cooperative spacecraft pose acquisition is a challenge in on-orbit service (OOS), especially for targets with unknown structures. A method for the pose measurement of non-cooperative spacecrafts based on the collaboration of binocular and time-of-flight (TOF) cameras is proposed in this study. The joint calibration is carried out to obtain the transformation matrix from the left camera coordinate system to the TOF camera system. The initial pose acquisition is mainly divided into feature point association and relative motion estimation. The initial value and key point information generated in stereo vision are yielded to refine iterative closest point (ICP) frame-to-frame registration. The final pose of the non-cooperative spacecraft is determined through eliminating the cumulative error based on the keyframes in the point cloud process. The experimental results demonstrate that the proposed method is able to track the target spacecraft during aerospace missions, which may provide a certain reference value for navigation systems.

A novel hand-eye calibration method of picking robot based on TOF camera.

Aiming at the stability of hand-eye calibration in fruit picking scene, a simple hand-eye calibration method for picking robot based on optimization combined with TOF (Time of Flight) camera is proposed. This method needs to fix the TOF depth camera at actual and calculated coordinates of the peach the end of the robot, operate the robot to take pictures of the calibration board from different poses, and record the current photographing poses to ensure that each group of pictures is clear and complete, so as to use the TOF depth camera to image the calibration board. Obtain multiple sets of calibration board depth maps and corresponding point cloud data, that is, "eye" data. Through the circle center extraction and positioning algorithm, the circle center points on each group of calibration plates are extracted, and a circle center sorting method based on the vector angle and the center of mass coordinates is designed to solve the circle center caused by factors such as mirror distortion, uneven illumination and different photographing poses. And through the tool center point of the actuator, the coordinate value of the circle center point on the four corners of each group of calibration plates in the robot end coordinate system is located in turn, and the "hand" data is obtained. Combined with the SVD method, And according to the obtained point residuals, the weight coefficients of the marker points are redistributed, and the hand-eye parameters are iteratively optimized, which improves the accuracy and stability of the hand-eye calibration. the method proposed in this paper has a better ability to locate the gross error under the environment of large gross errors. In order to verify the feasibility of the hand-eye calibration method, the indoor picking experiment was simulated, and the peaches were identified and positioned by combining deep learning and 3D vision to verify the proposed hand-eye calibration method. The JAKA six-axis robot and TuYang depth camera are used to build the experimental platform. The experimental results show that the method is simple to operate, has good stability, and the calibration plate is easy to manufacture and low in cost. work accuracy requirements.

Reconstruction of Non-Line-of-Sight Depth Data Using Genetic Algorithm-Lucy-Richardson Based on Time of Flight Camera

Reconstruction of Non-Line-of-Sight Depth Data Using Genetic Algorithm-Lucy-Richardson Based on Time of Flight Camera

DEVELOPMENT OF INTELLECTUAL SUPPORT’S ALGORITHM AT MANAGE DECISIONS MAKING IN THE FEEDING PROCESS BASED ON A MACHINE VISION SYSTEM’S ON A FARM

Analysis of information obtained from open sources has shown that feed’s abnormal overspending is up to 10% and occurs in 5% of cases. It also means that the farm loses the same share of milk yield, and with it profit. This problem solution can be achieved in various ways due diversity of both hardware (vision system) and software (methods and algorithms). It is possible cows’ feed on dairy farms malnutrition reducing with the help of software and hardware complex (PAC) being developed. This PAC will determine the remaining feed volume on the feed table’s distribution by the 3D vision system of obtaining data use analyzing. Based on this analysis, the PAC will draw conclusions and make recommendations to the farm workers, thereby feedback for the farm feeding system providing. To do this, it is necessary a vision system to choose, an algorithm for obtaining a point cloud to implement an algorithm for finding reference points and gluing images into one, so as an device algorithm for orientation in space to select and implement, an algorithm for calculating feed volume and hardware to develop and laboratorial and operational tests to conduct, as well as a database for development of recommendations to create. Two vision systems are considered for use - a stereo system and a ToF camera, and an algorithm finding for reference points based on 3D data is described.

DEVELOPMENT OF A FAULT INJECTION TOOL & DATASET FOR VERIFICATION OF CAMERA BASED PERCEPTION IN ROBOTIC SYSTEMS

Nowadays, camera-based perception is most popular topic in robotic systems. Verification of camera-based perception systems are crucial and difficult with current tools and methods. This study proposes Camera Fault Injection Tool (CamFITool), which enables different kind of fault injection methods to RGB and TOF cameras in order to perform verification and validation activities on robotic systems. Besides, Fault Injected Image Database which is created by CamFITool is introduced. In addition, the study guides to readers to create new datasets by injecting faults into existing image libraries or camera streams with CamFITool. As a result, CamFITool, an open-source fault injection tool, which is a critical tool for assessing of fault tolerant systems’ safety and security, is proposed. Also, a fault injected image dataset created by CamFITool for verification of camera-based perception studies in robotic systems is given.

Development of a Modular System for Digitalization of Appraisal Process

It is noted that the existing weighing platforms for weighing dairy cattle demonstrate certain shortcomings during the grading process. It was proposed to determine the fatness of animals by scanning their sacrum with optical systems, which allows monitoring their physiological state and, in case of deterioration, promptly change the diet. (Research purpose) To develop a modular system for the digitalization of appraisal process ensuring the autonomy and independence of modules. (Materials and methods) The emphasis is paid on the necessity of integrating into the whole farm system by using a unified Internet space. The development of a modular system for digitalization of appraisal was divided into 4 modules. A grading algorithm was worked out and compared with the layout diagram of the modules in order to assess the volume of work on the grading digitalization. Four strain gauges were installed in the weighing module. (Results and discussion) It was proved that the module provides high-precision weighing of animals (C3 accuracy), regardless of the external influence factors such as cow movements and the asymmetrical position of the cow inside the system box. The results can be displayed on the display of a single control unit. The camera of the vision module was placed at a height of 2,200 millimeters, which eliminates it being damaged by the animals and provides an unobstructed view for the 3D ToF camera. The transportation dimensions of the system prototype are as follows: length – 2,500 millimeters, width – 1,564, height – 2,118 millimeters. The total mass of the manufactured system, including the original platform, is approximately 620-640 kilograms. (Conclusions) A modular system for digitalization of appraisal work has been successfully developed and created. The weighing module was tested revealing an error of less than 1 percent, when testing the system with a reference mass of 655 kilograms and that of 1,200 kilograms.

Unsupervised Domain Adaptation of Deep Networks for ToF Depth Refinement.

Depth maps acquired with ToF cameras have a limited accuracy due to the high noise level and to the multi-path interference. Deep networks can be used for refining ToF depth, but their training requires real world acquisitions with ground truth, which is complex and expensive to collect. A possible workaround is to train networks on synthetic data, but the domain shift between the real and synthetic data reduces the performances. In this paper, we propose three approaches to perform unsupervised domain adaptation of a depth denoising network from synthetic to real data. These approaches are respectively acting at the input, at the feature and at the output level of the network. The first approach uses domain translation networks to transform labeled synthetic ToF data into a representation closer to real data, that is then used to train the denoiser. The second approach tries to align the network internal features related to synthetic and real data. The third approach uses an adversarial loss, implemented with a discriminator trained to recognize the ground truth statistic, to train the denoiser on unlabeled real data. Experimental results show that the considered approaches are able to outperform other state-of-the-art techniques and achieve superior denoising performances.

Relative Pose Estimation of Non-Cooperative Space Targets Using a TOF Camera

It is difficult to determine the accurate pose of non-cooperative space targets in on-orbit servicing (OOS). The visual camera is easily affected by the extreme light environment in space, and the scanning lidar will have motion distortion when the target moves at high speed. Therefore, we proposed a non-cooperative target pose-estimation system combining a registration and a mapping algorithm using a TOF camera. We first introduce the projection model of the TOF camera and proposed a new calibration method. Then, we introduce the three modules of the proposed method: the TOF data preprocessing module, the registration module and the model mapping module. We assembled the experimental platform to conduct semi-physical experiments; the results showed that the proposed method has the smallest translation error 8 mm and Euler angle error 1° compared with other classical methods. The total time consumption is about 100 ms, and the pose tracking frequency can reach 10 Hz. We can conclude that the proposed pose-estimation scheme can achieve the high-precision pose estimation of non-cooperative targets and meet the requirements necessary for aerospace applications.

A low cost collision avoidance system based on a ToF camera for SLAM approaches

Indoor positioning is a problem that has not yet been solved efficiently and accurately. In outdoors the most effective solution is the Global Position System (GPS), but it cannot be used indoors due to the weakening of the signal, so other solutions have been studied. These approaches could be applied to define a map for the guidance of blind people, tourism or navigation for autonomous robots. In this paper, the study, design, implementation and evaluation of a robust obstacle detection and mapping system is proposed. Thus, it can be used to alert of near objects presence and avoid possible collisions in an indoor navigation. The system is based on a Time-of-Flight (ToF) camera and a Single Board Computer (SBC) like Raspberry PI or NVIDIA Jetson Nano. In order to evaluate the system several real experiments were carried out. This kind of system can be integrated on a wheelchair and help the handicapped person to move indoors or take data from an indoor environment and recreate it in a 2D or 3D images.

Pseudo-Passive Time-of-Flight Imaging: Simultaneous Illumination, Communication, and 3D Sensing

The 3D time-of-flight (ToF) cameras have recently received a lot of attention due to their wide range of applications. Despite remarkable advancements in ToF imaging, state-of-the-art ToF cameras are still afflicted by the power hungriness of their illumination sources. To tackle this problem, we exploited existing lighting infrastructure, which ensures the ubiquitous presence of modulated light sources in indoor spaces that serve as opportunity illuminators. We explored the bistatic geometry for passive imaging using the pulse-based ToF approach. Our work is inspired by the recently introduced visible light communication (VLC) or light-fidelity (Li-Fi) infrastructure. VLC allows the infrastructure to provide indoor simultaneous illumination, communication, and sensing (SICS). To this end, we designed a bistatic geometry for the purpose of attaining passive 3D imaging. Such capabilities are achieved by exploiting the pulse shape of the autocorrelation function of real optical signals generated by VLC/Li-Fi modules (e.g., OpenVLC and LiFiMAX). We demonstrated passive imaging by means of matched filtering. In this work, we also studied different sampling strategies in the time-shift domain, including uniform, random, and sparse rulers, which is another step forward toward preserving high depth accuracy with a minimal number of measurements. The proposed methodology achieved successful depth reconstruction with negligible root-mean-square error (RMSE) for the low signal-to-noise ratio (SNR) of the measurements. Parametric models, such as Gaussian and sum of sines, are used to characterize the cross correlation functions and allow for robust parametric depth retrieval from a few measurements. Moreover, we attained a 20-mm worst case error for a target at 25 cm. The experiment proved that the bistatic passive depth reconstruction is feasible.

Growth parameter acquisition and geometric point cloud completion of lettuce.

The plant factory is a form of controlled environment agriculture (CEA) which is offers a promising solution to the problem of food security worldwide. Plant growth parameters need to be acquired for process control and yield estimation in plant factories. In this paper, we propose a fast and non-destructive framework for extracting growth parameters. Firstly, ToF camera (Microsoft Kinect V2) is used to obtain the point cloud from the top view, and then the lettuce point cloud is separated. According to the growth characteristics of lettuce, a geometric method is proposed to complete the incomplete lettuce point cloud. The treated point cloud has a high linear correlation with the actual plant height (R2 = 0.961), leaf area (R2 = 0.964), and fresh weight (R2 = 0.911) with a significant improvement compared to untreated point cloud. The result suggests our proposed point cloud completion method have has the potential to tackle the problem of obtaining the plant growth parameters from a single 3D view with occlusion.

Multi-line laser structured light fast visual positioning system with assist of TOF and CAD

As a reliable and high-precision three-dimensional (3D) vision method, line laser structured light vision has been widely used in the engineering field. However, to obtain overall 3D shape of target to be measured, scanning mechanisms need to be added. Multi-line laser structured light vision can obtain multi-line array 3D shape of workpiece without scanning; however, the encoding of multi-line lasers is a difficult task. This paper proposes a composite vision system, which is composed of time-of-flight(TOF) camera, color camera and vertical cross multi-line laser structured light. Color camera and multi-line laser form a multi-line array 3D vision system, while the coded signal of multi-line laser stripes is provided by TOF camera and corresponding coding algorithm is also proposed in this paper. Based on this system, automatic positioning of welding seam is realized by registration of CAD data and multi-line laser 3D data. The registration is divided into two steps, which are coarse registration and precise registration. Coarse registration is performed by using CAD point cloud data and original point cloud data of multi-line laser; while precise registration is performed with CAD welding seam point cloud data and welding seam point cloud data of multi-line laser. This paper also analyzes accuracy of vision system based on constructing structured light vision model. The measurement accuracy of vision system in x, y and z directions is 0.25, 0.06 and 0.29 mm respectively. This composite vision system can realize precise positioning of target within a single exposure time, which provides possibility for positioning under movement.

Evaluation and Compensation of the Effect of Dirt on Time-of-Flight 3D Imaging Systems

In the past few years, 3D imaging technology has received a lot of attention. Time-of-Flight (ToF) cameras based on the indirect ToF principle calculate depth information by measuring the phase shift between the emitted periodic signal and the reflected signal. When the illumination from the modulated light source reaches the same pixel through multiple paths, multipath interference will occur. Acquisitions at multiple modulation frequencies are then required to separate the interfering paths. Parametric spectral estimation methods, such as Prony’s method and its robust variants, have shown success in resolving multiple paths from multi-frequency measurements. With the widespread usage of ToF cameras in various fields, the operating environment of the camera should be taken into account in order to obtain accurate 3D images. In this work, we aim to study the effect of dirt on 3D imaging systems quantitatively. To this end, ten sheets with different transmittances, between 50% and 100% are used. To the best of our knowledge, for the first time we provide standardized methods for evaluating the effect of dirt on protective covers in the depth estimation of “flash lidar”. The accuracy of depth measurements obtained by multi-frequency multipath recovery in the absence of dust is up to 99%. However, when the light transmittance of the protective cover is less than 90%, the measurement result is already unreliable. The experiments have shown that the depth measurements obtained using the multi-frequency estimation method are more accurate in the presence of dust than the single-frequency estimation method.

Low-cost mobile mapping system solution for traffic sign segmentation using Azure Kinect

The mobile mapping system (MMS) could become the foundation of digital twins and 3D modeling, and is widely applicable in a variety of fields, such as infrastructure management, intelligent transportation systems, and smart cities. However, data collected by MMS is extensive and complex, making data processing difficult. We present a novel method for segmenting urban assets (specifically in this case study traffic signs) with a lower-cost Azure Kinect and automatic data processing workflows. First, it was necessary to verify the reliability of this approach using the Time of Flight (ToF) camera from Azure Kinect to detect road signs outdoors. Using the data generated by the ToF camera, we then extracted the Region of Interest (ROI) quickly and efficiently. After transforming the ROI to the RGB image, we obtained the traffic sign area through a hybrid color-shape based method. In addition, we calculated the distance between the traffic sign and Azure Kinect based on the depth image. The Coefficient of Variation cv averaged 1.1%. It is thus evident that Azure Kinect is reliable for outdoor traffic sign segmentation. Our algorithm has been compared with deep learning algorithms. According to our analysis, our algorithm has an accuracy of 0.8216, while the accuracy of deep learning is 0.7466, which indicates that our solution is more flexible and cost-effective.

Smartphone LiDAR Technologies for Surveying and Reality Modelling in Urban Scenarios: Evaluation Methods, Performance and Challenges

The aim of the research was to evaluate the performance of smartphone depth sensors (Time of Flight Camera(ToF) and Light Detection and Ranging (LiDAR)) from Android (Huawei P30 Pro) and iOS (iPhone 12 Pro and iPAD 2021 Pro) devices in order to build a 3D point cloud. In particular, the smartphones were tested in several case studies involving the scanning of several objects: 10 building material samples, a statue, an interior room environment and the remains of a Doric column in a major archaeological site. The quality of the point clouds was evaluated through visual analysis and using three eigenfeatures: surface variation, planarity and omnivariance. Based on this approach, some issues with the point clouds generated by smartphones were highlighted, such as surface splitting, loss of planarity and inertial navigation system drift problems. In addition, it can finally be deduced that, in the absence of scanning problems, the accuracies achievable from this type of scanning are ~1–3 cm. Therefore, this research intends to describe a method of quantifying anomalies occurring in smartphone scans and, more generally, to verify the quality of the point cloud obtained with these devices.

Imaging simulation of the AMCW ToF camera based on path tracking.

The time-of-flight (ToF) camera suffers from many error factors, such as multiple reflection or multipath interference and electronic and optical shot noise, making it difficult to simulate its imaging process. Aiming to test the ToF camera algorithm, it is important to obtain a depth image affected by these error factors. In order to model the light propagation behavior and the sensor effect in the imaging process of the ToF camera, an amplitude modulated continuous-wave (AMCW) ToF camera imaging simulation method based on path tracking is presented by deducing the path tracking algorithm model in the AMCW ToF camera theoretically and by realizing the physically based simulation by introducing the infrared bidirectional reflectance distribution function (BRDF) data of the actual materials. According to the constructed error evaluation indexes, the correctness of the imaging simulation method is verified based on the ground experiment. The mean absolute error (MAE) and root mean square error (RMSE) are 10.32 mm and 15.12 mm, respectively, which are less than the error results of the other two comparative simulation methods. The results show that the proposed method is reasonable and can provide reliable data support for AMCW ToF hardware development and algorithm testing.

Physics-Based TOF Imaging Simulation for Space Targets Based on Improved Path Tracing

Aiming at the application of close-up space measurement based on time-of-flight (TOF) cameras, according to the analysis of the characteristics of the space background environment and the imaging characteristics of the TOF camera, a physics-based amplitude modulated continuous wave (AMCW) TOF camera imaging simulation method for space targets based on the improved path tracing is proposed. Firstly, the microfacet bidirectional reflection distribution function (BRDF) model of several typical space target surface materials is fitted according to the measured BRDF data in the TOF camera response band to make it physics-based. Secondly, an improved path tracing algorithm is developed to adapt to the TOF camera by introducing a cosine component to characterize the modulated light in the TOF camera. Then, the imaging link simulation model considering the coupling effects of the BRDF of materials, the suppression of background illumination (SBI), optical system, detector, electronic equipment, platform vibration, and noise is established, and the simulation images of the TOF camera are obtained. Finally, ground tests are carried out, and the test shows that the relative error of the grey mean, grey variance, depth mean, and depth variance is 2.59%, 3.80%, 18.29%, and 14.58%, respectively; the MSE, SSIM, and PSNR results of our method are also better than those of the reference method. The ground test results verify the correctness of the proposed simulation model, which can provide image data support for the ground test of TOF camera algorithms for space targets.

Near-Infrared, Depth, Material: Towards a Trimodal Time-of-Flight Camera

Time-of-Flight cameras are active sensors that are able to capture both the light intensity reflected by each observed point in the scene and the distance between these points and the camera. Enhancing intensity images with a depth modality enables capturing surfaces in 3D and boosts the applicability of these sensors. Nevertheless, high-level information still needs to be extracted from the data stream in order to accomplish high-level tasks, like recognition or classification. Ideally, the semantic gap between sensor output and high-level requirements should be as small as possible, in order to reduce both computational cost and failure probability. An additional depth modality helps in this regard, but there are further cues that can be <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">seen</i> by a ToF sensor that have remained underexploited so far. In this paper we take the first steps towards a trimodal ToF camera, which adds a valuable <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">material modality</i> to the classical intensity and depth modalities. To this end, ToF raw data is used to obtain Fourier samples of the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">material impulse response function</i> (MIRF) to modulated illumination. The MIRF depends on the surface- and sub-surface-level scattering mechanisms of the material and, thus, can be used to identify materials of different nature. Consequently, distinctive feature vectors can be obtained from the Fourier measurements. Additionally, including the incidence angle in the feature vectors allows capturing the MIRF dependency on this parameter. Experimental validation confirms the feasibility of this approach. We also constructed a live demonstrator of our trimodal ToF camera.

TIMo-A Dataset for Indoor Building Monitoring with a Time-of-Flight Camera.

We present TIMo (Time-of-flight Indoor Monitoring), a dataset for video-based monitoring of indoor spaces captured using a time-of-flight (ToF) camera. The resulting depth videos feature people performing a set of different predefined actions, for which we provide detailed annotations. Person detection for people counting and anomaly detection are the two targeted applications. Most existing surveillance video datasets provide either grayscale or RGB videos. Depth information, on the other hand, is still a rarity in this class of datasets in spite of being popular and much more common in other research fields within computer vision. Our dataset addresses this gap in the landscape of surveillance video datasets. The recordings took place at two different locations with the ToF camera set up either in a top-down or a tilted perspective on the scene. Moreover, we provide experimental evaluation results from baseline algorithms.