Time-of-flight Camera Research Articles

Fast point cloud registration algorithm using parallel computing strategy

At present, the traditional ICP (Iterative Closest Point) algorithm used by TOF (Time-Of-Flight) cameras has the problem of high time-consuming in nearest point search process, resulting in low data update rate. Therefore, based on the characteristics of the data structure of the ICP algorithm, a multi-stage pipeline acceleration system is proposed, and the point cloud registration algorithm used in this system is studied. Firstly, design multiple units to store point cloud data and serve as the main search module. Secondly, make each group of units perform parallel operations, input the stored point clouds into their respective calculation modules, select a model point, traverse all query points, and perform subtraction. Then replace the model point and repeat the process until all model points are calculated. Finally, the query point corresponding to the minimum value in each model point difference data is selected and the model point is output as a point pair to the CPU to complete the final algorithm iteration. The experimental results show that at 30000 points, a single closest point search takes 25.65ms; at 20000 points, a single closest point search takes 11.65ms; at 10000 points, a single closest point search takes 2.88ms. It basically meets the requirements of TOF camera on-orbit measurement for the stability and reliability of point cloud registration and high data update rate.

Contactless Vital Sign Monitoring System for In-Vehicle Driver Monitoring Using a Near-Infrared Time-of-Flight Camera

We demonstrate a Contactless Vital Sign Monitoring (CVSM) system and road-test the system for in-cabin driver monitoring using a near-infrared indirect Time-of-Flight (ToF) camera. The CVSM measures both heart rate (HR) and respiration rate (RR) by leveraging the simultaneously measured grayscale and depth information from a ToF camera. For a camera-based driver monitoring system (DMS), key challenges from varying background illumination and motion-induced artifacts need to be addressed. In this study, active illumination and depth-based motion compensation are used to mitigate these two challenges. For HR measurements, active illumination allows the system to work under various lighting conditions, while our depth-based motion compensation has the advantage of directly measuring the motion of the driver without making prior assumptions about the motion artifacts. In addition, we can extract RR directly from the chest wall motion, circumventing the challenge of acquiring RR from the near-infrared photoplethysmography (PPG) signal of low signal quality. We investigate the system’s performance in various scenarios, including monitoring both drivers and passengers while driving on highways and local roads. Our results show that our CVSM system is ambient light agnostic, and the success rates of HR measurements on the highway are 82% and 71.9% for the passenger and driver, respectively. At the same time, we show that the system can measure RR on users driving on a highway with a mean deviation of −1.4 breaths per minute (BPM). With reliable HR and RR measurement in the vehicle, the CVSM system could one day be a key enabler to sudden sickness or drowsiness detection in DMS.

Self-supervised learning via cluster distance prediction for operating room context awareness.

Semantic segmentation and activity classification are key components to create intelligent surgical systems able to understand and assist clinical workflow. In the operating room, semantic segmentation is at the core of creating robots aware of clinical surroundings, whereas activity classification aims at understanding OR workflow at a higher level. State-of-the-art semantic segmentation and activity recognition approaches are fully supervised, which is not scalable. Self-supervision can decrease the amount of annotated data needed. We propose a new 3D self-supervised task for OR scene understanding utilizing OR scene images captured with ToF cameras. Contrary to other self-supervised approaches, where handcrafted pretext tasks are focused on 2D image features, our proposed task consists of predicting relative 3D distance of image patches by exploiting the depth maps. By learning 3D spatial context, it generates discriminative features for our downstream tasks. Our approach is evaluated on two tasks and datasets containing multiview data captured from clinical scenarios. We demonstrate a noteworthy improvement in performance on both tasks, specifically on low-regime data where utility of self-supervised learning is the highest. We propose a novel privacy-preserving self-supervised approach utilizing depth maps. Our proposed method shows performance on par with other self-supervised approaches and could be an interesting way to alleviate the burden of full supervision.

Time-of-Flight Imaging in Fog Using Polarization Phasor Imaging.

Due to the light scattered by atmospheric aerosols, the amplitude image contrast is degraded and the depth measurement is greatly distorted for time-of-flight (ToF) imaging in fog. The problem limits ToF imaging to be applied in outdoor settings, such as autonomous driving. To improve the quality of the images captured by ToF cameras, we propose a polarization phasor imaging method for image recovery in foggy scenes. In this paper, optical polarimetric defogging is introduced into ToF phasor imaging, and the degree of polarization phasor is proposed to estimate the scattering component. A polarization phasor imaging model is established, aiming at separating the target component from the signal received by ToF cameras to recover the amplitude and depth information. The effectiveness of this method is confirmed by several experiments with artificial fog, and the experimental results demonstrate that the proposed method significantly improves the image quality, with robustness in different thicknesses of fog.

Research on Depth Image Tracking Based on Aerial Refueling

Aiming at the problem of the current aerial refueling docking which is difficult to accurately identify the floating cone sleeve in a complex environment, the Meanshift algorithm can efficiently track moving targets in the plane, there is not very good solution strategy for the pose measurement of the cone sleeve and the movement in the optical axis direction. The Time-Of-Flight (TOF) in the depth camera can directly collect depth images and grayscale images, providing the distance information displayed by a flat camera. By using the Meanshift algorithm to iterate in the three-dimensional space, the Kalman filter is added to achieve robustness to the tracking of moving targets, effectively solve the problem of occlusion of the target, and provide the coordinate position of the target. In the experiment, 600 frames of images are extracted for comparison. The effect of the TOF camera has higher accuracy and stability than that of the flat camera. It still has a stable tracking effect after using the props to cover the cone model, which is better than the traditional plane Meanshift algorithm.

가축에 대한 3차원 형상 획득을 위한 측정시스템 개발

Smart farms are being actively developed to manage the growth of livestock. The quality of livestock is generally evaluated based on their body form and weight. Livestock of various sizes and appearances exist. Even within the same species, the appearance varies depending on the growth and pregnancy status. Live animals keep moving, and thus it is difficult to measure every single body part of them accurately. In this study, we propose a measurement system that can acquire the 3D shape of a moving livestock. A TOF (time of flight) camera is used to quickly measure the moving state, and multiple cameras are used to acquire the overall shape. A calibration work is required when using multiple cameras, and it is configured to be automatically calibrated using sphere targets. The accuracy and precision of the measurement system has been verified, and it is applied to pigs to obtain three-dimensional point cloud data.

InDepth

Mobile Augmented Reality (AR) demands realistic rendering of virtual content that seamlessly blends into the physical environment. For this reason, AR headsets and recent smartphones are increasingly equipped with Time-of-Flight (ToF) cameras to acquire depth maps of a scene in real-time. ToF cameras are cheap and fast, however, they suffer from several issues that affect the quality of depth data, ultimately hampering their use for mobile AR. Among them, scale errors of virtual objects - appearing much bigger or smaller than what they should be - are particularly noticeable and unpleasant. This article specifically addresses these challenges by proposing InDepth, a real-time depth inpainting system based on edge computing. InDepth employs a novel deep neural network (DNN) architecture to improve the accuracy of depth maps obtained from ToF cameras. The DNN fills holes and corrects artifacts in the depth maps with high accuracy and eight times lower inference time than the state of the art. An extensive performance evaluation in real settings shows that InDepth reduces the mean absolute error by a factor of four with respect to ARCore DepthLab. Finally, a user study reveals that InDepth is effective in rendering correctly-scaled virtual objects, outperforming DepthLab.

System Design for Pose Determination of Spacecraft Using Time-of-Flight Sensors

The pose determination between nanosatellites and the cooperative spacecraft is essential for swarm in-orbit services. Time-of–flight (ToF) sensors are one of the most promising sensors to achieve the tasks. This paper presented an end-to-end assessment of how these sensors were used for pose estimation. First, an embedded system was designed based on the ToF camera with lasers as a driven light source. Gray and depth images were collected to detect and match the cooperative spacecraft in real time, obtaining the pose information. A threshold-based segmentation was proposed to find a small set of the pixels belonging to reflector markers. Only operating on the defined active pixel set reduced computational resources. Then, morphological detection combined with an edge following-based ellipse detection extracted the centroid coordinate of the circular marker, while the center-of-heart rate was calculated as the recognition condition. Next, the marker matching was completed using a deterministic annealing algorithm, obtaining two sets of 3D coordinates. A singular value decomposition (SVD) algorithm estimated the relative pose between the nanosatellite and the spacecraft. In the experiments, the pose calculated by the TOF camera reached an accuracy of 0.13 degrees and 2 mm. It accurately identified the markers and determined the pose, verifying the feasibility of the ToF camera for rendezvous and docking.

3D Measurement of Welded Object Using ToF Camera

3D Measurement of Welded Object Using ToF Camera

A Control Method for SMA Robotic Actuators

Soft robots show remarkable benefits over conventional rigid robots due to their high energy density and other factors. We propose a circular soft robot integrating the control system, actuating it with several shape memory alloy (SMA) actuators. Our research methodology involved connecting the DC voltage supply and L298N module to provide uninterrupted power to the actuator and thence control the actuator via Arduino Uno MCU and TOF camera. We designed the controller to simultaneously complete the positioning and manipulation tasks. A novel method utilizing visual servo and closed-loop control algorithm was proposed and integrated into the controller. This method involves the implementation of multi-gait locomotion using SMA actuators. Additionally, the development of closed-loop dynamic controllers for a continuous soft robot is also evaluated. The proposed control model is designed and simulated on the MATLAB tool. To verify the efficiency of the proposed forward-feedback controller, simulations and experiments were conducted in the current study. A new control method using PID control based on the Kalman filtering algorithm and visual servo for the SMA actuator designed in this research is introduced. We conclude that applying spike excitation voltage would benefit the actuating performance. Overall, the experimental results demonstrated a promising future for the purposed control method.

Depth Restoration in Under-Display Time-of-Flight Imaging.

Under-display imaging has recently received considerable attention in both academia and industry. As a variation of this technique, under-display ToF (UD-ToF) cameras enable depth sensing for full-screen devices. However, it also brings problems of image blurring, signal-to-noise ratio and ranging accuracy reduction. To address these issues, we propose a cascaded deep network to improve the quality of UD-ToF depth maps. The network comprises two subnets, with the first using a complex-valued network in raw domain to perform denoising, deblurring and raw measurements enhancement jointly, while the second refining depth maps in depth domain based on the proposed multi-scale depth enhancement block (MSDEB). To enable training, we establish a data acquisition device and construct a real UD-ToF dataset by collecting real paired ToF raw data. Besides, we also build a large-scale synthetic UD-ToF dataset through noise analysis. The quantitative and qualitative evaluation results on public datasets and ours demonstrate that the presented network outperforms state-of-the-art algorithms and can further promote full-screen devices in practical applications.

ToF and Stereo Data Fusion Using Dynamic Search Range Stereo Matching

Time-of-Flight (ToF) sensors and stereo vision systems are both widely used for capturing depth data. They have some complementary strengths and limitations, which have been exploited in prior research to produce more accurate depth maps by fusing data from the two sources. However, among these diverse data fusion approaches, none of them provides an end-to-end neural network solution. In this work, we propose the first end-to-end ToF and stereo data fusion network using the coarse-to-fine matching framework, where the prior of ToF depth is integrated into the stereo matching process by constraining the search range of stereo matching within an interval around the ToF camera depth measurement. We adopt a dynamic search range for each pixel according to an estimated ToF error map, which is more efficient and effective than a constant one when handling various errors. The ToF error map is estimated by the ToF error estimator branching out from the stereo matching network. Both ToF error estimation and stereo matching are performed in a joint framework, with the two tasks assisting each other mutually. We also propose an upsampling module to replace the naive bilinear upsampling in the coarse-to-fine stereo matching network, which reduces the error caused by the upsampling. The proposed deep network is trained end-to-end on synthetic datasets and generalizable to real-world datasets without further fine-tuning. Experimental results show that our fusion method achieves higher accuracy than either ToF or stereo alone, and outperforms state-of-the-art fusion methods on both synthetic and real data.

Stray-Light Mitigation for Under-Display Time-of-Flight Imagers

The next big thing in smartphone development will be under-display cameras. Conventional color cameras, implemented under-display, starting to appear for consumer products. For secure face-authentication, additional information in the form of the third dimension is required. The time-of-flight (ToF) measurement principle is based on measuring the phase and consequently the delay of emitted and reflected light pulses. Unwanted reflections of the display hinder the implementation of under-display ToF cameras. These reflections introduce errors and circumvent the processing of the depth information. In this paper, we present novel measurement principles enabling under-display Time-of-Flight imaging. In contrast to traditional continuous wave ToF imaging, our measurement principles remove the the display introduced stray-light reflection from the captured images. We present two independent approaches. One is based on the combination of continuous wave measurement with coded modulation. The second method is based on measurements with two modulation frequencies. These methods require a non-traditional approach for depth and amplitude calculation. Furthermore, design considerations are presented, such as calibration and temperature dependency. Our extensive evaluation considers the reflection suppression performance and the fidelity of the captured depth data. The results reveal the suppression of the stray-light by more than 80%. We evaluate our methods with a detailed description of the advantages and disadvantages of the different methods. This work is a starting point for the implementation of under-display ToF imaging.

РАЗРАБОТКА АЛГОРИТМА ОЦЕНКИ ФИЗИОЛОГИЧЕСКОЙ УПИТАННОСТИ МОЛОЧНЫХ КОРОВ

Cow’s body condition scoring (BCS) based on neural network algorithms is necessary to monitor the health and productivity of the animals. To develop an algorithm for assessing the physiological fatness of dairy cows according to the parameters of the fermur heads, the sacrum and the hunger hollow, the authors studied eighty animals on a farm of the state unitary enterprise “Grigorievskoe”, which were divided into fatness groups (from 1 to 5). The studies were carried out in November 2021 during the morning milking. Data were collected using a 3D ToF camera O3D303. Previously, the installation of a three-dimensional camera on the farm was sim ulated, and an algorithm was developed that takes into account the cow height and the distance from the highest point of the spine to the three-dimensional camera. An algorithm for assessing the physiological fatness of dairy cows (BCS) has been developed to take into account the condition of the fermur heads, the sacrum, the hunger hollow, which determines the highest point of the withers, the proportions between the body length and width, as well as the depth of the hunger hollows and the severity of the caudal ligament. Software has been developed to register the unique number of a cow and determine the BCS, as well as show the dynamics of changes in the animal’s fatness. Data were processed in accordance with the developed algorithm. The image was processed using the regression method. Comparison results of the BCS of cows, obtained according to the developed algorithm, and the experts’ assessment showed that the algorithm error in the fatness range of 2…4 points averaged 10%. When determining the BCS of cows with borderline and limit fatness state (1 and 5 points), the measurement error by the proposed algorithm increased to 25%. Based on the results obtained, the authors recommend pre-setting a neural network for further research; determine the correction factor for fatness points 1 and 5; fi nalize the software, develop customized software and an automatic system for body condition scoring, and conduct tests.

USING A CONVOLUTIONAL NEURAL NETWORK TO DETERMINE THE TEAT LIMITS OF A COW’S UDDER

In dairy farming, there is an active use of systems for the accurate recognizing of cow’s teats, which is necessary for robotic milking. Object detection consists both in assessing the exact position of the object (localization of the object) and in determining the category to which it belongs (classification of the object). (Research purpose) The research purpose is developing a method for searching for clusters of points to determine the limits of the teat in three-dimensional maps obtained with a 3D ToF camera. (Materials and methods) An IFM O3D303 camera with a resolution of 352 × 264 pixels was used. Authors have developed a structure for analyzing static and dynamic scenes, the model consists of five blocks of convolutional layers, three fully connected layers of neurons and a classifier layer. The first two blocks contain two layers, the other three blocks contain three convolution layers. (Results and discussion) The software highlights the detected teats in color. The method is considered suitable if at 10 repetitions of the experiment the limits of four teats are recognized and there are no artifacts, the experiment can be considered successful, the permissible error of the experiment is no more than 10 percent. During the search for limits of teats, the algorithm repeatedly searches for points in the bounding cylinder of the refined cluster of points. (Conclusions) A method for searching for clusters of points to identify teats limits in three-dimensional maps processing data from a 3D ToF camera using a neural network with «anchor rectangles». Data from 36 animals and 6000 images were collected for training the neural network, field experiments confirmed the accuracy of the neural network in 90 percent. The proposed method allows determining length X, diameter Y and inclination angle Z of the teat in automatic mode.

Relative Pose Determination of Uncooperative Spacecraft Based on Circle Feature

This paper investigates the problem of spacecraft relative navigation with respect to an unknown target during the close-proximity operations in the on-orbit service system. The serving spacecraft is equipped with a Time-of-Flight (ToF) camera for object recognition and feature detection. A fast and robust relative navigation strategy for acquisition is presented without any extra information about the target by using the natural circle features. The architecture of the proposed relative navigation strategy consists of three ingredients. First, a point cloud segmentation method based on the auxiliary gray image is developed for fast extraction of the circle feature point cloud of the target. Secondly, a new parameter fitting method of circle features is proposed including circle feature calculation by two different geometric models and results’ fusion. Finally, a specific definition of the coordinate frame system is introduced to solve the relative pose with respect to the uncooperative target. In order to validate the efficiency of the segmentation, an experimental test is conducted based on real-time image data acquired by the ToF camera. The total time consumption is saved by 94%. In addition, numerical simulations are carried out to evaluate the proposed navigation algorithm. It shows good robustness under the different levels of noises.

Contactless Vital Sign Monitoring System for Heart and Respiratory Rate Measurements with Motion Compensation Using a Near-Infrared Time-of-Flight Camera

This study describes a contactless vital sign monitoring (CVSM) system capable of measuring heart rate (HR) and respiration rate (RR) using a low-power, indirect time-of-flight (ToF) camera. The system takes advantage of both the active infrared illumination as well as the additional depth information from the ToF camera to compensate for the motion-induced artifacts during the HR measurements. The depth information captures how the user is moving with respect to the camera and, therefore, can be used to differentiate where the intensity change in the raw signal is from the underlying heartbeat or motion. Moreover, from the depth information, the system can acquire respiration rate by directly measuring the motion of the chest wall during breathing. We also conducted a pilot human study using this system with 29 participants of different demographics such as age, gender, and skin color. Our study shows that with depth-based motion compensation, the success rate (system measurement within 10% of reference) of HR measurements increases to 75%, as compared to 35% when motion compensation is not used. The mean HR deviation from the reference also drops from 21 BPM to −6.25 BPM when we apply the depth-based motion compensation. In terms of the RR measurement, our system shows a mean deviation of 1.7 BPM from the reference measurement. The pilot human study shows the system performance is independent of skin color but weakly dependent on gender and age.

EDoF-ToF: extended depth of field time-of-flight imaging.

Conventional continuous-wave amplitude-modulated time-of-flight (CWAM ToF) cameras suffer from a fundamental trade-off between light throughput and depth of field (DoF): a larger lens aperture allows more light collection but suffers from significantly lower DoF. However, both high light throughput, which increases signal-to-noise ratio, and a wide DoF, which enlarges the system's applicable depth range, are valuable for CWAM ToF applications. In this work, we propose EDoF-ToF, an algorithmic method to extend the DoF of large-aperture CWAM ToF cameras by using a neural network to deblur objects outside of the lens's narrow focal region and thus produce an all-in-focus measurement. A key component of our work is the proposed large-aperture ToF training data simulator, which models the depth-dependent blurs and partial occlusions caused by such apertures. Contrary to conventional image deblurring where the blur model is typically linear, ToF depth maps are nonlinear functions of scene intensities, resulting in a nonlinear blur model that we also derive for our simulator. Unlike extended DoF for conventional photography where depth information needs to be encoded (or made depth-invariant) using additional hardware (phase masks, focal sweeping, etc.), ToF sensor measurements naturally encode depth information, allowing a completely software solution to extended DoF. We experimentally demonstrate EDoF-ToF increasing the DoF of a conventional ToF system by 3.6 ×, effectively achieving the DoF of a smaller lens aperture that allows 22.1 × less light. Ultimately, EDoF-ToF enables CWAM ToF cameras to enjoy the benefits of both high light throughput and a wide DoF.

기술융합 미디어아트를 위한 3D 터치스크린 제작법에 관한 연구

This study began with the author’s interest since the introduction of touchscreens using TOF cameras published in Siggraph 2005. Since the release of the “Khronos Projector” project in 2005, research and use of touchscreens using TOF cameras has continued to increase. However, hardware development such as thinning the screen and using infrared cameras instead of TOF cameras has been the mainstream of camera-based touchscreen research, and no further research has been conducted until recently to show the scalability of the production until recently. In this background, the purpose of this study is to enable new production by using a depth camera enabled touchscreen (hereinafter FTIR screen). Therefore, this paper paid attention to the 3D touch announced by “Apple” company and implemented research that applies this technology to FTIR screens.<BR> The 3D touch conducted in this study was implemented using the characteristics that the depth of touch and it’s surrounding area changes depending on the force exerted on the cloth screen. The contents and results of the study to utilize these features are as follows.<BR> Firstly, the author conducted research on depth cameras, fabrics and beam projectors.<BR> Secondly, he developed a software that tracks the position of touch along with the depth change of it’s surrounding area using a depth camera.<BR> Thirdly, a 3D touch function was tested using the software developed above with screens made by ourselves.<BR> Fourthly, it was used to produce technology-converged artworks using screens developed in this study. It was confirmed that the screen developed in this study worked stably as it was used for various art works in a total of three exhibitions. Based on these research results, the author will be studying on the use of FTIR screens and force exerted; therefore, foreseeing more diverse artworks using such methods in the future.

Application of Nanooptics in Photographic Imagery and Medical Imaging

Background. At present, with the continuous development of nanotechnology, great changes have taken place in people’s lives in medical treatment, production, daily leisure, and so on. Nanooptical technology is entirely based on nanotechnology that laser and visible light are limited to submicron structures (nanopores, nanoslits, and nanoneedles). Due to the great development potential of nanooptical technology in nanoscale sensors, TOF camera applications, THz imaging technology, and other imaging equipment materials and applications, people have been interested in it, recently. Scope and Approach. In this review, the importance of good practices for nanooptical technology used in equipment as both nanometer scale sensors and optical auxiliary equipment is described. Based on recent reports, this work discussed the development of nanooptical technology in daily photography and medical imaging from both the positive and the negative sides and compared the engineering techniques. Key Findings and Conclusions. As a kind of new optical technology, nanooptical technology can produce the plasmonic effect under the intense collision of atoms and electrons in nanostructures. It has significant effects in superresolution nanolithography, high-density data storage, near-field optics, and other fields. Although the current nanooptic technology is not extremely mature, the results obtained from current works are pointing out that nanooptical technology is the future of daily imaging and medical imaging, and it also will play a positive role in the improvement of people’s health and ecological environment quality. As a trend, nanooptical technology is developing in the direction of energy-saving, portability, high efficiency, and low pollution, and in the upsurge of environmental protection in the world, nanooptical technology will surely achieve amazing development in the field of daily photography and medical imaging. Under the huge market demand and innovation power, nanophotonics technology will cover all emerging technologies that share the same research field with it and take advantage of each technology (terahertz, cell and molecular microscopy, and nanoscale probes) to develop an unprecedented new century in nanoscience. The future trends of research contain finding new imaging equipment with nanostructure, designing nanooptical products, and improving engineering techniques.