Time-of-flight Sensors Research Articles

Neuron‐Inspired Time‐of‐Flight Sensing via Spike‐Timing‐Dependent Plasticity of Artificial Synapses

Time-of-Flight Sensing In article number 2100159, Kyusang Lee and co-workers demonstrate a neuron-like resistive timeof-flight sensor method for 3D sensing, with implications in light-detection and ranging (LiDAR),automotive vehicles, biomedical in vivo imaging, and augmented/virtual reality.

Unstable Landing Platform Pose Estimation Based on Camera and Range Sensor Homogeneous Fusion (CRHF)

Much research has been accomplished in the area of drone landing and specifically pose estimation. While some of these works focus on sensor fusion using GPS, or GNSS, we propose a method that uses sensors, including four Time of Flight (ToF) range sensors and a monocular camera. However, when the descending platform is unstable, for example, on ships in the ocean, the uncertainty will grow, and the tracking will fail easily. We designed an algorithm that includes four ToF sensors for calibration and one for pose estimation. The landing process was divided into two main parts, the rendezvous and the final landing. Two important assumptions were made for these two phases. During the rendezvous, the landing platform movement can be ignored, while during the landing phase, the drone is assumed to be stable and waiting for the best time to land. The current research modifies the landing part as a stable drone and an unstable landing platform, which is a Stewart platform, with a mounted AprilTag. A novel algorithm for calibration was used based on color thresholding, a convex hull, and centroid extraction. Next, using the homogeneous coordinate equations of the sensors’ touching points, the focal length in the X and Y directions can be calculated. In addition, knowing the plane equation allows the Z coordinates of the landmark points to be projected. The homogeneous coordinate equation was then used to obtain the landmark’s X and Y Cartesian coordinates. Finally, 3D rigid body transformation is engaged to project the landing platform transformation in the camera frame. The test bench used Software-in-the-Loop (SIL) to confirm the practicality of the method. The results of this work are promising for unstable landing platform pose estimation and offer a significant improvement over the single-camera pose estimation AprilTag detection algorithms (ATDA).

Probabilistic Modeling of Motion Blur for Time-of-Flight Sensors.

Synthetically creating motion blur in two-dimensional (2D) images is a well-understood process and has been used in image processing for developing deblurring systems. There are no well-established techniques for synthetically generating arbitrary motion blur within three-dimensional (3D) images, such as depth maps and point clouds since their behavior is not as well understood. As a prerequisite, we have previously developed a method for generating synthetic motion blur in a plane that is parallel to the sensor detector plane. In this work, as a major extension, we generalize our previously developed framework for synthetically generating linear and radial motion blur along planes that are at arbitrary angles with respect to the sensor detector plane. Our framework accurately captures the behavior of the real motion blur that is encountered using a Time-of-Flight (ToF) sensor. This work uses a probabilistic model that predicts the location of invalid pixels that are typically present within depth maps that contain real motion blur. More specifically, the probabilistic model considers different angles of motion paths and the velocity of an object with respect to the image plane of a ToF sensor. Extensive experimental results are shown that demonstrate how our framework can be applied to synthetically create radial, linear, and combined radial-linear motion blur. We quantify the accuracy of the synthetic generation method by comparing the resulting synthetic depth map to the experimentally captured depth map with motion. Our results indicate that our framework achieves an average Boundary F1 (BF) score of 0.7192 for invalid pixels for synthetic radial motion blur, an average BF score of 0.8778 for synthetic linear motion blur, and an average BF score of 0.62 for synthetic combined radial-linear motion blur.

A systematic non-uniformity correction method for correlation-based ToF imaging.

Correlation-based time-of-flight (ToF) imaging enables a diverse range of applications for its high frame rate, high resolution and low cost. However, the non-uniformity of the sensor significantly affects the flat-field accuracy of the ToF imaging system. In this paper, we analyze the sources of the non-uniformity and propose a systematic non-uniformity correction (NUC) method. The method utilizes the amplitude image, which can directly reflect the non-uniformity characteristics of the ToF sensor, to conduct NUC. Based on the established NUC system, the effectiveness and feasibility of the proposed NUC method are verified. Compared with the traditional methods, the RMSE was significantly reduced, while the SNR and PSNR were effectively improved. We believe this study provides new insights into the understanding of noise in the correlation-based ToF imaging system, and also provides effective references for the NUC of the three-dimensional measuring instruments.

Sensor Module Combining Time-of-Flight With Self-Capacitance Proximity and Tactile Sensors for Robot

Human collaborative robots (HCRs) are a useful future workforce because they can operate alongside people. Proximity and tactile sensing are important for HCRs to ensure safety. In this study, we propose a combined sensor module for HCRs using time-of-flight (ToF) as well as self-capacitance proximity and tactile sensors. The distance to an object in a noncontact range can be detected by a ToF sensor; however, the detection accuracy and the X-Y measurement range are lower for short ranges, with the Z axis as the measurement plane direction. The self-capacitance proximity and tactile sensors can detect objects and contact conditions in short ranges. For high-sensitivity proximity and contact measurements over a large distance, two of the aforementioned sensors can be combined. Prototypes of the sensor modules were prepared, and sixteen modules were mounted on a robot arm to detect the position and distance of an object at 300 mm or less, along with the pressure and position upon contact. In addition, the robot arm was controlled using information obtained from the sensors. The proposed sensor modules can thus be employed for proximity and tactile sensing in HCRs.

Neuron‐Inspired Time‐of‐Flight Sensing via Spike‐Timing‐Dependent Plasticity of Artificial Synapses

3D sensing is a primitive function that allows imaging with depth information generally achieved via the time‐of‐flight (ToF) principle. However, time‐to‐digital converters (TDCs) in conventional ToF sensors are usually bulky, complex, and exhibit large delay and power loss. To overcome these issues, a resistive time‐of‐flight (R‐ToF) sensor that can measure the depth information in an analog domain by mimicking the biological process of spike‐timing‐dependent plasticity (STDP) is proposed herein. The R‐ToF sensors based on integrated avalanche photodiodes (APDs) with memristive intelligent matters achieve a scan depth of up to 55 cm (≈89% accuracy and 2.93 cm standard deviation) and low power consumption (0.5 nJ/step) without TDCs. The in‐depth computing is realized via R‐ToF 3D imaging and memristive classification. This R‐ToF system opens a new pathway for miniaturized and energy‐efficient neuromorphic vision engineering that can be harnessed in light‐detection and ranging (LiDAR), automotive vehicles, biomedical in vivo imaging, and augmented/virtual reality.

Non-contact Breathing Rate Detection Based on Time of Flight Sensor.

There are a growing number of methods to detect a person's breathing rate, but most techniques still either require contact with body skin or are usually uncomfortable to wear, too expensive and unfriendly for daily monitoring. The massive adoption of smartphones in recent years has created many opportunities to improve daily health monitoring. In this work, we demonstrated that off-the-shelf ToF lens on smartphones can capture a person's breathing rate while still. In addition, we proposed a method for extracting breathing rate from ToF signal and compared it with actual breathing rate obtained from temperature sensor. We evaluated the breathing rate accuracy of 6 people at rest, with a mean absolute error of 0.009Hz when considering different mean breathing rate conditions. Moreover, the mean absolute error percentage is 3.56% and the root mean squared percentage error is 6.64%, which is smaller than other methods of non-contact breathing rate detection in recent works.

A 1.2-Mpixel Indirect Time-of-Flight Image Sensor With 4-Tap 3.5-μm Pixels for Peak Current Mitigation and Multi-User Interference Cancellation

A 1.2-Mpixel indirect time-of-flight (ToF) CMOS image sensor is presented to lower peak current and to cancel out multi-user interference. The proposed 4-tap 3.5- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mu \text{m}$ </tex-math></inline-formula> demodulation pixel is optimally designed to improve quantum efficiency (QE) and demodulation contrast (DC). A new “multiple-interleaving” scheme is proposed to reduce the peak current and self-compensate fixed-pattern phase noise (FPPN) with the minimal side effect of DC imbalance. To cancel out interference from the other ToF sensors, a multi-user interference cancellation (MUIC) scheme based on pseudorandom modulation is adopted. From the measurement results, we achieve a high QE of 38% at 940 nm and a high DC of 96% and 80% at 100- and 200-MHz modulation frequencies, respectively. With the superior pixel performance, the depth noise of <0.92% and <0.3% is achieved at the full and 2 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times $ </tex-math></inline-formula> 2 binning modes, respectively. The multiple-interleaving scheme suppresses the peak current to less than 0.9 A and removes the column FPPN with constant DC for the whole pixel array. The proposed MUIC proves that at least two interferers can be successfully rejected. The sensor chip is fabricated in a two-stack process with 65 nm for the top die and 65 nm for the bottom die.

Smart Surgical Light: Identification of Surgical Field States Using Time of Flight Sensors

This work presents a method for identifying surgical field states using time-of-flight (ToF) sensors equipped with a surgical light. It is important to understand the surgical field state in a smart surgical room. In this study, we aimed to identify surgical field states by using 28 ToF sensors with a surgical light installed on each. In the experimental condition, we obtained a sensor dataset by changing the number of people, posture, and movement state of a person under the surgical light. The identification accuracy of the proposed system was evaluated by applying machine learning techniques. This system can be realized simply by attaching ToF sensors to the surface of an existing surgical light.

Phase Unwrapping for Time-of-Flight Sensor Based on Image Segmentation

Phase unwrapping is a fundamental problem in Time-of-Flight (ToF) imaging, especially when high modulation frequency is used to achieve precise measurement accuracy. This paper proposes a novel double-frequency based phase unwrapping method for ToF sensors. The new method incorporates the idea of image segmentation to solve phase unwrapping region-by-region instead of pixel-by-pixel. The depth image is segmented based on the constraint between depth measurements and wrap numbers at two modulation frequencies. To avoid misclassification of the pixels around phase jump edges in the presence of inevitable noise, we employ a modified distance function to classify the edge pixels into the corresponding connected area. Furthermore, a graph model is used to accurately model the phase jumping relation between different connected areas. On this basis, we propose an MRF framework to fuse the amplitude and depth information to unwrap the phase. Experimental results on both synthetic and real-world data demonstrate that, the proposed method is robust to noise and outperforms state-of-the-art methods, while being highly efficient enabling real-time running.

Omnidirectional Proximity Sensor System for Drones Using Optical Time‐of‐Flight Sensors

In recent years, the use of unmanned aerial vehicles, including drones, to conduct inspections has been attracting increasing attention. When a drone is used for inspections, it often approaches objects and obstacles, and its collision risk increases the closer it gets. In this study, we developed an omnidirectional proximity sensor system for drones using optical time‐of‐flight (ToF) sensors to avoid collisions. The sensor system consists of 87 ToF sensors that can estimate the distance between the sensors and objects. The ToF sensors are mounted on both the propeller guard of the drone and its body. Thus, they can measure objects horizontally, at ±45° Z direction, and above the drone. Further, the sensor system can detect objects around the drone in a noncontact area. Accordingly, the drone can be controlled using the measured data. Our results show that drones using the proposed sensor system can avoid unexpected contacts and collisions during flights. Consequently, we believe that the sensor system will be highly useful in drone operation. © 2021 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

Hardhat Detection Using IR and Depth Frames

The construction industry is one of the largest sectors and 20% of the total deaths happen here. Safety hardhat helmets are one of the precautionary methods recommended and followed in a construction site. Detecting and classifying people wearing and not wearing helmets is an important task. Safety and Security is one of the most researched fields in recent years and acquiring 3D geometric information from any real environment is an important task for such applications. Well-known methods like stereo vision camera systems suffer from high time consumption or from the inability to match corresponding points inhomogeneous regions. Time of Flight (ToF)technology, being a recent development, fulfills features desired for real-time distance acquisition along with the compact size and higher frame rate. A safety application based on the ToF technology ad IR imaging is proposed in the paper. The ToF sensor provides depth information for each pixel as opposed to RGB values in case stereo camera-based systems. The depth sense cameras provide IR images along with the depth information. A YOLO framework is used to classify images. YOLO being faster than RCNN and the faster RCNN is much suitable for real-time classification. The model was trained on 50,000 images. Weights obtained during the 6000th epoch were chosen. A Mean Average Precision of 56% was obtained while testing.

Low attenuation soft and stretchable elastomeric optical waveguides

Low-attenuation polymeric waveguides transmit signals efficiently. Stretchable, cost-effective, and lightweight elastomeric optical waveguides (EOWs) were fabricated by one-step slow melt-extrusion, which induces slower crystallization kinetics and greater crystal nucleation to form uniform and greater defect-free crystals. Structurally homogeneous EOWs have attenuation coefficients of 0.194 dB/cm using a power meter and 0.268 dB/cm using time-of-flight sensor (ToFS). Higher AC in latter method is attributed to crosstalk and propagation of fewer modes in bent configuration. Softer and stretchable EOWs with higher optical transmittance could be used for wearable sensors and soft robotics.

A compact time-to-amplitude converter for single-photon time-of-flight measurement

This work proposes a compact time-to-amplitude converter (TAC) for single-photon time-of-flight (TOF) measurement. Based on the principle of current mirror integration, the regulated cascode output structure is used to improve the integration linearity. Fabricated in 180 nm standard CMOS technology, the test results show that the proposed TAC achieves an integral nonlinearity of less than 1.1 LSB, and a timing resolution of 78 ps in a full-scale range (FSR) of 20 ns. Moreover, the readout voltage only varies by 1% during 1 ms hold-off time. The simple circuit structure make TAC occupy a small area of 120 µm2 and consume low dynamic power of 36.8 µW. The TOF measurement error is less than 5.2 cm, and the measurement precision is up to 1.61 cm at 150 cm. The presented TAC provides a compact implementation solution to realize high-density TOF sensors.

Abdominal Stiffness Evaluation in Massage for Constipation.

According to the experience of nurses and physiotherapists, the abdomen of constipated people becomes softer after abdominal massage. However, the relationship between the decrease in abdominal stiffness and the benefits of abdominal massage has not been examined quantitatively and is unclear. Furthermore, devices for measuring stiffness have been designed to measure relatively hard areas such as the shoulders and do not take into account the lateral outflow of the target tissue, which can be a problem when measuring the stiffness of soft areas such as the abdomen. To address these issues, this study presents a stiffness sensor suitable for measuring abdominal stiffness and investigates the relationship between the reduction in abdominal stiffness and the benefits of abdominal massage. The solution to prevent the lateral outflow of the target is the realization of a stopper, including a contact detection device, which enables a wide-area contact around the targeted area. The sensor consists of a stopper, probe, spring, and time-of-flight (ToF) sensors. The probe and spring provide appropriate pressure and deformation to the abdomen, whereas the stopper prevents the probe from being pushed into the abdomen more than necessary. The ToF sensor measures the deformation length when the deformation is stopped by the stopper. The abdominal stiffness can be derived from the deformation length. The investigation results indicate that the reduction in abdominal stiffness corresponds to the improvement of the stool condition or the maintenance of a healthy stool condition, whereas the maintenance of abdominal stiffness indicates the maintenance or deterioration of the stool condition.

Contactless Elevator Call Based on TOF Laser Ranging and Floating Image Technologies

This paper introduces a method of applying TOF (Time of flight) laser ranging (TOFlrt) and floating image technologies for contactless elevator call. The first is to project the image of the elevator button into the air. The second is to touch the floating image with your finger to activate the TOF sensor to send a control signal to the elevator. In order to verify the accuracy and sensing range of the TOF laser sensor, a measurement experiment is carried out. The measurement results show that when the floating image height is from 30mm to 50mm above the device surface, the sensor can detect the finger accurately with low latency. This system has been successfully implemented to elevators in a number of office buildings in Shanghai and all implemented systems are currently in smooth operation.

High-Linearity High-Resolution Time-of-Flight Linear-Array Digital Image Sensor Using Time-Domain Feedback.

This paper presents a high-linearity high-resolution time-of-flight (ToF) linear-array digital image sensor using a time-domain negative feedback technique. A coarse ToF measurement loop uses a 5-bit digital-to-time converter (DTC) and a delayed gating-pulse generator for time-domain feedback to find the zero of the difference between ToF and the digital estimate of the gating-pulse delay while maintaining a constant operating point of the analog readout circuits. A fine ToF measurement uses a delta-sigma modulation (DSM) loop using the time-domain feedback with a bit-stream signal form. Because of the self-contained property of the DSM for low distortion and noise exploited by the oversampling signal processing, the proposed technique provides high-linearity and high-range resolution in the fine ToF measurement. A prototype ToF sensor of 16.8 × 16.8 μm2 two-tap pixels and fabricated in a 0.11 μm (1P4M) CMOS image sensors (CIS) process achieves +0.9%/−0.47% maximum nonlinearity error and a resolution of 0.24 mm (median) for the measurement range of 0–1.05 m. The ToF sensor produces an 11-bit fully digital output with a ToF measurement time of 22.4 ms.

Automated Detection of Real-World Falls: Modeled From People With Multiple Sclerosis.

Falls are a major health problem with one in three people over the age of 65 falling each year, oftentimes causing hip fractures, disability, reduced mobility, hospitalization and death. A major limitation in fall detection algorithm development is an absence of real-world falls data. Fall detection algorithms are typically trained on simulated fall data that contain a well-balanced number of examples of falls and activities of daily living. However, real-world falls occur infrequently, making them difficult to capture and causing severe data imbalance. People with multiple sclerosis (MS) fall frequently, and their risk of falling increases with disease progression. Because of their high fall incidence, people with MS provide an ideal model for studying falls. This paper describes the development of a context-aware fall detection system based on inertial sensors and time of flight sensors that is robust to imbalance, which is trained and evaluated on real-world falls in people with MS. The algorithm uses an auto-encoder that detects fall candidates using reconstruction error of accelerometer signals followed by a hyper-ensemble of balanced random forests trained using both acceleration and movement features. On a clinical dataset obtained from 25 people with MS monitored over eight weeks during free-living conditions, 54 falls were observed and our system achieved a sensitivity of 92.14%, and false-positive rate of 0.65 false alarms per day.

A Review of Single-Photon Avalanche Diode Time-of-Flight Imaging Sensor Arrays

Time-of-flight (ToF) sensors using light pulses or continuous waves allow accurate distance measurements. Three-dimensional imagers can be based on an array of timing or time-gated integration pixels. Single-photon avalanche diodes (SPADs) have been increasingly chosen as the pixel's photodetector device to develop fast, long-range ToF sensors. Solid-state ToF cameras are replacing other alternatives, showing attractive characteristics and bringing up new potential applications. This paper presents the technical evolution of SPAD ToF 3D imaging sensors, and provides insight into their development over the last few decades. Starting with the first SPAD imagers reported in the early 2000's, various direct and indirect arrays up to present day state-of-the-art prototypes are referenced. The existing methods, options and possible implementations are described, addressing their advantages and drawbacks, and showing the progress yet to be made. The performance of the different presented approaches are given and compared.

Vostok: 3D scanner simulation for industrial robot environments

Computer vision will drive the next wave of robot applications. Latest three-dimensional scanners provide increasingly realistic object reconstructions. We introduce an innovative simulator that allows interacting with those scanners within the operating environment, thus creating a powerful tool for developers, researchers and students. In particular, we present a novel approach for simulating structured-light and timeof-flight sensors. Qualitative results demonstrate the efficiency and reliability in industrial environments. By using the programmability of modern GPUs, it is now possible to make greater use of parallelized simulative approaches. Apart from the easy modification of sensor parameters, the main advantage in simulation is the opportunity of carrying out experiments under reproducible conditions, especially for dynamic scene setups. Moreover, thanks to a great computational power, it is possible to generate huge amounts of synthetic data which can be used as test datasets for training machine learning models.