Real-time 3D Shape Measurement Research Articles

High-Speed Phase Structured Light Integrated Architecture on FPGA

Phase-shifting fringe projection profilometry (FPP) in three-dimension (3D) structured light measurement is widely applied in high-accuracy 3D shape measurement scenarios. But motion objects or other dynamic scenarios make the traditional phase-shifting FPP hard to deal with due to its sizable CPU-based execution time. Therefore, an embedded high-speed and high-accuracy phase-shifting FPP system is designed and implemented on FPGAs. The system implements a three-frequency four-step phase-shifting hardware algorithm with RTL-level caching, multi-frame image ping-pong buffering, and data handling, etc. The system exhibits fully-parallel and pipeline processing while behaving with lower resource and power consumption. Measurement experiments indicate that the proposed method can reach high-accuracy real-time 3D shape measurement with a 3D decoding speed of 40.15 fps when the clock frequency is 200MHz, and the image size is 2432×2048.

Real-time 3D shape measurement of dynamic scenes using fringe projection profilometry: lightweight NAS-optimized dual frequency deep learning approach.

Achieving real-time and high-accuracy 3D reconstruction of dynamic scenes is a fundamental challenge in many fields, including online monitoring, augmented reality, and so on. On one hand, traditional methods, such as Fourier transform profilometry (FTP) and phase-shifting profilometry (PSP), are struggling to balance measuring efficiency and accuracy. On the other hand, deep learning-based approaches, which offer the potential for improved accuracy, are hindered by large parameter amounts and complex structures less amenable to real-time requirements. To solve this problem, we proposed a network architecture search (NAS)-based method for real-time processing and 3D measurement of dynamic scenes with rate equivalent to single-shot. A NAS-optimized lightweight neural network was designed for efficient phase demodulation, while an improved dual-frequency strategy was employed coordinately for flexible absolute phase unwrapping. The experiment results demonstrate that our method can effectively perform 3D reconstruction with a reconstruction speed of 58fps, and realize high-accuracy measurement of dynamic scenes based on deep learning for what we believe to be the first time with the average RMS error of about 0.08 mm.

High-performance phase measuring profilometry architecture based on Zynq SoC.

This paper presents a novel high-performance heterogeneous computation architecture, to the best of our knowledge, for stereo structure light using the phase measuring profilometry (PMP) algorithm based on a Zynq UltraScale+ system on chip (SoC). The proposed architecture aims to achieve real-time and high-accuracy 3D shape measurement. The experiment results indicate that the calculation time of a standard four-step PMP algorithm with a resolution of 1280×1024 is 14.11ms. It is nearly 51 times faster than the well-optimized software implementation running on a Raspberry Pi and nearly three times faster than a high-end PC, with 15 times less power consumption. Consequently, the proposed architecture is deemed suitable for real-time 3D measurements in embedded applications.

A novel phase unwrapping method for binocular structured light 3D reconstruction based on deep learning

Fringe projection has been prevalently adopted in the field of three-dimensional(3D) reconstruction. However, it is still a challenge to achieve high-precision and efficient measurement. To achieve real-time and efficient 3D shape measurement of the objects, we must extract the 3D coordinate data of the object from as few fringe patterns as possible. In the paper, we develop a new technique combining composite fringe coding and stair phase coding to solve the high-quality absolute phase and realize binocular 3D morphology reconstruction based on deep learning. This method only needs to project a composite fringe coding pattern to acquire the wrapped phase and a stair phase coding pattern to obtain the fringe order. Then we can recover the unwrapped phase maps of the captured fringe patterns. According to the calibration parameters of the binocular camera, the absolute phase is stereo matched to get the disparity map, and then converted into 3D spatial coordinates of the object. The influence of defocus and noise on the deviation between the wrapped phase and fringe order is solved by staggered phase unwrapping method. We demonstrated the validity and robustness of the novel method by experiments, which provides a promising research direction for 3D shape measurement based on binocular structured light.

Single-shot 3D profilometry using a color imaging polarimeter

We demonstrate a single-shot uniaxial three-dimensional (3D) profilometry system that illuminates a sample with a color structured polarization pattern with an axial chromatic aberration. We have previously shown that real-time 3D shape measurement can be performed without the occlusion problem using a polarization camera and a monochromatic polarized pattern. Here, we show that it is possible to double the depth range of the system, with no tradeoff in depth resolution, measurement speed, or accuracy, using color pattern projection and an RGB polarization camera. We present the measurement principles of the system and show the results of a real-time 3D profilometry experiment for a sample with a deep hole.

Real-time 3D shape measurement with dual-frequency composite grating and motion-induced error reduction

Phase-shifting profilometry has been increasingly sought and applied in dynamic three-dimensional (3D) shape measurement. However, the object motion will lead to extra phase shift error and thus measurement error. In this paper, a real-time 3D shape measurement method based on dual-frequency composite phase-shifting grating and motion-induced error reduction is proposed for a complex scene containing dynamic and static objects. The proposed method detects the motion region of a complex scene through the phase relations of the dual-frequency composite grating and reduces the motion-induced error with the combination of the phase calculated by a phase-shifting algorithm and the phase extracted by Fourier fringe analysis. It can correctly reconstruct the 3D shape of a complex dynamic scene and ensure high measurement accuracy of its static object as well. With the aid of the phase-shifting image ordering approach, the dynamic 3D shape of complex scenes can be reconstructed and the motion-induced error can also be suppressed in real time. Experimental results well proved that the proposed method is effective and practical.

A real-time 3D shape measurement with color texture using a monochromatic camera

High-speed real-time three-dimensional (3D) shape measurement with color texture has a large potential in various fields. The majority of the existing methods retrieve the 3D shape with color texture by employing an RGB camera or additional infrared camera; however, the RGB camera is more expensive and has lower sensitivity to light, and employing additional infrared camera requires complex alignment. Given this, a novel method is proposed to simultaneously retrieve the 3D shape and color texture. Specifically, compared with the existing methods, the proposed method only uses one monochromatic camera; as a result, the cameras alignment is not required, simplifying the 3D measurement system and reducing its cost. Three fringe patterns with different frequencies and three corresponding uniform gray-level patterns are set to different colors of red, green, and blue. Then Fourier transform profilometry (FTP) is employed to retrieve the 3D shape from the captured six images. While three uniform gray-level patterns can be used to eliminate the effects of the zero frequency term, they can be also used to generate the color texture of objects meanwhile. The proposed method only requires the capture of six monochromes, and the experimental results demonstrate the success of the proposed method.

Development of Quick Three-Dimensional Shape Measurement Projection Mapping System Using a Whole-Space Tabulation Method

A grating projection method is often used as a highly accurate 3D shape measurement method. A real-time 3D shape measurement system can also be applied to measure a wide and smooth curved surface, such as in sheet metal processing. In this case, operators take much effort to recognize the positions of some problem areas on an object from a measured result displayed on a monitor. This study develops a projection mapping system projecting an evaluation image, such as height, displacement, gradient, curvature factor, and area of defect, onto an object. These evaluation results are obtained from the measured 3D shape. The evaluation image should be deformed according to the 3D shape of the object because the camera and projector positions are different. Therefore, this study proposes a method to quickly produce a deformed evaluation image using a whole-space tabulation method. A coordinate transform table allowed the conversion of a camera pixel coordinate into a projector pixel coordinate by using reference planes to apply deformation to the evaluation image according to the measured 3D shape. The quick coordinate transformation from a camera pixel coordinate into a projector pixel coordinate was realized using the coordinate transform table. This is a key idea of this study. It was confirmed that the coordinate transformation from the camera pixel coordinate to the projector coordinate could be performed in 4.5 ms using the coordinate transform table. In addition, 3D shape measurement projection mapping was applied to a curved sheet metal with small deformation, and the deformation part was clearly shown by projecting the height distribution. The architecture and the experimental results are shown herein.

Large depth-of-field 3D shape measurement using an electrically tunable lens

The state-of-the-art 3D shape measurement system has rather shallow working volume due to the limited depth-of-field (DOF) of conventional lens. In this paper, we propose to use the electrically tunable lens to substantially enlarge the DOF. Specifically, we capture always in-focus phase-shifted fringe patterns by precisely synchronizing the tunable lens attached to the camera with the image acquisition and the pattern projection; we develop a phase unwrapping framework that fully utilizes the geometric constraint from the camera focal length setting; and we pre-calibrate the system under different focal distance to reconstruct 3D shape from unwrapped phase map. To validate the proposed idea, we developed a prototype system that can perform high-quality measurement for the depth range of approximately 1,000 mm (400 mm - 1400 mm) with the measurement error of 0.05%. Furthermore, we demonstrated that such a technique can be used for real-time 3D shape measurement by experimentally measuring moving objects.

Real-time 3D shape measurement using 3LCD projection and deep machine learning.

For 3D imaging and shape measurement, simultaneously achieving real-time and high-accuracy performance remains a challenging task in practice. In this paper, a fringe-projection-based 3D imaging and shape measurement technique using a three-chip liquid-crystal-display (3LCD) projector and a deep machine learning scheme is presented. By encoding three phase-shifted fringe patterns into the red, green, and blue (RGB) channels of a color image and controlling the 3LCD projector to project the RGB channels individually, the technique can synchronize the projector and the camera to capture the required fringe images at a fast speed. In the meantime, the 3D imaging and shape measurement accuracy is dramatically improved by introducing a novel phase determination approach built on a fully connected deep neural network (DNN) learning model. The proposed system allows performing 3D imaging and shape measurement of multiple complex objects at a real-time speed of 25.6fps with relative accuracy of 0.012%. Experiments have shown great promise for advancing scientific and engineering applications.

Three-dimensional shape measurement based on binary fringe conventional projection

A three-dimensional (3D) shape measurement method based on binary fringe conventional projection is proposed. In traditional 3D shape measurement methods based on binary fringe projection, the binary fringe is used as an approximate sinusoidal fringe by defocusing projection or even directly used as an approximate sinusoidal fringe by nonlinear error suppression feature of large-step phase measuring profilometry (PMP). In the proposed method (Prop), neither defocusing projection is needed nor large-step PMP must be used. When the binary fringe is conventionally projected on the measured object, just by filtering operation in spatial frequency domain from the captured deformed pattern, the nearly unbroken sinusoidal deformed pattern can be extracted efficiently. While N ( N≥3) frames binary fringes with 1/ N period misalignment between each adjacent fringe are conventionally projected onto the measured object sequentially, the corresponding nearly unbroken N-step sinusoidal deformed patterns extracted from the N frames captured deformed patterns can be used to reconstruct the 3D shape of the measured object with N-step PMP algorithm. The experimental results show the feasibility and validity of the Prop. It is more flexible for its optional fringe period and optional phase-shifting steps. It has potential application in the real-time 3D shape measurement.

Motion-artifact-free dynamic 3D shape measurement with hybrid Fourier-transform phase-shifting profilometry.

Fourier-transform profilometry (FTP) and phase-shifting profilometry (PSP) are two mainstream fringe projection techniques widely used for three-dimensional (3D) shape measurement. The former is well known for its single-shot nature and the latter for its higher measurement resolution and precision. However, when it comes to measuring the dynamic objects, neither approach is able to produce high-resolution, high-accuracy measurement results that are free from any depth ambiguities and motion-related artifacts. Furthermore, for scenes consisting of both static and dynamic objects, a trade-off between measurement precision and efficiency has to be made, suggesting that using a single approach can yield only suboptimal results. To this end, we propose a novel hybrid Fourier-transform phase-shifting profilometry method to integrate the advantages of both approaches. The motion vulnerability of multi-shot PSP can be overcome, or at least significantly alleviated, through the combination of single-shot FTP, while the high accuracy of PSP can also be preserved when the object is motionless. We design a phase-based, pixel-wise motion detection strategy that can accurately outline the moving object regions from their motionless counterparts. The final measurement result is obtained by fusing the determined regions where the PSP or FTP is applied correspondingly. To validate the proposed hybrid approach, we develop a real-time 3D shape measurement system for measuring multiple isolated moving objects. Experimental results demonstrate that our method achieves significantly higher precision and better robustness compared with conventional approaches where PSP or FTP is applied separately.

High-speed real-time 3D shape measurement based on adaptive depth constraint.

Stereo phase unwrapping (SPU) has been increasingly applied to high-speed real-time fringe projection profilometry (FPP) because it can retrieve the absolute phase or matching points in a stereo FPP system without projecting or acquiring additional fringe patterns. Based on a pre-defined measurement volume, artificial maximum/minimum phase maps can be created solely using geometric constraints of the FPP system, permitting phase unwrapping on a pixel-by-pixel basis. However, when high-frequency fringes are used, the phase ambiguities will increase which makes SPU unreliable. Several auxiliary techniques have been proposed to enhance the robustness of SPU, but their flexibility still needs to be improved. In this paper, we proposed an adaptive depth constraint (ADC) approach for high-speed real-time 3D shape measurement, where the measurement depth volume for geometric constraints is adaptively updated according to the current reconstructed geometry. By utilizing the spatio-temporal correlation of moving objects under measurement, a customized and tighter depth constraint can be defined, which helps enhance the robustness of SPU over a large measurement volume. Besides, two complementary techniques, including simplified left-right consistency check and feedback mechanism based on valid area, are introduced to further increase the robustness and flexibility of the ADC. Experimental results demonstrate the success of our proposed SPU approach in recovering absolute 3D geometries of both simple and complicated objects with only three phase-shifted fringe images.

High-dynamic-range 3D shape measurement utilizing the transitioning state of digital micromirror device

This paper introduces a novel method for 3D shape measurement of high-contrast surfaces in real-time by taking advantage of the transitioning state of digital micromirror device. We propose to project defocused 1-bit binary patterns for sinusoidal generation to bypass the rigid camera-projector synchronization requirement. Each pattern is capture twice in one projection cycle to obtain two fringe images with different brightness: the first fringe is captured when the camera is totally exposed within the projector’s bright period; the second fringe is captured when part of the projector’s dark period is within the camera exposure. The two fringes will be combined to alleviate the saturation problem. Experiments demonstrate the success of the proposed method by showing the real-time 3D shape measurement capability (image acquisition rate: 166 Hz; 3D frame rate: 14 Hz). Moreover, since 1-bit binary patterns are used, it is potentially applicable to high-speed measurements.

High-precision real-time 3D shape measurement based on a quad-camera system

Phase-shifting profilometry (PSP) based 3D shape measurement is well established in various applications due to its high accuracy, simple implementation, and robustness to environmental illumination and surface texture. In PSP, higher depth resolution generally requires higher fringe density of projected patterns which, in turn, lead to severe phase ambiguities that must be solved with additional information from phase coding and/or geometric constraints. However, in order to guarantee the reliability of phase unwrapping, available techniques are usually accompanied by increased number of patterns, reduced amplitude of fringe, and complicated post-processing algorithms. In this work, we demonstrate that by using a quad-camera multi-view fringe projection system and carefully arranging the relative spatial positions between the cameras and the projector, it becomes possible to completely eliminate the phase ambiguities in conventional three-step PSP patterns with high-fringe-density without projecting any additional patterns or embedding any auxiliary signals. Benefiting from the position-optimized quad-camera system, stereo phase unwrapping can be efficiently and reliably performed by flexible phase consistency checks. Besides, redundant information of multiple phase consistency checks is fully used through a weighted phase difference scheme to further enhance the reliability of phase unwrapping. This paper explains the 3D measurement principle and the basic design of quad-camera system, and finally demonstrates that in a large measurement volume of 200 mm × 200 mm × 400 mm, the resultant dynamic 3D sensing system can realize real-time 3D reconstruction at 60 frames per second with a depth precision of 50 μm.

Double-pattern triangular pulse width modulation technique for high-accuracy high-speed 3D shape measurement.

Using 1-bit binary patterns for three-dimensional (3D) shape measurement has been demonstrated as being advantageous over using 8-bit sinusoidal patterns in terms of achievable speeds. However, the phase quality generated by binary pattern(s) typically are not high if only a small number of phase-shifted patterns are used. This paper proposes a method to improve the phase quality by representing each pattern with the difference of two binary patterns: the first binary pattern is generated by triangular pulse width modulation (TPWM) technique, and the second being π shifted from the first pattern that is also generated by TPWM technique. The phase is retrieved by applying a three-step phase-shifting algorithm to the difference patterns. Through optimizing the modulation frequency of the triangular carrier signal, we demonstrate that a high-quality phase can be generated for a wide range of fringe periods (e.g., from 18 to 1140 pixels) with only six binary patterns. Since only 1-bit binary patterns are required for 3D shape measurement, this paper will present a real-time 3D shape measurement system that can achieve 30 Hz.

Real-time microscopic 3D shape measurement based on optimized pulse-width-modulation binary fringe projection

In recent years, tremendous progress has been made in 3D measurement techniques, contributing to the realization of faster and more accurate 3D measurement. As a representative of these techniques, fringe projection profilometry (FPP) has become a commonly used method for real-time 3D measurement, such as real-time quality control and online inspection. To date, most related research has been concerned with macroscopic 3D measurement, but microscopic 3D measurement, especially real-time microscopic 3D measurement, is rarely reported. However, microscopic 3D measurement plays an important role in 3D metrology and is indispensable in some applications in measuring micro scale objects like the accurate metrology of MEMS components of the final devices to ensure their proper performance. In this paper, we proposed a method which effectively combines optimized binary structured patterns with a number–theoretical phase unwrapping algorithm to realize real-time microscopic 3D measurement. A slight defocusing of our optimized binary patterns can considerably alleviate the measurement error based on four-step phase-shifting FPP, providing the binary patterns with a comparable performance to ideal sinusoidal patterns. The static measurement accuracy can reach 8 μm, and the experimental results of a vibrating earphone diaphragm reveal that our system can successfully realize real-time 3D measurement of 120 frames per second (FPS) with a measurement range of in lateral and 8 mm in depth.

High-precision real-time 3D shape measurement using a bi-frequency scheme and multi-view system.

High-speed and high-precision 3D shape measurement plays a central role in diverse applications such as automatic online inspection, robotics control, and human-computer interaction. Conventional multi-frame phase-shifting-based fringe projection profilometry techniques face inherent trade-offs between the speed and measurement precision, which are fundamentally limited by the fringe density and extra pattern projections used for de-ambiguity of fringe orders. Increasing the frequency of the projection fringes can obviously improve the measurement precision; however, it creates difficulties in the subsequent phase unwrapping. For this reason, to date, the frequency of the fringes in typical real-time 3D shape measurement techniques is generally less than 30 to guarantee a reasonable reliability of phase unwrapping. To overcome this limitation, a bi-frequency phase-shifting technique based on a multi-view fringe projection system is proposed, which significantly enhances the measurement precision without compromising the measurement speed. Based on the geometric constraints in a multi-view system, the unwrapped phase of the low-frequency (10-period) fringes can be obtained directly, which serves as a reference to unwrap the high-frequency phase map with a total number of periods of up to 160. Besides, the proposed scheme with 10-period and 160-period fringes is suitable for slightly defocusing projection, allowing a higher projection rate and measurement speed. Experiments on both static and dynamic scenes are performed, verifying that our method can achieve high-speed and high-precision 3D measurement at 300 frames per second with a precision of about 50μm.

High dynamic range real-time 3D shape measurement.

This paper proposes a method that can measure high-contrast surfaces in real-time without changing camera exposures. We propose to use 180-degree phase-shifted (or inverted) fringe patterns to complement regular fringe patterns. If not all of the regular patterns are saturated, inverted fringe patterns are used in lieu of original saturated patterns for phase retrieval, and if all of the regular fringe patterns are saturated, both the original and inverted fringe patterns are all used for phase computation to reduce phase error. Experimental results demonstrate that three-dimensional (3D) shape measurement can be achieved in real time by adopting the proposed high dynamic range method.

Optimized design of composite grating in real-time three-dimensional shape measurement

A new method for improving the measuring precision in real-time three-dimensional (3D) shape measurement by optimizing structural parameters of composite grating is proposed. By analyzing the formation mechanism of composite grating, in which three phase-shifting sinusoidal gratings with equal phase-shifts 2π/3 are modulated by three carrier gratings with different carrier frequencies along the orthogonal direction respectively, it is found that the measuring precision can be improved by optimizing the period of phase-shifting sinusoidal gratings inside the composite grating when the frequencies of carrier gratings are selected properly. Three typical surfaces (cone, peaks-function, saddle) are selected according to the complexity of the measured object to design optimized composite gratings. Experimental results show that the proposed method can improve the measuring precision in real-time 3D shape measurement.