High-speed 3D Shape Measurement Research Articles

High-speed 3D shape measurement using efficient moiré-assisted three-frequency heterodyne phase unwrapping algorithm

The conventional three-frequency heterodyne phase unwrapping (CTHPU) algorithm works very well in measuring isolated objects or discontinuous surfaces. However, it is unsuitable for high-speed applications due to many exposures to reconstruct one 3D profile. To this end, we propose a moiré-assisted three-frequency heterodyne phase unwrapping (MTHPU) algorithm, in which only four fringe patterns are needed to be projected to update one 3D result in dynamic scenes. In the proposed method, the wrapped phase distributions of the highest-frequency fringe patterns are calculated by three-step phase-shifting technique to ensure accuracy. And, the heterodyne phase maps, which are used to fulfill the absolute phase unwrapping, are extracted by a novel complex computational moiré algorithm from the other two fringe patterns and the highest-frequency wrapped phase. Moreover, the number of exposures can be further reduced to four to produce a 3D measurement by multiplexing the projecting sequence. Therefore, the proposed MTHPU can achieve high accuracy and high-efficiency measurement in dynamic scenes. Experiments demonstrate that our method can achieve a high 3D reconstruction frame rate of 2381 Hz.

High-Speed Vision and its Applications Toward High-Speed Intelligent Systems

Currently, high-speed vision based on parallel processing exists, and its various applications as high-speed intelligent systems have been proposed and implemented. The basic goal of high-speed vision is to realize vision capabilities and systems that operate at speeds necessary for intelligent systems, in which intelligence operating at the speed inherently required by the application system is achieved. This paper described the vision chip and parallel image processing architectures, presented outlines of system architectures, image-processing algorithms, and related peripheral technologies; described the concepts required to configure high-speed intelligent systems, such as hierarchical parallel distributed architecture, parallel decomposition, orthogonal decomposition, dynamics matching, latency minimization, high-speed 3D shape measurement, active vision, tracking vision, dynamic compensation, and dynamic projection mapping; and discussed a wide range of application systems in a systematic manner.

High-efficiency and robust binary fringe optimization for superfast 3D shape measurement.

By utilizing 1-bit binary fringe patterns instead of conventional 8-bit sinusoidal patterns, binary defocusing techniques have been successfully applied for high-speed 3D shape measurement. However, simultaneously achieving high accuracy and high speed remains challenging. To overcome this limitation, we propose a high-efficiency and robust binary fringe optimization method for superfast 3D shape measurement, which consists of 1D optimization and 2D modulation. Specifically, for 1D optimization, the three-level OPWM technique is introduced for high-order harmonics elimination, and an optimization framework is presented for generating the 'best' three-level OPWM pattern especially for large fringe periods. For 2D modulation, a single-pattern three-level OPWM strategy is proposed by utilizing all the dimensions for intensity modulation to decrease the required projection patterns. Thus, the proposed method essentially belongs to the 2D modulation technique, yet iterative optimization is carried out along one dimension, which drastically improves the computational efficiency while ensuring high accuracy. With only one set of optimized patterns, both simulations and experiments demonstrate that high-quality phase maps can be consistently generated for a wide range of fringe periods (e.g., from 18 to 1140 pixels) and different amounts of defocusing, and it can achieve superfast and high-accuracy 3D shape measurement.

Digital image correlation assisted absolute phase unwrapping.

This paper presents an absolute phase unwrapping method for high-speed three-dimensional (3D) shape measurement. This method uses three phase-shifted patterns and one binary random pattern on a single-camera, single-projector structured light system. We calculate the wrapped phase from phase-shifted images and determine the coarse correspondence through the digital image correlation (DIC) between the captured binary random pattern of the object and the pre-captured binary random pattern of a flat surface. We then developed a computational framework to determine fringe order number pixel by pixel using the coarse correspondence information. Since only one additional pattern is used, the proposed method can be used for high-speed 3D shape measurement. Experimental results successfully demonstrated that the proposed method can achieve high-speed and high-quality measurement of complex scenes.

High-speed three-dimensional shape measurement with inner shifting-phase fringe projection profilometry

Fringe projection profilometry (FPP) has been extensively studied in the field of three-dimensional (3D) measurement. Although FPP always uses high-frequency fringes to ensure high measurement accuracy, too many patterns are projected to unwrap the phase, which affects the speed of 3D reconstruction. We propose a high-speed 3D shape measurement method using only three high-frequency inner shifting-phase patterns (70 periods), which satisfies both high precision and high measuring speed requirements. Besides, our proposed method obtains the wrapped phase and the fringe order simultaneously without any other information and constraints. The proposed method has successfully reconstructed moving objects with high speed at the camera’s full frame rate (1700 frames per second).

High-efficiency dynamic three-dimensional shape measurement based on misaligned Gray-code light

The combination of Gray-code method with phase-shifting algorithm has been widely used in three-dimensional (3D) shape measurement. However, the non-uniform surface reflectance, the defocus of the projector and the object motion will cause order jump error at the boundary of adjacent Gray-code. This paper proposes a 3D shape measurement method based on misaligned Gray-code (MGC) light to avoid this kind of error. In this method, all the traditional Gray-code patterns are pre-moved by half fringe period before projection so that the edges of the decoding order and the wrapped phase are staggered exactly, and then each period of decoding order is divided into four sub-regions to construct correct phase order and unwrap the wrapped phase. In addition, the fringe period number can be further expanded by introducing the virtual plane to improve the measurement efficiency and accuracy simultaneously. Two sets of comparative experiments on static scenes confirm that our method can successfully avoid the order jump error without projecting additional Gray-code pattern. And the dynamic experimental results demonstrate that proposed method can especially realize high-efficiency and high-speed 3D shape measurement at a rate of 2381 fps by labeling 32-period fringe patterns with only 4 misaligned Gray-code patterns.

Improved Ternary Gray-Code Phase Unwrapping Algorithm for 3D Measurement Using a Binary Defocusing Technique

By using one-bit binary patterns instead of eight-bit sinusoidal ones, the binary defocusing techniques have been widely applied for high-speed 3D shape measurement. As projector defocusing is required, the phase unwrapping process of these techniques remains challenging. A recently proposed ternary Gray-code method can effectively increase the measurement speed by reducing the number of acquired coding patterns. However, it still has limitations, including the measuring range and a noise problem. To improve these, a new ternary Gray-code method is proposed by utilizing 2D modulation instead of the 1D modulation used in the conventional method. Simulations were conducted to compare these two methods, and the results verify that the proposed method reduces the variance in the introduced intermediate gray value by nearly 90%. Experiments were also carried out with the results verifying the superiority of the proposed method.

High-speed and accurate 3D shape measurement using DIC-assisted phase matching and triple-scanning

Structured-light (SL) three-dimensional (3D) shape measurements technique has been extensively studied and applied in various fields, but the real-time capability of SL system is limited by the number of projected SL patterns. This paper presents a digital image correlation (DIC) assisted phase matching and triple-scanning method for high-speed and accurate 3D shape measurements based on the combined projection of three-step phase-shifting patterns and one speckle pattern. Benefiting from the triple-view geometric constraints and DIC technique, stereo correspondence and absolute fringe order of each valid pixel in the left wrapped phase map can be explored without requiring spatial phase unwrapping process or any priori information, the initial disparity map and absolute phase map can be generated. Subsequently, an efficient disparity refinement scheme based on connectivity area segmentation and neighbor disparity check is applied to the initial disparity map and absolute phase map to compensate the pixels with wrong disparities. Several efficient strategies are developed in the pipeline of data processing to speed up the algorithm. To improve the measurement accuracy, stereo SL model is proposed to make adequate use of triple-view information to calculate 3D coordinates using the disparity map and absolute phase map, and the measurement error has been reduced by 33% compared with the conventional methods. The experimental results demonstrate that the proposed method can perform high-speed and accurate 3D shape measurements for arbitrary surfaces and dynamic scenes.

A DIC-assisted fringe projection profilometry for high-speed 3D shape, displacement and deformation measurement of textured surfaces

High-speed three-dimensional (3D) shape measurement techniques based on fringe projection profilometry (FPP) have undergone huge advances over the past two decades. However, accurate 3D displacement mapping and deformation analysis of dynamic scenes using FPP remains an unsolved problem. Because fringe patterns are projected rather than attached on the tested surfaces, the full-field point-to-point correspondence cannot be accurately established between any two 3D shape results. To deal with this challenge, a DIC-assisted FPP for high-speed 3D shape, displacement and deformation measurement of textured surfaces is proposed. Firstly, a high-speed 3D shape measurement system is adopted using our recently proposed robust and efficient Gray-coded coding strategy, which can accurately reconstruct full-field shape of discontinuous surfaces with rich texture information. Then, the modulation-based method is proposed to retrieve high-quality texture map from three phase-shifting fringe patterns, which can eliminate the adverse influence of the nonuniform and time-varying ambient light. By matching the retrieved surface texture images at difference states using DIC, accurate point tracking between the measured 3D shape data can be fulfilled, leading to precise 3D displacement and deformation measurement. Experiments have verified that the proposed method can achieve 3D shape, displacement and deformation measurement of dynamic scenes at a frame rate of 542 fps without any extra expense of hardware or calibration for the FPP system. The presented method is reliable and promising for further 3D displacement mapping and deformation analysis of dynamic scenes using FPP.

High-speed 3D shape measurement using a rotary mechanical projector.

In this paper, a fast rotary mechanical projector (RMP) is designed and manufactured for high-speed 3D shape measurement. Compared with the common high-speed projectors, RMP has a good performance in high-speed projection, which can obtain high quality projected fringes with shorter camera exposure time by using the error diffusion binary coding method and chrome plating technology. The magnitude, acceptability of systemic projection error is analyzed and quantified in detail. For the quantified error, the probability distribution function (PDF) algorithm is introduced to correct the error. Corrected projection error is reduced to more than one third of the original error. Subsequently, a monocular measurement system composed of the RMP and a single camera is constructed. The combination of the RMP device and PDF algorithm ensure the accuracy of a corresponding 3D shape measurement system. Experiments have demonstrated that the proposed solution has a good performance for the 3D measurement of high-speed scenes.

FPGA-assisted high-precision, high-speed 3D shape measurement

In order to achieve three dimensional (3D) shape measurement, improve measurement accuracy, a high-speed and high-precision FPGA architecture for phase measurement profilometry (PMP) algorithm is proposed. The system uses 12 sinusoidal fringes to achieve 3D shape measurement and corrects the inverse-phase of the projector to improve measurement accuracy. In addition, the coordinate rotation digital computer (CORDIC) algorithm is used to realize the arctangent function calculation, which not only improves the calculation speed, but also improves the phase calculation accuracy. The system utilizes the parallelism of FPGA and the superior performance of pipeline processing. It also redesigns the modules to greatly improve the speed of 3D reconstruction such as data buffer, phase wrapping, phase unwrapping, and point cloud. The results of experiments show that the method we proposed is faster than conventional methods.

Point-to-point compensation method for color coupling and imbalance in color-fringe pattern profilometry

Color-fringe pattern profilometry is widely developed for high-speed 3D shape measurement due to more information can be extracted through a single color pattern. However, previous researches show that the color coupling and imbalance of the system cause significant errors, which severely influence the accuracy of the color-fringe pattern profilometry. We proposed a point-to-point method to compensate the errors caused by color coupling and imbalance. Different from previous methods, the proposed method bases on the correspondence between camera pixels and projector pixels, which achieves point-to-point compensation of the full field. Experimental results verify that the proposed easy-to-operated method reduces the measurement errors significantly.

High-Speed Optical 3D Measurement Sensor for Industrial Application

3D optical sensors are becoming more and more popular in vision guidance of industrial robots and other industrial automation applications. Although research on high-speed 3D shape measurement (or imaging) has experienced tremendous growth over the past decades, simultaneously achieving high-speed and high-accuracy performance remains a major challenge in industrial practice. This paper presents a new FPGA architecture for the PMP algorithm and designs an high-speed embedded binocular structured light 3D measurement sensor. The sensor mainly contains a DLP projector, two cameras and a Xilinx Zynq-7000 UltraScale which make real-time computation possible. Experiments showed the time consuming of processing a set of 9 ×2 images with a resolution of 2048 ×1536 using the sensor proposed in this paper is 31.47 ms that has a better performance than 255ms on GPU. To the best of our knowledge, this is the first high-speed FPGA-based embedded binocular structured light 3D measurement sensor. This proposed 3D sensor can obtain ultra high quality 3D information in real time, it can be widely applied to robot random pin-picking and other industrial applications.

High-speed high dynamic range 3D shape measurement based on deep learning

For many three-dimensional (3D) measurement techniques based on fringe projection profilometry (FPP), measuring the objects with a large variation range of surface reflectivity is always a very tricky problem due to the limited dynamic range of camera. Many high dynamic range (HDR) 3D measurement methods are developed for static scenes, which are fragile for dynamic objects. In this paper, we address the problem of phase information loss in HDR scenes, in order to enable 3D reconstruction from saturated or dark images by deep learning. By using a specifically designed convolutional neural network (CNN), we can accurately extract phase information in both the low signal-to-noise ratio (SNR) and saturation situations after proper training. Experimental results demonstrate the success of our network in 3D reconstruction for both static and dynamic HDR objects. Our method can improve the dynamic range of three-step phase-shifting by a factor of 4.8 without any additional projected images or hardware adjustment during measurement. And the final 3D measurement speed of our method is about 13.89 Hz (off-line).

Multilevel symmetric pattern design and optimization for high-speed and high-accuracy 3D shape measurement

The binary defocusing technique has enabled speed breakthroughs for 3D shape measurement, yet simultaneously achieving high accuracy and high speed remains difficult. To overcome this limitation, we propose to utilize multilevel symmetric pattern for high-speed and high-accuracy 3D shape measurement. Compared to conventional binary patterns, multilevel pattern could bring more flexibility for eliminating undesired high-frequency harmonics, thus has the potential to greatly enhance the phase quality and measurement accuracy. In this paper, the symmetric pattern design principle and related optimization procedure are presented in order to find the “best” multilevel fringe patterns. Both simulation and experiments verify that with respect to conventional methods, the proposed method could consistently generate better fringe patterns for a wide range of fringe periods. Furthermore, we developed an absolute 3D shape measurement system with the speed of 667 Hz, verifying that the proposed method is applicable for high-speed, high-accuracy applications.

High Speed 3D Shape Measurement with Temporal Fourier Transform Profilometry

A novel high-speed 3D shape measurement technology called temporal Fourier transform profilometry (TFTP for short) is proposed by combining the merits of Fourier transform profilometry (FTP) and phase-measuring profilometry (PMP). Instead of using the digital light projector, a mechanical projector is employed to generate multi-period phase-shifting fringe patterns sequentially. During the reconstruction process, the phase value of each pixel is calculated independently along the temporal axis and no spectrum filtering operation is performed in a spatial domain. Therefore, high-frequency components containing the detailed information of the measured object effectively remain. The proposed method is suitable for measuring isolated dynamic objects. Only one frame of deformed fringe pattern is required to retrieve one 3D shape of the measured object, so it has the obvious advantage if measuring the dynamic scene at a high speed. A low-cost self-made mechanical projector with fast projection speed is developed to execute the principle-proof experiments, whose results demonstrate the feasibility of measuring isolated dynamic objects.

High-speed three-dimensional shape measurement based on shifting Gray-code light.

The measuring technique combining a phase-shifting algorithm and Gray-code light has been widely used in three-dimensional (3D) shape measurement for static scenes because of its high robustness and anti-noise ability. However, in the high-speed measurement, phase unwrapping errors occur easily on the boundaries of adjacent Gray-code words because of the defocus of the projector, the motion of the objects and the non-uniform reflectivity of the surface. To overcome this challenge, a high-speed 3D shape measurement method based on shifting Gray-code light has been proposed in this paper. Firstly, the average intensity of three captured phase-shifting fringe images are used as a pixel-wise threshold to binarize the Gray codes and to eliminate most phase unwrapping errors caused by the non-uniform reflectivity, ambient light variations, and the defocus of projector. Then, the shifting Gray-code (SGC) coding strategy is proposed to avoid the remaining errors of phase unwrapping on the edge of the code words. In this strategy, no additional patterns are projected, and two sets of decoding words with staggered boundaries are built in the temporal sequences for one wrapped phase. Finally, the simple, efficient, and robust phase unwrapping can be achieved in the high-speed dynamic measurement. This proposed method has been applied to reconstruct 3D shape of randomly collapsing objects in a large depth range, and the experimental results demonstrate that it can reliably obtain high-quality shape and texture information at 310 frames per second.

High-resolution few-pattern method for 3D optical measurement

Accurately and quickly obtaining the three-dimensional (3D) shape information of objects has become increasingly important in various scientific fields. However, simultaneously achieving the high-resolution and high-speed 3D shape measurement of unknown objects remains challenging in practice. In this Letter, we propose a novel variant shifting-phase method for 3D optical measurement. Based on a digital fringe projection system, the method performs a point-wise high-resolution measurement of objects by projecting only four intensity-coded patterns. We can retrieve the wrapped phases and their corresponding fringe orders simultaneously from these four patterns, and do not require any pre-acquired information of the object. The experimental results successfully demonstrate the effectiveness of the easy-to-operate method.

High-speed 3D shape measurement using the optimized composite fringe patterns and stereo-assisted structured light system.

In this paper, we propose a high-speed 3D shape measurement technique based on the optimized composite fringe patterns and stereo-assisted structured light system. Stereo phase unwrapping, as a new-fashioned method for absolute phase retrieval based on the multi-view geometric constraints, can eliminate the phase ambiguities and obtain a continuous phase map without projecting any additional patterns. However, in order to ensure the stability of phase unwrapping, the period of fringe is generally around 20, which limits the accuracy of 3D measurement. To solve this problem, we develop an optimized method for designing the composite pattern, in which the speckle pattern is embedded into the conventional 4-step phase-shifting fringe patterns without compromising the fringe modulation, and thus the phase measurement accuracy. We also present a simple and effective evaluation criterion for the correlation quality of the designed speckle pattern in order to improve the matching accuracy significantly. When the embedded speckle pattern is demodulated, the periodic ambiguities in the wrapped phase can be eliminated by combining the adaptive window image correlation with geometry constraint. Finally, some mismatched regions are further corrected based on the proposed regional diffusion compensation technique (RDC). These proposed techniques constitute a complete computational framework that allows to effectively recover an accurate, unambiguous, and distortion-free 3D point cloud with only 4 projected patterns. Experimental results verify that our method can achieve high-speed, high-accuracy, robust 3D shape measurement with dense (64-period) fringe patterns at 5000 frames per second.

High-speed three-dimensional shape measurement based on cyclic complementary Gray-code light.

The binary defocusing technique has been widely used in high-speed three-dimensional (3D) shape measurement because it breaks the bottlenecks in high-speed fringe projection and the projector's nonlinear response. However, it is challenging for this method to realize a two- or multi-frequency phase-shifting algorithm because it is difficult to simultaneously generate high-quality sinusoidal fringe patterns with different periods under the same defocusing degree. To bypass this challenge, we proposed a high-speed 3D shape measurement technique for dynamic scenes based on cyclic complementary Gray-code (CCGC) patterns. In this proposed method, the projected phase-shifting sinusoidal fringes kept one same frequency, which is beneficial to ensure the optimum defocusing degree for binary dithering technique. The wrapped phase can be calculated by phase-shifting algorithm and unwrapped with the aid of complementary Gray-code (CGC) patterns in a simple and robust way. Then, the cyclic coding strategy further extends the unambiguous phase measurement range and improves the measurement accuracy compared with the traditional Gray-coding strategy under the condition of the same number of projected patterns. High-quality 3D results of three complex dynamic scenes-including a cooling fan and a standard ceramic ball with a free-falling table tennis, collapsing building blocks, and impact of the Newton's cradle-were demonstrated at a frame rate of 357 fps. This verified the proposed method's feasibility and validity.