Accurate 3D Shape Measurement Research Articles

Calibration-free structured-light-based 3D scanning system in laparoscope for robotic surgery.

Accurate 3D shape measurement is crucial for surgical support and alignment in robotic surgery systems. Stereo cameras in laparoscopes offer a potential solution; however, their accuracy in stereo image matching diminishes when the target image has few textures. Although stereo matching with deep learning has gained significant attention, supervised learning requires a large dataset of images with depth annotations, which are scarce for laparoscopes. Thus, there is a strong demand to explore alternative methods for depth reconstruction or annotation for laparoscopes. Active stereo techniques are a promising approach for achieving 3D reconstruction without textures. In this study, a 3D shape reconstruction method is proposed using an ultra-small patterned projector attached to a laparoscopic arm to address these issues. The pattern projector emits a structured light with a grid-like pattern that features node-wise modulation for positional encoding. To scan the target object, multiple images are taken while the projector is in motion, and the relative poses of the projector and a camera are auto-calibrated using a differential rendering technique. In the experiment, the proposed method is evaluated by performing 3D reconstruction using images obtained from a surgical robot and comparing the results with a ground-truth shape obtained from X-rayCT.

Measuring 3D shape and deformation in the presence of extremely strong ambient light and thermal radiation with a single time-gated camera.

Monochromatic light-illuminated active-imaging stereo-digital image correlation (stereo-DIC) has been extensively used for measuring the surface deformation of materials and structures at elevated temperatures. Despite the improvements in the image acquisition techniques or devices, it is still challenging to measure the 3D deformation of materials and structures in the presence of strong, time-varying ambient light and thermal radiation. In this study, we present what we believe to be a novel dual-filtering single-camera stereo-DIC technique for full-field 3D high-temperature deformation measurement, even in the case of extremely intense ambient light and thermal radiation. In contrast to conventional active-imaging stereo-DIC that only suppresses the thermal radiations in the spectral domain, the proposed technique utilized a dual-filtering strategy (i.e., narrow bandpass optical filtering and ultrashort exposing) to suppress the strong ambient light and thermal radiation in both time and spectral domains. Besides, a four-mirror adapter is adopted to realize 3D shape and deformation measurement using a compact single time-gated camera. Experimental verifications, including assessments with laboratory experiments and validations on real thermal deformation tests under transient aerodynamic heating and direct ohmic heating, convincingly demonstrated that the proposed single-camera dual-filtering stereo-DIC method can achieve accurate 3D shape, motion and deformation measurement, even with strong light and thermal radiation from the quartz lamps and the heated sample.

Single-Shot Three-Dimensional Measurement by Fringe Analysis Network

Fringe projection profilometry (FPP) has been broadly applied in three-dimensional (3D) measurements, but the existing multi-shot methods, which mostly utilize phase-shifting techniques, are heavily affected by the disturbance of vibration and cannot be used in dynamic scenes. In this work, a single-shot 3D measurement method using a deep neural network named the Fringe Analysis Network (FrANet) is proposed. The FrANet is composed of a phase retrieval subnetwork, phase unwrapping subnetwork, and refinement subnetwork. The combination of multiple subnetworks can help to recover long-range information that is missing for a single U-Net. A two-stage training strategy in which the FrANet network is pre-trained using fringe pattern reprojection and fine-tuned using ground truth phase maps is designed. Such a training strategy lowers the number of ground truth phase maps in the data set, saves time during data collection, and maintains the accuracy of supervised methods in real-world setups. Experimental studies were carried out on a setup FPP system. In the test set, the mean absolute error (MAE) of the refined absolute phase maps was 0.0114 rad, and the root mean square error (RMSE) of the 3D reconstruction results was 0.67 mm. The accuracy of the proposed method in dynamic scenes was evaluated by measuring moving standard spheres. The measurement of the sphere diameter maintained a high accuracy of 84 μm at a speed of 0.759 m/s. Two-stage training only requires 8800 fringe images in data acquisition, while supervised methods require 96,000 fringe images for the same number of iterations. Ablation studies verified the effectiveness of two training stages and three subnetworks. The proposed method achieved accurate single-shot 3D measurements comparable to those obtained using supervised methods and has a high data efficiency. This enables the accurate 3D shape measurement of moving or vibrating objects in industrial manufacturing and allows for further exploration of network architecture and training strategy with few training samples for single-shot 3D measurement.

Specular Surface Shape Measurement with Orthogonal Dual-Frequency Fourier Transform Deflectometry

Three-dimensional (3D) shape measurement for specular surfaces is becoming increasingly important in various applications. A novel orthogonal dual-frequency fringe is proposed in the specular surface shape measurement to overcome the phase jumping and discontinuities in spatial phase unwrapping. The fringe recalibrated high-accuracy phase information from its high-frequency fringe component with low-ambiguity phase information from its low-frequency fringe component. An improved Fourier transform deflectometry method based on the orthogonal dual-frequency fringe is proposed to measure 3D specular surface shapes. Simulation results showed that the orthogonal dual-frequency Fourier transform deflectometry (ODD) method could precisely reconstruct flat surfaces with an error of 2.16 nm rms, and concave surfaces with an error of 1.86 μm rms. Experimental results showed that the reconstructed shapes of both the flat mirror and the concave mirror measured by the ODD measurement system were highly comparable to those obtained by the phase-measuring deflectometry (PMD) method. This new fringe provides a distinctive approach to structured pattern construction and reduces the phase unwrapping ambiguities in specular surface shape measurement. The ODD method can achieve accurate 3D shape measurement for specular surfaces by sampling only one fringe, providing a possible basis for future real-time measurement of specular surfaces.

Self-Unwrapping Phase-Shifting for Fast and Accurate 3-D Shape Measurement

Phase unwrapping is one of the most important procedures in three-dimensional phase measurement profilometry and has attracted great research interest in the past few decades. However, existing phase unwrapping methods require additional patterns or pre-assumptions to determine the fringe orders for further absolute phase retrieval, which dramatically affect the measurement efficiency. To address this problem, we propose a generic self-unwrapping phase-shifting algorithm that retrieves the absolute phase without external information or priors. To this end, we firstly embed a novel space-varying phase shift that uniquely determines the fringe order information into sinusoidal patterns, and then extract it to retrieve the absolute phase by pixel-wise calculation. Our method achieves higher efficiency over previous methods while preserving the measurement precision, and the minimum number of patterns required is only four. All theoretical, simulation and extensive experimental results demonstrate its superiority in fast and accurate 3D shape measurement.

A combined calibration method of a mobile robotic measurement system for large-sized components

Product quality control of large-sized components with complex supports requires automatic and accurate 3D shape measurement. For accurate and efficient measurements of key local features (KLFs) on the mounting surface of multiple supports, we propose a flexible measurement method combining a laser tracker, a 3D scanner, and a mobile robot system. As for improving the overall measurement accuracy, we also introduce a calibration method based on a multidimensional combined collaboration target (MCCT) to establish correlation constraints between global and local calibration. Specifically, a global calibration method based on multidimensional geometric constraints provides a global measurement field. Additionally, a local calibration method derived from combination optimization model achieves accurate calibration of extrinsic parameters. We constructed the corresponding mobile robotic measurement system (MRMS) and conducted experiments. The experimental results demonstrate that the maximum and mean calibration errors are reduced from 0.045/0.0261 mm to 0.0231/0.0122 mm within 7 m, which features high accuracy.

High-speed and accurate 3D shape measurement using DIC-assisted phase matching and triple-scanning

Structured-light (SL) three-dimensional (3D) shape measurements technique has been extensively studied and applied in various fields, but the real-time capability of SL system is limited by the number of projected SL patterns. This paper presents a digital image correlation (DIC) assisted phase matching and triple-scanning method for high-speed and accurate 3D shape measurements based on the combined projection of three-step phase-shifting patterns and one speckle pattern. Benefiting from the triple-view geometric constraints and DIC technique, stereo correspondence and absolute fringe order of each valid pixel in the left wrapped phase map can be explored without requiring spatial phase unwrapping process or any priori information, the initial disparity map and absolute phase map can be generated. Subsequently, an efficient disparity refinement scheme based on connectivity area segmentation and neighbor disparity check is applied to the initial disparity map and absolute phase map to compensate the pixels with wrong disparities. Several efficient strategies are developed in the pipeline of data processing to speed up the algorithm. To improve the measurement accuracy, stereo SL model is proposed to make adequate use of triple-view information to calculate 3D coordinates using the disparity map and absolute phase map, and the measurement error has been reduced by 33% compared with the conventional methods. The experimental results demonstrate that the proposed method can perform high-speed and accurate 3D shape measurements for arbitrary surfaces and dynamic scenes.

Accurate and dynamic 3D shape measurement with digital image correlation-assisted phase shifting

Phase-shifting profilometry (PSP) has been widely used in structured-light (SL) systems for three-dimensional (3D) shape measurements, but the speed of the PSP technique is limited by increased phase-shifting patterns. This paper proposes an accurate and dynamic 3D shape measurement method by projecting only four patterns, including three-step phase-shifting patterns and one speckle pattern. Three-step phase-shifting images are used to obtain the initial unwrapped phase map with phase ambiguity. Based on the principle of digital image correlation and multi-view geometry, the absolute phase can be recovered reliably without requiring any embedded features or pre-defined information of the object. To improve the measurement accuracy, the projector coordinate is used as the measuring coordinate to establish a novel stereo SL system model. By solving a least square solution using the triple-view information, accurate 3D surface data can be reconstructed. The experimental results indicate that the proposed method can perform high-speed and accurate 3D shape measurements with an accuracy of 10.64 μm, which is superior to conventional methods and has certain instructive significance for 3D profilometry and measurement engineering.

High-resolution sequential thermal fringe projection technique for fast and accurate 3D shape measurement of transparent objects.

Three-dimensional (3D) shape measurement systems based on diffuse reflection of projected structured light do not deliver reliable data when measuring glossy, transparent, absorbent, or translucent objects. In recent years, we have developed a method based on stereo recording with infrared cameras and projection of areal aperiodic sinusoidal thermal patterns to detect such objects. However, the measurements took longer than 10 s, up to minutes; moreover, the measurement accuracy was improvable. Now, we have succeeded in both drastically reducing measurement time and significantly increasing measurement quality. This finally provides a technique for reliably measuring transparent objects, e.g., in series production. We demonstrate measurement examples achieved within 1 s and with 3D standard deviations less than 10 µm.

Hybrid calibration procedure for fringe projection profilometry based on stereo vision and polynomial fitting.

The key to accurate 3D shape measurement in fringe projection profilometry (FPP) is the proper calibration of the measurement system. Current calibration techniques rely on phase-coordinate mapping (PCM) or back-projection stereo vision (SV) methods. PCM methods are cumbersome to implement as they require precise positioning of the calibration target relative to the FPP system, but they produce highly accurate measurements within the calibration volume. SV methods generally do not achieve the same accuracy level. However, the calibration is more flexible in that the calibration target can be arbitrarily positioned. In this work, we propose a hybrid calibration method that leverages the SV calibration approach using a PCM method to achieve higher accuracy. The method has the flexibility of SV methods, is robust to lens distortions, and has a simple relation between the recovered phase and the metric coordinates. Experimental results show that the proposed hybrid method outperforms the SV method in terms of accuracy and reconstruction time due to its low computational complexity.

Development of Quick Three-Dimensional Shape Measurement Projection Mapping System Using a Whole-Space Tabulation Method

A grating projection method is often used as a highly accurate 3D shape measurement method. A real-time 3D shape measurement system can also be applied to measure a wide and smooth curved surface, such as in sheet metal processing. In this case, operators take much effort to recognize the positions of some problem areas on an object from a measured result displayed on a monitor. This study develops a projection mapping system projecting an evaluation image, such as height, displacement, gradient, curvature factor, and area of defect, onto an object. These evaluation results are obtained from the measured 3D shape. The evaluation image should be deformed according to the 3D shape of the object because the camera and projector positions are different. Therefore, this study proposes a method to quickly produce a deformed evaluation image using a whole-space tabulation method. A coordinate transform table allowed the conversion of a camera pixel coordinate into a projector pixel coordinate by using reference planes to apply deformation to the evaluation image according to the measured 3D shape. The quick coordinate transformation from a camera pixel coordinate into a projector pixel coordinate was realized using the coordinate transform table. This is a key idea of this study. It was confirmed that the coordinate transformation from the camera pixel coordinate to the projector coordinate could be performed in 4.5 ms using the coordinate transform table. In addition, 3D shape measurement projection mapping was applied to a curved sheet metal with small deformation, and the deformation part was clearly shown by projecting the height distribution. The architecture and the experimental results are shown herein.

3D measurement method based on S-shaped segmental phase encoding

Phase-based encoding has good immunity and robustness due to its insensitivity to ambient light, camera noise, and surface reflectance of measured objects. However, the number of code words is limited using traditional phase encoding method. In order to solve this problem, this paper proposes a new approach named S-shaped segmental phase encoding for 3D measurement. Specifically, by using a segmentation mechanism, the proposed method can, firstly, expand the range of code words whose number is limited using traditional phase encoding method. Secondly, the measurement accuracy can be improved. In this paper, Gray code is used to encode the phase encoding segment number, such that the fringe order judgment accuracy can be guaranteed. The experimental results demonstrated more accurate and efficient 3D shape measurement than the traditional methods, especially regarding the measurement of complex and isolated objects.

High-accuracy 3D shape measurement of translucent objects by fringe projection profilometry.

Acquiring complete and accurate 3D shape measurement results of translucent objects by fringe projection profilometry (FPP) is difficult because of the subsurface scattering effect. The phase offset introduces geometric errors, and the degraded image contrast leads to incomplete measurement results and random errors. In this research, a high-accuracy 3D shape measurement method for translucent objects based on phase-shifting FPP is proposed. The relationship between fringe period and phase error is investigated to determine the fringe periods. Random errors are suppressed by temporal noise reduction, and the robustness of multi-frequency heterodyne phase unwrapping is improved by increasing the interval of fringe periods along with temporal noise reduction. Geometric errors are compensated for by projecting multi-frequency fringe patterns to establish the relationship between fringe period and depth offset. Experimental results show that the proposed method can acquire complete measurement results and significantly reduce the overall error for measuring translucent objects.

Fast phase-measuring profilometry through composite color-coding method

Phase-measuring profilometry (PMP) has become one of the most effective means of three-dimensional (3D) shape measurement. Owing to the low robustness of the phase-unwrapping algorithm and the large number of projection patterns, the off-the-shelf PMP methods have a limited speed and accuracy for the measurement of discontinuous objects. To improve the measurement speed and accuracy, a composite color-coding method is presented in this paper. The main idea of the coding is embedding sinusoidal stripes, phase-coding stripes, and Gray code into the red, blue, and green channels of a color pattern, respectively. The Gray code is used for identifying the order of the sub-stair phase, which is quantized into m levels with a stair height of 2π/m. Each stair corresponds to the 2π phase-changed period of the sinusoidal fringes. The complexity of the phase unwrapping is low, and the proposed method needs fewer patterns for measurement than conventional methods. The experimental results indicate that the proposed method can achieve more accurate and efficient 3D shape measurement than the traditional phase-coding methods and is suitable for measuring discontinuous objects with complex shapes.

Accurate 3D Shape, Displacement and Deformation Measurement Using a Smartphone.

The stereo-digital image correlation technique using two synchronized industrial-grade cameras has been extensively used for full-field 3D shape, displacement and deformation measurements. However, its use in resource-limited institutions and field settings is inhibited by the need for relatively expensive, bulky and complicated experimental set-ups. To mitigate this problem, we established a cost-effective and ultra-portable smartphone-based stereo-digital image correlation system, which only uses a smartphone and an optical attachment. This optical attachment is composed of four planar mirrors and a 3D-printed mirror support, and can split the incoming scene into two sub-images, simulating a stereovision system using two virtual smartphones. Although such a mirror-based system has already been used for stereo-image correlation, this is the first time it has been combined with a commercial smartphone. This publication explores the potential and limitations of such a configuration. We first verified the effectiveness and accuracy of this system in 3D shape and displacement measurement through shape measurement and in-plane and out-of-plane translation tests. Severe thermal-induced virtual strains (up to 15,000 με) were found in the measured results due to the smartphone heating. The mechanism for the generation of the temperature-dependent errors in this system was clearly and reasonably explained. After a simple preheating process, the smartphone-based system was demonstrated to be accurate in measuring the strain on the surface of a loaded composite specimen, with comparable accuracy to a strain gauge. Measurements of 3D deformation are illustrated by tracking the deformation on the surface of a deflating ball. This cost-effective and ultra-portable smartphone-based system not only greatly decreases the hardware investment in the system construction, but also increases convenience and efficiency of 3D deformation measurements, thus demonstrating a large potential in resource-limited and field settings.

Vison-Based 3D Shape Measurement System for Transparent Microdefect Characterization

The main task of vision-based industrial defect inspection is to implement efficient non-contact visual quality control, i.e., to detect if there is a defect and to achieve an accurate 3D shape measurement of such a defect, and this kind of vision defect inspection system has been widely applied in various industrial application. However, it is still not the case in the inspection of transparent microdefect on the polarizer (which is the most important part of an LCD screen). Optical measurement devices (such as confocal microscopy) are often utilized to fulfil this task. To solve problems lied in the current confocal microscopy inspection system, such as expensive and non-real-time processing, this research aims to develop a novel vision-based 3D shape measurement system for polarizer transparent microdefect characterization. The innovation of this system, which has been verified by our optical model simulation, is that the 3D sizes of microdefect have a monotonically relation to the grayscale of the microdefect image. Hence, a microdefect imaging system, which could acquire defect image accurately, is first well designed and implemented. Then, a support vector regression (SVR) algorithm is derived by the trained data, i.e., 100 acquired defect images and its corresponding 3D shape value by confocal microscopy. Characterized 3D measurement of microdefect is thereby obtained by this SVR algorithm. 30 polarizer microdefect samples have been imaged and measured by our proposed system, and several important performance indicators, including processing speed, accuracy and system reproducibility, have been elaborately tested. The experimental results show that the proposed system could achieve a high-accuracy measurement but in a much faster and more efficient way than the confocal microscopy. Besides, this developed imaging system has been evaluated in real applications, and over 300 samples have been detected, which also validate the effectiveness of the proposed system.

A Robot-Driven 3D Shape Measurement System for Automatic Quality Inspection of Thermal Objects on a Forging Production Line

The three-dimensional (3D) geometric evaluation of large thermal forging parts online is critical to quality control and energy conservation. However, this online 3D measurement task is extremely challenging for commercially available 3D sensors because of the enormous amount of heat radiation and complexity of the online environment. To this end, an automatic and accurate 3D shape measurement system integrated with a fringe projection-based 3D scanner and an industrial robot is presented. To resist thermal radiation, a double filter set and an intelligent temperature control loop are employed in the system. In addition, a time-division-multiplexing trigger is implemented in the system to accelerate pattern projection and capture, and an improved multi-frequency phase-shifting method is proposed to reduce the number of patterns required for 3D reconstruction. Thus, the 3D measurement efficiency is drastically improved and the exposure to the thermal environment is reduced. To perform data alignment in a complex online environment, a view integration method is used in the system to align non-overlapping 3D data from different views based on the repeatability of the robot motion. Meanwhile, a robust 3D registration algorithm is used to align 3D data accurately in the presence of irrelevant background data. These components and algorithms were evaluated by experiments. The system was deployed in a forging factory on a production line and performed a stable online 3D quality inspection for thermal axles.

Comparative Analysis of Warp Function for Digital Image Correlation-Based Accurate Single-Shot 3D Shape Measurement.

Digital image correlation (DIC)-based stereo 3D shape measurement is a kind of single-shot method, which can achieve high precision and is robust to vibration as well as environment noise. The efficiency of DIC has been greatly improved with the proposal of inverse compositional Gauss-Newton (IC-GN) operators for both first-order and second-order warp functions. Without the algorithm itself, both the registration accuracy and efficiency of DIC-based stereo matching for shapes with different complexities are closely related to the selection of warp function, subset size, and convergence criteria. Understanding the similarity and difference of the impacts of prescribed subset size and convergence criteria on first-order and second-order warp functions, and how to choose a proper warp function and set optimal subset size as well as convergence criteria for different shapes are fundamental problems in realizing efficient and accurate 3D shape measurement. In this work, we present a comparative analysis of first-order and second-order warp functions for DIC-based 3D shape measurement using IC-GN algorithm. The effects of subset size and convergence criteria of first-order and second-order warp functions on the accuracy and efficiency of DIC are comparatively examined with both simulation tests and real experiments. Reference standards for the selection of warp function for different kinds of 3D shape measurement and the setting of proper convergence criteria are recommended. The effects of subset size on the measuring precision using different warp functions are also concluded.

Three-dimensional shape measurement technique for shiny surfaces by adaptive pixel-wise projection intensity adjustment

Conventional methods based on analyses of the absolute gray levels of pixels in fringe pattern images are affected by the problems of image saturation, interreflection, and high sensitivity to noise when obtaining three-dimensional (3D) shape measurements of shiny surfaces. This study presents a robust, adaptive, and fast 3D shape measurement technique, which adaptively adjusts the pixel-wise intensity of the projected patterns, thus it avoids image saturation and has a high signal to noise ratio (SNR) during 3D shape measurement for shiny surfaces. Compared with previous time-consuming methods using multiple exposures and the projection of fringe patterns with multiple intensities, where a large number of fringe pattern images need to be captured, the proposed technique needs to capture far fewer pattern images for measurement. In addition, it can greatly reduce the time costs to obtain the optimal projection intensities by the fusion of uniform gray level patterns and coordinates mapping. Our experimental results demonstrate that the proposed technique can achieve highly accurate and efficient 3D shape measurement for shiny surfaces.

Pre-calibration-free 3D shape measurement method based on fringe projection.

This paper presents a pre-calibration-free 3D shape measurement method based on fringe projection. Unlike ordinary methods, it performs calibration and 3D shape measurement concurrently. The captured phase-coded fringe images are utilized to obtain homogenous control points from two camera viewpoints, and the rough 3D structure of these points can be retrieved. Further, a constrained non-linear least-squares optimization model is established to determine the in situ geometry of the optical components, and then, the 3D scene is reconstructed. This method provides an accurate 3D shape measurement capability even during disturbance of the optical geometry. Moreover, not requiring a preliminary calibration process makes the system ultra-flexible. The performance of this method was verified by experiments.