Fringe Images Research Articles

Visual Three-Dimensional Reconstruction Based on Spatiotemporal Analysis Method

To accurately reconstruct the three-dimensional (3D) surface of dynamic objects, we proposed a wrapped phase extraction method for spatiotemporal analysis based on 3D wavelet transform (WT). Our proposed method uses a 2D spatial fringe image combined with the time dimension and forms a 3D image sequence. The encoded fringe image sequence’s wrapped phase information was extracted by 3D WT and complex Morlet wavelet, and we improved the wrapped phase extraction’s accuracy by using the characteristics of spatiotemporal analysis and a multi-scale analysis of 3D WT, then we reconstructed the measured object by wrapped phase unwrapping and phase height transformation. Our simulation experiment results show that our proposed method can further filter the noise in the time dimension, and its accuracy is better than that of the one- (1D) and two-dimensional (2D) WT wrapped phase extraction method and the 3D Fourier transform wrapped phase extraction method because the reconstructed spherical crown’s RMSE value does not exceed 0.25 and the PVE value is less than 0.95. Our results show that the proposed method can be applied to the dynamic 3D reconstruction of a real human thoracic and abdominal surface, which fluctuates slowly with respiration movement, further verifying its effectiveness.

High-Accuracy Three-Dimensional Deformation Measurement System Based on Fringe Projection and Speckle Correlation.

Fringe projection profilometry (FPP) and digital image correlation (DIC) are widely applied in three-dimensional (3D) measurements. The combination of DIC and FPP can effectively overcome their respective shortcomings. However, the speckle on the surface of an object seriously affects the quality and modulation of fringe images captured by cameras, which will lead to non-negligible errors in the measurement results. In this paper, we propose a fringe image extraction method based on deep learning technology, which transforms speckle-embedded fringe images into speckle-free fringe images. The principle of the proposed method, 3D coordinate calculation, and deformation measurements are introduced. Compared with the traditional 3D-DIC method, the experimental results show that this method is effective and precise.

Deep learning-enabled anti-ambient light approach for fringe projection profilometry.

Achieving high-quality surface profiles under strong ambient light is challenging in fringe projection profilometry (FPP) since ambient light inhibits functional illumination from exhibiting sinusoidal stripes with high quantization levels. Conventionally, large-step phase shifting approaches are presented to enhance the anti-interference capability of FPP, but the image acquisition process in these approaches is highly time-consuming. Inspired by the promising performance of deep learning in optical metrology, we propose a deep learning-enabled anti-ambient light (DLAL) approach that can help FPP extract phase distributions from a single fringe image exposed to unbalanced lighting. In this work, the interference imposed by ambient light on FPP is creatively modeled as ambient light-induced phase error (ALPE). Guided by the ALPE model, we generate the dataset by precisely adjusting the stripe contrast before performing active projection, overcoming the challenge of collecting a large sample of fringe images with various illumination conditions. Driven by the novel dataset, the generated deep learning model can effectively suppress outliers among surface profiles in the presence of strong ambient light, thereby implementing high-quality 3D surface imaging. Experimentally, we verify the effectiveness and adaptability of the proposed DLAL approach in both indoor and outdoor scenarios with strong irradiation.

High-Precision Wafer Bonding Alignment Mark Using Moiré Fringes and Digital Grating.

This paper investigates a moiré-based mark for high-precision wafer bonding alignment. During alignment, the mark is combined with digital grating, which has the benefits of high precision and small size. A digital grating is superimposed on the mark to generate moiré fringes. By performing a phase calculation on the moiré fringe images corresponding to the upper and lower wafers, the relative offset of the upper and lower wafers can be accurately calculated. These moiré fringes are exceptionally stable, thereby enhancing the alignment stability. In this study, through practical experiments, we tested the rationality and practicability of the mark.

Modeling and correction of fringe patterns in Doppler asymmetric spatial heterodyne interferometry.

Doppler asymmetric spatial heterodyne (DASH) interferometry is a novel concept for observing atmospheric winds. This paper discusses a numerical model for the simulation of fringe patterns and a methodology to correct fringe images for extracting Doppler information from ground-based DASH measurements. Based on the propagation of optical waves, the fringe pattern was modeled considering different angular deviations and optical aberrations. A dislocation between two gratings can introduce an additional spatial modulation associated with the diffraction order, which was seen in laboratory measurements. A phase correction is proposed to remove phase differences between different row interferograms, which is the premise for calculating the average interferogram to improve the signal-to-noise ratio. Laboratory tests, simulation results, and Doppler velocity measurements indicate that a matrix determined in the laboratory can be applied to correct interferograms obtained from ground-based DASH measurements.

3D reconstruction method based on the optimal projection intensity of a polarization system.

The measurement of shiny objects is a major challenge for structured light 3D measurement. Polarization systems can be used to measure shiny objects. However, the polarizer may attenuate the light intensity of the captured image, reducing the SNR of the image, and the measurement accuracy will be affected. To improve the measurement accuracy and efficiency, this paper proposes a 3D measurement method based on polarization optimal projection intensity. By establishing the camera response function under the polarization system, the desired optimal projection intensity can be directly estimated and used to compensate for the image light intensity attenuated by the additional polarizer. This method does not require rotating polarizers and multiple exposure times. In addition, the optimal fringe images are obtained by applying image fusion algorithms. The experimental results show that the method improves pixels in the image that are too dark, while significantly reducing the saturated pixels on the shiny surface. This method can effectively improve the image contrast, get better fringe images, extract more effective information, and reconstruct a more complete 3D point cloud.

Non-Invasive Tear Break-Up Detection with the Kowa DR-1α and Its Relationship to Dry Eye Clinical Severity.

The purpose of this study is to compare visual versus software detection of non-invasive tear break-up with the KOWA DR-1α tear interferometer and investigate the relationship between non-invasive tear break-up time (NIBUT) and dry eye clinical severity. Tear interferometry with the KOWA DR-1α, together with a standardized comprehensive ocular surface/tear evaluation, was performed on 348 consecutive eyes. Investigator visually detected or software detected non-invasive tear break-up and NIBUT were measured and compared on a subset of these examinations. The relationship between software-detected NIBUT and categorical dry eye severity based on irritation symptoms and corneal and conjunctival dye staining scores was determined. The sensitivity of visual (frame-by-frame) or software detected non-invasive tear break-up in eyes with tear instability (FBUT < 10) was similar (range 63−69%). NIBUT, measured visually or by software, had a correlation coefficient of 0.87. NIBUT was significantly lower in severity levels 2 and 3 compared to levels 0 + 1, and level 3 was significantly lower than level 2. In conclusion, there is a good correlation between investigator visually detected and software-detected tear break-up and tear break-up time in the KOWA DR-1α interferometric fringe images. Software-detected NIBUT is a clinically relevant measure of dry eye clinical severity.

Error compensation for phase retrieval in deflectometry based on deep learning

Fringe patterns are widely applied in optical metrology, and phase retrieval is an essential process for decoding surface information. In the field of phase measuring deflectometry (PMD), phase errors in the phase retrieval process have more significant effects for PMD is a slope-based technique and is more sensitive to low-frequency errors. The main factors affecting the quality of the captured fringe images include the gamma effect of the liquid crystal display screen, the random noise from the charge-coupled device camera, and the random noise amplified by the defocused fringe patterns. Conventional methods compensated the phase errors of these factors separately with different methods, which are inefficient in handling the errors from coupling factors effectively. In this paper, we propose a deep neural network to compensate for the phase errors resulting from the combination of the factors. Experimental results demonstrate that the proposed network can significantly suppress the errors in phase retrieval with non-ideal fringe images. The phase errors can be reduced in both simulated and authentic data for deflectometry, which verifies the robustness and effectiveness of the proposed method.

A Self-Adaptive Multiple Exposure Image Fusion Method for Highly Reflective Surface Measurements

Fringe projection profilometry (FPP) has been extensively applied in various fields for its superior fast speed, high accuracy and high data density. However, measuring objects with highly reflective surfaces or high dynamic range surfaces remains challenging when using FPP. A number of multiple exposure image fusion methods have been proposed and successfully improved measurement performance for these kinds of objects. Normally, these methods have a relatively fixed sequence of exposure settings determined by practical experiences or trial and error experiments, which may decrease the efficiency of the entire measurement process and may have less robustness with regard to various environmental lighting conditions and object reflective properties. In this paper, a novel self-adaptive multiple exposure image fusion method is proposed with two areas of improvement relating to adaptively optimizing the initial exposure and the exposure sequence. First, by introducing the theory of information entropy, combined with an analysis of the characterization of fringe image entropy, an adaptive initial exposure searching method is proposed. Then, an exposure sequence generation method based on dichotomy is further described. On the basis of these two improvements, a novel self-adaptive multiple exposure image fusion method for FPP as well as its detailed procedures are provided. Experimental results validate the performance of the proposed self-adaptivity multiple exposure image fusion method via the measurement of objects with differences in surface reflectivity under different ambient lighting conditions.

Adaptive Phase Error Suppression Concerning 3D surface Deformation Measurement on Color Digital Fringe Projection Profilometry

3D surface measurement based on phase-shifting profilometry (PSP) has been actively developed in recent years. Three color channels of RGB that are modulated to generate a one-shot PSP method is a concept of color digital fringe pattern profilometry (CDFPP). The CDFPP is a promising technique for the 3D imaging profile of dynamic surface deformation if several phase errors in the one-shot PSP method can be suppressed. This work proposes a processing scheme for phase error suppression schemes (PESS) based on retrieving the modulated sinusoidal fringe and color fringe normalization in PSP using RGB color channel. The processing of PESS consists of tunable bandpass filtering (BPF) followed by fringe normalization. The initial BPF function is defined based on a smoothing spline data set of frequency and power spectrum from the baseline color fringe image. The predefine BPF function could be tunable during the imaging process by considering each frame's condition and RGB channel spectrum mapping. The corrected fringe images are then normalized from the color imbalance, and the phase shift is calculated using the conventional three-step PSP. For evaluation, PESS is performed to reconstruct simulator membrane deformation from four different static profiles and tested to observe the 3D surface of continuous membrane deformation. The PESS could suppress the phase errors of less than 30% less absolute errors than the conventional method and successfully reconstruct the 3D surface for low-frequency continuous membrane deformation with minimizing phase errors.

Active projection nonlinear γ correction method for fringe projection profilometry.

In fringe projection profilometry (FPP), the luminance nonlinearity generated by the superimposed γ effect of the projector and camera can lead to distortion of the intensity of the sinusoidal phase-shift fringe, resulting in a reduction of measurement precision and resolution. Traditional phase error compensation and γ-correction methods need to focus on the projector's optimal performance. However, commercial projectors often have huge apertures and are, therefore, unable to project a perfectly focused sinusoidal fringe image. This paper proposes an easy-to-implement active projection error correction method with high precision that is insensitive to projector defocus. After calibrating the projector to establish the nonlinear γ-response model of the optical measurement system, inverse γ compensation is performed. By generating and projecting a set of precorrected sinusoidal fringes, the camera can capture the high-quality sinusoidal fringe image and decrease the phase measurement error caused by the nonlinear γ effect of the FPP system. Computer simulations and experiments verify the effectiveness and feasibility of the proposed method for estimating and correcting the nonlinear γ distortion of the FPP system. The experimental results show that using the proposed active projection method to compensate for the error of the three-step phase-shift algorithm can achieve a high-precision measurement, and the influence of the system's nonlinear γ effect on the measurement accuracy is significantly suppressed.

Image signal denoising method of grating linear displacement sensor based on NLM

To improve the measurement resolution and accuracy of the grating displacement sensor, this paper proposes an image signal denoising method of grating linear displacement sensors based on NLM. Firstly, the optical system is designed to load the displacement information into the fringe image based on the grating pitch; Then, the principle of NLM image denoising is studied, and the inverse ratio kernel NLM denoising algorithm is designed; Thirdly, the effect of denoising algorithm is compared with the typical kernel function; Finally, the optimized image is used for displacement demodulation. The experimental results show that the SNR of the image is 30.1370db after denoising by this algorithm, and the high-precision displacement stage is set at 20 nm, 40 nm, and, 60 nm. Using the method in this paper, the positioning can be calculated to 22.7 nm, 38.2 nm, and 63.3 nm, and the sub-pixel positioning of the image and the nanopositioning measurement of the displacement are completed.

Three-dimensional measurement for specular reflection surface based on deep learning and phase measuring profilometry

ObjectivePhase measuring profilometry (PMP) is a non-contact 3D measurement method. However，it is still a very challenging task when it is applied to an object with a specular reflection surface. In this paper, a 3D measurement method for specular reflection surface based on a deep learning framework is proposed to reduce the measurement error caused by highlights. MethodsFirstly, a dual-channel U-Net network (DC-UNet) is used to identify the highlight areas in the fringe image. Secondly, the dilated residual networks (DRN) combined with the features of the neighborhood of the highlight pixels are used to correct the highlight pixels, and the predicted fringe image without highlight can be obtained. Thirdly, the predicted fringe image is blended with the original image, regarded as the final result of the highlight removal. Fourthly, phase unwrapping is implemented using the fringe image of highlight removal, and the 3D coordinates values can be obtained by combining the calibrated parameters with the phase values unwrapped. ResultsSimulated results show that the absolute maximum deviation, absolute mean error and standard deviation of reconstructed 3D height coordinates are reduced by 98.27 %, 68.70 %, and 91.11 %, respectively. The real experiment results show that the maximum error of the absolute phase is reduced by more than 87.00 %, and the mean error of the absolute phase is reduced by more than 92.72 %. Finally, more than 99.14 % of the pixels with intensity saturated caused by highlights can be corrected by the proposed method.

Wavefront-coded phase measuring deflectometry for the all-focused measurement.

Phase measuring deflectometry is a powerful measuring method for complex optical surfaces, which captures the reflected fringe images encoded on the screen under the premise of focusing the measured specular surface. Due to the limited depth of field of the camera, the captured images and the measured surface cannot be focused at the same time. To solve the position-angle uncertainty issue, in this Letter, the wavefront coding technology is used to modulate the imaging wavefront of the deflectometry, thereby making the measuring system insensitive to the defocus and other low-order aberration including astigmatism, field curvature, and so on. To obtain the accurate phase, the captured fringe images are deconvoluted using the modulated point spread function to reduce the phase error. Demonstrated with a highly curved spherical surface, the measurement accuracy can be improved by four times. Experiments demonstrate that the proposed method can successfully reconstruct the complex surfaces defocusing the captured images, which can greatly release the focusing requirement and improve measurement accuracy.

Phase Target-Based Calibration of Projector Radial Chromatic Aberration for Color Fringe 3D Measurement Systems.

The camera and projector are indispensable hardware parts of a color fringe projection 3D measurement system. Chromatic aberration between different color channels of the projector and camera has an impact on the measurement accuracy of the color fringe projection 3D profile measurement. There are many studies on camera calibration, but the chromatic aberration of the projector remains a question deserving of further investigation. In view of the complex system architecture and theoretical derivation of the traditional projector radial chromatic aberration method, a phase target based on projector radial chromatic aberration measurement and the correction method are proposed in this paper. This method uses a liquid crystal display with a holographic projection film as the phase target. The liquid crystal display sequentially displays red, green, and blue horizontal and vertical sinusoidal fringe images. The projector projects red, green, and blue horizontal and vertical sinusoidal fringe images to the phase target in turn, and calculates the absolute phases of the display fringes and reflection fringes, respectively. Taking the green channel as the reference channel, a phase coordinate system is established based on the phases of the vertical and horizontal directions displayed on the display screen, using the phase of the reflection fringes on the display screen as the ideal phase value of the phase point. Then, the phase coordinate system of the red and blue channels is transferred to the green phase coordinate system to calculate the chromatic aberration of the red-green channels and the blue-green channels, and pre-compensation is conducted. Experimental results prove that this method can measure and calibrate the radial chromatic aberration of the projector without being affected by the image quality of the camera. The correction effect of this method is that the maximum chromatic aberration of the red-green channel decreases from 1.9591/pixel to 0.5759/pixel, and the average chromatic aberration decreases from 0.2555/pixel to 0.1865/pixel. In addition, blue-green channel maximum chromatic aberration decreased from 1.8906/pixel to 0.5938/pixel, and the average chromatic aberration decreased from 0.2347/pixel to 0.1907/pixel. This method can improve the projection quality for fringe projection 3D profile measurement technology.

Fast and accurate 3D face reconstruction based on facial geometry constraints and fringe projection without phase unwrapping

Phase unwrapping is a critical process and a challenging step in fringe projection profilometry. Methods employing additional auxiliary patterns or embedding signals in the projected fringe pattern can successfully address this issue, but most of them suffer from increased number of structured patterns or depressed signal-to-noise ratio (SNR) of fringe images, which are not expected in real-time high-accuracy application such as 3D face reconstruction. In this paper, a 3D face reconstruction method that completely sidesteps the phase unwrapping process is proposed. Phase order lines (POLs) are first extracted from the left and right wrapped phase map of the stereo vision system respectively, and then the left and right POLs are matched by automatically drawing polylines on the face with facial geometry constraints as prior knowledge. Finally, based on the matched POL pairs, fine matching is performed directly employing the wrapped phase to generate a dense disparity map. Compared with the conventional, the processing time of the proposed method is further improved without sacrificing accuracy. Experiments demonstrate the effectiveness and robustness of the method.

The elimination of errors caused by shadow in fringe projection profilometry based on deep learning

The fringe projection profilometry (FPP) has been regarded as a classical and mature technique for 3D shape measurement. However, in practical applications, shadow is un-avoided in the imaging process and causes errors in many FPP systems. In this paper, the errors in FPP systems caused by shadow are first analyzed. Then, a direction-aware spatial context module based network is proposed for detecting the shadow regions of a fringe image. Further, a repairment method is developed based on a generative adversarial network combining some simple processes. The training datasets are rendered by a graphic software to easy the training of the networks. The proposed method can repair the shadow regions successfully with only one fringe image and so it can be applied in varieties of FPP systems. The feasibility and the accuracy of improved by the proposed method have been illustrated by abundant experiments.

Intrinsic stress determination based on the combination of photoelasticity and digital gradient sensing

A compact stress measurement system for transparent objects was proposed based on the combination of photoelascity and DGS methods. The interferometry fringe and speckle image of the deformed specimen were captured from perpendicular and slant directions, respectively. The fundamentals of the stress calculation and imaging model of oblique camera viewing were analyzed mathematically. After phase-shifting computation, image correlation, and geometric transformation, full-field distributions of the major and minor intrinsic principal stresses were obtained simultaneously. A validation experiment was conducted on a circular disk under diametral compression by the proposed method. Comparison between the experimental resulsts and theoretical data calculated according to the theory of elasticity verified the feasibility and accuracy. Moreover, the combined method was also successfully applied for the intrinsic principal stresses determination on transparent wrinkling thin film with defects subjected to shearing forces.

Wind imaging using simultaneous fringe sampling with field-widened Michelson interferometers.

The first, to our knowledge, successful laboratory implementation of an approach to image winds using simultaneous (as opposed to sequential) fringe imaging of suitable isolated spectral emission lines is described. Achieving this in practice has been a long-standing goal for wind imaging using airglow. It avoids the aliasing effects of source irradiance variations that are possible with sequential fringe sampling techniques. Simultaneous fringe imaging is accomplished using a field-widened Michelson interferometer by depositing phase steps on four quadrants of one of the mirrors and designing an optical system so that four images of the scene of interest, each at a different phase, are simultaneously produced. In this paper, the instrument characteristics, its characterization, and the analysis algorithms necessary for use of the technique for this type of interferometer are described for the first time, to the best of our knowledge. The large throughput associated with field-widened Michelson interferometers is sufficient for the spatial resolutions and temporal cadences necessary for ground based imaging of gravity waves in wind and irradiance to be achieved. The practical demonstration of this technique also validates its use for proposed monolithic satellite instruments for wind measurements using airglow on the Earth and Mars.

Single-exposure height-recovery structured illumination microscopy based on deep learning.

Modulation-based structured illumination microscopy (SIM) is performed to reconstruct three-dimensional (3D) surface topography. Generally speaking, modulation decoding algorithms mainly include a phase-shift (PS) method and frequency analysis technique. The PS method requires at least three images with fixed PSs, which leads to low efficiency. Frequency methods could decode modulation from a single image, but the loss of high-frequency information is inevitable. In addition, these methods all need to calculate the mapping relationship between modulation and height to recover the 3D shape. In this paper, we propose a deep learning enabled single-exposure surface measurement method. With only one fringe image, this method can directly restore the height information of the object. Processes such as denoising, modulation calculation, and height mapping are all included in the neural network. Compared with traditional Fourier methods, our method has higher accuracy and efficiency. Experimental results demonstrate that the proposed method can provide accurate and fast surface measurement for different structures.