Photometric Stereo Method Research Articles

Adaptive Weighted Data Fusion for Line Structured Light and Photometric Stereo Measurement System.

Line structured light (LSL) measurement systems can obtain high accuracy profiles, but the overall clarity relies greatly on the sampling interval of the scanning process. Photometric stereo (PS), on the other hand, is sensitive to tiny features but has poor geometrical accuracy. Cooperative measurement with these two methods is an effective way to ensure precision and clarity results. In this paper, an LSL-PS cooperative measurement system is brought out. The calibration methods used in the LSL and PS measurement system are given. Then, a data fusion algorithm with adaptive weights is proposed, where an error function that contains the 3D point cloud matching error and normal vector error is established. The weights, which are based on the angles of adjacent normal vectors, are also added to the error function. Afterward, the fusion results can be obtained by solving linear equations. From the experimental results, it can be seen that the proposed method has the advantages of both the LSL and PS methods. The 3D reconstruction results have the merits of high accuracy and high clarity.

Deep Learning Methods for Calibrated Photometric Stereo and Beyond.

Photometric stereo recovers the surface normals of an object from multiple images with varying shading cues, i.e., modeling the relationship between surface orientation and intensity at each pixel. Photometric stereo prevails in superior per-pixel resolution and fine reconstruction details. However, it is a complicated problem because of the non-linear relationship caused by non-Lambertian surface reflectance. Recently, various deep learning methods have shown a powerful ability in the context of photometric stereo against non-Lambertian surfaces. This paper provides a comprehensive review of existing deep learning-based calibrated photometric stereo methods utilizing orthographic cameras and directional light sources. We first analyze these methods from different perspectives, including input processing, supervision, and network architecture. We summarize the performance of deep learning photometric stereo models on the most widely-used benchmark data set. This demonstrates the advanced performance of deep learning-based photometric stereo methods. Finally, we give suggestions and propose future research trends based on the limitations of existing models.

Photometric Stereo Method Used for Woven Fabric Density Measurement Based on 3D Surface Structure

The measurement of the density of woven fabrics based on the vision method has been widely developed. This study used a photometric stereo method to measure the warp and weft density of woven fabrics based on the 3D surface structure. Six 2D images of the fabric were recorded, each with a different lighting direction. The six images were then reconstructed using the unbiased photometric stereo algorithm to produce the three-dimensional surface structure. The reconstructed image was used to detect and correct the skew angle with the Hough transform. For each image, a depth profile was made toward the x-axis to get the weft curve and towards the y-axis to get the warped curve. The two depth curves were filtered using a locally weighted smoothing (LOESS) filter. This study successfully measured the density of woven fabric with an average error for warp and weft of 0.64% and 0.45%, respectively.

A sparse-point guided photometric stereo method for the metal complex surfaces measurement and high-fidelity geometry recovery

Accurate recovery of complex surfaces with dense points and normals often requires informative measurement data, posing a great challenge for measurement. In this paper, a contact sensor and a photometric stereo (PS) vision device are integrated together to recover the geometry with high-fidelity details. A two-branch deep PS neural network is developed and the Gaussian process regression (GPR) is used to reconstruct a global geometric shape from some sparse points captured by the contact probe. This prior geometry is sent to the proposed cross-convolution attention residual block (CCARB) based network to guide the PS for better normal recovery. The resulting normals are further fused with the sparse points by a normal integral model to obtain dense and high-accuracy points. Synthetic datasets are constructed to train the network. Experiments are conducted on both synthetic and real data, and show the superiority in the high-fidelity reconstruction for the measurement of complex surfaces.

Event fusion photometric stereo network

Photometric stereo methods typically rely on RGB cameras and are usually performed in a dark room to avoid ambient illumination. Ambient illumination poses a great challenge in photometric stereo due to the restricted dynamic range of the RGB cameras. To address this limitation, we present a novel method, namely Event Fusion Photometric Stereo Network (EFPS-Net), which estimates the surface normals of an object in an ambient light environment by utilizing a deep fusion of RGB and event cameras. The high dynamic range of event cameras provides a broader perspective of light representations that RGB cameras cannot provide. Specifically, we propose an event interpolation method to obtain ample light information, which enables precise estimation of the surface normals of an object. By using RGB-event fused observation maps, our EFPS-Net outperforms previous state-of-the-art methods that depend only on RGB frames, resulting in a 7.94% reduction in mean average error. In addition, we curate a novel photometric stereo dataset by capturing objects with RGB and event cameras under numerous ambient light environments.

A novel image registration-based dynamic photometric stereo method for online defect detection in aluminum alloy castings

The adoption of three-dimensional (3D) measurement technology for parts surface defect detection can improve inspection reliability. For online inspection purposes, 3D measurement technologies must possess the characteristics of high speed and high efficiency. The photometric stereo method is a potential 3D measurement method with high speed and low cost. However, the traditional photometric stereo method is unsuitable for dynamic scenes due to its initial design for static scenes. In this paper, we propose a novel dynamic photometric stereo method based on an image registration method. To achieve fast speed and high efficiency, we reduce the computational cost by automatically generating regions of interest (ROI). Additionally, we innovatively map the depth information (the surface normal vectors) to a mean curvature map of the surface and use it to detect defects, which combines the robustness of 3D methods and the fast speed of 2D methods. We designed experiments and the results showed that our method can detect defects on the surfaces of aluminum alloy castings accurately and robustly in an online manner. This paper also aims to reveal the importance of utilizing multidimensional information in high-speed online inspections.

RMAFF-PSN: A Residual Multi-Scale Attention Feature Fusion Photometric Stereo Network

Predicting accurate normal maps of objects from two-dimensional images in regions of complex structure and spatial material variations is challenging using photometric stereo methods due to the influence of surface reflection properties caused by variations in object geometry and surface materials. To address this issue, we propose a photometric stereo network called a RMAFF-PSN that uses residual multiscale attentional feature fusion to handle the “difficult” regions of the object. Unlike previous approaches that only use stacked convolutional layers to extract deep features from the input image, our method integrates feature information from different resolution stages and scales of the image. This approach preserves more physical information, such as texture and geometry of the object in complex regions, through shallow-deep stage feature extraction, double branching enhancement, and attention optimization. To test the network structure under real-world conditions, we propose a new real dataset called Simple PS data, which contains multiple objects with varying structures and materials. Experimental results on a publicly available benchmark dataset demonstrate that our method outperforms most existing calibrated photometric stereo methods for the same number of input images, especially in the case of highly non-convex object structures. Our method also obtains good results under sparse lighting conditions.

Light invariant photometric stereo.

An imaging setup that enables unsynchronized photometric stereo (PS) for Lambertian objects based on modulated light sources is presented. Knowing the specific frequency of the modulated light source allows to filter out any other light in the scene. This creates an image that depends only on the particular light source while ignoring the ambient light. Moreover, if the scene is illuminated by multiple modulated sources with different frequencies, repeating this process for every frequency will produce a sequence of images with the corresponding illumination. This sequence is then used as an input to the PS algorithm for 3D reconstruction. The proposed approach, named Light Invariant Photometric Stereo (LIPS), was verified on both synthetic and real-world data. LIPS eliminates the need for synchronization between the sources and the camera and significantly outperformed the classical PS method in an illuminated environment.

DPPS: A deep-learning based point-light photometric stereo method for 3D reconstruction of metallic surfaces

Three-dimensional (3D) measurement provides essential geometric information for quality control and process monitoring in many manufacturing applications. Photometric stereo is one of the potential solutions for in-process metrology and active geometry compensation, which takes multiple images of an object under different illuminations as inputs and recovers its surface normal map based on a reflectance model. Deep learning approaches have shown their potential in solving the highly nonlinear problem for photometric stereo, but the main challenge preventing their practical application in process metrology lies in the difficulties in the generation of a comprehensive dataset for training the deep learning model. This paper presents a new Deep-learning based Point-light Photometric Stereo method, DPPS, which utilizes a multi-channel deep convolutional neural network (CNN) to achieve end-to-end prediction for both the surface normal and height maps in a semi-calibrated fashion. The key contribution is a new dataset generation method combining both physics-based and data-driven approaches, which minimizes the training cost and enables DPPS to handle reflective metal surfaces with unknown surface roughness. Even trained only with fully synthetic and high-fidelity dataset, our DPPS surpasses the state-of-the-art with an accuracy better than 0.15 cm over a 10 cm × 10 cm area and its real-life experimental results are on par with commercial 3D scanners. The demonstrated results provide guidance on improving the generalizability and robustness of deep-learning based computer vision metrology with minimized training cost as well as show the potential for in-process 3D metrology in advanced manufacturing processes.

Photometric-Stereo-Based Defect Detection System for Metal Parts.

Automated inspection technology based on computer vision is now widely used in the manufacturing industry with high speed and accuracy. However, metal parts always appear in high gloss or shadow on the surface, resulting in the overexposure of the captured images. It is necessary to adjust the light direction and view to keep defects out of overexposure and shadow areas. However, it is too tedious to adjust the position of the light direction and view the variety of parts' geometries. To address this problem, we design a photometric-stereo-based defect detection system (PSBDDS), which combines the photometric stereo with defect detection to eliminate the interference of highlights and shadows. Based on the PSBDDS, we introduce a photometric-stereo-based defect detection framework, which takes images captured in multiple directional lights as input and obtains the normal map through the photometric stereo model. Then, the detection model uses the normal map as input to locate and classify defects. Existing learning-based photometric stereo methods and defect detection methods have achieved good performance in their respective fields. However, photometric stereo datasets and defect detection datasets are not sufficient for training and testing photometric-stereo-based defect detection methods, thus we create a photometric stereo defect detection (PSDD) dataset using our PSBDDS to eliminate gaps between learning-based photometric stereo and defect detection methods. Furthermore, experimental results prove the effectiveness of the proposed PSBBD and PSDD dataset.

Patch-Based Uncalibrated Photometric Stereo Under Natural Illumination.

This paper presents a photometric stereo method that works with unknown natural illumination without any calibration objects or initial guess of the target shape. To solve this challenging problem, we propose the use of an equivalent directional lighting model for small surface patches consisting of slowly varying normals, and solve each patch up to an arbitrary orthogonal ambiguity. We further build the patch connections by extracting consistent surface normal pairs via spatial overlaps among patches and intensity profiles. Guided by these connections, the local ambiguities are unified to a global orthogonal one through Markov Random Field optimization and rotation averaging. After applying the integrability constraint, our solution contains only a binary ambiguity, which could be easily removed. Experiments using both synthetic and real-world datasets show our method provides even comparable results to calibrated methods.

Minifying photometric stereo via knowledge distillation-based feature translation.

Photometric stereo (PS) estimates the surface normals of an object by utilizing multiple images captured under different light conditions. To obtain accurate surface normals, a large number of input images is often required. Therefore, a huge effort is needed to capture images and calibrate light directions together with a heavy computational cost. Therefore, in this paper, we propose a robust photometric stereo method even when the number of input images is very small. To this end, we design a feature translation module (FTM) that enriches features having scarce information. In particular, we insert FTMs between the layers of the baseline backbone PS network. Then, activations of each FTM are supervised by distillation loss. For computing distillation loss, we utilize a teacher PS network trained by taking lots of images as inputs. As a result, our PS network requires very few input images but produces a similar quality of output surface normals with the teacher PS network. The proposed method is applicable to both calibrated and uncalibrated PS. We show the effectiveness of the proposed method not only when the number of input images is small but also in various input conditions.

ReDDLE-Net: Reflectance Decomposition for Directional Light Estimation

The surfaces of real objects can visually appear to be glossy, matte, or anywhere in between, but essentially, they display varying degrees of diffuse and specular reflectance. Diffuse and specular reflectance provides different clues for light estimation. However, few methods simultaneously consider the contributions of diffuse and specular reflectance for light estimation. To this end, we propose ReDDLE-Net, which performs Reflectance Decomposition for Directional Light Estimation. The primary idea is to take advantage of diffuse and specular clues and adaptively balance the contributions of estimated diffuse and specular components for light estimation. Our method achieves a superior performance advantage over state-of-the-art directional light estimation methods on the DiLiGenT benchmark. Meanwhile, the proposed ReDDLE-Net can be combined with existing calibrated photometric stereo methods to handle uncalibrated photometric stereo tasks and achieve state-of-the-art performance.

A generic topography reconstruction method based on multi‐detector backscattered electron images

AbstractSurface topographies can be reconstructed from backscattered electron (BSE) images captured from different detector orientations. This article presents a very general approach to this problem, in the spirit of photometric stereo methods, allowing for arbitrary BSE detector number (at least 3) and shapes. The general idea is to both determine the (non‐linear) model parameters and compute the surface topography so that the modelled images match at best the acquired ones. Three samples are used for validation of the measured topography with respect to atomic force microscopy (AFM) measurements. Root mean square (RMS) errors in the range of 10–35 nm, or 1–1.5% of total sampleheight, are obtained.

Portrait imaging relighting system based on a simplified photometric stereo method.

This study proposes a portrait image relighting system based on a simplified photometric stereo method. The system, comprising a controllable digital single lens reflex camera and five polarized flashlights, can obtain a color shade-less image and synthesize a normal map from shaded images. When calibrating the photometric stereo, the normal map is taken as a linear combination of shaded images and clamped with respect to specific normal directions on a white-coated sphere. The relit images were generated through inverse rendering in a predefined virtual environment. To evaluate personal preference, 24 adult subjects were recruited to conduct subjective assessments comparing the deep portrait relighting method results. From experiments regarding different scenarios, we concluded that the proposed system based on a simplified photometric stereo performs acceptably for relighting portrait images.

Example-Based Multispectral Photometric Stereo for Multi-Colored Surfaces.

A photometric stereo needs three images taken under three different light directions lit one by one, while a color photometric stereo needs only one image taken under three different lights lit at the same time with different light directions and different colors. As a result, a color photometric stereo can obtain the surface normal of a dynamically moving object from a single image. However, the conventional color photometric stereo cannot estimate a multicolored object due to the colored illumination. This paper uses an example-based photometric stereo to solve the problem of the color photometric stereo. The example-based photometric stereo searches the surface normal from the database of the images of known shapes. Color photometric stereos suffer from mathematical difficulty, and they add many assumptions and constraints; however, the example-based photometric stereo is free from such mathematical problems. The process of our method is pixelwise; thus, the estimated surface normal is not oversmoothed, unlike existing methods that use smoothness constraints. To demonstrate the effectiveness of this study, a measurement device that can realize the multispectral photometric stereo method with sixteen colors is employed instead of the classic color photometric stereo method with three colors.

Measurement of three-dimensional track profiles on CR-39s based on the photometric stereo method.

To obtain more information about incident particles, a new method for measuring three-dimensional track profiles formed on CR-39s based on the photometric stereo method was developed. A new optical microscope system with 16 lasers and a complementary metal-oxide-semiconductor camera was built to automatically capture the reflecting track images illuminated by the laser beams from different angles, and the track profiles were three-dimensionally reconstructed using a self-developed software. To verify the reconstruction results of the track profiles, both the openings and depth were measured with an atomic force microscope. The results showed that the relative deviations between the two methods of the openings were about 5.5% and the deviations of the depth were about 8.0%. At present, the reconstruction speed of a three-dimensional track profile is a factor of 400 greater than that of the atomic force microscope. The new method shows great potential for rapid reconstruction of numerous track morphologies. It is expected to be helpful for further studies on the energy and angle discrimination of incident particles in the field of nuclear measurements.

A Novel Learning Based Non-Lambertian Photometric Stereo Method for Pixel-Level Normal Reconstruction of Polished Surfaces

High-quality reconstruction of polished surfaces is a promising yet challenging task in the industrial field. Due to its extreme reflective properties, state-of-the-art methods have not achieved a satisfying trade-off between retaining texture and removing the effects of specular outliers. In this paper, we propose a learning based pixel-level photometric stereo method to estimate the surface normal. A feature fusion convolutional neural network is used to extract the features from the normal map solved by the least square method and from the original images respectively, and combine them to regress the normal map. The proposed network outperforms the state-of-the-art methods on the DiLiGenT benchmark dataset. Meanwhile, we use the polished rail welding surface to verify the generalization of our method. To fit the complex geometry of the rails, we design a flexible photometric stereo information collection hardware with multi-angle lights and multi-view cameras, which can collect the light and shade information of the rail surface for photometric stereo. The experimental results indicate that the proposed method is able to reconstruct the normal of the polished surface at the pixel level with abundant texture information.

DRM-Based Colour Photometric Stereo Using Diffuse-Specular Separation for Non-Lambertian Surfaces.

This paper presents a photometric stereo (PS) method based on the dichromatic reflectance model (DRM) using colour images. The proposed method estimates surface orientations for surfaces with non-Lambertian reflectance using diffuse-specular separation and contains two steps. The first step, referred to as diffuse-specular separation, initialises surface orientations in a specular invariant colour subspace and further separates the diffuse and specular components in the RGB space. In the second step, the surface orientations are refined by first initialising specular parameters via solving a log-linear regression problem owing to the separation and then fitting the DRM using Levenburg-Marquardt algorithm. Since reliable information from diffuse reflection free from specularities is adopted in the initialisations, the proposed method is robust and feasible with less observations. At pixels where dense non-Lambertian reflectances appear, signals from specularities are exploited to refine the surface orientations and the additionally acquired specular parameters are potentially valuable for more applications, such as digital relighting. The effectiveness of the newly proposed surface normal refinement step was evaluated and the accuracy in estimating surface orientations was enhanced around on average by including this step. The proposed method was also proven effective in an experiment using synthetic input images comprised of twenty-four different reflectances of dielectric materals. A comparison with nine other PS methods on five representative datasets further prove the validity of the proposed method.

Discrete Search Photometric Stereo for Fast and Accurate Shape Estimation.

We consider the problem of estimating surface normals of a scene with spatially varying, general bidirectional reflectance distribution functions (BRDFs) observed by a static camera under varying distant illuminations. Unlike previous approaches that rely on continuous optimization of surface normals, we cast the problem as a discrete search problem over a set of finely discretized surface normals. In this setting, we show that the expensive processes can be precomputed in a scene-independent manner, resulting in accelerated inference. We discuss two variants of our discrete search photometric stereo (DSPS), one working with continuous linear combinations of BRDF bases and the other working with discrete BRDFs sampled from a BRDF space. Experiments show that DSPS has comparable accuracy to state-of-the-art exemplar-based photometric stereo methods while achieving 10-100x acceleration.