Photometric Stereo Research Articles

Automatic Detection and Classification of Defective Areas on Metal Parts by Using Adaptive Fusion of Faster R-CNN and Shape From Shading

Computer vision and deep learning approaches have an important role in industrial inspection systems. Computer vision technology is essential for fast, defect-free control of products in the production line. The importance of the computer vision concept is recognized when the problems of the classical methods are taken into consideration. Metallic defect detection is a challenging problem as metal surfaces are easily affected by environmental factors such as lighting and light reflection. Since traditional detection algorithms are inefficient in complex problems, we propose a novel method to detect and classify metal surface defects, such as cracks, scratches, inclusion, etc. The type and location of defects were detected by the Faster Regional Convolutional Neural Network (Faster R-CNN), combined with the Shape From Shading (SFS) method, which can extract surface characteristics. The Northeastern University (NEU) surface defect database was used for defective samples. The proposed algorithm has also been tested on an unlabeled dataset (KolektorSDD2/KSDD2) to show labeling performance. The results on both labeled and unlabeled datasets have demonstrated state-of-the-art performance in automatic defect detection, classification, and labeling. The proposed method has satisfactory results for the detection of defects on the metal surface, and the mean average precision is 0.83. The average precision of crazing, pitted surface, patches, scratches, inclusion, and rolled-in scale are 0.96, 0.93, 0,89, 0.80, 0.72, and 0.71, respectively.

Self-Calibrating Sparse Far-Field Photometric Stereo With Collocated Light

Photometric stereo is one common surface normal estimation technique, which can be utilized for various industrial tasks such as 3-D surface detail reconstruction and defect inspection. To further satisfy the requirement of the industrial inspection, the uncalibrated photometric stereo is needed. This article proposes a collocated light method for uncalibrated far-field photometric stereo to achieve accurate surface normal estimation under a small number of lights. Collocated light is the light close to the camera, with the angle between the lighting direction of the collocated light and the viewing direction of the camera being less than 10°. The image under the collocated light could be utilized to extract features related to the shadings and the shadows that are the key to the light self-calibration. Thus, the collocated light is introduced to realize the light self-calibration, with the accuracy being less influenced by the number of lights. Then, the normal estimation network with collocated light is integrated to give precise surface normal estimations under a small number of lights. Both synthetic and real experiments validate that the proposed method can achieve the state-of-the-art performance.

Accurate Analysis of the Spatial Pattern of Reflected Light and Surface Orientations Based on Color Illumination

3D Recovery approaches require a variety of clues to obtain shape information. The shape from shading (SFS) method uses shading information in images to estimate depth maps. Although shading contains detailed information, it causes some well-known ambiguities such as convex-concave ambiguity. In this study, a system installation, using red, green, and blue illumination, and an algorithm, processing reflections on the surface, were proposed for the accurate analysis of surface orientations, and ambiguity problems. Surface orientations, erroneously predicted by six different methods, were improved by implementing the proposed system. Consequently, the correct orientation of the surface points was determined by removing the ambiguities in images taken without considering the location of illumination, and all the tested methods provided successful results using the proposed system.

A Comprehensive Real-World Photometric Stereo Dataset for Unsupervised Anomaly Detection

Unsupervised anomaly detection is a technology that trains a model to detect anomalies using only normal data. One of the main challenges of anomaly detection is detecting defects that are not visible or obscured by lighting conditions. However, most existing methods use a single image as an input to detect anomalies in the target object, thus it is difficult to properly cope with such a challenge. In this paper, we explore an anomaly detection model using multiple images taken under different lighting conditions. For this purpose, we propose a new anomaly detection dataset based on the photometric stereo (PS) setup, and demonstrate its characteristics and advantages. In addition, we apply existing anomaly detection methods to our new dataset to verify their performance and present benchmark results. Furthermore, we present a technique for fusing multi-image and 3D surface information. Anomaly detection based on our new setup using multiple images achieves dramatic improvement of +10.15%, +11.68% and +4.38% over baseline setup using a single image for Skip-GANomaly, MKDAD and DifferNet models, respectively. Our PS-based anomaly detection dataset and analysis reports will be publicly available for future researches.

Stereo Vision-based Calibration Approach for Position Parameters of LED in Photometric Stereo

Stereo Vision-based Calibration Approach for Position Parameters of LED in Photometric Stereo

Discrete Search Photometric Stereo for Fast and Accurate Shape Estimation.

We consider the problem of estimating surface normals of a scene with spatially varying, general bidirectional reflectance distribution functions (BRDFs) observed by a static camera under varying distant illuminations. Unlike previous approaches that rely on continuous optimization of surface normals, we cast the problem as a discrete search problem over a set of finely discretized surface normals. In this setting, we show that the expensive processes can be precomputed in a scene-independent manner, resulting in accelerated inference. We discuss two variants of our discrete search photometric stereo (DSPS), one working with continuous linear combinations of BRDF bases and the other working with discrete BRDFs sampled from a BRDF space. Experiments show that DSPS has comparable accuracy to state-of-the-art exemplar-based photometric stereo methods while achieving 10-100x acceleration.

Evaluation of Photometric Stereo and Elastomeric Sensor Imaging for the Non-Destructive 3D Analysis of Questioned Documents

Evaluation of Photometric Stereo and Elastomeric Sensor Imaging for the Non-Destructive 3D Analysis of Questioned Documents

Underwater Optical 3-D Reconstruction of Photometric Stereo Considering Light Refraction and Attenuation

Photometric stereo is a widely used optical 3-D reconstruction method, which has advantages for high-resolution and well-detailed 3-D reconstruction even in weak texture regions. However, in underwater photometric stereo, the light refraction and attenuation using the camera and point light sources usually result in significant errors in shape recovery. In this article, we present a novel solution to address these challenges for improving the performance of underwater photometric stereo by combining it with underwater laser triangulation. First, we employ underwater laser triangulation as references to correct the global surface-shape distortion caused by nonuniform close-range illumination. Second, we propose to use an underwater camera refractive model to eliminate nonlinear refraction distortion. Third, we build a device implementing the proposed method for 3-D seabed reconstruction in on-site sea trials. Experimental results demonstrate that the proposed approach is able to produce accurate 3-D reconstruction results in the underwater environment.

Stud Pose Detection Based on Photometric Stereo and Lightweight YOLOv4

There are hundreds of welded studs in a car. The posture of a welded stud determines the quality of the body assembly thus affecting the safety of cars. It is crucial to detect the posture of the welded studs. Considering the lack of accurate method in detecting the position of welded studs, this paper aims to detect the weld stud’s pose based on photometric stereo and neural network. Firstly, a machine vision-based stud dataset collection system is built to achieve the stud dataset labeling automatically. Secondly, photometric stereo algorithm is applied to estimate the stud normal map which as input is fed to neural network. Finally, we improve a lightweight YOLOv4 neural network which is applied to achieve the detection of stud position thus overcoming the shortcomings of traditional testing methods. The research and experimental results show that the stud pose detection system designed achieves rapid detection and high accuracy positioning of the stud. This research provides the foundation combining the photometric stereo and deep learning for object detection in industrial production.

A deep-shallow and global–local multi-feature fusion network for photometric stereo

Recovering 3D surfaces based on the photometric stereo is a challenging task, due to the non-Lambertian surface of real-world objects. Although much effort has been made to address this issue, existing photometric stereo methods based on deep learning did not fully consider the influence of global–local features and deep-shallow features on the training process. How to combine multi-feature into a framework effectively to overcome their drawbacks has not been explored. Therefore, we propose a novel multi-feature fusion photometric stereo network (MF-PSN), focusing on both local–global and deep-shallow features fusion. Global–local feature fusion maintains the features under different illuminations and the most salient features of all illuminations, thereby effectively uses the information of each input image. Deep-shallow feature fusion keeps the features from deep and shallow layers with different receptive fields, which effectively improves the accuracy and robustness of the model. Experiments show that multi-feature fusion can make full use of the information of the input image to achieve a better reconstruction of surface normals of the object. Extensive ablation studies and experiments on the widely used DiLiGenT benchmark dataset have well verified the effectiveness of our proposed method. In addition, testing on the Gourd & Apple dataset and Light Stage Data Gallery verifies the generalization of our method.

Near-field photometric stereo using a ring-light imaging device

Traditional photometric stereo reconstructs the 3D surface shape of the target in fine details but assumes distant/directional lighting and orthogonal imaging. Yet, systematic errors occur in practical applications where the lights are divergent and near to the object. In this paper, we present a new post-processing method to correct the bias using a ring-light imaging device. We demonstrate and put forward a bias regulation process in ring-light equipment using traditional photometric stereo, where an approximate linear deviation exists in the surface gradient and a related quadratic deviation exists in the estimated surface height. Based on this low-frequency bias regulation, the proposed method corrects the systematic errors in traditional photometric stereo by using a set of sparse known real 3D points of the object surface, in order to fit the quadratic height error model and obtain more accurate 3D heights estimation. The proposed method has been evaluated using both simulated and real data to verify its accuracy and effectiveness, and has achieved significant performance. In addition, we integrated ring-light sources in homemade equipment to conduct real-world experiments, which demonstrates significant bias reduction and higher reconstruction quality compared to the existing methods. The source code of our method is available at https://github.com/fanhao/ring-light-photometric-stereo.

3D Fourier ghost imaging via semi-calibrated photometric stereo.

We achieved three-dimensional (3D) computational ghost imaging with multiple photoresistors serving as single-pixel detectors using the semi-calibrated lighting approach. We performed imaging in the spatial frequency domain by having each photoresistor obtain the Fourier spectrum of the scene at a low spectral coverage ratio of 5%. To retrieve a depth map of a scene, we inverted, apodized, and applied semi-calibrated photometric stereo (SCPS) to the spectra. At least 93.5% accuracy was achieved for the 3D results of the apodized set of images applied with SCPS in comparison with the ground truth. Furthermore, intensity error map statistics obtained at least 97.0% accuracy for the estimated surface normals using our method. Our system does not need special calibration objects or any additional optical components to perform accurate 3D imaging, making it easily adaptable. Our method can be applied in current imaging systems where multiple detectors operating at any wavelength are used for two-dimensional (2D) imaging, such as imaging cosmological objects. Employing the idea of changing light patterns to illuminate a target scene and having stored information about these patterns, the data retrieved by one detector will give the 2D information while the multiple-detector system can be used to get a 3D profile.

Automated image segmentation of air voids in hardened concrete surface using photometric stereo method

ABSTRACT The accumulative freeze-thaw cycles in cold regions could cause deteriorations to concrete pavements. The measurement of air-void system inside hardened concrete is important for estimating the freeze-thawing performance of a concrete pavement. The segmentation of air voids from the concrete surface image is one of the most essential and challenging steps for automated air-void measurement. Conventional automated air-voids segmentation methods rely on a color enhancement step to manually create a contrast between air voids and non-air-voids. This study aims to investigate the use of a photometric stereo method for automated segmentation of air voids in hardened concrete surface without a need for contrast enhancement. In the study, the photometric stereo system with a Charge-Coupled Device (CCD) camera and six illumination lights was designed to collect images of polished concrete specimen surfaces under various lighting directions. Two air-void segmentation approaches were developed in the study. One approach interpreted the air voids by the surface depth. The other approach interpreted the air voids by the surface normal. Consequently, the segmentation results were refined by a set of image processing procedures. The results showed that the proposed automated air-void segmentation procedures could detect air voids in concrete surface with a satisfactory accuracy.

Research on Character Computer Intelligent Recognition of Automobile Front Bar Bracket Based on Machine Vision

Aiming at the problem that the characters in the superimposed character area on the surface of the front bumper bracket of the car are difficult to recognize, a machine vision-based automobile front bumper bracket character recognition method is proposed, use Python, OpenCV and Halcon computer vision library for software system design. For eight images collected from different angles of light source directions, an innovative method of polynomial fitting gray value can reducethe uneven illumination of the images. The photometric stereo algorithm is used to obtain a high-contrast character image, and the separation of the two types of characters is simultaneously achieved. After the image filtering, opening and closing processes canremove background interference, use a morphological improvement algorithm based on scanning algorithm to complete character positioning, and then the improved algorithm of projection dichotomy based on the connected domain size is used to complete the character segmentation, and finally the support vector machine is used for character feature recognition. The experimental results claim that these methods can quickly separate superimposed characters and complete the recognition of non-color difference convex characters and inkjet characters, and the average recognition accuracy rate is over 96%, which meets the expected recognition requirements.

Learning conditional photometric stereo with high-resolution features

Photometric stereo aims to reconstruct 3D geometry by recovering the dense surface orientation of a 3D object from multiple images under differing illumination. Traditional methods normally adopt simplified reflectance models to make the surface orientation computable. However, the real reflectances of surfaces greatly limit applicability of such methods to real-world objects. While deep neural networks have been employed to handle non-Lambertian surfaces, these methods are subject to blurring and errors, especially in high-frequency regions (such as crinkles and edges), caused by spectral bias: neural networks favor low-frequency representations so exhibit a bias towards smooth functions. In this paper, therefore, we propose a self-learning conditional network with multi-scale features for photometric stereo, avoiding blurred reconstruction in such regions. Our explorations include: (i) a multi-scale feature fusion architecture, which keeps high-resolution representations and deep feature extraction, simultaneously, and (ii) an improved gradient-motivated conditionally parameterized convolution (GM-CondConv) in our photometric stereo network, with different combinations of convolution kernels for varying surfaces. Extensive experiments on public benchmark datasets show that our calibrated photometric stereo method outperforms the state-of-the-art.

A multi-scale methodology of turbine blade surface recovery based on photometric stereo through fast calibrations

This paper proposed a multi-scale measurement methodology for turbine blade, which is able to evaluate the geometrical features of turbine surface in different scales. A turbine blade measurement system is designed by using LED illuminators in this work. To ensure the efficiency and the accuracy of our system during the scale transition, a fast light source calibration method is introduced with a special designed calibration target, where with only one camera snapshot, the position and the orientation of a LED can be calibrated precisely and simultaneously. A surface reconstruction algorithm via iteration is improved by analyzing the LED radiant intensity distribution. The simulation is carried out with a synthetic hyperboloid structure, where the root mean square error (RMSE) of rebuilding is lower than 1.7 µm, and the mean angular error(MAE) of normal estimation is 0.5°. System calibration is performed by using a USAF 1951 calibration target with 3 different scales with the minimum deviation below 4 µm. Experiments are implemented on the concave and the tenon of a turbine blade from civil aero engine, where the profile, the size of the cooling hole and the roughness are retrieved successfully in different scales, which shows the great potential of the proposed methodology in blade inspection applications.

Profile measurement of non-Lambertian surfaces by integrating fringe projection profilometry with near-field photometric stereo

Fringe projection profilometry (FPP) performs poorly when measuring non-Lambertian surfaces, since the measured points are missing in specular regions. The existing solutions need to make a trade-off between precision and efficiency, because no extra information about specular regions is available. To tackle the specular reflection problem of FPP, this paper proposes a multi-sensor measurement method by integrating FPP with near-field photometric stereo (NFPS). Firstly, the point clouds in non-specular regions are measured by FPP. Then, a deep neural network is applied to estimate the normal map of the surface by NFPS. Finally, the intact point clouds are reconstructed by minimizing the depth gradient error calculated from the normal map. The proposed method can achieve high-precision measurement without any human intervention. In the experiment of measuring an aluminum alloy free-form surface, the deviation of the profile error between the proposed method and a coordinate measuring machine is only 6.9 µm.

Learning inter- and intraframe representations for non-Lambertian photometric stereo

Photometric stereo provides an important method for high-fidelity 3D reconstruction based on multiple intensity images captured under different illumination directions. In this paper, we present a complete framework, including a multilight source illumination and acquisition hardware system and a two-stage convolutional neural network (CNN) architecture, to construct inter- and intraframe representations for accurate normal estimation of non-Lambertian objects. We experimentally investigate numerous network design alternatives for identifying the optimal scheme to deploy inter- and intraframe feature extraction modules for the photometric stereo problem. Moreover, we propose utilizing the easily obtained object mask to eliminate adverse interference from invalid background regions in intraframe spatial convolutions, thus effectively improving the accuracy of normal estimation for surfaces made of dark materials or with cast shadows. Experimental results demonstrate that the proposed masked two-stage photometric stereo CNN model (MT-PS-CNN) performs favourably against state-of-the-art photometric stereo techniques in terms of both accuracy and efficiency. In addition, the proposed method is capable of predicting accurate and rich surface normal details for non-Lambertian objects of complex geometry and performs stably given inputs captured in both sparse and dense lighting distributions.

Recreating the Relief of the Temple of the Jaguars: Exploring Digital and Analogue 2.5D Printing of Mesoamerican Imagery

Digital and analogue printing methods are studied for reconstructing a Mayan decorative relief from the ancient temple of the Jaguars. Height maps, image files encoding height information as intensity values, were produced in commercial image editing software from early 20th century photographic records. Physical height was generated in different materials through analogue printing processes including casting and embossing from photo exposed polymer plates, and digital processes including 2.5D elevated printing and stereolithography. The surface geometry of the outcomes from the analogue processes was recorded using photometric stereo to obtain feedback on the translation of the grayscale range onto relief. 2.5D prints produced with Canon Elevated Printing and stereolithography were used for comparison and qualitive assessment. Differences, parallels and sustainability of the processes are discussed.

Taylor-rider-based deep convolutional neural network for image forgery detection in 3D lighting environment

PurposeWith the advancements in photo editing software, it is possible to generate fake images, degrading the trust in digital images. Forged images, which appear like authentic images, can be created without leaving any visual clues about the alteration in the image. Image forensic field has introduced several forgery detection techniques, which effectively distinguish fake images from the original ones, to restore the trust in digital images. Among several forgery images, spliced images involving human faces are more unsafe. Hence, there is a need for a forgery detection approach to detect the spliced images.Design/methodology/approachThis paper proposes a Taylor-rider optimization algorithm-based deep convolutional neural network (Taylor-ROA-based DeepCNN) for detecting spliced images. Initially, the human faces in the spliced images are detected using the Viola–Jones algorithm, from which the 3-dimensional (3D) shape of the face is established using landmark-based 3D morphable model (L3DMM), which estimates the light coefficients. Then, the distance measures, such as Bhattacharya, Seuclidean, Euclidean, Hamming, Chebyshev and correlation coefficients are determined from the light coefficients of the faces. These form the feature vector to the proposed Taylor-ROA-based DeepCNN, which determines the spliced images.FindingsExperimental analysis using DSO-1, DSI-1, real dataset and hybrid dataset reveal that the proposed approach acquired the maximal accuracy, true positive rate (TPR) and true negative rate (TNR) of 99%, 98.88% and 96.03%, respectively, for DSO-1 dataset. The proposed method reached the performance improvement of 24.49%, 8.92%, 6.72%, 4.17%, 0.25%, 0.13%, 0.06%, and 0.06% in comparison to the existing methods, such as Kee and Farid's, shape from shading (SFS), random guess, Bo Peng et al., neural network, FOA-SVNN, CNN-based MBK, and Manoj Kumar et al., respectively, in terms of accuracy.Originality/valueThe Taylor-ROA is developed by integrating the Taylor series in rider optimization algorithm (ROA) for optimally tuning the DeepCNN.