Photometric Stereo Research Articles

Revealing ‘invisible’ poetry by W. H. Auden through computer vision: Using photometric stereo to visualize indented impressions

Abstract This article explores the use of computer-vision technologies in the context of digitally editing and researching letters and literary papers by the British-American poet W. H. Auden. Two documents in the previously inaccessible ‘Auden Musulin Papers’ contain colourless indented typewriter impressions of poetry. These impressions result from the papers’ original use as ‘backing sheets’, inserted into a typewriter below those sheets of paper on which Auden typed his poetry. Subsequently, these backing sheets were reused in the poet’s ‘working correspondence’ to Welsh-Austrian writer Stella Musulin. While standard image-digitization technologies fail to capture these 3D indented impressions, they can successfully be represented by means of Photometric Stereo, which has been fruitfully employed in the research of 3D cultural-heritage objects. Following a detailed outline of this method, this article demonstrates how Photometric Stereo can help to reconstruct poetry that has survived only in the form of indented impressions. Thus, the case study illustrates how computer vision can contribute to our understanding both of ‘poetic’ practices of composition and revision as well as of ‘material’ writing practices. It also has wide-ranging implications for re-conceptualizing sheets of paper as 3D objects in the research of literary documents from the twentieth century.

KNOWLEDGE AND DOCUMENTATION OF RENAISSANCE WORKS OF ART: THE REPLICA OF THE “ANNUNCIATION” BY BEATO ANGELICO

Abstract. The Annunciation by Guido di Pietro from Mugello, known as Beato Angelico, is a wide tempera painting with some fine gold foil placed on a wooden support, today hosted at the Museum of the Basilica of Santa Maria delle Grazie, in San Giovanni Valdarno. On the occasion of the exhibition “Masaccio e Angelico. Dialogo sulla verità nella pittura”, the museum asked to the Department of Architecture at the University of Bologna to develop a digital high-resolution surrogate to favour deep investigations, to plan restoration and to simply tell the stories behind the artwork. Two tasks were accomplished: to let visitors discover the secrets in the painting and to let scholars study the artwork, to better understand the masterpiece. This paper introduces the outcomes of the research developed to digitize the Annunciation, following a dedicated pipeline developed to improve the fruition of its digital replica, originated from different input sources, and surrogating the user experience on the real object. This work presents a method for the 3D reconstruction of the surfaces based on different techniques for elements with different depth resolutions (i.e., the painting and the wooden frame) which combine photogrammetry and photometric stereo exploiting both procedures and pushing forward the boundaries of Gigapixel Imaging and photogrammetric-based 3D model representation.

A measure of perceived quality to secure the polishability of new paint systems

The automotive industry continuously strives to reduce their environmental impact. For the paint shop it means to introduce more sustainable paint concepts, while maintaining the production rate and retain the right surface appearance that is crucial for the vehicle’s perceived quality. Today most painted parts are visually inspected and, if needed, manually repaired by abrasive polishing to eliminate spot defects. The repair process consists of one sanding step to remove the defect, and one or two rubbing/polishing steps to restore the surface, but still it tends to be a non-reliable process leaving patterns or clusters of shallow micro scratches seen as three-dimensional shapes moving over the surface when viewed from different angles like holograms. These so called ‘polishing roses’ are hard to detect in artificial light but clearly visible in Sunlight and therefore they constitute a constant quality issue. Accurate polishing procedures in combination with more objective inspection techniques would secure a high surface quality—but what is ‘accurate’? The overall scope of the study was to deepen the knowledge of paint systems to develop test routines for the polishability of coatings already during the development stage, and thereby ease the implementation of new coating systems in production. The study was based on collected process data from professional polishers to define a process window based on key parameters for successful end-of-line repairs of coated surfaces, i.e. strategies minimizing the occurrence of visible polishing traces. A CNC-machine was built up for the purpose to systematically test and evaluate new coating systems and repair procedures. The surface estimation was made by visual inspections as well as by a further developed photometric stereo system providing quantitative images of remaining repair traces.

Paper bending stiffness and web tension measurement from a running web using a vacuum and computer imaging

A novel method for measuring the bending stiffness of paper online during manufacturing is intro-duced. The method uses photometric stereo imaging to detect the shape of the deflection surface caused by a controllable pressure difference over the paper’s surfaces. The hardware for the measurement is based on a combination of two existing sensors, which has accelerated and facilitated the development of the implementation. The deflection and loads are tied together by the governing differential equation for the bending of an orthotropic elastic plate with selected simplifications. An approach to resolve material parameters and in-plane loads without knowledge of traditional boundary conditions is suggested. The presented method was tested in a paper mill during manufacturing. For bending stiffnesses measured online, correlation coefficients 0.88 and 0.92 were obtained compared to state-of-the-art laboratory measurements. However, the results gained from a moving web are noisy and the data requires correction of the slope and an offset. Although several issues remain to be resolved before the method can be considered as an accurate measurement for industrial use, the theoretical background, the performance of online implementation, and the results are promising. Possible causes for the observed discrepancies and the future development of the method are discussed.

RMAFF-PSN: A Residual Multi-Scale Attention Feature Fusion Photometric Stereo Network

Predicting accurate normal maps of objects from two-dimensional images in regions of complex structure and spatial material variations is challenging using photometric stereo methods due to the influence of surface reflection properties caused by variations in object geometry and surface materials. To address this issue, we propose a photometric stereo network called a RMAFF-PSN that uses residual multiscale attentional feature fusion to handle the “difficult” regions of the object. Unlike previous approaches that only use stacked convolutional layers to extract deep features from the input image, our method integrates feature information from different resolution stages and scales of the image. This approach preserves more physical information, such as texture and geometry of the object in complex regions, through shallow-deep stage feature extraction, double branching enhancement, and attention optimization. To test the network structure under real-world conditions, we propose a new real dataset called Simple PS data, which contains multiple objects with varying structures and materials. Experimental results on a publicly available benchmark dataset demonstrate that our method outperforms most existing calibrated photometric stereo methods for the same number of input images, especially in the case of highly non-convex object structures. Our method also obtains good results under sparse lighting conditions.

Evaluation of photometric stereo and elastomeric sensor imaging for the non-destructive 3D analysis of questioned documents – A pilot study

Photometric Stereo and Elastomeric Sensor Imaging were assessed for measuring the 3-dimensional (3D) morphology of questioned document samples. Photometric stereo is shown to be a powerful non-contact technique for revealing micron level detail of the samples examined. Elastomeric Sensor Imaging is shown to complement photometric stereo by yielding equivalent results. Additionally, this technique allows quantification of the morphological depth information. The techniques were applied to 2 different types of questioned document sample. Firstly, written signatures were examined. Both techniques were able to reveal characteristic features that could be used to infer stroke direction and ink line application sequence. Secondly toner/ink intersections were examined. Both techniques allowed visualisation of 3D features which were used to infer the sequence of application.

An efficient calibration model for fringe projection profilometry using inclined Lambertian plane based on photometric stereo

An efficient calibration model for fringe projection profilometry using inclined Lambertian plane based on photometric stereo

Continuous material reflectance map for deep photometric stereo.

Solving calibrated photometric stereo under a sparse set of lights is of great interest for real-world applications. Since neural networks show advantages in dealing with material appearance, this paper proposes a bidirectional reflectance distribution function (BRDF) representation, which is based on reflectance maps for a sparse set of lights and can handle various types of BRDFs. We discuss the optimal way to compute these BRDF-based photometric stereo maps regarding the shape, size, and resolution, and experimentally investigate the contribution of these maps to normal map estimation. The training dataset was analyzed to establish the BRDF data to use between the measured and parametric BRDFs. The proposed method was compared to state-of-the-art photometric stereo algorithms for different datasets from numerical rendering simulations, DiliGenT, and our two acquisition systems. The results show that our representation outperforms the observation maps as BRDF representation for a neural network for various surface appearances on specular and diffuse areas.

Light invariant photometric stereo.

An imaging setup that enables unsynchronized photometric stereo (PS) for Lambertian objects based on modulated light sources is presented. Knowing the specific frequency of the modulated light source allows to filter out any other light in the scene. This creates an image that depends only on the particular light source while ignoring the ambient light. Moreover, if the scene is illuminated by multiple modulated sources with different frequencies, repeating this process for every frequency will produce a sequence of images with the corresponding illumination. This sequence is then used as an input to the PS algorithm for 3D reconstruction. The proposed approach, named Light Invariant Photometric Stereo (LIPS), was verified on both synthetic and real-world data. LIPS eliminates the need for synchronization between the sources and the camera and significantly outperformed the classical PS method in an illuminated environment.

DPPS: A deep-learning based point-light photometric stereo method for 3D reconstruction of metallic surfaces

Three-dimensional (3D) measurement provides essential geometric information for quality control and process monitoring in many manufacturing applications. Photometric stereo is one of the potential solutions for in-process metrology and active geometry compensation, which takes multiple images of an object under different illuminations as inputs and recovers its surface normal map based on a reflectance model. Deep learning approaches have shown their potential in solving the highly nonlinear problem for photometric stereo, but the main challenge preventing their practical application in process metrology lies in the difficulties in the generation of a comprehensive dataset for training the deep learning model. This paper presents a new Deep-learning based Point-light Photometric Stereo method, DPPS, which utilizes a multi-channel deep convolutional neural network (CNN) to achieve end-to-end prediction for both the surface normal and height maps in a semi-calibrated fashion. The key contribution is a new dataset generation method combining both physics-based and data-driven approaches, which minimizes the training cost and enables DPPS to handle reflective metal surfaces with unknown surface roughness. Even trained only with fully synthetic and high-fidelity dataset, our DPPS surpasses the state-of-the-art with an accuracy better than 0.15 cm over a 10 cm × 10 cm area and its real-life experimental results are on par with commercial 3D scanners. The demonstrated results provide guidance on improving the generalizability and robustness of deep-learning based computer vision metrology with minimized training cost as well as show the potential for in-process 3D metrology in advanced manufacturing processes.

A novel objective fabric pilling evaluation method based on the photometric stereo algorithm

The pilling grade is one of the important indexes to evaluate the quality of fabric; the traditional method for pilling evaluation is based on manual operation, which is subjective. In this paper, a self-developed system is established and a photometric stereo-based three-dimensional (3D) acquisition method for pilling evaluation has been demonstrated. Firstly, a 3D surface model is obtained by using photometric stereo, mapping a 3D model to a two-dimensional image; the gray value is used to represent the depth value of the fabric surface and the pilling segmentation is realized by the Otsu threshold method after filtering the fabric texture by the relative total variance algorithm. The pilling number and pilling area are used as eigenvectors for pilling classification using the support vector machine. Finally, the classification accuracy of the pilling grade was 95.9%. For pilling evaluation, the developed system and method can be considered reliable and effective according to the experimental results.

FFT-BASED FILTERING APPROACH TO FUSE PHOTOGRAMMETRY AND PHOTOMETRIC STEREO 3D DATA

Abstract. Image-based 3D reconstruction has been successfully employed for micro-measurements and industrial quality control purposes. However, obtaining a highly-detailed and reliable 3D reconstruction and inspection of non-collaborative surfaces is still an open issue. Photometric stereo (PS) offers the high spatial frequencies of the surface, but the low frequency is erroneous due to the mathematical model's assumptions and simplifications on how light interacts with the object surface. Photogrammetry, on the other hand, gives precise low-frequency information but fails to utilize high frequencies. As a result, in this research, we present a fusion strategy in Fourier domain to replace the low spatial frequencies of PS with the corresponding photogrammetric frequencies in order to have correct low frequencies while maintaining high frequencies from PS. The proposed method was tested on three different objects. Different cloud-to-cloud comparisons were provided between reference data and the 3D points derived from the proposed method to evaluate high and low frequency information. The obtained 3D findings demonstrated how the proposed methodology generates a high-detail 3D reconstruction of the surface topography (below 20 µm) while maintaining low-frequency information (0.09 µm on average for three different testing objects) by fusing photogrammetric and PS depth data with the proposed FFT-based method.

Underwater 3D reconstruction using a photometric stereo with illuminance estimation.

Underwater image sensing is directly affected by the change in illuminance towards a camera caused by the refraction of light in different media. In this study, the convergence of a near-field point light source in water is analyzed using a light propagation model. The photometric stereo (PS) formula is determined based on an accurate estimation of the illuminance entering the camera. An underwater PS system is designed to verify the proposed method's feasibility. The experimental results demonstrate improved accuracy in normal calculation. This helps achieve accurate underwater 3D reconstruction of objects that is suitable for underwater surface microdefect detection.

Detection algorithm for bearing roller end surface defects based on improved YOLOv5n and image fusion

For the current problems of low accuracy and poor reliability of defect detection for bearing roller end surfaces in industrial production, this paper proposes a bearing roller end surface defect detection algorithm based on improved YOLOv5n and the fusion of gamma-corrected maps and curvature maps. First, this paper uses photometric stereo vision to reconstruct the three-dimensional shape of the surface and proposes an improved Frankot–Chellappa integration algorithm to solve the problem of reconstructing surface deformation. Secondly, the DenseFuse network is used to fuse gamma-corrected maps and curvature maps to generate an image dataset that combines the strengths of both images to enhance defect features and improve the precision of target detection. Finally, the improved target detection model YOLOv5n is proposed to detect defects in the end surfaces of bearing rollers. The experimental results show that by using fused images for training, detection models with higher mean average precision(mAP) than traditional images can be obtained, and the improved YOLOv5n algorithm maintains the high real-time performance of the original algorithm while the mAP0.5 and mAP0.5:0.95 of improved YOLOv5n are 98.6% and 87.4%, respectively, which are respectively 0.9% and 2.8% higher than YOLOv5n.

SGVM: Semantic-Guided Variational Model for Sealing Nail Defect Extraction Within Albedo Domain via Photometric Stereo

SGVM: Semantic-Guided Variational Model for Sealing Nail Defect Extraction Within Albedo Domain via Photometric Stereo

Accurate Normal Measurement of Non-Lambertian Complex Surface Based on Photometric Stereo

Accurate Normal Measurement of Non-Lambertian Complex Surface Based on Photometric Stereo

Image-to-Height Domain Translation for Synthetic Aperture Sonar

Synthetic aperture sonar (SAS) intensity statistics are dependent upon the sensing geometry at the time of capture. Estimating bathymetry from acoustic surveys is challenging. While several methods have been proposed to estimate seabed relief via intensity, we develop the first large-scale study that relies on deep learning models. In this work, we pose bathymetric estimation from SAS surveys as a domain translation problem of translating intensity to height. Since no dataset of coregistered seabed relief maps and sonar imagery previously existed to learn this domain translation, we produce the first large simulated dataset containing coregistered pairs of seabed relief and intensity maps from two unique sonar data simulation techniques. We apply four types of models, with varying complexity, to translate intensity imagery to seabed relief: a shape-from-shading (SFS) approach, a Gaussian Markov random field (GMRF) approach, a conditional Generative Adversarial Network (cGAN), and UNet architectures. Each model is applied to datasets containing sand ripples, rocky, mixed, and flat sea bottoms. Methods are compared in reference to the coregistered simulated datasets using L1 error. Additionally, we provide results on simulated and real SAS imagery. Our results indicate that the proposed UNet architectures outperform an SFS, a GMRF, and a pix2pix cGAN model.

MASR-PSN: a low-resolution photometric stereo images-relevant deep learning model for high-resolution surface normal reconstruction

MASR-PSN: a low-resolution photometric stereo images-relevant deep learning model for high-resolution surface normal reconstruction

Multilayer Perceptron-Based Fusion Method for Metal Surface Measurement Data by Multi-Sensors Incorporating Photometric Stereo and Structured Light

Multilayer Perceptron-Based Fusion Method for Metal Surface Measurement Data by Multi-Sensors Incorporating Photometric Stereo and Structured Light

Application of Deep Learning Technology to Photometric Stereo Three-dimensional Reconstruction

Application of Deep Learning Technology to Photometric Stereo Three-dimensional Reconstruction