Photometric Stereo Research Articles

Automated Robot-based Large-Scale 3D Surface Imaging

Abstract This work develops a robot-based automated 3D imaging system for large-scale surface measurement at high resolution. The system has the advantages of allowing 1) high-resolution 3D surface imaging based on photometric stereo, and 2) automatic stitching of multiple images collected by a robot for large-scale surface measurement. We developed a dome-shaped image acquisition system with 16 individually controlled lights, mounted on a robot (Kuka iiwa lbr). A photometric stereo with a lighting selection mechanism is used for the reconstruction of local surface regions. To allow image stitching for large-scale surface measurement, one challenge arises from the robot arm’s limited encoder precision and accuracy, which is about ±150 µm for its repeatability and even lower for its nominal accuracy. This is unsuitable for the applications of surface metrology or inspection. To compensate the errors introduced by the chained robot arm’s encoders, for image stitching, we experimented with two feature descriptors extracted from the normal and the curvature space respectively, and performed comparative studies with standard feature descriptors from the standard grey-scale intensity space. The normal-based feature descriptor demonstrated advantages of illumination invariance while the curvature-based feature descriptor demonstrated clear advantages of rotation invariance, and feasibility of aligning multiple images with high accuracy.

3D Face Reconstruction From Single 2D Image Using Distinctive Features

3D face reconstruction is considered to be a useful computer vision tool, though it is difficult to build. This paper proposes a 3D face reconstruction method, which is easy to implement and computationally efficient. It takes a single 2D image as input, and gives 3D reconstructed images as output. Our method primarily consists of three main steps: feature extraction, depth calculation, and creation of a 3D image from the processed image using a Basel face model (BFM). First, the features of a single 2D image are extracted using a two-step process. Before distinctive-features extraction, a face must be detected to confirm whether one is present in the input image or not. For this purpose, facial features like eyes, nose, and mouth are extracted. Then, distinctive features are mined by using scale-invariant feature transform (SIFT), which will be used for 3D face reconstruction at a later stage. Second step comprises of depth calculation, to assign the image a third dimension. Multivariate Gaussian distribution helps to find the third dimension, which is further tuned using shading cues that are obtained by the shape from shading (SFS) technique. Thirdly, the data obtained from the above two steps will be used to create a 3D image using BFM. The proposed method does not rely on multiple images, lightening the computation burden. Experiments were carried out on different 2D images to validate the proposed method and compared its performance to those of the latest approaches. Experiment results demonstrate that the proposed method is time efficient and robust in nature, and it outperformed all of the tested methods in terms of detail recovery and accuracy.

Multiscale Convolutional Fusion Network for Non-Lambertian Photometric Stereo

One of the key issues in photometric stereo is the extension of its application in real world objects which shows non-Lambertian reflectance. This letter proposes a multi-scale weighted convolutional fusion network with deep learning architecture to realize high-precision perception of non-Lambertian surfaces under arbitrary illumination conditions. A multi-scale convolutional fusion module is designed to strengthen the photometric physics and the utilization of the neighborhood features at the same time so as to overcome shadows and distinguish multiple materials. In order to further deal with the problem of arbitrary illumination conditions, a multi-resolution polar coordinate division method is proposed to integrate the input image information and fully utilizes the multi-scale convolution. Both syntheses and real-world experiments verifies the performance of the proposed method in recovery accuracy and computational efficiency.

Hybrid Framework for Haptic Texture Modeling and Rendering

We present a hybrid framework for the modeling and rendering of both homogeneous and inhomogeneous textures with high realism by combining force and vibration feedback. Given a real texture sample, we first capture a force-feedback model using photometric stereo, which is one of the most accurate algorithms for 3D surface reconstruction in computer vision. This method allows us to model the micro-geometry of the surface and represent it by a two-dimensional (2D) height map in a very high resolution to the order of 10$\mu \text{m}$ . We also provide a new force rendering algorithm adequate for such an accurate and fine texture model. By a user study, we verify that the proposed force modeling and rendering algorithms are suitable for our hybrid texture framework. Second, we model the homogeneous and stationary vibrational characteristics of the real texture sample using the prevalent LPC (linear predictive coding)-based method. Third, we merge the force and vibration feedback models appropriately into a hybrid framework on the basis of their spectral characteristics. Last, another user study that assessed the perceived similarity between real textures and their virtual simulations demonstrates that our hybrid framework can achieve a high level of realism. To our knowledge, our framework is unique in that it can model inhomogeneous (in additional to homogeneous) real textures and render them in a virtual environment with high realism by providing both force and vibrational sensory cues.

Research and Application of Label Defect Detection Method Based on Machine Vision

For the problems of scratches and false detections such as wrinkles and other defects in the printing process, the HALCON software is used for programming, and the product label defects are detected in real time based on the photometric stereo vision technology. Firstly, by dimming the product labels of four different orientations, different orientation maps are obtained, the illumination map is calculated to obtain the albedo map, and then the four-source photometric stereo technology is used on the basis of the traditional photometric stereo technology. After obtaining the surface gradient information of the tag to be tested, the measurement image is subjected to morphological processing using the gradient information to obtain a curvature image. The curvature image is then image processed. Finally, the defective portion is filtered to effectively segment the surface defect area of the label. This method is implemented using HALCON software and is implemented in conjunction with C# mixed programming.

Multi-scale measurement of high-reflective surfaces by integrating near-field photometric stereo with touch trigger probe

Precision measurement of large scale surface topography is a challenging task for high-reflective parts. This paper presents a multi-sensor system to address this issue by integrating the technical merits of touch trigger probe and photometric stereo. The probe is used to extract a set of high-precision sparse points for an initial surface construction, which is then applied to drive a near-field photometric stereo for recovering dense surface normal of the surface. By fusing the sparse discrete points with the dense surface normal, high-resolution surface topography can be obtained accordingly. Experiments confirm the state-of-the-art performance of the proposed method.

Multi-View Photometric Stereo: A Robust Solution and Benchmark Dataset for Spatially Varying Isotropic Materials.

We present a method to capture both 3D shape and spatially varying reflectance with a multi-view photometric stereo (MVPS) technique that works for general isotropic materials. Our algorithm is suitable for perspective cameras and nearby point light sources. Our data capture setup is simple, which consists of only a digital camera, some LED lights, and an optional automatic turntable. From a single viewpoint, we use a set of photometric stereo images to identify surface points with the same distance to the camera. We collect this information from multiple viewpoints and combine it with structure-from-motion to obtain a precise reconstruction of the complete 3D shape. The spatially varying isotropic bidirectional reflectance distribution function (BRDF) is captured by simultaneously inferring a set of basis BRDFs and their mixing weights at each surface point. In experiments, we demonstrate our algorithm with two different setups: a studio setup for highest precision and a desktop setup for best usability. According to our experiments, under the studio setting, the captured shapes are accurate to 0.5 millimeters and the captured reflectance has a relative root-mean-square error (RMSE) of 9%. We also quantitatively evaluate state-of-the-art MVPS on a newly collected benchmark dataset, which is publicly available for inspiring future research.

Leveraging Shape, Reflectance and Albedo From Shading for Face Presentation Attack Detection

Presentation attack detection is a challenging problem that aims at exposing an impostor user seeking to deceive the authentication system. In facial biometrics systems, this kind of attack is performed using a photograph, video, or 3D mask containing the biometric information of a genuine identity. In this paper, we propose a novel approach to detecting face presentation attacks based on intrinsic properties of the scene such as albedo, depth, and reflectance properties of the facial surfaces, which were recovered through a shape-from-shading (SfS) algorithm. To extract meaningful patterns from the different maps obtained with the SfS algorithm, we designed a novel shallow CNN architecture for learning features useful to the presentation attack detection (PAD). We performed several experiments considering the intra- and inter-dataset evaluation protocols. The obtained results showed the effectiveness of the proposed method considering several types of photo- and video-based presentation attacks, and in the cross-sensor scenario, besides achieving competitive results for the inter-dataset evaluation protocol.

3D-FLARE: A Touchless Full-3D Fingerprint Recognition System Based on Laser Sensing

“The friction ridge pattern is a 3D structure which, in its natural state, is not deformed by contact with a surface”. Building upon this rather trivial observation, the present work constitutes a first solid step towards a paradigm shift in fingerprint recognition from its very foundations. We explore and evaluate the feasibility to move from current technology operating on 2D images of elastically deformed impressions of the ridge pattern, to a new generation of systems based on full-3D models of the natural non-deformed ridge pattern itself. There are already a small number of previous studies that have already started scratching the surface of 3D fingerprint recognition and that should not go overlooked. However, the vast majority of these few successful approaches published so far, are based on the reconstruction of fingerprints from multiple 2D images acquired with different lighting conditions (photometric stereo 3D reconstruction) or acquired from different angles (stereo vision 3D reconstruction). Such reconstruction methods lead in general to 2D fingerprints wrapped over the overall volume of the finger. These volumetric fingerprints have shown some promising performance, but still miss the real depth information of the ridge pattern, which, in the best case scenario, is coarsely estimated during the error-prone reconstruction process. In the present work we take one step further, directly acquiring for the first time in a consistent and repeatable manner, full-3D fingerprint models stored as point-clouds, where each point is defined by its $[x,y,z]$ coordinates. This way, the 3D data is directly measured by the sensor, with no post-processing reconstruction stage required. The complete recognition system developed represents as well an alternative to traditional technology based on minutiae detection. It shows that image-based processing algorithms and descriptors can be successfully applied to the new full-3D data, reaching very competitive results and confirming the high distinctiveness of the models.

Non-Lambertian Photometric Stereo Network based on Inverse Reflectance Model with Collocated Light.

Current non-Lambertian photometric stereo methods generally require a large number of images to ensure accurate surface normal estimation. To achieve accurate surface normal recovery under a sparse set of lights, this paper proposes a non-Lambertian photometric stereo network based on a derived inverse reflectance model with collocated light. The model is deduced using monotonicity of isotropic reflectance and the univariate property of collocated light to decouple the surface normal from the reflectance function. Thus, the surface normal can be estimated by three steps, i.e., model fitting, shadow rejection, and normal estimation. We leverage a supervised deep learning technique to enhance the shadow rejection ability and the flexibility of the inverse reflectance model. Shadows are handled through max-pooling. Information from a neighborhood image patch is utilized to improve the flexibility to various reflectances. Experiments using both synthetic and real images demonstrate that the proposed method achieves state-of-the-art accuracy in surface normal estimation.

Photometric computer vision-aided system for psoriasis severity scoring: a preclinical study based on a mouse model of psoriasis

Psoriasis is an inflammatory cutaneous disease of unknown origin, characterized by the appearance of red, itchy, and scaly plaques of abnormal skin. One of the most important issues that guides treatment of psoriasis is to evaluate the degree of the illness determining the psoriasis and area severity index (PASI). Dermatologists usually use visual and tactile senses to assess lesion severity, involving a subjective judgment depending on the medical practitioner. For this purpose, we propose an image processing system based on photometric stereo acquisition technique and adapted analysis criteria to evaluate skin parameters. This diagnosis aided system gives an objective and accurate evaluation of skin erythema (redness), skin thickness, and scaling, which are the three clinical parameters, with the area of lesioned skin, that determine PASI score. Thus, we estimate the intensity of erythema from the albedo map and thickness from the three-dimensional estimation of the skin surface. Finally, the combination of color and geometry results allows identifying and quantifying the skin scaling. A validation protocol with an image database that covers all parameter variation ranges is proposed, and the results show a high correlation with dermatologist scores.

Image-Based Inspection Technique of a Machined Metal Surface for an Unmanned Lapping Process

This paper presents a new machine vision framework for the efficient examination and classification of surface textures on medium- and large-sized mold products, such as used for automobiles, TVs, and refrigerators. Existing techniques, which are based on the hands and eyes of skilled workers, are inconsistent and time-consuming. Although there are many types of precise surface inspection and measurement methods, most are difficult to apply at industrial sites or by finishing robots due to problems such as speed, setup limitations, and robustness. This paper proposes two techniques based on image processing that aims to automate surface inspection during an unmanned lapping process that is mainly employed to eliminate milling tool marks. First, both the shape of the reflected light and the intensity of the captured near-field contrast image right after the reflected specular are used to determine the machined surface state, and the presence of tool marks as the line light source scans counter-clockwise. Second, the photometric stereo technique is used to detect surface scratches through the normal map that recovers the surface. The proposed techniques show localized machined patterns and classify them with high accuracy.

Single Day Outdoor Photometric Stereo.

Photometric Stereo (PS) under outdoor illumination remains a challenging, ill-posed problem due to insufficient variability in illumination. Months-long capture sessions are typically used in this setup, with little success on shorter, single-day time intervals. In this paper, we investigate the solution of outdoor PS over a single day, under different weather conditions. First, we investigate the relationship between weather and surface reconstructability in order to understand when natural lighting allows existing PS algorithms to work. Our analysis reveals that partially cloudy days improve the conditioning of the outdoor PS problem while sunny days do not allow the unambiguous recovery of surface normals from photometric cues alone. We demonstrate that calibrated PS algorithms can thus be employed to reconstruct Lambertian surfaces accurately under partially cloudy days. Second, we solve the ambiguity arising in clear days by combining photometric cues with prior knowledge on material properties, local surface geometry and the natural variations in outdoor lighting through a CNN-based, weakly-calibrated PS technique. Given a sequence of outdoor images captured during a single sunny day, our method robustly estimates the scene surface normals with unprecedented quality for the considered scenario. Our approach does not require precise geolocation and significantly outperforms several state-of-the-art methods on images with real lighting, showing that our CNN can combine efficiently learned priors and photometric cues available during a single sunny day.

A non-Lambertian photometric stereo under perspective projection

Under the perspective projection assumption, non-Lambertian photometric stereo is a highly non-linear problem. In this study, we present an optimized framework for reconstructing the surface normal and depth with non-Lambertian reflection models under perspective projection. By decomposing the images into diffuse and specular components, we compute the surface normal and reflectance simultaneously. We also propose a variational formulation that is robust and useful for surface reconstruction. The experiments show that our method accurately reconstructs both the surface shape and reflectance of colorful objects with non-Lambertian surfaces.

A dual-cue network for multispectral photometric stereo

Estimating pixel-wise surface normal from a single image is a challenging task but offers great values to computer vision and robotics applications. By using the spectrally and spatially variant illumination, multispectral photometric stereo can produce pixel-wise surface normal from just one image. But multispectral photometric stereo methods may encounter the tangle problem of illumination, surface reflectance and camera response, which lead to an under-determined system. Existing approaches rely on either extra depth information or material calibration strategies, assuming a Lambertian surface condition which limits their application in practical systems. Previous learning-based methods employ fully-connected or CNN architectures to estimate surface normal. Compared with fully-connected framework, CNN takes advantage of the information embedded in the neighborhood of a surface point, but losing high-frequency surface normal details. In this paper, we present a new method that addresses this task by designing two stacked deep network. We first apply a CNN-based structural cue network to approximate coarse surface normal on small patches. Then, we use a pixel level fully-connected photometric cue network to further refine surface normal details and correct errors from the first step. The fused network is robust to non-Lambertian surfaces and complex illumination environments, such as ambient light and variant light directions. Experimental results show that our dual-cue fused network outperforms existing methods.

Automating RTI: Automatic light direction detection and correcting non-uniform lighting for more accurate surface normals

Reflectance Transformation Imaging (RTI) (Malzbender et al., 2001) is a photometric stereo technique that enables the interactive relighting of the object of interest from novel lighting directions, and an estimation of surface topography through the calculation of surface normal vectors. We propose a novel, fully automated technique for correcting common lighting errors in RTI and markedly improve the accuracy of surface normal estimation, as well as increasing the legibility of low relief surface variations. This moves RTI from the qualitative domain (e.g. enabling the reading of weathered inscriptions) into the quantitative domain of computer vision. RTI assumes only light direction, and not received intensity, changes as the object is imaged. Like other authors we show that this assumption is false and propose a novel method to correct for it. However, we estimate the lighting directions automatically, unlike other proposed correction techniques. Our method also requires no calibration equipment, meaning it can be easily retrofitted to any existing stack of RTI photographs. We increase the simplicity of the standard highlight RTI method by automatically detecting lighting directions and maintain its appeal to non-imaging professionals.

An integrated photogrammetric and photoclinometric approach for illumination-invariant pixel-resolution 3D mapping of the lunar surface

High-resolution 3D information on lunar and planetary surfaces is crucial for planetary exploration missions and science. Photogrammetry is a state-of-the-art technology for generating 3D topographic models of surfaces such as digital elevation models (DEMs). The performance of the photogrammetry and the resulting DEMs is affected by image matching. However, most matching algorithms fail when the images have large differences in illumination and subtle textures. This problem can be addressed by integrating photoclinometry into the photogrammetric process. This paper presents an integrated photogrammetric and photoclinometric approach that is able to generate pixel-resolution DEMs of the lunar surface and is illumination invariant. The incorporation of photoclinometry into photogrammetry involves two main steps. First, a photoclinometry assisted image matching (PAM) approach is developed by integrating photometric stereo analysis in the image matching to create pixel-wise matches, even for images with large illumination differences. Second, the DEM derived from photogrammetry using the matching results is further refined to pixel-wise resolution using photoclinometry with a shadow constraint. The proposed approach has been used for high-resolution topographic mapping at the Chang’E-4 and Chang’E-5 landing sites using Lunar Reconnaissance Orbiter Camera (LROC) Narrow Angle Camera (NAC) images acquired under different illumination conditions. The results indicate that the proposed approach is robust to severe inconsistencies in illumination and subtle textures in cases where the conventional approaches fail. The approach is able to achieve geometric accuracies comparable to photogrammetry but more small-scale topographic details. The proposed approach can also be used for high-resolution topographic mapping of other planetary bodies such as Mercury or asteroids, and provides a useful reference for similar topographic mapping on Earth.

Detailed Surface Geometry and Albedo Recovery from RGB-D Video under Natural Illumination.

This article presents a novel approach for depth map enhancement from an RGB-D video sequence. The basic idea is to exploit the photometric information in the color sequence to resolve the inherent ambiguity of shape from shading problem. Instead of making any assumption about surface albedo or controlled object motion and lighting, we use the lighting variations introduced by casual object movement. We are effectively calculating photometric stereo from a moving object under natural illuminations. One of the key technical challenges is to establish correspondences over the entire image set. We, therefore, develop a lighting insensitive robust pixel matching technique that out-performs optical flow method in presence of lighting variations. An adaptive reference frame selection procedure is introduced to get more robust to imperfect lambertian reflections. In addition, we present an expectation-maximization framework to recover the surface normal and albedo simultaneously, without any regularization term. We have validated our method on both synthetic and real datasets to show its superior performance on both surface details recovery and intrinsic decomposition.

Improved Visual Inspection Through 3D Image Reconstruction of Defects Based on the Photometric Stereo Technique.

Visual inspections of nuclear power plant (NPP) reactors are important for understanding current NPP conditions. Unfortunately, the existing visual inspection methods only provide limited two-dimensional (2D) information due to a loss of depth information, which can lead to errors identifying defects. However, the high cost of developing new equipment can be avoided by using advanced data processing technology with existing equipment. In this study, a three-dimensional (3D) photometric stereo (PS) reconstruction technique is introduced to recover the lost depth information in NPP images. The system uses conventional inspection equipment, equipped with a camera and four light-emitting diodes (LEDs). The 3D data of the object surface are obtained by capturing images under multiple light sources oriented in different directions. The proposed method estimates the light directions and intensities for various image pixels in order to reduce the limitation of light calibration, which results in improved performance. This novel technique is employed to test specimens with various defects under laboratory conditions, revealing promising results. This study provides a new visual inspection method for NPP reactors.

Development of a device for 3D surface reconstruction of thermal shrinkage of flame-resistant fabrics

Thermal shrinkage of flame-retardant fabrics under high temperature exposure would result in the change of air layer under clothing, which can affect the thermal protective performance of clothing. This paper introduces a new image-based device to characterize the thermal shrinkage of the fabrics. The device was aimed to reconstruct the three-dimensional (3D) surface morphology of fabrics based on grayscale images. The imaging collecting system was designed to obtain the grayscale images of thermal-shrinkage fabrics. Then, MATLAB programming of the 3D reconstruction algorithm based on photometric stereo was developed. A comparison of the results of the 3D reconstruction method with the 3D scanning method showed that the 3D reconstruction method was practical and repeatable. The research provides a simple and effective method to characterize the 3D morphology of thermal-shrinkage fabrics, which will help in the quantification of thermal shrinkage of fabrics and the prediction of the thermal protective performance of clothing.