Multi-view Photometric Stereo Research Articles

Multi-View Photometric Stereo: A Robust Solution and Benchmark Dataset for Spatially Varying Isotropic Materials.

We present a method to capture both 3D shape and spatially varying reflectance with a multi-view photometric stereo (MVPS) technique that works for general isotropic materials. Our algorithm is suitable for perspective cameras and nearby point light sources. Our data capture setup is simple, which consists of only a digital camera, some LED lights, and an optional automatic turntable. From a single viewpoint, we use a set of photometric stereo images to identify surface points with the same distance to the camera. We collect this information from multiple viewpoints and combine it with structure-from-motion to obtain a precise reconstruction of the complete 3D shape. The spatially varying isotropic bidirectional reflectance distribution function (BRDF) is captured by simultaneously inferring a set of basis BRDFs and their mixing weights at each surface point. In experiments, we demonstrate our algorithm with two different setups: a studio setup for highest precision and a desktop setup for best usability. According to our experiments, under the studio setting, the captured shapes are accurate to 0.5 millimeters and the captured reflectance has a relative root-mean-square error (RMSE) of 9%. We also quantitatively evaluate state-of-the-art MVPS on a newly collected benchmark dataset, which is publicly available for inspiring future research.

Three-dimensional reconstruction based on multi-view photometric stereo fusion technology in movies special-effect

3D movies have become very popular in recent years. But there are vertical disparities between left and right views of 3D frames due to the lack of accuracy of the mechanical alignment during the shooting. In order to improve accuracy of reconstruction, a three-dimensional reconstruction technology based on multi-view photometric stereo fusion algorithm in movies special-effect production is present in this paper. The original normal is firstly replaced with the surface normal in the average normal, and the reconstructed normal is optimized so as to reduce the deviation of the original surface normal. And then, a reference-plane-based approach is applied to estimate the principle optical axis of each light source as well as its principle radiant energy. For each surface point on the target, the direction and intensity of its incident light ray can be precisely determined by the calibration parameters and the quasi-point light model. Finally, 3D reconstruction of the surface with a quasi-point light source is also implemented in two steps. By estimating the mean value of depth in the iterative process, the surface depth is projected into the physical coordinates. Qualitative and quantitative experimental results show that higher accuracy surface normal as well as better 3D reconstruction quality can be obtained by the proposed approach in comparison with conventional reconstruction methods.

High-fidelity facial reflectance and geometry inference from an unconstrained image

We present a deep learning-based technique to infer high-quality facial reflectance and geometry given a single unconstrained image of the subject, which may contain partial occlusions and arbitrary illumination conditions. The reconstructed high-resolution textures, which are generated in only a few seconds, include high-resolution skin surface reflectance maps, representing both the diffuse and specular albedo, and medium- and high-frequency displacement maps, thereby allowing us to render compelling digital avatars under novel lighting conditions. To extract this data, we train our deep neural networks with a high-quality skin reflectance and geometry database created with a state-of-the-art multi-view photometric stereo system using polarized gradient illumination. Given the raw facial texture map extracted from the input image, our neural networks synthesize complete reflectance and displacement maps, as well as complete missing regions caused by occlusions. The completed textures exhibit consistent quality throughout the face due to our network architecture, which propagates texture features from the visible region, resulting in high-fidelity details that are consistent with those seen in visible regions. We describe how this highly underconstrained problem is made tractable by dividing the full inference into smaller tasks, which are addressed by dedicated neural networks. We demonstrate the effectiveness of our network design with robust texture completion from images of faces that are largely occluded. With the inferred reflectance and geometry data, we demonstrate the rendering of high-fidelity 3D avatars from a variety of subjects captured under different lighting conditions. In addition, we perform evaluations demonstrating that our method can infer plausible facial reflectance and geometric details comparable to those obtained from high-end capture devices, and outperform alternative approaches that require only a single unconstrained input image.

Multi-view photometric stereo using surface deformation

This paper presents a hybrid approach for 3D reconstruction by fusing photometric stereo and multi-view stereo. The 3D surface is obtained by capturing a set of images taken from different viewpoints under time-varying illuminations. Key factors in the reconstruction process are surface normals that are obtained from photometric stereo. The surface is initialized by integrating the normals and then refined by performing iterative deformations on the initial surface and thereby optimizing image and normal consistency in multiple views. Benefiting from the employment of the deformation approach, we are able to perform image and normal consistency optimization without using matching windows. Instead, always the complete surface is back-projected. This makes the proposed approach much simpler and more robust compared to window-based approaches, which typically require global optimization with constraints on neighboring windows. Experiments on real-world data and ground-truth data show that for diffuse midsized objects without large depth discontinuities our approach improves the accuracy of the reconstructions compared to exiting approaches.

Robust Multiview Photometric Stereo Using Planar Mesh Parameterization.

We propose a robust uncalibrated multiview photometric stereo method for high quality 3D shape reconstruction. In our method, a coarse initial 3D mesh obtained using a multiview stereo method is projected onto a 2D planar domain using a planar mesh parameterization technique. We describe methods for surface normal estimation that work in the parameterized 2D space that jointly incorporates all geometric and photometric cues from multiple viewpoints. Using an estimated surface normal map, a refined 3D mesh is then recovered by computing an optimal displacement map in the same 2D planar domain. Our method avoids the need of merging view-dependent surface normal maps that is often required in conventional methods. We conduct evaluation on various real-world objects containing surfaces with specular reflections, multiple albedos, and complex topologies in both controlled and uncontrolled settings and demonstrate that accurate 3D meshes with fine geometric details can be recovered by our method.

R(2)OBBIE-3D, a Fast Robotic High-Resolution System for Quantitative Phenotyping of Surface Geometry and Colour-Texture.

While recent imaging techniques provide insights into biological processes from the molecular to the cellular scale, phenotypes at larger scales remain poorly amenable to quantitative analyses. For example, investigations of the biophysical mechanisms generating skin morphological complexity and diversity would greatly benefit from 3D geometry and colour-texture reconstructions. Here, we report on R2OBBIE-3D, an integrated system that combines a robotic arm, a high-resolution digital colour camera, an illumination basket of high-intensity light-emitting diodes and state-of-the-art 3D-reconstruction approaches. We demonstrate that R2OBBIE generates accurate 3D models of biological objects between 1 and 100 cm, makes multiview photometric stereo scanning possible in practical processing times, and enables the capture of colour-texture and geometric resolutions better than 15 μm without the use of magnifying lenses. R2OBBIE has the potential to greatly improve quantitative analyses of phenotypes in addition to providing multiple new applications in, e.g., biomedical science.

センサフュージョンによる効率的な３次元モデルの推定と表現

We propose a novel method for 3d modeling using a fusion of a laser range sensor, a camera, and a flashlight. This combination provides dense normals and surface colors that can be mapped on a 3d model, whereas conventional sensors only output point clouds of the 3d geometry. Furthermore, the fusion enables formulations to be made simply and practically. Multi-view photometric stereo is used for estimating the fine normal distribution with a basic shape measured by the laser range sensor. Our photometric stereo can easily handle near-light formulation and specularity. Detailed surfaces can be shown by applying the normal map as bump mapping to the basic shape. Robust estimation and clustering are used for estimating reflection parameters. Results demonstrate that our method can estimate highly accurate reflection parameters and provide fine surface appearances using only a small amount of data. The effectiveness of our method is shown with an application of 3d content.

Dynamic shape capture using multi-view photometric stereo

We describe a system for high-resolution capture of moving 3D geometry, beginning with dynamic normal maps from multiple views. The normal maps are captured using active shape-from-shading (photometric stereo), with a large lighting dome providing a series of novel spherical lighting configurations. To compensate for low-frequency deformation, we perform multi-view matching and thin-plate spline deformation on the initial surfaces obtained by integrating the normal maps. Next, the corrected meshes are merged into a single mesh using a volumetric method. The final output is a set of meshes, which were impossible to produce with previous methods. The meshes exhibit details on the order of a few millimeters, and represent the performance over human-size working volumes at a temporal resolution of 60Hz.

Joint Estimation of Shape and Reflectance using Multiple Images with Known Illumination Conditions

We propose a generative model based method for recovering both the shape and the reflectance of the surface(s) of a scene from multiple images, assuming that illumination conditions and cameras calibration are known in advance. Based on a variational framework and via gradient descents, the algorithm minimizes simultaneously and consistently a global cost functional with respect to both shape and reflectance. The motivations for our approach are threefold. (1) Contrary to previous works which mainly consider specific individual scenarios, our method applies indiscriminately to a number of classical scenarios; in particular it works for classical stereovision, multiview photometric stereo and multiview shape from shading. It works with changing as well as static illumination. (2) Our approach naturally combines stereo, silhouette and shading cues in a single framework. (3) Moreover, unlike most previous methods dealing with only Lambertian surfaces, the proposed method considers general dichromatic surfaces. We verify the method using various synthetic and real data sets.

Multiview Photometric Stereo

This paper addresses the problem of obtaining complete, detailed reconstructions of textureless shiny objects. We present an algorithm which uses silhouettes of the object, as well as images obtained under changing illumination conditions. In contrast with previous photometric stereo techniques, ours is not limited to a single viewpoint but produces accurate reconstructions in full 3D. A number of images of the object are obtained from multiple viewpoints, under varying lighting conditions. Starting from the silhouettes, the algorithm recovers camera motion and constructs the object's visual hull. This is then used to recover the illumination and initialise a multi-view photometric stereo scheme to obtain a closed surface reconstruction. There are two main contributions in this paper: Firstly we describe a robust technique to estimate light directions and intensities and secondly, we introduce a novel formulation of photometric stereo which combines multiple viewpoints and hence allows closed surface reconstructions. The algorithm has been implemented as a practical model acquisition system. Here, a quantitative evaluation of the algorithm on synthetic data is presented together with complete reconstructions of challenging real objects. Finally, we show experimentally how even in the case of highly textured objects, this technique can greatly improve on correspondence-based multi-view stereo results.