Multi-view Photometric Stereo with Spatially Varying Isotropic Materials

We present a method to capture both 3D shape and spatially varying reflectance with a multi-view photometric stereo (MVPS) technique that works for general isotropic materials. Our algorithm is suitable for perspective cameras and nearby point light sources. Our data capture setup is simple, which consists of only a digital camera, some LED lights, and an optional automatic turntable. From a single viewpoint, we use a set of photometric stereo images to identify surface points with the same distance to the camera. We collect this information from multiple viewpoints and combine it with structure-from-motion to obtain a precise reconstruction of the complete 3D shape. The spatially varying isotropic bidirectional reflectance distribution function (BRDF) is captured by simultaneously inferring a set of basis BRDFs and their mixing weights at each surface point. In experiments, we demonstrate our algorithm with two different setups: a studio setup for highest precision and a desktop setup for best usability. According to our experiments, under the studio setting, the captured shapes are accurate to 0.5 millimeters and the captured reflectance has a relative root-mean-square error (RMSE) of 9%. We also quantitatively evaluate state-of-the-art MVPS on a newly collected benchmark dataset, which is publicly available for inspiring future research.

Multi-view photometric stereo (MVPS) is a preferred method for detailed and precise 3D acquisition of an object from images. Although popular methods for MVPS can provide outstanding results, they are often complex to execute and limited to isotropic material objects. To address such limitations, we present a simple, practical approach to MVPS, which works well for isotropic as well as other object material types such as anisotropic and glossy. The proposed approach in this paper exploits the benefit of uncertainty modeling in a deep neural network for a reliable fusion of photometric stereo (PS) and multi-view stereo (MVS) network predictions. Yet, contrary to the recently proposed state-of-the-art, we introduce neural volume rendering methodology for a trustworthy fusion of MVS and PS measurements. The advantage of introducing neural volume rendering is that it helps in the reliable modeling of objects with diverse material types, where existing MVS methods, PS methods, or both may fail. Furthermore, it allows us to work on neural 3D shape representation, which has recently shown outstanding results for many geometric processing tasks. Our suggested new loss function aims to fit the zero level set of the implicit neural function using the most certain MVS and PS network predictions coupled with weighted neural volume rendering cost. The proposed approach shows state-of-the-art results when tested extensively on several benchmark datasets.

Reflectance properties of real-world opaque materials can be represented by bidirectional reflectance distribution function (BRDF). Non-parametric BRDF becomes the main aspect nowadays because of its verisimilitude and generality. This paper introduces our BRDF capture equipment and a new compact non-parametric model based on curves for isotropic BRDF. 3D isotropic BRDF data are captured and assembled into a 2D matrix, then decomposed to several 1D curves through twice matrix decomposition, each curve has its own feature, such as specular shape curve, diffuse color curve, etc. The data are compressed effectively, and then reconstructed by these 1D curves during real-time rendering. More important, artist can edit the captured reflectance by adjusting one or more meaningful 1D curve intuitively. Experimental results show that, our method can capture the reflectance data, model and edit BRDF effectively.

We present a reliable and accurate global optimization framework for estimating parameters of isotropic analytical bidirectional reflectance distribution function (BRDF) models. This approach is based on a branch and bound strategy with linear programming and interval analysis. Conventional local optimization is often very inefficient for BRDF estimation since its fitting quality is highly dependent on initial guesses due to the nonlinearity of analytical BRDF models. The algorithm presented in this paper employs L1-norm error minimization to estimate BRDF parameters in a globally optimal way and interval arithmetic to derive our feasibility problem and lower bounding function. Our method is developed for the Cook-Torrance model but with several normal distribution functions such as the Beckmann, Berry, and GGX functions. Experiments have been carried out to validate the presented method using 100 isotropic materials from the MERL BRDF database, and our experimental results demonstrate that the L1-norm minimization provides a more accurate and reliable solution than the L2-norm minimization.

We present a generative model for isotropic bidirectional reflectance distribution functions (BRDFs) based on acquired reflectance data. Instead of using analytical reflectance models, we represent each BRDF as a dense set of measurements. This allows us to interpolate and extrapolate in the space of acquired BRDFs to create new BRDFs. We treat each acquired BRDF as a single high-dimensional vector taken from a space of all possible BRDFs. We apply both linear (subspace) and non-linear (manifold) dimensionality reduction tools in an effort to discover a lower-dimensional representation that characterizes our measurements. We let users define perceptually meaningful parametrization directions to navigate in the reduced-dimension BRDF space. On the low-dimensional manifold, movement along these directions produces novel but valid BRDFs.

We present a generative model for isotropic bidirectional reflectance distribution functions (BRDFs) based on acquired reflectance data. Instead of using analytical reflectance models, we represent each BRDF as a dense set of measurements. This allows us to interpolate and extrapolate in the space of acquired BRDFs to create new BRDFs. We treat each acquired BRDF as a single high-dimensional vector taken from a space of all possible BRDFs. We apply both linear (subspace) and non-linear (manifold) dimensionality reduction tools in an effort to discover a lower-dimensional representation that characterizes our measurements. We let users define perceptually meaningful parametrization directions to navigate in the reduced-dimension BRDF space. On the low-dimensional manifold, movement along these directions produces novel but valid BRDFs.

This paper presents a novel approach for efficient sampling of isotropic Bidirectional Reflectance Distribution Functions (BRDFs). Our approach builds upon a new parameterization, the Projected Deviation Vector parameterization, in which isotropic BRDFs can be described by two 1D functions. We show that BRDFs can be efficiently and accurately measured in this space using simple mechanical measurement setups. To demonstrate the utility of our approach, we perform a thorough numerical evaluation and show that the BRDFs reconstructed from measurements along the two 1D bases produce rendering results that are visually comparable to the reference BRDF measurements which are densely sampled over the 4D domain described by the standard hemispherical parameterization.

We introduce a novel parametric bidirectional reflectance distribution function (BRDF) model that can accurately encode a wide variety of real-world isotropic BRDFs with a small number of parameters. The key observation we make is that a BRDF may be viewed as a statistical distribution on a unit hemisphere. We derive a novel directional statistics distribution, which we refer to as the hemispherical exponential power distribution, and model real-world isotropic BRDFs as mixtures of it. We derive a canonical probabilistic method for estimating the parameters, including the number of components, of this novel directional statistics BRDF model. We show that the model captures the full spectrum of real-world isotropic BRDFs with high accuracy, but a small footprint. We also demonstrate the advantages of the novel BRDF model by showing its use for reflection component separation and for exploring the space of isotropic BRDFs.

Highly accurate 3D volumetric reconstruction is still an open research topic where the main difficulty is usually related to merging some rough estimations with high frequency details. One of the most promising methods is the fusion between multi-view stereo and photometric stereo images. Beside the intrinsic difficulties that multi-view stereo and photometric stereo in order to work reliably, supplementary problems arise when considered together. In this work, we present a volumetric approach to the multi-view photometric stereo problem. The key point of our method is the signed distance field parameterisation and its relation to the surface normal. This is exploited in order to obtain a linear partial differential equation which is solved in a variational framework, that combines multiple images from multiple points of view in a single system. In addition, the volumetric approach is naturally implemented on an octree, which allows for fast ray-tracing that reliably alleviates occlusions and cast shadows. Our approach is evaluated on synthetic and real data-sets and achieves state-of-the-art results.

We consider the problem of reconstructing the shape of a surface with an arbitrary, spatially varying isotropic bidirectional reflectance distribution function (BRDF), and introduce a novel, stratified photometric stereo method. By using a particular configuration of lights, it is possible to use symmetry in the image measurements resulting from BRDF isotropy to estimate at each point a plane containing the surface normal. For differentiable surfaces, this allows us to recover the isocontours of the depth map, but not the actual depth associated with each contour. The isocontour structure provides topological information about the surface (critical points, Reeb graph, etc.). By using additional cues in the image data or imposing additional constraints on the surface (e.g., shadows, specular highlights, Helmholtz reciprocity, uniform BRDF), the unknown height of each isocontour can be estimated and the metric structure is resolved. We validate this technique on real and synthetic data by successfully recovering the isocontours of the depth map from images.

The bidirectional reflectance distribution function (BRDF) is critical for rendering, and accurate material representation requires data-driven reflectance models. However, isotropic BRDFs are 3D functions, and measuring the reflectance of a flat sample can require a million incident and outgoing direction pairs, making the use of measured BRDFs impractical. In this paper, we address the problem of reconstructing a measured BRDF from a limited number of samples. We present a novel mapping of the BRDF space, allowing for extraction of descriptive principal components from measured databases, such as the MERL BRDF database. We optimize for the best sampling directions, and explicitly provide the optimal set of incident and outgoing directions in the Rusinkiewicz parameterization for n = {1, 2, 5, 10, 20} samples. Based on the principal components, we describe a method for accurately reconstructing BRDF data from these limited sets of samples. We validate our results on the MERL BRDF database, including favorable comparisons to previous sets of industry-standard sampling directions, as well as with BRDF measurements of new flat material samples acquired with a gantry system. As an extension, we also demonstrate how this method can be used to find optimal sampling directions when imaging a sphere of a homogeneous material; in this case, only two images are often adequate for high accuracy.

We investigate the expressiveness of the microfacet model for isotropic bidirectional reflectance distribution functions (BRDFs) measured from real materials by introducing a non-parametric factor model that represents the model’s functional structure but abandons restricted parametric formulations of its factors. We propose a new objective based on compressive weighting that controls rendering error in high-dynamic-range BRDF fits better than previous factorization approaches. We develop a simple numerical procedure to minimize this objective and handle dependencies that arise between microfacet factors. Our method faithfully captures a more comprehensive set of materials than previous state-of-the-art parametric approaches yet remains compact (3.2KB per BRDF). We experimentally validate the benefit of the microfacet model over a naïve orthogonal factorization and show that fidelity for diffuse materials is modestly improved by fitting an unrestricted shadowing/masking factor. We also compare against a recent data-driven factorization approach [Bilgili et al. 2011] and show that our microfacet-based representation improves rendering accuracy for most materials while reducing storage by more than 10 ×.

We propose a method for accurate 3D shape reconstruction using uncalibrated multiview photometric stereo. A coarse mesh reconstructed using multiview stereo is first parameterized using a planar mesh parameterization technique. Subsequently, multiview photometric stereo is performed in the 2D parameter domain of the mesh, where all geometric and photometric cues from multiple images can be treated uniformly. Unlike traditional methods, there is no need for merging view-dependent surface normal maps. Our key contribution is a new photometric stereo based mesh refinement technique that can efficiently reconstruct meshes with extremely fine geometric details by directly estimating a displacement texture map in the 2D parameter domain. We demonstrate that intricate surface geometry can be reconstructed using several challenging datasets containing surfaces with specular reflections, multiple albedos and complex topologies.

This paper presents a simple and effective solution to the longstanding classical multi-view photometric stereo (MVPS) problem. It is well-known that photometric stereo (PS) is excellent at recovering high-frequency surface details, whereas multi-view stereo (MVS) can help remove the low-frequency distortion due to PS and retain the global geometry of the shape. This paper proposes an approach that can effectively utilize such complementary strengths of PS and MVS. Our key idea is to combine them suitably while considering the per-pixel uncertainty of their estimates. To this end, we estimate per-pixel surface normals and depth using an uncertainty-aware deep-PS network and deep-MVS network, respectively. Uncertainty modeling helps select reliable surface normal and depth estimates at each pixel which then act as a true representative of the dense surface geometry. At each pixel, our approach either selects or discards deep-PS and deep-MVS network prediction depending on the prediction uncertainty measure. For dense, detailed, and precise inference of the object's surface profile, we propose to learn the implicit neural shape representation via a multilayer perceptron (MLP). Our approach encourages the MLP to converge to a natural zero-level set surface using the confident prediction from deep-PS and deep-MVS networks, providing superior dense surface reconstruction. Extensive experiments on the DiLiGenT-MV benchmark dataset show that our method provides high-quality shape recovery with a much lower memory footprint while outperforming almost all of the existing approaches.

Multi-view photometric stereo is well established for the shape recovery of static objects. However, it is difficult to align motion images under varying illumination so as to perform photometric stereo reconstruction for dynamic objects. To tackle this issue, this paper presents an optical flow estimation approach which works under periodically varying illuminations, and in cooperation with photometric stereo, enables high-quality 3D reconstruction of dynamic objects. Firstly, multi-view images of the moving object are captured under periodically varying illumination by the multi-camera multi-light system. Then, the optical flow is estimated to synthesize images under different illuminations for each viewpoint. Finally, the multi-view photometric stereo technique is employed to get a high accurate 3D model for each time instant. Experimental results on motion actors demonstrate that temporal successive images under varying illuminations are effectively registered, permitting accurate photometric reconstruction for moving objects.