Spatially Varying Bidirectional Reflectance Distribution Function Research Articles

Single-image SVBRDF estimation with auto-adaptive high-frequency feature extraction

In this paper, we address the task of estimating spatially-varying bi-directional reflectance distribution functions (SVBRDF) of a near-planar surface from a single flash-lit image. Disentangling SVBRDF from the material appearance by deep learning has proven a formidable challenge. This difficulty is particularly pronounced when dealing with images lit by a point light source because the uneven distribution of irradiance in the scene interacts with the surface, leading to significant global luminance variations across the image. These variations may be overemphasized by the network and wrongly baked into the material property space. To tackle this issue, we propose a high-frequency path that contains an auto-adaptive subband “knob”. This path aims to extract crucial image textures and details while eliminating global luminance variations present in the original image. Furthermore, recognizing that color information is ignored in this path, we design a two-path strategy to jointly estimate material reflectance from both the high-frequency path and the original image. Extensive experiments on a substantial dataset have confirmed the effectiveness of our method. Our method outperforms state-of-the-art methods across a wide range of materials.

Deep SVBRDF Acquisition and Modelling: A Survey

AbstractHand in hand with the rapid development of machine learning, deep learning and generative AI algorithms and architectures, the graphics community has seen a remarkable evolution of novel techniques for material and appearance capture. Typically, these machine‐learning‐driven methods and technologies, in contrast to traditional techniques, rely on only a single or very few input images, while enabling the recovery of detailed, high‐quality measurements of bi‐directional reflectance distribution functions, as well as the corresponding spatially varying material properties, also known as Spatially Varying Bi‐directional Reflectance Distribution Functions (SVBRDFs). Learning‐based approaches for appearance capture will play a key role in the development of new technologies that will exhibit a significant impact on virtually all domains of graphics. Therefore, to facilitate future research, this State‐of‐the‐Art Report (STAR) presents an in‐depth overview of the state‐of‐the‐art in machine‐learning‐driven material capture in general, and focuses on SVBRDF acquisition in particular, due to its importance in accurately modelling complex light interaction properties of real‐world materials. The overview includes a categorization of current methods along with a summary of each technique, an evaluation of their functionalities, their complexity in terms of acquisition requirements, computational aspects and usability constraints. The STAR is concluded by looking forward and summarizing open challenges in research and development toward predictive and general appearance capture in this field. A complete list of the methods and papers reviewed in this survey is available at computergraphics.on.liu.se/star_svbrdf_dl/.

Optical and Electromechanical Design and Implementation of an Advanced Multispectral Device to Capture Material Appearance.

The application of materials with changing visual properties with lighting and observation directions has found broad utility across diverse industries, from architecture and fashion to automotive and film production. The expanding array of applications and appearance reproduction requirements emphasizes the critical role of material appearance measurement and surface characterization. Such measurements offer twofold benefits in soft proofing and product quality control, reducing errors and material waste while providing objective quality assessment. Some image-based setups have been proposed to capture the appearance of material surfaces with spatial variations in visual properties in terms of Spatially Varying Bidirectional Reflectance Distribution Functions (SVBRDF) and Bidirectional Texture Functions (BTF). However, comprehensive exploration of optical design concerning spectral channels and per-pixel incident-reflection direction calculations, along with measurement validation, remains an unexplored domain within these systems. Therefore, we developed a novel advanced multispectral image-based device designed to measure SVBRDF and BTF, addressing these gaps in the existing literature. Central to this device is a novel rotation table as sample holder and passive multispectral imaging. In this paper, we present our compact multispectral image-based appearance measurement device, detailing its design, assembly, and optical considerations. Preliminary measurements showcase the device's potential in capturing angular and spectral data, promising valuable insights into material appearance properties.

Towards Scalable Multi-View Reconstruction of Geometry and Materials.

In this paper, we propose a novel method for joint recovery of camera pose, object geometry and spatially-varying Bidirectional Reflectance Distribution Function (svBRDF) of 3D scenes that exceed object-scale and hence cannot be captured with stationary light stages. The input are high-resolution RGB-D images captured by a mobile, hand-held capture system with point lights for active illumination. Compared to previous works that jointly estimate geometry and materials from a hand-held scanner, we formulate this problem using a single objective function that can be minimized using off-the-shelf gradient-based solvers. To facilitate scalability to large numbers of observation views and optimization variables, we introduce a distributed optimization algorithm that reconstructs 2.5D keyframe-based representations of the scene. A novel multi-view consistency regularizer effectively synchronizes neighboring keyframes such that the local optimization results allow for seamless integration into a globally consistent 3D model. We provide a study on the importance of each component in our formulation and show that our method compares favorably to baselines. We further demonstrate that our method accurately reconstructs various objects and materials and allows for expansion to spatially larger scenes. We believe that this work represents a significant step towards making geometry and material estimation from hand-held scanners scalable.

An attention-embedded GAN for SVBRDF recovery from a single image

Learning-based approaches have made substantial progress in capturing spatially-varying bidirectional reflectance distribution functions (SVBRDFs) from a single image with unknown lighting and geometry. However, most existing networks only consider per-pixel losses which limit their capability to recover local features such as smooth glossy regions. A few generative adversarial networks use multiple discriminators for different parameter maps, increasing network complexity. We present a novel end-to-end generative adversarial network (GAN) to recover appearance from a single picture of a nearly-flat surface lit by flash. We use a single unified adversarial framework for each parameter map. An attention module guides the network to focus on details of the maps. Furthermore, the SVBRDF map loss is combined to prevent paying excess attention to specular highlights. We demonstrate and evaluate our method on both public datasets and real data. Quantitative analysis and visual comparisons indicate that our method achieves better results than the state-of-the-art in most cases.

Metappearance

There currently exist two main approaches to reproducing visual appearance using Machine Learning (ML): The first is training models that generalize over different instances of a problem, e.g., different images of a dataset. As one-shot approaches, these offer fast inference, but often fall short in quality. The second approach does not train models that generalize across tasks, but rather over-fit a single instance of a problem, e.g., a flash image of a material. These methods offer high quality, but take long to train. We suggest to combine both techniques end-to-end using meta-learning: We over-fit onto a single problem instance in an inner loop, while also learning how to do so efficiently in an outer-loop across many exemplars. To this end, we derive the required formalism that allows applying meta-learning to a wide range of visual appearance reproduction problems: textures, Bidirectional Reflectance Distribution Functions (BRDFs), spatially-varying BRDFs (svBRDFs), illumination or the entire light transport of a scene. The effects of meta-learning parameters on several different aspects of visual appearance are analyzed in our framework, and specific guidance for different tasks is provided. Metappearance enables visual quality that is similar to over-fit approaches in only a fraction of their runtime while keeping the adaptivity of general models.

Data Driven SVBRDF Estimation Using Deep Embedded Clustering

Photo-realistic representation in user-specified view and lighting conditions is a challenging but high-demand technology in the digital transformation of cultural heritages. Despite recent advances in neural renderings, it is still necessary to capture high-quality surface reflectance from photography in a controlled environment for real-time applications such as VR/AR and digital arts. In this paper, we present a deep embedding clustering network for spatially-varying bidirectional reflectance distribution function (SVBRDF) estimation. Our network is designed to simultaneously update the reflectance basis and its linear manifold in the spatial domain of SVBRDF. We show that our dual update scheme excels in optimizing the rendering loss in terms of the convergence speed and visual quality compared to the current iterative SVBRDF update methods.

Spatially Varying BRDF Map Filtering

In recent years, the use of spatially varying bidirectional reflectance distribution function (SVBRDF) textures has been steadily increasing in real-time rendering. To reduce aliasing, it is necessary to prefilter SVBRDF textures before rendering. A comprehensive filtering algorithm is proposed that considers the contribution of each component of the SVBRDF texels. First, the weight of each texel is defined as the sum of the luminance of its diffuse albedo and specular albedo. Next, the model of mixture of von Mises-Fisher (vMF) is examined and weighting mechanism is added to it. Finally, by heuristically choosing the initial values for the <italic>k</italic>-means algorithm, an improved method for calculating SVBRDF Mipmap in a bottom-up fashion is proposed. To evaluate the algorithm, 4 SVBRDF materials with different microscopic normal distributions are crafted by hand. When rendering complex SVBRDF materials, the proposed method could achieve an RMSE value 25.5% lower than that of the reference algorithm. When changing the number of fitted lobes, the proposed algorithm can obtain a better result with 2 vMF lobes than previous methods with 4 lobes.

SVBRDF Estimation Using a Normal Sorting Technique

Spatially varying bidirectional reflectance distribution functions (SVBRDFs) play an important role in appearance modeling of real-world surfaces. Automatic capture of these surface properties is highly desirable, but many techniques only partially capture these properties or use complicated setups to do so. Micro surface roughness variations are especially difficult to capture using image-based methods. In this paper, we propose a novel approach towards estimating the complete SVBRDFs of surfaces using a portable projector-camera system made of standard consumer-grade components. Our approach uses insights about the relations between the illumination and viewing geometry and captured image statistics to estimate surface reflectance properties. Our technique should be of great value to practitioners seeking to model and render the geometric and reflectance properties of complex real-world surfaces.

An Inverse Procedural Modeling Pipeline for SVBRDF Maps

Procedural modeling is now the de facto standard of material modeling in industry. Procedural models can be edited and are easily extended, unlike pixel-based representations of captured materials. In this article, we present a semi-automatic pipeline for general material proceduralization. Given Spatially Varying Bidirectional Reflectance Distribution Functions (SVBRDFs) represented as sets of pixel maps, our pipeline decomposes them into a tree of sub-materials whose spatial distributions are encoded by their associated mask maps. This semi-automatic decomposition of material maps progresses hierarchically, driven by our new spectrum-aware material matting and instance-based decomposition methods. Each decomposed sub-material is proceduralized by a novel multi-layer noise model to capture local variations at different scales. Spatial distributions of these sub-materials are modeled either by a by-example inverse synthesis method recovering Point Process Texture Basis Functions (PPTBF) [ 30 ] or via random sampling. To reconstruct procedural material maps, we propose a differentiable rendering-based optimization that recomposes all generated procedures together to maximize the similarity between our procedural models and the input material pixel maps. We evaluate our pipeline on a variety of synthetic and real materials. We demonstrate our method’s capacity to process a wide range of material types, eliminating the need for artist designed material graphs required in previous work [ 38 , 53 ]. As fully procedural models, our results expand to arbitrary resolution and enable high-level user control of appearance.

Adaptive view and illumination planning for SVBRDF measurement of complicated three-dimensional objects

Appearance attributes of a three-dimensional (3-D) object, including reflection and color, are modeled as a spatially varying bidirectional reflectance distribution function (SVBRDF). However, a considerable number of images of the object are required to accurately define an SVBRDF when it exhibits complicated geometric components, thereby leading to a considerable amount of time to acquire, store, and process them. We propose a method to considerably decrease the number of images for SVBRDF modeling while maintaining the quality of the model that is achieved with several images. We introduce an acquisition score and its corresponding objective function by considering the local geometry and reflectance properties to estimate the optimal view and illumination directions for the image acquisition in which an object is measured with a reduced number of images. The captured images are subsequently processed to obtain an SVBRDF of the object. The experimental results demonstrate that the proposed method measures the appearance of a 3-D object with complicated shapes by using a few hundred images and efficiently produces an SVBRDF of the object with complicated shapes. The results indicate that the proposed method photorealistically renders an object with fewer artifacts than existing methods when any arbitrary view and illumination directions are given.

Practical SVBRDF acquisition of 3D objects with unstructured flash photography

Capturing spatially-varying bidirectional reflectance distribution functions (SVBRDFs) of 3D objects with just a single, hand-held camera (such as an off-the-shelf smartphone or a DSLR camera) is a difficult, open problem. Previous works are either limited to planar geometry, or rely on previously scanned 3D geometry, thus limiting their practicality. There are several technical challenges that need to be overcome: First, the built-in flash of a camera is almost colocated with the lens, and at a fixed position; this severely hampers sampling procedures in the light-view space. Moreover, the near-field flash lights the object partially and unevenly. In terms of geometry, existing multiview stereo techniques assume diffuse reflectance only, which leads to overly smoothed 3D reconstructions, as we show in this paper. We present a simple yet powerful framework that removes the need for expensive, dedicated hardware, enabling practical acquisition of SVBRDF information from real-world, 3D objects with a single, off-the-shelf camera with a built-in flash. In addition, by removing the diffuse reflection assumption and leveraging instead such SVBRDF information, our method outputs high-quality 3D geometry reconstructions, including more accurate high-frequency details than state-of-the-art multiview stereo techniques. We formulate the joint reconstruction of SVBRDFs, shading normals, and 3D geometry as a multi-stage, iterative inverse-rendering reconstruction pipeline. Our method is also directly applicable to any existing multiview 3D reconstruction technique. We present results of captured objects with complex geometry and reflectance; we also validate our method numerically against other existing approaches that rely on dedicated hardware, additional sources of information, or both.

Single-image SVBRDF capture with a rendering-aware deep network

Texture, highlights, and shading are some of many visual cues that allow humans to perceive material appearance in single pictures. Yet, recovering spatially-varying bi-directional reflectance distribution functions (SVBRDFs) from a single image based on such cues has challenged researchers in computer graphics for decades. We tackle lightweight appearance capture by training a deep neural network to automatically extract and make sense of these visual cues. Once trained, our network is capable of recovering per-pixel normal, diffuse albedo, specular albedo and specular roughness from a single picture of a flat surface lit by a hand-held flash. We achieve this goal by introducing several innovations on training data acquisition and network design. For training, we leverage a large dataset of artist-created, procedural SVBRDFs which we sample and render under multiple lighting directions. We further amplify the data by material mixing to cover a wide diversity of shading effects, which allows our network to work across many material classes. Motivated by the observation that distant regions of a material sample often offer complementary visual cues, we design a network that combines an encoder-decoder convolutional track for local feature extraction with a fully-connected track for global feature extraction and propagation. Many important material effects are view-dependent, and as such ambiguous when observed in a single image. We tackle this challenge by defining the loss as a differentiable SVBRDF similarity metric that compares the renderings of the predicted maps against renderings of the ground truth from several lighting and viewing directions. Combined together, these novel ingredients bring clear improvement over state of the art methods for single-shot capture of spatially varying BRDFs.

Simulating painted appearance of BTF materials

Bidirectional texture function (BTF) provides a realistic representation for real-world materials; however, the representation does not allow users to append additional rendering effects to the BTF data in a simple and effective way. In this paper, we present an approach for generating the painted appearance on a material that is represented by a BTF. We first convert a BTF into a physical representation composed of a height-field and a spatially varying bidirectional reflectance distribution function (SVBRDF). This representation allows us to simulate the appearance of paints on the surface of a material by computing the reflectance and the height field of the painted material. During rendering, we combine an existing BTF rendering approach and the modified material data to simulate the painted appearance of the BTF materials. Our experiments show that the proposed approach can simulate the painted appearance of a material while preserving the characteristic appearance of the original BTF. Our approach is mainly suitable for less specular BTF materials that allow depth reconstruction.

Non-parametric modeling and editing for captured SVBRDF

Reflectance properties of real-world objects can be represented by spatially varying bidirectional reflectance distribution function (SVBRDF). Non-parametric BRDF becomes the main aspect nowadays because of its verisimilitude and generality. We present a new method for capturing, modeling and editing non-parametric SVBRDF. Our method seeks to achieve high reconstruction accuracy, compactness and editability of representation, and meanwhile to speed up the SVBRDF modeling processes. For a planar surface, we (1) design a capturing device to acquire reflectance samples of the surface; (2) propose a Laplacian-based angular interpolation scheme for a 2D slice of BRDF at a given surface location, and a kernel Nyström method for SVBRDF data matrix reconstruction; (3) propose a practical algorithm to extract linearly independent basis BRDFs, and to calculate blending weights by projecting reconstructed reflectance onto these basis BRDFs; (4) decompose these basis BRDFs into groups of 1D curves for editing intuitively. Our experiment results demonstrate that our approach can model real-world reflectance with both high accuracy and high visual fidelity for real-time virtual environment rendering.

A Statistical Method for SVBRDF Approximation from Video Sequences in General Lighting Conditions

AbstractWe present a statistical method for the estimation of the Spatially Varying Bidirectional Reflectance Distribution Function (SVBRDF) of an object with complex geometry, starting from video sequences acquired with fixed but general lighting conditions. The aim of this work is to define a method that simplifies the acquisition phase of the object surface appearance and allows to reconstruct an approximated SVBRDF. The final output is suitable to be used with a 3D model of the object to obtain accurate and photo‐realistic renderings. The method is composed by three steps: the approximation of the environment map of the acquisition scene, using the same object as a probe; the estimation of the diffuse color of the object; the estimation of the specular components of the main materials of the object, by using a Phong model. All the steps are based on statistical analysis of the color samples projected by the video sequences on the surface of the object. Although the method presents some limitations, the trade‐off between the easiness of acquisition and the obtained results makes it useful for practical applications.

Printing spatially-varying reflectance

Although real-world surfaces can exhibit significant variation in materials --- glossy, diffuse, metallic, etc. --- printers are usually used to reproduce color or gray-scale images. We propose a complete system that uses appropriate inks and foils to print documents with a variety of material properties. Given a set of inks with known Bidirectional Reflectance Distribution Functions (BRDFs), our system automatically finds the optimal linear combinations to approximate the BRDFs of the target documents. Novel gamut-mapping algorithms preserve the relative glossiness between different BRDFs, and halftoning is used to produce patterns to be sent to the printer. We demonstrate the effectiveness of this approach with printed samples of a number of measured spatially-varying BRDFs.

Measurement of bidirectional reflectance distribution function with a linear light source

This paper proposes a practical measurement system for bidirectional reflectance distribution functions (BRDFs) of a three-dimensional (3D) object with a linear light source. Using the linear light source, the proposed system can reduce the number of image acquisitions which are necessary for an estimation of the spatially-varying BRDFs of the object. Furthermore, the size of the proposed system is much smaller than a conventional system which uses a parallel light. In this proposed system, the light field of the linear light source is previously measured to determine direction and radiance of incident rays to each point of the object, because the direction and radiance are not constant at each point. Using the proposed system, the BRDF of a point of flat objects was experimentally measured, and results showed validation of the estimation accuracy of the proposed system. Measurement efficiency of the proposed system was also evaluated by comparing reflectance model parameters estimated by the conventional and proposed systems. For the estimation, the reflectance function of a 3D object was measured by both systems. The estimation accuracy of the proposed method was also evaluated by comparing among a real image and rendered 3D objects of the conventional and proposed methods.

Inverse shade trees for non-parametric material representation and editing

Recent progress in the measurement of surface reflectance has created a demand for non-parametric appearance representations that are accurate, compact, and easy to use for rendering. Another crucial goal, which has so far received little attention, is editability: for practical use, we must be able to change both the directional and spatial behavior of surface reflectance (e.g., making one material shinier, another more anisotropic, and changing the spatial "texture maps" indicating where each material appears). We introduce an Inverse Shade Tree framework that provides a general approach to estimating the "leaves" of a user-specified shade tree from high-dimensional measured datasets of appearance. These leaves are sampled 1- and 2-dimensional functions that capture both the directional behavior of individual materials and their spatial mixing patterns. In order to compute these shade trees automatically, we map the problem to matrix factorization and introduce a flexible new algorithm that allows for constraints such as non-negativity, sparsity, and energy conservation. Although we cannot infer every type of shade tree, we demonstrate the ability to reduce multi-gigabyte measured datasets of the Spatially-Varying Bidirectional Reflectance Distribution Function (SVBRDF) into a compact representation that may be edited in real time.