Light Transport Matrix Research Articles

Slope Disparity Gating: System and Applications

Active illumination systems which have the ability to selectively image photons that arrive from a specified surface geometry some distance away have several applications in robotics, autonomous vehicles, and surveillance. In this paper, we present a new technique for sloped disparity gating utilizing a synchronized projector-camera system consisting of a raster scanning laser projector and a rolling shutter camera. This system enables disparity-based triangulation of light from specific sloped planar surfaces in the imaging volume. We demonstrate how to control the slope and thickness of these planar surfaces using hardware parameters of pixel clock, synchronization delay, and exposure. Using our system, we present a new decomposition of the light transport matrix in a scene as a function of disparity. We perform applications including real-time image masking and imaging and relighting in scattering media. We also demonstrate applications of the proposed light transport probing technique like novel disparity-dependent relighting, light transport-aware foreground/background separation and green screening, and improved visualization and relighting in scattering media.

Light stage super-resolution

The light stage has been widely used in computer graphics for the past two decades, primarily to enable the relighting of human faces. By capturing the appearance of the human subject under different light sources, one obtains the light transport matrix of that subject, which enables image-based relighting in novel environments. However, due to the finite number of lights in the stage, the light transport matrix only represents a sparse sampling on the entire sphere. As a consequence, relighting the subject with a point light or a directional source that does not coincide exactly with one of the lights in the stage requires interpolation and resampling the images corresponding to nearby lights, and this leads to ghosting shadows, aliased specularities, and other artifacts. To ameliorate these artifacts and produce better results under arbitrary high-frequency lighting, this paper proposes a learning-based solution for the "super-resolution" of scans of human faces taken from a light stage. Given an arbitrary "query" light direction, our method aggregates the captured images corresponding to neighboring lights in the stage, and uses a neural network to synthesize a rendering of the face that appears to be illuminated by a "virtual" light source at the query location. This neural network must circumvent the inherent aliasing and regularity of the light stage data that was used for training, which we accomplish through the use of regularized traditional interpolation methods within our network. Our learned model is able to produce renderings for arbitrary light directions that exhibit realistic shadows and specular highlights, and is able to generalize across a wide variety of subjects. Our super-resolution approach enables more accurate renderings of human subjects under detailed environment maps, or the construction of simpler light stages that contain fewer light sources while still yielding comparable quality renderings as light stages with more densely sampled lights.

Camera-free three-dimensional dual photography.

We report camera-free three-dimensional (3D) dual photography. Inspired by the linkage between fringe projection profilometry (FPP) and dual photography, we propose to implement coordinate mapping to simultaneously sense the direct component of the light transport matrix and the surface profiles of 3D objects. By exploiting Helmholtz reciprocity, dual photography and scene relighting can thus be performed on 3D images. To verify the proposed imaging method, we have developed a single-pixel imaging system based on two digital micromirror devices (DMDs). Binary cyclic S-matrix patterns and binary sinusoidal fringe patterns are loaded on each DMD for scene encoding and virtual fringe projection, respectively. Using this system, we have demonstrated viewing and relighting 3D images at user-selectable perspectives. Our work extends the conceptual scope and the imaging capability of dual photography.

Interferometric transmission probing with coded mutual intensity

We introduce a new interferometric imaging methodology that we term interferometry with coded mutual intensity, which allows selectively imaging photon paths based on attributes such as their length and endpoints. At the core of our methodology is a new technical result that shows that manipulating the spatial coherence properties of the light source used in an interferometric system is equivalent, through a Fourier transform, to implementing light path probing patterns. These patterns can be applied to either the coherent transmission matrix, or the incoherent light transport matrix describing the propagation of light in a scene. We test our theory by building a prototype inspired by the Michelson interferometer, extended to allow for programmable phase and amplitude modulation of the illumination injected in the interferometer. We use our prototype to perform experiments such as visualizing complex fields, capturing direct and global transport components, acquiring light transport matrices, and performing anisotropic descattering, both in steady-state imaging and, by combining our technique with optical coherence tomography, in transient imaging.

Dual Imaging–Can Virtual Be Better Than Real?

In this paper, a dual imaging technique is used for high-precision reconstruction of an observed 3D scene. In contrast to stereo vision, dual imaging systems use a camera and a projector instead of a camera pair. We propose a multiresolution approach based on the sum-to-one transform, coupled with compressive sensing principles, for efficient estimation of the light transport matrix (LTM). The LTM contains information on both optical systems and the 3D scene. In our setup, the camera sensor is intentionally chosen to be low resolution to prove the future use of inexpensive sensors in nonvisible regions of the light spectrum, as well as the potential design of simplified multiview and light field acquisition systems. We show that a high-precision estimation of the LTM from a reduced set of measurements is possible. Virtual measurements, instead of physical, are conducted to obtain the 3D reconstruction. We show that 3D scene reconstruction from the proposed virtual measurements corresponds with the actual physical acquisition. Moreover, this approach provides much more detail in the reconstruction. The computational complexity of the proposed methods is reduced to such a level that practical implementations are feasible.

Computational Imaging on the Electric Grid.

Night beats with alternating current (AC) illumination. By passively sensing this beat, we reveal new scene information which includes: the type of bulbs in the scene, the phases of the electric grid up to city scale, and the light transport matrix. This information yields unmixing of reflections and semi-reflections, nocturnal high dynamic range, and scene rendering with bulbs not observed during acquisition. The latter is facilitated by a dataset of bulb response functions for a range of sources, which we collected and provide. To do all this, we built a novel coded-exposure high-dynamic-range imaging technique, specifically designed to operate on the grid's AC lighting.

Image-based relighting using image segmentation and bootstrap strategy

Image-based relighting technologies enable us to recover the illumination effects of modeled scenes under new light conditions without complicated geometrical information. However, most of them are troubled by specialized devices and tedious sampling work. In this study, we propose an efficient and accurate image-based relighting method for the estimation of the light transport matrix of modeled scene, starting from a small number of images acquired with a fixed viewpoint and with lighting sampled over a uniform 2D grid. Especially, the image space is segmented based on the position and average color value of each pixel using K-means. The local coherence among the pixels can be considered to associate with pixel position and pixels’ albedo. The pixels of each cluster can be trained by several neural networks and the training scene datasets can be chosen using the bootstrap strategy. These tricks improve the regression performance. We validate our method with light transport data of several scenes containing complex lighting effects. The obtained results show that the proposed method is useful for practical applications and we can get more plausible rendered images with fewer input images in comparison to related techniques.

Homogeneous light transport matrix estimation based 3D shape measurement

We introduce homogeneous light transport (HLT) matrix for efficient 3D measurements. Which is an extension of the light transport (LT) matrix. The LT Matrix is a model of light transportation between a projector and a camera. It describes weights for all combinations of the projector pixels and the camera pixels. The HLT Matrix includes background lights information with LT Matrix. By using the HLT Matrix, we can estimate LT Matrix without backgrounds measurement. In this paper, we demonstrate that 3D points can be measured by using '1-pixel projection' and considering the epipolar geometry, for measurement targets which are difficult to measure by traditional methods. This measurement is the same as the measurement using the LT Matrix approach. We use a compressive sensing method for estimation to reduce the number of projection and capture.

Homogeneous light transport matrix estimation based 3D shape measurement

We introduce homogeneous light transport (HLT) matrix for efficient 3D measurements. Which is an extension of the light transport (LT) matrix. The LT Matrix is a model of light transportation between a projector and a camera. It describes weights for all combinations of the projector pixels and the camera pixels. The HLT Matrix includes background lights information with LT Matrix. By using the HLT Matrix, we can estimate LT Matrix without backgrounds measurement. In this paper, we demonstrate that 3D points can be measured by using '1-pixel projection' and considering the epipolar geometry, for measurement targets which are difficult to measure by traditional methods. This measurement is the same as the measurement using the LT Matrix approach. We use a compressive sensing method for estimation to reduce the number of projection and capture.

Handheld Reflectance Acquisition of Paintings

Relightable photographs are alternatives to traditional photographs as they provide a richer viewing experience. However, the complex acquisition systems of existing techniques have restricted its usage to specialized setups. We introduce an easy-to-use and affordable solution for using smartphones to acquire the reflectance of paintings and similar almost-planar objects like tablets, engravings and textile. Our goal is to enable interactive relighting of such artifacts by everyone. In our approach, we nonuniformly sample the reflectance functions by moving the LED light of a smartphone, and simultaneously, tracking the position of the smartphone by using its camera. We then propose a compressive-sensing-based approach for reconstructing the light transport matrix from the nonuniformly sampled data. As shown with experiments, we accurately reconstruct the light transport matrix that can then be used to create relightable photographs.

VITRAIL: Acquisition, Modeling, and Rendering of Stained Glass.

Stained glass windows are designed to reveal their powerful artistry under diverse and time-varying lighting conditions; virtual relighting of stained glass, therefore, represents an exceptional tool for the appreciation of this age old art form. However, as opposed to most other artifacts, stained glass windows are extremely difficult if not impossible to analyze using controlled illumination because of their size and position. In this paper, we present novel methods built upon image based priors to perform virtual relighting of stained glass artwork by acquiring the actual light transport properties of a given artifact. In a preprocessing step, we build a material-dependent dictionary for light transport by studying the scattering properties of glass samples in a laboratory setup. We can now use the dictionary to recover a light transport matrix in two ways: under controlled illuminations the dictionary constitutes a sparsifying basis for a compressive sensing acquisition, while in the case of uncontrolled illuminations the dictionary is used to perform sparse regularization. The proposed basis preserves volume impurities and we show that the retrieved light transport matrix is heterogeneous, as in the case of real world objects. We present the rendering results of several stained glass artifacts, including the Rose Window of the Cathedral of Lausanne, digitized using the presented methods.

Paper] Radiometric Compensation Using Color-Mixing Matrix Reformed from Light Transport Matrix

Paper] Radiometric Compensation Using Color-Mixing Matrix Reformed from Light Transport Matrix

Image based relighting using neural networks

We present a neural network regression method for relighting realworld scenes from a small number of images. The relighting in this work is formulated as the product of the scene's light transport matrix and new lighting vectors, with the light transport matrix reconstructed from the input images. Based on the observation that there should exist non-linear local coherence in the light transport matrix, our method approximates matrix segments using neural networks that model light transport as a non-linear function of light source position and pixel coordinates. Central to this approach is a proposed neural network design which incorporates various elements that facilitate modeling of light transport from a small image set. In contrast to most image based relighting techniques, this regression-based approach allows input images to be captured under arbitrary illumination conditions, including light sources moved freely by hand. We validate our method with light transport data of real scenes containing complex lighting effects, and demonstrate that fewer input images are required in comparison to related techniques.

On the Duality of Forward and Inverse Light Transport

Inverse light transport seeks to undo global illumination effects, such as interreflections, that pervade images of most scenes. This paper presents the theoretical and computational foundations for inverse light transport as a dual of forward rendering. Mathematically, this duality is established through the existence of underlying Neumann series expansions. Physically, it can be shown that each term of our inverse series cancels an interreflection bounce, just as the forward series adds them. While the convergence properties of the forward series are well known, we show that the oscillatory convergence of the inverse series leads to more interesting conditions on material reflectance. Conceptually, the inverse problem requires the inversion of a large light transport matrix, which is impractical for realistic resolutions using standard techniques. A natural consequence of our theoretical framework is a suite of fast computational algorithms for light transport inversion--analogous to finite element radiosity, Monte Carlo and wavelet-based methods in forward rendering--that rely at most on matrix-vector multiplications. We demonstrate two practical applications, namely, separation of individual bounces of the light transport and fast projector radiometric compensation, to display images free of global illumination artifacts in real-world environments.

From the Rendering Equation to Stratified Light Transport Inversion

Recent advances in fast light transport acquisition have motivated new applications for forward and inverse light transport. While forward light transport enables image relighting, inverse light transport provides new possibilities for analyzing and cancelling interreflections, to enable applications like projector radiometric compensation and light bounce separation. With known scene geometry and diffuse reflectance, inverse light transport can be easily derived in closed form. However, with unknown scene geometry and reflectance properties, we must acquire and invert the scene's light transport matrix to undo the effects of global illumination. For many photometric setups such as that of a projector-camera system, the light transport matrix often has a size of 105×105 or larger. Direct matrix inversion is accurate but impractical computationally at these resolutions. In this work, we explore a theoretical analysis of inverse light transport, relating it to its forward counterpart, expressed in the form of the rendering equation. It is well known that forward light transport has a Neumann series that corresponds to adding bounces of light. In this paper, we show the existence of a similar inverse series, that zeroes out the corresponding physical bounces of light. We refer to this series solution as stratified light transport inversion, since truncating to a certain number of terms corresponds to cancelling the corresponding interreflection bounces. The framework of stratified inversion is general and may provide insight for other problems in light transport and beyond, that involve large-size matrix inversion. It is also efficient, requiring only sparse matrix-matrix multiplications. Our practical application is to radiometric compensation, where we seek to project patterns onto real-world surfaces, undoing the effects of global illumination. We use stratified light transport inversion to efficiently invert the acquired light transport matrix for a static scene, after which interreflection cancellation is a simple matrix-vector multiplication to compensate the input image for projection.

Precomputed compressive sensing for light transport acquisition

In this article, we propose an efficient and accurate compressive-sensing-based method for estimating the light transport characteristics of real-world scenes. Although compressive sensing allows the efficient estimation of a high-dimensional signal with a sparse or near-to-sparse representation from a small number of samples, the computational cost of the compressive sensing in estimating the light transport characteristics is relatively high. Moreover, these methods require a relatively smaller number of images than other techniques although they still need 500–1000 images to estimate an accurate light transport matrix. Precomputed compressive sensing improves the performance of the compressive sensing by providing an appropriate initial state. This improvement is achieved in two steps: 1) pseudo-single-pixel projection by multiline projection and 2) regularized orthogonal matching pursuit (ROMP) with initial signal. With these two steps, we can estimate the light transport characteristics more accurately, much faster, and with a lesser number of images.

Combining global and local virtual lights for detailed glossy illumination

Accurately rendering glossy materials in design applications, where previewing and interactivity are important, remains a major challenge. While many fast global illumination solutions have been proposed, all of them work under limiting assumptions on the materials and lighting in the scene. In the presence of many glossy (directionally scattering) materials, fast solutions either fail or degenerate to inefficient, brute-force simulations of the underlying light transport. In particular, many-light algorithms are able to provide fast approximations by clamping elements of the light transport matrix, but they eliminate the part of the transport that contributes to accurate glossy appearance. In this paper we introduce a solution that separately solves for the global (low-rank, dense) and local (highrank, sparse) illumination components. For the low-rank component we introduce visibility clustering and approximation, while for the high-rank component we introduce a local light technique to correct for the missing illumination. Compared to competing techniques we achieve superior gloss rendering in minutes, making our technique suitable for applications such as industrial design and architecture, where material appearance is critical.

Sparsely Precomputing The Light Transport Matrix for Real‐Time Rendering

AbstractPrecomputation‐based methods have enabled real‐time rendering with natural illumination, all‐frequency shadows, and global illumination. However, a major bottleneck is the precomputation time, that can take hours to days. While the final real‐time data structures are typically heavily compressed with clustered principal component analysis and/or wavelets, a full light transport matrix still needs to be precomputed for a synthetic scene, often by exhaustive sampling and raytracing. This is expensive and makes rapid prototyping of new scenes prohibitive. In this paper, we show that the precomputation can be made much more efficient by adaptive and sparse sampling of light transport. We first select a small subset of “dense vertices”, where we sample the angular dimensions more completely (but still adaptively). The remaining “sparse vertices” require only a few angular samples, isolating features of the light transport. They can then be interpolated from nearby dense vertices using locally low rank approximations. We demonstrate sparse sampling and precomputation 5 × faster than previous methods.

Kernel Nyström method for light transport

We propose a kernel Nyström method for reconstructing the light transport matrix from a relatively small number of acquired images. Our work is based on the generalized Nyström method for low rank matrices. We introduce the light transport kernel and incorporate it into the Nyström method to exploit the nonlinear coherence of the light transport matrix. We also develop an adaptive scheme for efficiently capturing the sparsely sampled images from the scene. Our experiments indicate that the kernel Nyström method can achieve good reconstruction of the light transport matrix with a few hundred images and produce high quality relighting results. The kernel Nyström method is effective for modeling scenes with complex lighting effects and occlusions which have been challenging for existing techniques.

All-frequency shadows using non-linear wavelet lighting approximation

We present a method, based on pre-computed light transport, for real-time rendering of objects under all-frequency, time-varying illumination represented as a high-resolution environment map. Current techniques are limited to small area lights, with sharp shadows, or large low-frequency lights, with very soft shadows. Our main contribution is to approximate the environment map in a wavelet basis, keeping only the largest terms (this is known as a non-linear approximation ). We obtain further compression by encoding the light transport matrix sparsely but accurately in the same basis. Rendering is performed by multiplying a sparse light vector by a sparse transport matrix, which is very fast. For accurate rendering, using non-linear wavelets is an order of magnitude faster than using linear spherical harmonics, the current best technique.