Relighting Of Objects Research Articles

SGformer: Boosting transformers for indoor lighting estimation from a single image

Estimating lighting from standard images can effectively circumvent the need for resource-intensive high-dynamic-range (HDR) lighting acquisition. However, this task is often ill-posed and challenging, particularly for indoor scenes, due to the intricacy and ambiguity inherent in various indoor illumination sources. We propose an innovative transformer-based method called SGformer for lighting estimation through modeling spherical Gaussian (SG) distributions—a compact yet expressive lighting representation. Diverging from previous approaches, we explore underlying local and global dependencies in lighting features, which are crucial for reliable lighting estimation. Additionally, we investigate the structural relationships spanning various resolutions of SG distributions, ranging from sparse to dense, aiming to enhance structural consistency and curtail potential stochastic noise stemming from independent SG component regressions. By harnessing the synergy of local-global lighting representation learning and incorporating consistency constraints from various SG resolutions, the proposed method yields more accurate lighting estimates, allowing for more realistic lighting effects in object relighting and composition. Our code and model implementing our work can be found at https://github.com/junhong-jennifer-zhao/SGformer.

Complex object relighting via split-then-composition by semantics and materials

Complex object illumination transfer is a special challenge in computer vision. In our paper, we put forward a method for complex object illumination transfer. Firstly, the input object image was divided into object components by semantic analysis, to find the reference object images consistent with the object component material in the physical world by material analysis. Material has a great influence on the illumination transfer of object image, so the use of material analysis can greatly reduce the influence of material on the illumination transfer in the later stage. Next, a block matching algorithm was used to deform each reference object image and made it match with each component shape of the input object image. Then, each component of the input object image and each warped reference object image were illuminated by local and global transfer model. Finally, semantic analysis was used to synthesize the re-illumination components of the input object image to obtain the re-illumination input object image. The experimental results prove that the method could make a good effect on the illumination transfer. Our main contribution is the use of semantic and material analysis to split complex objects into simple objects, and skillfully combine semantic and material parsing and composition, block matching algorithm, local and global light migration model to achieve the relighting of complex objects.

Illumination consistency based on single low dynamic range images

In this paper, a light consistency solution for generating high dynamic range (HDR) images based on a single low dynamic range image(LDR) is proposed, and the virtual object is rendered by illumination. The solution can reduce the time of image acquisition and processing, and solve the problems caused by the limitations of image acquisition equipment. The solution is divided into three stages: image preprocessing, high dynamic range image generation and virtual object relighting. Firstly, in the stage of image pretreatment, the wavelet noise reduction method based on a Gaussian mixture model is used to remove image noise and avoid image detail distortion. The inverse camera response function is utilized to linearize the image, the pixel brightness range is expanded based on the inverse tone mapping function, and the threshold segmentation method is combined with flooding Gaussian smoothing to calculate the highlight spread diagram to compensate for scene highlights lost during camera shooting. Then, the extended dynamic range image is interpolated linearly by using the specular expansion image to get the high dynamic range image. Based on the analysis and experimental simulation, compared with other methods, it is found that using a single low-dynamic-range image can greatly reduce the time of image acquisition and processing and reduce the limitations of image acquisition equipment, while maintaining good light fusion. Based on the simulation results, the efficiency of light consistency processing is improved.

An Approximate Shading Model with Detail Decomposition for Object Relighting

We present an object relighting system that allows an artist to select an object from an image and insert it into a target scene. Through simple interactions, the system can adjust illumination on the inserted object so that it appears naturally in the scene. To support image-based relighting, we build object model from the image, and propose a \emph{perceptually-inspired} approximate shading model for the relighting. It decomposes the shading field into (a) a rough shape term that can be reshaded, (b) a parametric shading detail that encodes missing features from the first term, and (c) a geometric detail term that captures fine-scale material properties. With this decomposition, the shading model combines 3D rendering and image-based composition and allows more flexible compositing than image-based methods. Quantitative evaluation and a set of user studies suggest our method is a promising alternative to existing methods of object insertion.

Image-based 3D photography using opacity hulls

We have built a system for acquiring and displaying high quality graphical models of objects that are impossible to scan with traditional scanners. Our system can acquire highly specular and fuzzy materials, such as fur and feathers. The hardware set-up consists of a turntable, two plasma displays, an array of cameras, and a rotating array of directional lights. We use multi-background matting techniques to acquire alpha mattes of the object from multiple viewpoints. The alpha mattes are used to construct an opacity hull. The opacity hull is a new shape representation, defined as the visual hull of the object with view-dependent opacity. It enables visualization of complex object silhouettes and seamless blending of objects into new environments. Our system also supports relighting of objects with arbitrary appearance using surface reflectance fields, a purely image-based appearance representation. Our system is the first to acquire and render surface reflectance fields under varying illumination from arbitrary viewpoints. We have built three generations of digitizers with increasing sophistication. In this paper, we present our results from digitizing hundreds of models.