Arbitrary Lighting Conditions Research Articles

Color calibration of moist soil images captured under irregular lighting conditions

Soil color is a valuable indicator that visually reflects soil properties. In the field of soil science, soil color has been widely used for soil identification and property assessment. Soil color is influenced not only by soil properties but also by lighting conditions; therefore, adequately considering the effect of lighting conditions for the color-based soil assessment is essential. This paper proposes a color calibration method for moist soil images captured under irregular lighting conditions. Four samples of weathered granite soil with varying water contents were captured under nine different lighting conditions. The effect of lighting conditions on the color of moist soil was analyzed using linear regression analysis in the CIELAB color space. The findings demonstrate a linear relationship between soil color and lighting conditions. Furthermore, the slopes of the linear regression equations remained consistent across all moist soil samples, irrespective of soil type and water content. Based on this, calibration equations were derived to adjust the color of moist soil images captured under arbitrary lighting conditions to those under the desired lighting conditions. The proposed method was validated using soil images captured under natural lighting conditions, which show high accuracy in color calibration except for cases with illuminance exceeding 70,000 lux. This color calibration method provides a reliable approach to evaluating soil properties through accurate color representation and presents new possibilities for developing digital image-based soil property prediction techniques that are less sensitive to lighting conditions.

Detachable image decomposition and illumination mapping search for low-light image enhancement

Low-light image enhancement is a fundamental low-level computer vision task that has a wide range of applications, such as night monitoring, all-weather automatic driving, back-light imaging, and so on. In this paper, we propose a novel four-stage detachable framework for low-light image enhancement, where each module can be trained and fine-tuned separately. More importantly, the proposed framework can effectively be adapted to unseen datasets. In the first stage, we propose an image decomposition network that can extract a pair of a reflectance map and an illumination map given an image with arbitrary lighting conditions. The proposed loss function ensures that images of the same scene with different lighting conditions share the same reflectance map. Instead of using gamma correction, we establish an automatic searching scheme in the second stage to find an explicit mapping relationship between illumination maps in different lighting conditions. In the third stage, we use an efficient and effective unsupervised training method to find the best parameter set for the mapping. Finally, a normal-light image is obtained by composing together its reflectance map and transformed illumination map. A series of experiments are conducted on several popular datasets, and the results demonstrate the superiority of our proposed method on unseen datasets. Our framework surpasses other state-of-the-art methods and is applicable to enhance images in a wide range of lighting conditions.

NeRFactor

We address the problem of recovering the shape and spatially-varying reflectance of an object from multi-view images (and their camera poses) of an object illuminated by one unknown lighting condition. This enables the rendering of novel views of the object under arbitrary environment lighting and editing of the object's material properties. The key to our approach, which we call Neural Radiance Factorization (NeRFactor), is to distill the volumetric geometry of a Neural Radiance Field (NeRF) [Mildenhall et al. 2020] representation of the object into a surface representation and then jointly refine the geometry while solving for the spatially-varying reflectance and environment lighting. Specifically, NeRFactor recovers 3D neural fields of surface normals, light visibility, albedo, and Bidirectional Reflectance Distribution Functions (BRDFs) without any supervision, using only a re-rendering loss, simple smoothness priors, and a data-driven BRDF prior learned from real-world BRDF measurements. By explicitly modeling light visibility, NeRFactor is able to separate shadows from albedo and synthesize realistic soft or hard shadows under arbitrary lighting conditions. NeRFactor is able to recover convincing 3D models for free-viewpoint relighting in this challenging and underconstrained capture setup for both synthetic and real scenes. Qualitative and quantitative experiments show that NeRFactor outperforms classic and deep learning-based state of the art across various tasks. Our videos, code, and data are available at people.csail.mit.edu/xiuming/projects/nerfactor/.

Deep Photometric Stereo Networks for Determining Surface Normal and Reflectances

This article presents a photometric stereo method based on deep learning. One of the major difficulties in photometric stereo is designing an appropriate reflectance model that is both capable of representing real-world reflectances and computationally tractable for deriving surface normal. Unlike previous photometric stereo methods that rely on a simplified parametric image formation model, such as the Lambert's model, the proposed method aims at establishing a flexible mapping between complex reflectance observations and surface normal using a deep neural network. In addition, the proposed method predicts the reflectance, which allows us to understand surface materials and to render the scene under arbitrary lighting conditions. As a result, we propose a deep photometric stereo network (DPSN) that takes reflectance observations under varying light directions and infers the surface normal and reflectance in a per-pixel manner. To make the DPSN applicable to real-world scenes, a dataset of measured BRDFs (MERL BRDF dataset) has been used for training the network. Evaluation using simulation and real-world scenes shows the effectiveness of the proposed approach in estimating both surface normal and reflectances.

Objective assessment in digital images of skin erythema caused by radiotherapy.

Skin toxicity caused by radiotherapy has been visually classified into discrete grades. The present study proposes an objective and continuous assessment method of skin erythema in digital images taken under arbitrary lighting conditions, which is the case for most clinical environments. The purpose of this paper is to show the feasibility of the proposed method. Clinical data were gathered from six patients who received carbon beam therapy for lung cancer. Skin condition was recorded using an ordinary compact digital camera under unfixed lighting conditions; a laser Doppler flowmeter was used to measure blood flow in the skin. The photos and measurements were taken at 3 h, 30, and 90 days after irradiation. Images were decomposed into hemoglobin and melanin colors using independent component analysis. Pixel values in hemoglobin color images were compared with skin dose and skin blood flow. The uncertainty of the practical photographic method was also studied in nonclinical experiments. The clinical data showed good linearity between skin dose, skin blood flow, and pixel value in the hemoglobin color images; their correlation coefficients were larger than 0.7. It was deduced from the nonclinical that the uncertainty due to the proposed method with photography was 15%; such an uncertainty was not critical for assessment of skin erythema in practical use. Feasibility of the proposed method for assessment of skin erythema using digital images was demonstrated. The numerical relationship obtained helped to predict skin erythema by artificial processing of skin images. Although the proposed method using photographs taken under unfixed lighting conditions increased the uncertainty of skin information in the images, it was shown to be powerful for the assessment of skin conditions because of its flexibility and adaptability.

Faces as light probes for relighting

A light probe is commonly used for measuring the illumination of a real scene. Instead of equipping a man-made light probe such as a mirror ball, we propose to use a face in images as a natural light probe. To that end, we construct a statistical reflectance model for faces and use this model to extract the lighting and the reflectance field of an input face. With an iterative procedure, we can obtain the lighting condition from an unknown face image. As a byproduct of this procedure, we also estimate the reflectance field of the same face. By identifying the lighting condition of scene, we can provide an effective solution for various practical appli- cations. First, we can insert a virtual object seamlessly into a real scene by illuminating the virtual object under the lighting present in the real scene. Second, we can relight a face under an arbitrary lighting condition using the estimated reflectance fields. Third, we can swap two unknown faces using the estimates of both lighting and reflectance fields. Based on var- ious experiments, we show that the proposed algorithm is an effective tool for many practical applications: inserting a virtual object into a real scene, face relighting, and face swapping. © 2012 Society of Photo-Optical Instrumentation

Novel‐View Synthesis of Outdoor Sport Events Using an Adaptive View‐Dependent Geometry

AbstractWe propose a novel fully automatic method for novel‐viewpoint synthesis. Our method robustly handles multi‐camera setups featuring wide‐baselines in an uncontrolled environment. In a first step, robust and sparse point correspondences are found based on an extension of the Daisy features [TLF10]. These correspondences together with back‐projection errors are used to drive a novel adaptive coarse to fine reconstruction method, allowing to approximate detailed geometry while avoiding an extreme triangle count.To render the scene from arbitrary viewpoints we use a view‐dependent blending of color information in combination with a view‐dependent geometry morph. The view‐dependent geometry compensates for misalignments caused by calibration errors. We demonstrate that our method works well under arbitrary lighting conditions with as little as two cameras featuring wide‐baselines. The footage taken from real sports broadcast events contains fine geometric structures, which result in nice novel‐viewpoint renderings despite of the low resolution in the images.

A Simple Framework for Face Photo‐Sketch Synthesis

This paper proposes a simple framework for face photo‐sketch synthesis. We first describe the shadow details on faces and extract the prominent facial feature by two‐scale decomposition using bilateral filtering. Then, we enhance the hair and some unapparent facial feature regions by combining the edge map and hair color similarity map. Finally, we obtain the face photo sketch by adding the results of the two processes. Compared with current methods, the proposed framework demands non feature localization, training or iteration process, creating vivid hair in sketch synthesis, and process arbitrary lighting conditions of input images, especially for complex self‐shadows. And more importantly, it can be easily expanded to natural scene. The effectiveness of the presented framework is evaluated on a variety of databases.

Photometric approach to surface reconstruction of artist paintings

We propose a method for surface reconstruction of artist paintings. In order to reproduce the appearance of a painting, including color, surface texture, and glossiness, it is essential to acquire the pixel-wise light reflection property and orientation of the surface and render an image under an arbitrary lighting condition. A photometric approach is used to estimate bidirectional reflectance distribution functions (BRDFs) and surface normals from a set of images photographed by a fixed camera with sparsely distributed point light sources. A robust and computationally less expensive nonlinear optimization algorithm is proposed that optimizes the small number of parameters to simultaneously determine all of the specular BRDF, diffuse albedo, and surface normal. The proposed method can be applied to moderately glossy surfaces without separating captured images into diffuse and specular reflections beforehand. Experiments were conducted using oil paintings with different surface glossiness. The effectiveness of the proposed method is validated by comparing captured and rendered images.

Face Relighting from a Single Image under Arbitrary Unknown Lighting Conditions

In this paper, we present a new method to modify the appearance of a face image by manipulating the illumination condition, when the face geometry and albedo information is unknown. This problem is particularly difficult when there is only a single image of the subject available. Recent research demonstrates that the set of images of a convex Lambertian object obtained under a wide variety of lighting conditions can be approximated accurately by a low-dimensional linear subspace using a spherical harmonic representation. Moreover, morphable models are statistical ensembles of facial properties such as shape and texture. In this paper, we integrate spherical harmonics into the morphable model framework by proposing a 3D spherical harmonic basis morphable model (SHBMM). The proposed method can represent a face under arbitrary unknown lighting and pose simply by three low-dimensional vectors, i.e., shape parameters, spherical harmonic basis parameters, and illumination coefficients, which are called the SHBMM parameters. However, when the image was taken under an extreme lighting condition, the approximation error can be large, thus making it difficult to recover albedo information. In order to address this problem, we propose a subregion-based framework that uses a Markov random field to model the statistical distribution and spatial coherence of face texture, which makes our approach not only robust to extreme lighting conditions, but also insensitive to partial occlusions. The performance of our framework is demonstrated through various experimental results, including the improved rates for face recognition under extreme lighting conditions.

Automatic illumination and color compensation using mean shift and sigma filter

We present a novel framework for automatic illumination and color compensation algorithm using mean shift and the sigma filter (ICCMS) to restore distorted images taken under the arbitrary lighting conditions. The proposed method is effective for appropriate illumination compensation, vivid color restoration, artifacts suppression, automatic parameter estimation, and low computational cost for HW implementation. We show the efficiency of the mean shift filter and sigma filter for illumination compensation with small sized kernel while considering the processing time and removing the artifacts such as HALO and noise amplification. The proposed color restoration function can restore the natural color and correct color noise artifact more perceptually compared with conventional methods. For the automatic processing, the image statistics analysis estimates suitable parameter and all constants are pre-defined. We also introduce the ROI-based parameter estimation dealing with small shadow area against spacious well-exposed background in an image for the touch-screen camera. The object evaluation is performed by CMC, CIEde2000, PSNR, SSIM, and 3D CIELAB gamut with state-of-the-art research and existing commercial solutions.

NONITERATIVE 3D FACE RECONSTRUCTION BASED ON PHOTOMETRIC STEREO

3D face reconstruction is a popular area within the computer vision domain. 3D face reconstruction should ideally be achieved easily and cost-effectively, without requiring specialized equipment to estimate 3D shapes. As a result of this, many techniques for retrieving 3D shapes from 2D images have been proposed. In this paper, a novel method for 3D face reconstruction based on photometric stereo, which estimates the surface normal from shading information in multiple images, hence recovering the 3D shape of a face, is proposed. In order to overcome the problems of previous approaches related to prior-knowledge regarding lighting conditions and iterative algorithms, the exemplar is synthesized with known lighting conditions from at least three images, under arbitrary lighting conditions and using an illumination reference. Experiments in 3D face reconstruction were made by verifying the proposed approach using the illumination subset of the Max-Planck Institute face database and Yale face database B. Experimental results demonstrate that the proposed method is effective for 3D shape reconstruction of faces from 2D images.

Pose-Encoded Spherical Harmonics for Face Recognition and Synthesis Using a Single Image

Face recognition under varying pose is a challenging problem, especially when illumination variations are also present. In this paper, we propose to address one of the most challenging scenarios in face recognition. That is, to identify a subject from a test image that is acquired under dierent pose and illumination condition from only one training sample (also known as a gallery image) of this subject in the database. For example, the test image could be semifrontal and illuminated by multiple lighting sources while the corresponding training image is frontal under a single lighting source. Under the assumption of Lambertian reflectance, the spherical harmonics representation has proved to be effective in modeling illumination variations for a fixed pose. In this paper, we extend the spherical harmonics representation to encode pose information. More specifically, we utilize the fact that 2D harmonic basis images at different poses are related by close-form linear transformations, and give a more convenient transformation matrix to be directly used for basis images. An immediate application is that we can easily synthesize a different view of a subject under arbitrary lighting conditions by changing the coefficients of the spherical harmonics representation. A more important result is an efficient face recognition method, based on the orthonormality of the linear transformations, for solving the above-mentioned challenging scenario. Thus, we directly project a nonfrontal view test image onto the space of frontal view harmonic basis images. The impact of some empirical factors due to the projection is embedded in a sparse warping matrix; for most cases, we show that the recognition performance does not deteriorate after warping the test image to the frontal view. Very good recognition results are obtained using this method for both synthetic and challenging real images.

Seeing People in Different Light-Joint Shape, Motion, and Reflectance Capture

By means of passive optical motion capture, real people can be authentically animated and photo-realistically textured. To import real-world characters into virtual environments, however, surface reflectance properties must also be known. We describe a video-based modeling approach that captures human shape and motion as well as reflectance characteristics from a handful of synchronized video recordings. The presented method is able to recover spatially varying surface reflectance properties of clothes from multiview video footage. The resulting model description enables us to realistically reproduce the appearance of animated virtual actors under different lighting conditions, as well as to interchange surface attributes among different people, e.g., for virtual dressing. Our contribution can be used to create 3D renditions of real-world people under arbitrary novel lighting conditions on standard graphics hardware.

A Low-dimensional Illumination Space Representation of Human Faces for Arbitrary Lighting Conditions

The proposed method for low-dimensional illumination space representation (LDISR) of human faces can not only synthesize a virtual face image when given lighting conditions but also estimate lighting conditions when given a face image. The LDISR is based on the observation that 9 basis point light sources can represent almost arbitrary lighting conditions for face recognition application and different human faces have a similar LDISR. The principal component analysis (PGA) and the nearest neighbor clustering method are adopted to obtain the 9 basis point light sources. The 9 basis images under the 9 basis point light sources are then used to construct an LDISR which can represent almost all face images under arbitrary lighting conditions. Illumination ratio image (IRI) is employed to generate virtual face images under different illuminations. The LDISR obtained from face images of one person can be used for other people. Experimental results on image reconstruction and face recognition indicate the efficiency of LDISR.

Face recognition under arbitrary illumination using illuminated exemplars

Recently, the importance of face recognition has been increasingly emphasized since popular CCD cameras are distributed to various applications. However, facial images are dramatically changed by lighting variations, so that facial appearance changes caused serious performance degradation in face recognition. Many researchers have tried to overcome these illumination problems using diverse approaches, which have required a multiple registered images per person or the prior knowledge of lighting conditions. In this paper, we propose a new method for face recognition under arbitrary lighting conditions, given only a single registered image and training data under unknown illuminations. Our proposed method is based on the illuminated exemplars which are synthesized from photometric stereo images of training data. The linear combination of illuminated exemplars can represent the new face and the weighted coefficients of those illuminated exemplars are used as identity signature. We make experiments for verifying our approach and compare it with two traditional approaches. As a result, higher recognition rates are reported in these experiments using the illumination subset of Max-Planck Institute face database and Korean face database.

Three-dimensional object detection under arbitrary lighting conditions

A novel method of 3D object recognition independent of lighting conditions is presented. The recognition model is based on a vector space representation using an orthonormal basis generated by the Lambertian reflectance functions obtained with distant light sources. Changing the lighting conditions corresponds to multiplying the elementary images by a constant factor and because of that, all possible lighting views will be elements that belong to that vector space. The recognition method proposed is based on the calculation of the angle between the vector associated with a certain illuminated 3D object and that subspace. We define the angle in terms of linear correlations to get shift and illumination-invariant detection.

Illumination invariant face recognition

The appearance of a face will vary drastically when the illumination changes. Variations in lighting conditions make face recognition an even more challenging and difficult task. In this paper, we propose a novel approach to handle the illumination problem. Our method can restore a face image captured under arbitrary lighting conditions to one with frontal illumination by using a ratio-image between the face image and a reference face image, both of which are blurred by a Gaussian filter. An iterative algorithm is then used to update the reference image, which is reconstructed from the restored image by means of principal component analysis (PCA), in order to obtain a visually better restored image. Image processing techniques are also used to improve the quality of the restored image. To evaluate the performance of our algorithm, restored images with frontal illumination are used for face recognition by means of PCA. Experimental results demonstrate that face recognition using our method can achieve a higher recognition rate based on the Yale B database and the Yale database. Our algorithm has several advantages over other previous algorithms: (1) it does not need to estimate the face surface normals and the light source directions, (2) it does not need many images captured under different lighting conditions for each person, nor a set of bootstrap images that includes many images with different illuminations, and (3) it does not need to detect accurate positions of some facial feature points or to warp the image for alignment, etc.

Light source position and reflectance estimation from a single view without the distant illumination assumption

Several techniques have been developed for recovering reflectance properties of real surfaces under unknown illumination. However, in most cases, those techniques assume that the light sources are located at inifinity, which cannot be applied safely to, for example, reflectance modeling of indoor environments. In this paper, we propose two types of methods to estimate the surface reflectance property of an object, as well as the position of a light source from a single view without the distant illumination assumption, thus relaxing the conditions in the previous methods. Given a real image and a 3D geometric model of an object with specular reflection as inputs, the first method estimates the light source position by fitting to the Lambertian diffuse component, while separating the specular and diffuse components by using an iterative relaxation scheme. Our second method extends that first method by using as input a specular component image, which is acquired by analyzing multiple polarization images taken from a single view, thus removing its constraints on the diffuse reflectance property. This method simultaneously recovers the reflectance properties and the light source positions by optimizing the linearity of a log-transformed Torrance-Sparrow model. By estimating the object's reflectance property and the light source position, we can freely generate synthetic images of the target object under arbitrary lighting conditions with not only source direction modification but also source-surface distance modification. Experimental results show the accuracy of our estimation framework.

Measuring bidirectional texture reflectance with a kaleidoscope

We describe a new technique for measuring the bidirectional texture function (BTF) of a surface that requires no mechanical movement, can measure surfaces in situ under arbitrary lighting conditions, and can be made small, portable and inexpensive. The enabling innovation is the use of a tapered kaleidoscope, which allows a camera to view the same surface sample simultaneously from many directions. Similarly, the surface can be simultaneously illuminated from many directions, using only a single structured light source. We describe the techniques of construction and measurement, and we show experimental results.