Illumination Estimation Research Articles

RIS-GAN: Explore Residual and Illumination with Generative Adversarial Networks for Shadow Removal

Residual images and illumination estimation have been proved very helpful in image enhancement. In this paper, we propose a general and novel framework RIS-GAN which explores residual and illumination with Generative Adversarial Networks for shadow removal. Combined with the coarse shadow-removal image, the estimated negative residual images and inverse illumination maps can be used to generate indirect shadow-removal images to refine the coarse shadow-removal result to the fine shadow-free image in a coarse-to-fine fashion. Three discriminators are designed to distinguish whether the predicted negative residual images, shadow-removal images, and the inverse illumination maps are real or fake jointly compared with the corresponding ground-truth information. To our best knowledge, we are the first one to explore residual and illumination for shadow removal. We evaluate our proposed method on two benchmark datasets, i.e., SRD and ISTD, and the extensive experiments demonstrate that our proposed method achieves the superior performance to state-of-the-arts, although we have no particular shadow-aware components designed in our generators.

Enhancing Underexposed Photos Using Perceptually Bidirectional Similarity

Although remarkable progress has been made, existing methods for enhancing underexposed photos tend to produce visually unpleasing results due to the existence of visual artifacts (e.g., color distortion, loss of details and uneven exposure). We observed that this is because they fail to ensure the perceptual consistency of visual information between the source underexposed image and its enhanced output. To obtain high-quality results free of these artifacts, we present a novel underexposed photo enhancement approach that is able to maintain the perceptual consistency. We achieve this by proposing an effective criterion, referred to as perceptually bidirectional similarity, which explicitly describes how to ensure the perceptual consistency. Particularly, we adopt the Retinex theory and cast the enhancement problem as a constrained illumination estimation optimization, where we formulate perceptually bidirectional similarity as constraints on illumination and solve for the illumination which can recover the desired artifact-free enhancement results. In addition, we describe a video enhancement framework that adopts the presented illumination estimation for handling underexposed videos. To this end, a probabilistic approach is introduced to propagate illuminations of sampled keyframes to the entire video by tackling a Bayesian Maximum A Posteriori problem. Extensive experiments demonstrate the superiority of our method over the state-of-the-art methods.

An alternative approach to preserve naturalness with non-uniform illumination estimation for images enhancement using normalized $$L_2$$-Norm based on Retinex

In recent works, the illumination estimation based on Retinex is utilized to enhance the image details. However, to enhance the image using illumination estimation without affecting the naturalness with non-uniform illumination is very difficult. The usual algorithms used for illumination estimation failed to include these constraints such as sharp edges on illumination boundaries, spatial smoothness, and the limited range of illumination while preserving the naturalness. In this paper, an illumination estimation algorithm using normalized $$L_2$$ -Norm with the joint edge-preservation filter is proposed. The proposed algorithm efficiently estimates illumination and fulfils all the above-mentioned constraints. The normalized $$L_2$$ -Norm is used to approximate the illumination to overcome the block effect in patch-wise illumination. Then, the overall structure is refined by using the joint edge-preservation filter. Experimental results show the quality of the smoothness of illumination beyond the edges and ensure the range of the estimated illumination in comparison with the other state-of-the-art algorithms.

Deep Learning-Based Illumination Estimation Using Light Source Classification

Color constancy is one of the key steps in the process of image formation in digital cameras. Its goal is to process the image so that there is no influence of illumination color on the colors of objects and surfaces. To capture the target scene colors as accurately as possible, it is crucial to estimate the illumination vector with high accuracy. Unfortunately, the illumination estimation is an ill-posed problem, and solving it most often relies on assumptions. To date, various assumptions have been proposed, which resulted in a wide variety of illumination estimation methods. Statistics-based methods have shown to be appropriate for hardware implementation, but learning-based methods achieve state-of-the-art results, especially those that use deep neural networks. The large learning capacities and generalization abilities of deep neural networks can be used to develop the illumination estimation methods, which are more general and precise. This approach avoids introducing many new assumptions, which often only work in some specific situations. In this paper, a new method for illumination estimation based on light source classification is proposed. In the first step, the set of possible illuminations is reduced by classifying the input image in one of three classes. The classes include images captured in outdoor scenes under natural illuminations, images captured in outdoor scenes under artificial illuminations, and images captured in indoor scenes under artificial illuminations. In the second step, a deep illumination estimation network, which is trained exclusively on images in the class that was predicted in the first step, is applied to the input image. Dividing the illumination space into smaller regions makes the training of illumination estimation networks simpler because the distribution of image scenes and illuminations is less diverse. The experiments on the Cube+ image dataset have shown the median illumination estimation error of 1.27°, which is an improvement of more than 25% compared to the use of the single network for all illuminations.

The Cube++ Illumination Estimation Dataset

Computational color constancy has the important task of reducing the influence of the scene illumination on the object colors. As such, it is an essential part of the image processing pipelines of most digital cameras. One of the important parts of the computational color constancy is illumination estimation, i.e. estimating the illumination color. When an illumination estimation method is proposed, its accuracy is usually reported by providing the values of error metrics obtained on the images of publicly available datasets. However, over time it has been shown that many of these datasets have problems such as too few images, inappropriate image quality, lack of scene diversity, absence of version tracking, violation of various assumptions, GDPR regulation violation, lack of additional shooting procedure info, etc. In this paper, a new illumination estimation dataset is proposed that aims to alleviate many of the mentioned problems and to help the illumination estimation research. It consists of 4890 images with known illumination colors as well as with additional semantic data that can further make the learning process more accurate. Due to the usage of the SpyderCube color target, for every image there are two ground-truth illumination records covering different directions. Because of that, the dataset can be used for training and testing of methods that perform single or two-illuminant estimation. This makes it superior to many similar existing datasets. The datasets, it's smaller version SimpleCube++, and the accompanying code are available at https://github.com/Visillect/CubePlusPlus/.

Bag of Color Features for Color Constancy

In this paper, we propose a novel color constancy approach, called Bag of Color Features (BoCF), building upon Bag-of-Features pooling. The proposed method substantially reduces the number of parameters needed for illumination estimation. At the same time, the proposed method is consistent with the color constancy assumption stating that global spatial information is not relevant for illumination estimation and local information ( edges, etc.) is sufficient. Furthermore, BoCF is consistent with color constancy statistical approaches and can be interpreted as a learning-based generalization of many statistical approaches. To further improve the illumination estimation accuracy, we propose a novel attention mechanism for the BoCF model with two variants based on self-attention. BoCF approach and its variants achieve competitive, compared to the state of the art, results while requiring much fewer parameters on three benchmark datasets: ColorChecker RECommended, INTEL-TUT version 2, and NUS8.

Multi-Cue Semi-Supervised Color Constancy With Limited Training Samples

Color constancy is one of the fundamental tasks in computer vision. Many supervised methods, including recently proposed Convolutional Neural Networks (CNN)-based methods, have been proved to work well on this problem, but they often require a sufficient number of labeled data. However, it is expensive and time-consuming to collect a large number of labeled training images with accurately measured illumination. In order to reduce the dependence on labeled images and leverage unlabeled ones without measured illumination, we propose a novel semi-supervised framework with limited training samples for illumination estimation. Our key insight is that the images with similar features from different cues will share similar lighting conditions. Consequently, three graphs based on three visual cues, low-level RGB color distribution, mid-level initial illuminant estimates and high-level scene content, are constructed to represent the relationship among different images. Then a multi-cue semi-supervised color constancy method (MSCC) is proposed after integrating these three graphs into a unified model. Extensive experiments on benchmark datasets demonstrate that our proposed MSCC method outperforms nearly all the existing supervised methods with limited labeled samples. Even with no unlabeled samples, MSCC still obtains better performance and stableness than most supervised methods.

Color Constancy by Reweighting Image Feature Maps.

In this study, a novel illuminant color estimation framework is proposed for computational color constancy, which incorporates the high representational capacity of deep-learningbased models and the great interpretability of assumptionbased models. The well-designed building block, feature map reweight unit (ReWU), helps to achieve comparative accuracy on benchmark datasets with respect to prior state-of-the-art deep learning based models while requiring more compact model size and cheaper computational cost. In addition to local color estimation, a confidence estimation branch is also included such that the model is able to simultaneously produce point estimate and its uncertainty estimate, which provides useful clues for local estimates aggregation and multiple illumination estimation. The source code and the dataset have been made available1.

Canonical Illumination Decomposition and Its Applications

Raw data acquired by imaging devices are converted into digital images by post-processing algorithms. However, these algorithms are significantly affected by numerous illumination conditions. Particularly, in the case in which illumination conditions depend on canonical light sources, unwanted light sources that locally illuminate the scene or mixed light from several light sources are recognized as the spatially varying illumination conditions. These complex illumination conditions cause several artifacts by affecting the digital image acquisition process. For example, optical aberrations are generated by refraction of complex light, or illumination estimation is significantly affected by the different color temperatures of multiple light sources. To overcome these problems, this study proposes an algorithm that decomposes the complex illumination from several light sources. First, the spatially varying illumination condition is discussed, and the artifacts generated by the conditions, such as false colors and aberrations, are analyzed. Second, mixed light from several canonical light sources is decomposed based on the imaging devices and spectral information of the canonical light sources. The proposed method has low complexity and increased applicability. Furthermore, the improvement scheme based on the proposed method has a parallelized structure and can be easily applied to various types of algorithms, such as color constancy, deconvolution, denoising, and contrast enhancement. The algorithms employing the improvement scheme show the potential of the proposed method for solving the problems associated with multiple light sources.

Illuminant Estimation Through Reverse Calibration of an Auto White-Balanced Image That Contains Displays

This study proposes an illuminant estimation method that reproduces the original illuminant of a scene using a mobile display as a target. The original lighting environment of an auto white-balancing (AWB) photograph is obtained through reverse calibration, using the white point of a display in the photograph. This reproduces the photograph before AWB processed, and we can obtain the illuminant information using Gray World computation. The study consists of two sessions. In Session 1, we measured the display's white points under varying illuminants to prove that display colors show limited changes under any light conditions. Then, in Session 2, we generated the estimations and assessed the performance of display-based illuminant estimation by comparing the result with the optically measured values in the real situation. Overall, the proposed method is a satisfactory way to estimate the less chromatic illuminants under 6300 K that we experience as indoor light in our daily lives.

Evaluating Colour Constancy on the new MIST dataset of Multi-Illuminant Scenes

A new image test set of synthetically generated, full-spectrum images with pixelwise ground truth has been developed to aid in the evaluation of illumination estimation methods for colour constancy. The performance of 9 illumination methods is reported for this dataset along and compared to the optimal single-illuminant estimate. None of the methods specifically designed to handle multi-illuminant scenes is found to perform any better than the optimal single-illuminant case based on completely uniform illumination.

Adaptive Single Image Dehazing Using Joint Local-Global Illumination Adjustment

Haze has a serious impact on the outdoor optical imaging systems, and it will result in image blurring, color shift, and saturation reduction. Recently, many single image dehazing algorithms have been proposed for practical applications, such as surveillance. However, since the widely-used global atmospheric light in image dehazing fails to well describe the local illumination differences of images, these algorithms fail to well adapt to scenes with different haze concentrations and lighting conditions. Therefore, this paper proposes an adaptive single image dehazing algorithm using joint local-global illumination adjustment. A local illumination estimation for hazy image is proposed to replace the global atmospheric light constant in the atmospheric scattering model, and it can better adapt to the local differences of image illumination. Correspondingly, the global atmospheric light constant is proposed to be utilized to adaptively compensate the illumination intensity, which may better overcome the dark illumination problem within the dehazed image. The experimental results demonstrate that the proposed algorithm can outperform the state-of-the-art algorithms in terms of not only the dehazing effect but also the adaptability.

Dual Illumination Estimation for Robust Exposure Correction

AbstractExposure correction is one of the fundamental tasks in image processing and computational photography. While various methods have been proposed, they either fail to produce visually pleasing results, or only work well for limited types of image (e.g., underexposed images). In this paper, we present a novel automatic exposure correction method, which is able to robustly produce high‐quality results for images of various exposure conditions (e.g., underexposed, overexposed, and partially under‐ and over‐exposed). At the core of our approach is the proposed dual illumination estimation, where we separately cast the under‐and over‐exposure correction as trivial illumination estimation of the input image and the inverted input image. By performing dual illumination estimation, we obtain two intermediate exposure correction results for the input image, with one fixes the underexposed regions and the other one restores the overexposed regions. A multi‐exposure image fusion technique is then employed to adaptively blend the visually best exposed parts in the two intermediate exposure correction images and the input image into a globally well‐exposed image. Experiments on a number of challenging images demonstrate the effectiveness of the proposed approach and its superiority over the state‐of‐the‐art methods and popular automatic exposure correction tools.

Illumination estimation for augmented reality based on a global illumination model

With the rapid development of 3D technology, the illumination consistency plays an important role in realistic rendering of virtual objects which are superimposed into the real scene. In this paper we proposed a new approach to estimate illumination, given the images of a scene. Our algorithm is designed from principles in optics, which is based on a global illumination model. First, we get the images of a scene and perform scene reconstruction using images. Then, a photon emission hemispherical model is built for emitting photons into the scene. The photons are traced, meanwhile the result is stored in multiple photon maps. Finally, the estimated illumination is obtained by estimating the photon radiance value in the photon emission hemispherical model. Experiments on our newly collected real scene databases and virtual scene databases show the accuracy of our method subjectively and objectively. By comparing with related work, it is found that more accurate results can be obtained using the global illumination inversely.

Deep reinforcement learning-based patch selection for illuminant estimation

Previous deep learning based approaches to illuminant estimation either resized the raw image to lower resolution or randomly cropped image patches for the deep learning model. However, such practices would inevitably lead to information loss or the selection of noisy patches that would affect estimation accuracy. In this paper, we regard patch selection in neural network based illuminant estimation as a controlling problem of selecting image patches that could help remove noisy patches and improve estimation accuracy. To achieve this, we construct a selection network (SeNet) to learn a patch selection policy. Based on data statistics and the learning progression state of the deep illuminant estimation network (DeNet), the SeNet decides which training patches should be input to the DeNet, which in turn gives feedback to the SeNet for it to update its selection policy. To achieve such interactive and intelligent learning, we utilize a reinforcement learning approach termed policy gradient to optimize the SeNet. We show that the proposed learning strategy can enhance the illuminant estimation accuracy, speed up the convergence and improve the stability of the training process of DeNet. We evaluate our method on two public datasets and demonstrate our method outperforms state-of-the-art approaches.

Autonomous robot navigation using Retinex algorithm for multiscale image adaptability in low-light environment

This paper proposes an improved Retinex theory based on a weighted guided filter method to enhance images in low-light conditions. The captured images under low illumination can cause dimness, distortion or details loss. We use the weighted guided filter method to perform illumination estimation and the original image is regarded as the guidance image, which can avoid color distortion and over-enhancement. It can adjust the regularization parameter adaptively based on the image content. Perceptual contrast is improved by using an illumination enhancement method with dynamic adjustment. To test the validness of our algorithm, the weighted guided filter method proposed in this paper is compared with bilateral filter and the guided filter method. Finally, experiment under low illumination is implemented on a NAO robot by using the proposed weighted guided filter method based on EKF-SLAM. The experiment result demonstrates that the proposed weighted guided filter method is feasible and effective in low-light environment.

TDCC: top‐down semantic aggregation for colour constancy

Images obtained from an illuminated scene often have their original colour contaminated. Colour constancy is a study considering how to restore them. Substantial progress on colour constancy has been made in recent years due to the development of a convolutional neural network (CNN). In a CNN structure, high-level features contain semantic information while low-level features show local details. If both are taken into account, they would help achieve a more accurate illuminant estimation. However, previous works paid little attention to the latter for lack of frameworks, which can combine those two kinds of features together. Inspired by the pyramid model, a top-down network that successively propagates high-level information to low-level layers is proposed. This network, named top-down semantic aggregation for colour constancy (TDCC), takes full advantage of the multi-scale representations with strong semantics. As a result, objects with intrinsic colours are captured and a better estimation is obtained. Experiments on three benchmark datasets demonstrate that TDCC significantly outperforms state-of-the-art colour constancy methods.

Providing a Single Ground-Truth for Illuminant Estimation for the ColorChecker Dataset.

The ColorChecker dataset is one of the most widely used image sets for evaluating and ranking illuminant estimation algorithms. However, this single set of images has at least 3 different sets of ground-truth (i.e., correct answers) associated with it. In the literature it is often asserted that one algorithm is better than another when the algorithms in question have been tuned and tested with the different ground-truths. In this short correspondence we present some of the background as to why the 3 existing ground-truths are different and go on to make a new single and recommended set of correct answers. Experiments reinforce the importance of this work in that we show that the total ordering of a set of algorithms may be reversed depending on whether we use the new or legacy ground-truth data.

Computational colour constancy based on convolutional neural networks with a cross‐level architecture

Computational colour constancy refers to the problem of obtaining reliable illuminant colour so that the image can be rectified to generate a new image under the canonical illumination by illumination estimation. This study researches on convolutional neural networks (CNNs) with cross-level architecture for colour constancy. This new methodology obtains the illumination estimation by combining widely-used AlexNet model with cross-level strategy, in which, the cross-level strategy is upgraded to a more adaptive and intelligent level for colour constancy problem. The proposed method is based on the assumption of uniform illumination on the scene, and the colour constancy is approached as an illumination classification problem on real illuminants by training the authors' cnns. Furthermore, the output of the networks could be directly used for illumination computation of the scene. Finally, the performance of the proposed methodology is verified by comparing with the networks without cross-level architecture or with similar strategies, including multiple paths architecture, multi-scale architecture and ConvNet structure, to guarantee the feasibility and effectiveness. Experimental results on the test datasets indicate that up to around 26% error could be reduced via the proposed methodology for colour constancy.

DeepLight: light source estimation for augmented reality using deep learning

This paper presents a novel method for illumination estimation from RGB-D images. The main focus of the proposed method is to enhance visual coherence in augmented reality applications by providing accurate and temporally coherent estimates of real illumination. For this purpose, we designed and trained a deep neural network which calculates a dominant light direction from a single RGB-D image. Additionally, we propose a novel method for real-time outlier detection to achieve temporally coherent estimates. Our method for light source estimation in augmented reality was evaluated on the set of real scenes. Our results demonstrate that the neural network can successfully estimate light sources even in scenes which were not seen by the network during training. Moreover, we compared our results with illumination estimates calculated by the state-of-the-art method for illumination estimation. Finally, we demonstrate the applicability of our method on numerous augmented reality scenes.