Intrinsic Image Research Articles

Adaptive Fusion of Heterogeneous Manifolds for Subspace Clustering.

Multiview clustering (MVC) has recently received great interest due to its pleasing efficacy in combining the abundant and complementary information to improve clustering performance, which overcomes the drawbacks of view limitation existed in the standard single-view clustering. However, the existing MVC methods are mostly designed for vectorial data from linear spaces and, thus, are not suitable for multiple dimensional data with intrinsic nonlinear manifold structures, e.g., videos or image sets. Some works have introduced manifolds' representation methods of data into MVC and obtained considerable improvements, but how to fuse multiple manifolds efficiently for clustering is still a challenging problem. Particularly, for heterogeneous manifolds, it is an entirely new problem. In this article, we propose to represent the complicated multiviews' data as heterogeneous manifolds and a fusion framework of heterogeneous manifolds for clustering. Different from the empirical weighting methods, an adaptive fusion strategy is designed to weight the importance of different manifolds in a data-driven manner. In addition, the low-rank representation is generalized onto the fused heterogeneous manifolds to explore the low-dimensional subspace structures embedded in data for clustering. We assessed the proposed method on several public data sets, including human action video, facial image, and traffic scenario video. The experimental results show that our method obviously outperforms a number of state-of-the-art clustering methods.

A deep learning system to obtain the optimal parameters for a threshold-based breast and dense tissue segmentation

Background and ObjectiveBreast cancer is the most frequent cancer in women. The Spanish healthcare network established population-based screening programs in all Autonomous Communities, where mammograms of asymptomatic women are taken with early diagnosis purposes. Breast density assessed from digital mammograms is a biomarker known to be related to a higher risk to develop breast cancer.It is thus crucial to provide a reliable method to measure breast density from mammograms. Furthermore the complete automation of this segmentation process is becoming fundamental as the amount of mammograms increases every day. Important challenges are related with the differences in images from different devices and the lack of an objective gold standard.This paper presents a fully automated framework based on deep learning to estimate the breast density. The framework covers breast detection, pectoral muscle exclusion, and fibroglandular tissue segmentation. MethodsA multi-center study, composed of 1785 women whose “for presentation” mammograms were segmented by two experienced radiologists. A total of 4992 of the 6680 mammograms were used as training corpus and the remaining (1688) formed the test corpus. This paper presents a histogram normalization step that smoothed the difference between acquisition, a regression architecture that learned segmentation parameters as intrinsic image features and a loss function based on the DICE score. ResultsThe results obtained indicate that the level of concordance (DICE score) reached by the two radiologists (0.77) was also achieved by the automated framework when it was compared to the closest breast segmentation from the radiologists. For the acquired with the highest quality device, the DICE score per acquisition device reached 0.84, while the concordance between radiologists was 0.76. ConclusionsAn automatic breast density estimator based on deep learning exhibits similar performance when compared with two experienced radiologists. It suggests that this system could be used to support radiologists to ease its work.

Illuminance Compensation and Texture Enhancement via the Hodge Decomposition

Image brightness in color representation has caught active research interests, while its influence to texture features is relatively rarely studied. In this paper, we address the issue of illuminance, or brightness, interference to texture descriptors, especially to the Gabor filter based approaches. Firstly, we reveal the fact that the Gabor filter response linearly drifts with the illuminance. Secondly, the Hodge decomposition is introduced, and a linear regression model is presented to interpret the linear drift phenomenon. An interesting connection with the intrinsic image decomposition is established. Finally, a partial differential equation based method is proposed to compensate the drift issue. Consequently, the texture of the compensated image is enhanced. Extensive experiments are conducted to verify the proposed statements. The proposed linear regression model is verified empirically, and the compensation performances are demonstrated. Classification results on various databases indicate that the compensation can effectively improve the capacity of many existing texture description methods. Especially, the results show that the boosting effect on some deep learning based methods is also valid.

Image Memorability Prediction Using Depth and Motion Cues

Memorability is one of the intrinsic image properties, which enables one to quantify to what extent images are memorable to the human cognitive system. Many works have shed light on various visual factors that influence the image memorability. Recently, one such study showed the influence of image depth and motion information on image memorability [3] . Based on these findings, this article proposes a deep learning-based prediction model, which utilizes depth and motion cues to predict the image memorability scores. The proposed model contains three deep CNN networks. Each of these three networks is individually trained to utilize one of the three visual factors: 1) visual depth information; 2) optical flow information; and 3) fine-tuned scene- and object-related features. In the end, all three networks are ensembled to predict the final memorability scores for the given image. An extensive set of experiments are conducted on large scale image memorability data set LaMem. From the experimental results, it is observed that the proposed model performs better than the current state-of-the-art model [3] by 2.44%.

An unsupervised image segmentation method combining graph clustering and high-level feature representation

Image segmentation is one of the most important assignments in computer vision. In this paper, we present an unsupervised segmentation method that combines graph-based clustering and high-level semantic features. We over-segment the given image into a collection of superpixels. Various low-level features assemble a descriptor of each superpixel. Besides the intrinsic image features such as color, texture and gradient, we add image saliency into the low-level visual features as prior knowledge of human perception. Instead of using the low-level features directly, we design a graph-based method to segment the image by clustering the high-level semantic features learned from a neural network. We test the proposed method on two well-known datasets. The experimental evaluation validates that our approach can provide consistent and meaningful segmentation.

PDE Learning of Filtering and Propagation for Task-Aware Facial Intrinsic Image Analysis.

Filtering and propagation are two basic operations in image analysis and rendering, and they are also widely used in computer graphics and machine learning. However, the models of filtering and propagation were based on diverse mathematical formulations, which have not been fully understood. This article aims to explore the properties of both filtering and propagation models from a partial differential equation (PDE) learning perspective. We propose a unified PDE learning framework based on nonlinear reaction-diffusion with a guided map, graph Laplacian, and reaction weight. It reveals that: 1) the guided map and reaction weight determines whether the PDE produces filtering or propagation diffusion and 2) the kernel of graph Laplacian controls the diffusion pattern. Based on the proposed PDE framework, we derive the mathematical relations between different models, including learning to diffusion (LTD) model, label propagation, edit propagation, and edge-aware filter. In practical verification, we apply the PDE framework to design diffusion operations with the adaptive kernel to tackle the ill-posed problem of facial intrinsic image analysis (FIIA). A flexible task-aware FIIA system is built to achieve various facial rendering effects, such as face image relighting and delighting, artistic illumination transfer, illumination-aware face swapping, or transfiguring. Qualitative and quantitative experiments show the effectiveness and flexibility of task-aware FIIA and provide new insights on PDE learning for visual analysis and rendering.

Hyperspectral Anomaly Detection Using Dual Window Density

Hyperspectral anomaly detection is one of the most active topics in hyperspectral image (HSI) analysis. The fine spectral information of HSIs allows us to uncover anomalies with very high accuracy. Recently, an intrinsic image decomposition (IID) model has been introduced for low-rank IID (LRIID) in multispectral images. Inspired by the LRIID, which is able to effectively recover the reflectance and shading components of the multispectral image, this article adapts the LRIID for obtaining the reflectance component of HSIs (which is the key feature for the discrimination of different objects). In order to exploit the reflectance component, we also propose a new dual window density (DWD)-based detector for anomaly detection, which is based on the idea that anomalies are usually rare pixels and, thus, exhibit low density in the image. The density analysis of DWD is intended not only to circumvent the Gaussian assumption regarding the distribution of HSI data, but also to mitigate the contamination of background statistics caused by anomalies. The dual window operation of our DWD is specifically designed to adaptively calculate the density of each pixel under test, so as to identify anomalies with nonspecific sizes. Our experimental results, obtained on a database of real HSIs including Airport, Beach, and Urban scenes, demonstrate the superiority of the proposed method in terms of detection performance when compared to other widely used anomaly detection methods.

Intrinsic image decomposition as two independent deconvolution problems

In this paper, a novel method to decompose an image into “intrinsic images” is introduced. This decomposition is based on the dichromatic model which separates the influence of specular and diffuse reflections. The separation of these components is very important in several applications of image analysis including segmentation, classification, recoloring, and specularity removal. The proposed method is based on two simple deconvolution steps. The method aims to be generic: it is applicable to any kind of image (i.e. RGB as well as multispectral) and does not rely on a learning step. The method is applied to three datasets including multispectral and RGB images. The algorithm is compared to recent algorithms from the literature and gives similar or better results than the state of the art.

Remote Sensing Image Denoising via Low-Rank Tensor Approximation and Robust Noise Modeling

Noise removal is a fundamental problem in remote sensing image processing. Most existing methods, however, have not yet attained sufficient robustness in practice, due to more or less neglecting the intrinsic structures of remote sensing images and/or underestimating the complexity of realistic noise. In this paper, we propose a new remote sensing image denoising method by integrating intrinsic image characterization and robust noise modeling. Specifically, we use low-Tucker-rank tensor approximation to capture the global multi-factor correlation within the underlying image, and adopt a non-identical and non-independent distributed mixture of Gaussians (non-i.i.d. MoG) assumption to encode the statistical configurations of the embedded noise. Then, we incorporate the proposed image and noise priors into a full Bayesian generative model and design an efficient variational Bayesian algorithm to infer all involved variables by closed-form equations. Moreover, adaptive strategies for the selection of hyperparameters are further developed to make our algorithm free from burdensome hyperparameter-tuning. Extensive experiments on both simulated and real multispectral/hyperspectral images demonstrate the superiority of the proposed method over the compared state-of-the-art ones.

Separate in Latent Space: Unsupervised Single Image Layer Separation

Many real world vision tasks, such as reflection removal from a transparent surface and intrinsic image decomposition, can be modeled as single image layer separation. However, this problem is highly ill-posed, requiring accurately aligned and hard to collect triplet data to train the CNN models. To address this problem, this paper proposes an unsupervised method that requires no ground truth data triplet in training. At the core of the method are two assumptions about data distributions in the latent spaces of different layers, based on which a novel unsupervised layer separation pipeline can be derived. Then the method can be constructed based on the GANs framework with self-supervision and cycle consistency constraints, etc. Experimental results demonstrate its successfulness in outperforming existing unsupervised methods in both synthetic and real world tasks. The method also shows its ability to solve a more challenging multi-layer separation task.

Leveraging Multi-View Image Sets for Unsupervised Intrinsic Image Decomposition and Highlight Separation

We present an unsupervised approach for factorizing object appearance into highlight, shading, and albedo layers, trained by multi-view real images. To do so, we construct a multi-view dataset by collecting numerous customer product photos online, which exhibit large illumination variations that make them suitable for training of reflectance separation and can facilitate object-level decomposition. The main contribution of our approach is a proposed image representation based on local color distributions that allows training to be insensitive to the local misalignments of multi-view images. In addition, we present a new guidance cue for unsupervised training that exploits synergy between highlight separation and intrinsic image decomposition. Over a broad range of objects, our technique is shown to yield state-of-the-art results for both of these tasks.

Inhomogeneous morphological PDEs for robust and adaptive image shock filters

Classical morphological filters suffer from well performing in a noisy environment, and intrinsic image structures are not taken into account. The authors propose here an alternative to overcome such weaknesses, by properly using robust shock filters and inhomogeneity. Thus, they obtain multiscale morphological operators by using image edge functions as local weights in inhomogeneous Hamiltonians in classical multiscale dilations/erosions formulated with partial differential equations (PDEs). They provide the equivalent sup-inf-based formulations, and derive sharpening/enhancement methods. In addition, they establish the PDE associated with the asymptotical iterations of the proposed robust and adaptive filters. The good behaviours of the proposed sup-inf and PDE-based methods are illustrated on synthetic, greyscale, and colour images; results are analysed both qualitatively and quantitatively.

3D face mask presentation attack detection based on intrinsic image analysis

Face presentation attacks have become a major threat against face recognition systems and many countermeasures have been proposed over the past decade. However, most of them are devoted to 2D face presentation attack detection, rather than 3D face masks. Unlike the real face, the 3D face mask is usually made of resin materials and has a smooth surface, resulting in reflectance differences. Therefore, in this study, the authors propose a novel 3D face mask presentation attack detection method based on analysis of image reflectance. In the proposed method, the face image is first processed with intrinsic image decomposition algorithm to compute its reflectance image. Then, the intensity distribution histograms are extracted from three orthogonal planes to represent the intensity differences of reflectance images between the real face and 3D face mask. After that, given that the reflectance image of a smooth surface is more sensitive to illumination changes, 1D convolutional neural network is used to characterise how different materials or surfaces react differently to illumination changes. Extensive experiments with the public available 3DMAD database demonstrate the effectiveness of the proposed method for distinguishing a face mask from the real one and show that the detection performance outperforms other state-of-the-art methods.

A Sparse Representation-Based Sample Pseudo-Labeling Method for Hyperspectral Image Classification

Hyperspectral image classification methods may not achieve good performance when a limited number of training samples are provided. However, labeling sufficient samples of hyperspectral images to achieve adequate training is quite expensive and difficult. In this paper, we propose a novel sample pseudo-labeling method based on sparse representation (SRSPL) for hyperspectral image classification, in which sparse representation is used to select the purest samples to extend the training set. The proposed method consists of the following three steps. First, intrinsic image decomposition is used to obtain the reflectance components of hyperspectral images. Second, hyperspectral pixels are sparsely represented using an overcomplete dictionary composed of all training samples. Finally, information entropy is defined for the vectorized sparse representation, and then the pixels with low information entropy are selected as pseudo-labeled samples to augment the training set. The quality of the generated pseudo-labeled samples is evaluated based on classification accuracy, i.e., overall accuracy, average accuracy, and Kappa coefficient. Experimental results on four real hyperspectral data sets demonstrate excellent classification performance using the new added pseudo-labeled samples, which indicates that the generated samples are of high confidence.

Intrinsic RGB and multispectral images recovery by independent quadratic programming.

This work introduces a method to estimate reflectance, shading, and specularity from a single image. Reflectance, shading, and specularity are intrinsic images derived from the dichromatic model. Estimation of these intrinsic images has many applications in computer vision such as shape recovery, specularity removal, segmentation, or classification. The proposed method allows for recovering the dichromatic model parameters thanks to two independent quadratic programming steps. Compared to the state of the art in this domain, our approach has the advantage to address a complex inverse problem into two parallelizable optimization steps that are easy to solve and do not require learning. The proposed method is an extension of a previous algorithm that is rewritten to be numerically more stable, has better quantitative and qualitative results, and applies to multispectral images. The proposed method is assessed qualitatively and quantitatively on standard RGB and multispectral datasets.

PBR-Net: Imitating Physically Based Rendering using Deep Neural Network.

Physically based rendering has been widely used to generate photo-realistic images, which greatly impacts industry by providing appealing rendering, such as for entertainment and augmented reality, and academia by serving large scale high-fidelity synthetic training data for data hungry methods like deep learning. However, physically based rendering heavily relies on ray-tracing, which can be computational expensive in complicated environment and hard to parallelize. In this paper, we propose an end-to-end deep learning based approach to generate physically based rendering efficiently. Our system consists of two stacked neural networks, which effectively simulates the physical behavior of the rendering process and produces photo-realistic images. The first network, namely shading network, is designed to predict the optimal shading image from surface normal, depth and illumination; the second network, namely composition network, learns to combine the predicted shading image with the reflectance to generate the final result. Our approach is inspired by intrinsic image decomposition, and thus it is more physically reasonable to have shading as intermediate supervision. Extensive experiments show that our approach is robust to noise thanks to a modified perceptual loss and even outperforms the physically based rendering systems in complex scenes given a reasonable time budget.

Uneven Illumination Surface Defects Inspection Based on Saliency Detection and Intrinsic Image Decomposition

Surface defect detection based on computer vision remains a challenging task due to the uneven illumination, low contrast and miscellaneous patterns of defects. Current methods usually present undesirable detection accuracy and lack adaptability for the various scenes. In the paper, the novel uneven illumination surface defects inspection (UISDI) method is proposed to address these issues. First, the multi-scale saliency detection (MSSD) method is proposed to construct a coarse defect map and obtain the corresponding background regions. Second, a novel background similarity prior-based intrinsic image decomposition model (BSIID) is applied to divide the defect image into a non-defective shading layer and a defective reflectance layer. An accelerated optimization solution is proposed to solve the minimization problem of the intrinsic image decomposition model. Last, the enhanced defect image is obtained by filtering the reflectance image and is then utilized to accurately segment the defect region from the coarse defect map. The experiments conducted using four real-world defect datasets demonstrate that the proposed method outperforms state-of-the-art methods.

Deep intrinsic decomposition trained on surreal scenes yet with realistic light effects

Estimation of intrinsic images still remains a challenging task due to weaknesses of ground-truth datasets, which either are too small or present non-realistic issues. On the other hand, end-to-end deep learning architectures start to achieve interesting results that we believe could be improved if important physical hints were not ignored. In this work, we present a twofold framework: (a) a flexible generation of images overcoming some classical dataset problems such as larger size jointly with coherent lighting appearance; and (b) a flexible architecture tying physical properties through intrinsic losses. Our proposal is versatile, presents low computation time, and achieves state-of-the-art results.

Shadow effect weakening based on intrinsic image extraction with effective projection of logarithmic domain for road scene

The accuracy of road region extraction is an important factor for the safety and reliability of intelligent driving system. Due to the shadow greatly affects the result of road region extraction in practical applications, the shadow effect weakening on a single image by the optical principles and theories is meaningful. To weaken the shadow effect, we extract the intrinsic images of road scenes based on logarithm domain projection. The intrinsic image has an advantage to remove the shadow effect. Thus, we propose effective projection angle calculation methods in logarithmic domain based on simple statistic, which can eliminate the impact of the direction of camera features. Furthermore, the best whole time projection angle with practical application can be obtained for rapid acquisition of intrinsic image. In the experiment, the proposed methods can weaken the shadow effect for road scene images. In order to evaluate intrinsic image extracted by the proposed methods, the same road detection method is implemented to extract road region. The results demonstrate that the road region detection accuracy based on intrinsic image extracted by our methods are better than the compared methods.

An eigendecomposition method based on deep learning and probabilistic graph model

With the rapid development of computer, computer vision derived from computer vision has also made important progress in the field of image research. The extraction of image information is the most basic work in the field of image research. However, in the current environment, there is still a lack of effective methods to understand more complex image problems, such as image shape, material and illumination distribution in the environment. Eigenimage decomposition can be achieved by obtaining albedo eigenvalues and luminance eigenvalues. The color and illumination information of the image can be obtained more intuitively. Based on this, this paper proposes an intrinsic image decomposition method based on depth learning and probability graph model, in order to extract image information more accurately. Firstly, a deep convolution neural network is trained to decompose reflectivity image and shadow image. Then the conditional random field is used to optimize the reflectivity image and shadow image. The convolutional neural network designed in this paper obtains preliminary results through multi-scale architecture, deep supervision, step-by-step refinement of synthetic images and multi-stage training, which has been significantly improved compared with previous algorithms. Then the essential image and the corresponding gradient image are further optimized by conditional random field, and the eigenvalue image with richer details and clearer boundary can be obtained.