Photometric Invariants Research Articles

A Survey on Underwater Computer Vision

Underwater computer vision has attracted increasing attention in the research community due to the recent advances in underwater platforms such as of rovers, gliders, autonomous underwater vehicles (AUVs) , and the like, that now make possible the acquisition of vast amounts of imagery and video for applications such as biodiversity assessment, environmental monitoring, and search and rescue. Despite growing interest, underwater computer vision is still a relatively under-researched area, where the attention in the literature has been paid to the use of computer vision techniques for image restoration and reconstruction, where image formation models and image processing methods are used to recover colour corrected or enhanced images. This is due to the notion that these methods can be used to achieve photometric invariants to perform higher-level vision tasks such as shape recovery and recognition under the challenging and widely varying imaging conditions that apply to underwater scenes. In this paper, we review underwater computer vision techniques for image reconstruction, restoration, recognition, depth, and shape recovery. Further, we review current applications such as biodiversity assessment, management and protection, infrastructure inspection and AUVs navigation, amongst others. We also delve upon the current trends in the field and examine the challenges and opportunities in the area.

Physics-based shading reconstruction for intrinsic image decomposition

We investigate the use of photometric invariance and deep learning to compute intrinsic images (albedo and shading). We propose albedo and shading gradient descriptors which are derived from physics-based models. Using the descriptors, albedo transitions are masked out and an initial sparse shading map is calculated directly from the corresponding RGB image gradients in a learning-free unsupervised manner. Then, an optimization method is proposed to reconstruct the full dense shading map. Finally, we integrate the generated shading map into a novel deep learning framework to refine it and also to predict corresponding albedo image to achieve intrinsic image decomposition. By doing so, we are the first to directly address the texture and intensity ambiguity problems of the shading estimations. Large scale experiments show that our approach steered by physics-based invariant descriptors achieve superior results on MIT Intrinsics, NIR-RGB Intrinsics, Multi-Illuminant Intrinsic Images, Spectral Intrinsic Images, As Realistic As Possible, and competitive results on Intrinsic Images in the Wild datasets while achieving state-of-the-art shading estimations.

Outdoor Inverse Rendering From a Single Image Using Multiview Self-Supervision.

In this paper we show how to perform scene-level inverse rendering to recover shape, reflectance and lighting from a single, uncontrolled image using a fully convolutional neural network. The network takes an RGB image as input, regresses albedo, shadow and normal maps from which we infer least squares optimal spherical harmonic lighting coefficients. Our network is trained using large uncontrolled multiview and timelapse image collections without ground truth. By incorporating a differentiable renderer, our network can learn from self-supervision. Since the problem is ill-posed we introduce additional supervision. Our key insight is to perform offline multiview stereo (MVS) on images containing rich illumination variation. From the MVS pose and depth maps, we can cross project between overlapping views such that Siamese training can be used to ensure consistent estimation of photometric invariants. MVS depth also provides direct coarse supervision for normal map estimation. We believe this is the first attempt to use MVS supervision for learning inverse rendering. In addition, we learn a statistical natural illumination prior. We evaluate performance on inverse rendering, normal map estimation and intrinsic image decomposition benchmarks.

Modeling of Radiation Fields of Uniform Anisotropically Scattering Slab of Aarbitrary Optical Thickness

Modeling of radiation fields of an arbitrary optical thickness uniform slab in the case of a weak absorption and strongly elongated phase functions has been carried. The simple modification of classical Ambarzumian’s — Chandrasekhar’s invariance principle indispensable for the receiving of new non-linear integral equations connected with azimuthal Fourier harmonics of generalized unified photometric function and photometric invariants has been used. These values join upgoing and downgoing radiation fields intensities making use of simple linear manner at arbitrary optical levels in mirror vision directions including fixed azimuthal angles and solar zenith distance. Parametrizations of obtained non-linear integral equations have demonstrated in the absence of reflecting underlying surfaces, placed at the lowest level of considered uniform slab, the possibility to express angular-spatial properties of unified photometrical function and appropriate photometrical invariants taking into account the phase functions strongly elongation near small scattering angles and small slab’s absorption with the help of primary scattered radiation field intensities and adaptive fitting multipliers. These functional adaptive corrections have been stipulated by uniform slab’s multiple light scattering and possess the clear physical interpretation. The use of mirror reflection (symmetry) principle, elaborated by the author, and conception of unified photometric function allows one to estimate the above-mentioned peculiarities of real environment’s phase functions in the framework of photometric invariants numerical modeling. An analysis of appropriate radiative modeling results has shown a dominating influence of primary light scattering in the formation of anisotropically scattered radiation fields of an arbitrary optical thickness uniform slab in the case of weak radiation absorption and strongly elongated phase functions.

Rotative maximal pattern: A local coloring descriptor for object classification and recognition

Inspired by the photometric invariance of color space, this paper proposes a simple yet powerful descriptor for object detection and recognition, called Rotative Maximal Pattern (RMP). The effectiveness of RMP comes from the two components: Rotatable Couple Templates (RCTs) with max pooling, and Normalized Histogram Intersection (NHI) with the theoretical guarantee. More concretely, RCTs are the combination of two templates to code the possible rotations. NHI serves as the similarity between two color histograms. We have conducted extensive experiments on INRIA pedestrian and Pascal VOC2007 data sets for object detection tasks; we also show that our approach leads to a promising performance on Caltech 101, Scene 15, UIUCsport and Stanford 40 action data sets.

Numerical Modeling of Uniform Slab Brightness Coefficients based on the Mirror Reflection Principle and Linear Singular Integral Equations Solutions

Based on the mirror reflection principle and solutions of modified linear singular integral equations, the numerical modeling of the unified exit function of the outside radiation field and photometrical invariants of brightness coefficients for a uniform slab of finite optical thickness has been carried out. The efficacy of applying the angular discretization method for problems of numerical modeling of outer radiation fields in the «atmosphere — underlying surface» system has been proved. This new approach allows generalizing the basic results in the particular case of a semi-infinite uniform slab. In this connection the main mathematical aspects and computational peculiarity of the numerical realization of the angular discretization method have been considered. Due to linearity of the used basic integral equations, the conducted analysis can be generalized to the case of scalar and polarized inner radiation fields taking into account the multiple anisotropic scattering of photons and their reflection from an arbitrary horizontally uniform underlying surface.

Robust lifelong visual tracking using compact binary feature with color attributes

In this paper, we address the problem of visual tracking where the target object undergo appearance variations due to illumination variation, occlusion, motion, background clutter and deformation. To deal with the significant appearance variations, we introduce color attributes into traditional shape feature at cell level, the new feature representation takes into consideration both its photometric invariance as well as its discriminative power. We construct compact binary code for the shape-color feature to reduce the high dimensions, and update the hash function in an online manner. A discriminative lifelong learning model is built to construct an appearance model that optimally separates the object from its surrounds. The lifelong learner uses the shared latent basis to transfer historical observations to simple classifier while a new frame arrives. Experimental results on tracking benchmark demonstrate that the proposed tracking algorithm outperforms state-of-the-art methods.

Visualizing Deep Convolutional Neural Networks Using Natural Pre-images

Image representations, from SIFT and bag of visual words to Convolutional Neural Networks (CNNs) are a crucial component of almost all computer vision systems. However, our understanding of them remains limited. In this paper we study several landmark representations, both shallow and deep, by a number of complementary visualization techniques. These visualizations are based on the concept of "natural pre-image", namely a natural-looking image whose representation has some notable property. We study in particular three such visualizations: inversion, in which the aim is to reconstruct an image from its representation, activation maximization, in which we search for patterns that maximally stimulate a representation component, and caricaturization, in which the visual patterns that a representation detects in an image are exaggerated. We pose these as a regularized energy-minimization framework and demonstrate its generality and effectiveness. In particular, we show that this method can invert representations such as HOG more accurately than recent alternatives while being applicable to CNNs too. Among our findings, we show that several layers in CNNs retain photographically accurate information about the image, with different degrees of geometric and photometric invariance.

A Single-Lobe Photometric Stereo Approach for Heterogeneous Material

Shape from shading with multiple light sources is an active research area, and a diverse range of approaches have been proposed in recent decades. However, devising a robust reconstruction technique still remains a challenging goal, as the image acquisition process is highly nonlinear. Recent Photometric Stereo variants rely on simplifying assumptions in order to make the problem solvable: light propagation is still commonly assumed to be uniform, and the Bidirectional Reflectance Distribution Function is assumed to be diffuse, with limited interest for specular materials. In this work, we introduce a well-posed formulation based on partial differential equations (PDEs) for a unified reflectance function that can model both diffuse and specular reflections. We base our derivation on ratio of images, which makes the model independent from photometric invariants and yields a well-posed differential problem based on a system of quasi-linear PDEs with discontinuous coefficients. In addition, we directly solve...

Realistic photometric stereo using partial differential irradiance equation ratios

Shape from shading with multiple light sources is an active research area and a diverse range of approaches have been proposed in the last decades. However, devising a robust reconstruction technique still remains a challenging goal due to several highly non-linear physical factors being involved in the image acquisition process. Recent attempts at tackling the problem via photometric stereo rely on simplified hypotheses in order to make the problem solvable. Light propagation is still commonly assumed to be uniformly oriented, and the BRDF assumed to be diffuse, with limited interest for materials giving specular reflection. Taking into account realistic point light sources, in this paper we introduce a well-posed formulation based on partial differential equations for both diffuse and specular shading models. We base our derivation on the popular approach of image ratios, which makes the model independent from photometric invariants. The practical effectiveness of our method is confirmed with a wide range of experiments on both synthetic and real data, where we compare favorably to the state of the art.

Improvements of the Environment Radiation Fields Modeling Numerical Methods on the Basis of Mirror Symmetry Principle

Fundamental properties of the angular-spatial symmetry of radiation fields in the uniform slab of a finite optical thickness are used for improvement of the numerical methods and algorithms of the classical radiative transfer theory. A new notion of so called photometrical invariants is introduced. The basic boundary-value problem of the radiative transfer theory is reformulated in new terms for the subsequent simplification of algorithms of numerical modeling methods such as spherical harmonics, discrete ordinates, Gauss-Seidel, Case and Hunt-Grant methods. This simplification leads to two-fold decrease of the ranks of linear algebraic equations with simultaneous reduction of numerical modeling intervals connected with angular and spatial variables.

Matching by Tone Mapping: Photometric Invariant Template Matching

A fast pattern matching scheme termed matching by tone mapping (MTM) is introduced which allows matching under nonlinear tone mappings. We show that, when tone mapping is approximated by a piecewise constant/linear function, a fast computational scheme is possible requiring computational time similar to the fast implementation of normalized cross correlation (NCC). In fact, the MTM measure can be viewed as a generalization of the NCC for nonlinear mappings and actually reduces to NCC when mappings are restricted to be linear. We empirically show that the MTM is highly discriminative and robust to noise with comparable performance capability to that of the well performing mutual information, but on par with NCC in terms of computation time.

On Differential Photometric Reconstruction for Unknown, Isotropic BRDFs

This paper presents a comprehensive theory of photometric surface reconstruction from image derivatives in the presence of a general, unknown isotropic BRDF. We derive precise topological classes up to which the surface may be determined and specify exact priors for a full geometric reconstruction. These results are the culmination of a series of fundamental observations. First, we exploit the linearity of chain rule differentiation to discover photometric invariants that relate image derivatives to the surface geometry, regardless of the form of isotropic BRDF. For the problem of shape-from-shading, we show that a reconstruction may be performed up to isocontours of constant magnitude of the gradient. For the problem of photometric stereo, we show that just two measurements of spatial and temporal image derivatives, from unknown light directions on a circle, suffice to recover surface information from the photometric invariant. Surprisingly, the form of the invariant bears a striking resemblance to optical flow; however, it does not suffer from the aperture problem. This photometric flow is shown to determine the surface up to isocontours of constant magnitude of the surface gradient, as well as isocontours of constant depth. Further, we prove that specification of the surface normal at a single point completely determines the surface depth from these isocontours. In addition, we propose practical algorithms that require additional initial or boundary information, but recover depth from lower order derivatives. Our theoretical results are illustrated with several examples on synthetic and real data.

Image region description using orthogonal combination of local binary patterns enhanced with color information

Visual content description is a key issue for machine-based image analysis and understanding. A good visual descriptor should be both discriminative and computationally efficient while possessing some properties of robustness to viewpoint changes and lighting condition variations. In this paper, we propose a new operator called the orthogonal combination of local binary patterns (denoted as OC-LBP) and six new local descriptors based on OC-LBP enhanced with color information for image region description. The aim is to increase both discriminative power and photometric invariance properties of the original LBP operator while keeping its computational efficiency. The experiments in three different applications show that the proposed descriptors outperform the popular SIFT, CS-LBP, HOG and SURF, and achieve comparable or even better performances than the state-of-the-art color SIFT descriptors. Meanwhile, the proposed descriptors provide complementary information to color SIFT, because a fusion of these two kinds of descriptors is found to perform clearly better than either of the two separately. Moreover, the proposed descriptors are about four times faster to compute than color SIFT.

Feature Detection by Color Tensor-Based Photometric Invariants

Although the majority of images are recorded in color format nowadays, computer vision research is still mostly restricted to luminance-based feature detection. In this paper, we combine the features based on the color tensor with photometric invariant derivatives to arrive at photometric invariant features. The combination of the photometric invariance theory and tensor based features allows for detection of a variety of features such as photometric invariant edges, corners. Experiments show that the proposed features are robust to scene incidental events and perform well in real-world scene.

Illumination-robust variational optical flow using cross-correlation

We address the problem of variational optical flow for video processing applications that need fast operation and robustness to drastic variations in illumination. Recently, a solution [1] has been proposed based on the photometric invariants of the dichromatic reflection model [2]. However, this solution is only applicable to colour videos with brightness variations. Greyscale videos, or colour videos with colour illumination changes cannot be adequately handled. We propose two illumination-robust variational methods based on cross-correlation that are applicable to colour and grey-level sequences and robust to brightness and colour illumination changes. First, we present a general implicit nonlinear scheme that assumes no particular analytical form of energy functional and can accommodate different image components and data metrics, including cross-correlation. We test the nonlinear scheme on standard synthetic data with artificial brightness and colour effects added and conclude that cross-correlation is robust to both kinds of illumination changes. Then we derive a fast linearised numerical scheme for cross-correlation based variational optical flow. We test the linearised algorithm on challenging data and compare it to a number of state-of-the-art variational flow methods.

A Solution of the Dichromatic Model for Multispectral Photometric Invariance

In this paper, we address the problem of photometric invariance in multispectral imaging making use of an optimisation approach based upon the dichromatic model. In this manner, we cast the problem of recovering the spectra of the illuminant, the surface reflectance and the shading and specular factors in a structural optimisation setting. Making use of the additional information provided by multispectral imaging and the structure of image patches, we recover the dichromatic parameters of the scene. To do this, we formulate a target cost function combining the dichromatic error and the smoothness priors for the surfaces under study. The dichromatic parameters are recovered through minimising this cost function in a coordinate descent manner. The algorithm is quite general in nature, admitting the enforcement of smoothness constraints and extending in a straightforward manner to trichromatic settings. Moreover, the objective function is convex with respect to the subset of variables to be optimised in each alternating step of the minimisation strategy. This gives rise to an optimal closed-form solution for each of the iterations in our algorithm. We illustrate the effectiveness of our method for purposes of illuminant spectrum recovery, skin recognition, material clustering and specularity removal. We also compare our results to a number of alternatives.

Approximation-free running SVD and its application to motion detection

In different tasks such as adaptive background modelling, the Singular Value Decomposition (SVD) has to be applied in running fashion. Typically, this happens when the SVD is used in a sliding spatial or temporal data window. Each time the window moves on, the SVD should be calculated in the batch mode from scratch, or re-calculated using the previous solution. When the data matrix is relatively small, the batch mode is fast enough. When the matrix is large, the batch mode is prohibitive and fast re-calculation is needed. In background modelling for video surveillance, the data matrix is formed by consecutive frames and can be fairly large. The existing PCA or SVD based approaches use approximate solutions and divide the frame into small blocks to speed up the re-calculation. We present a fast approximation-free solution for running SVD and apply it to moving object detection in video. Photometric invariants are used for dynamic background with frequent shading. The proposed approach is compared on challenging sequences to the Multiple Gaussian Method which is a standard in adaptive background modelling.

Color Subspaces as Photometric Invariants

Complex reflectance phenomena such as specular reflections confound many vision problems since they produce image ‘features’ that do not correspond directly to intrinsic surface properties such as shape and spectral reflectance. A common approach to mitigate these effects is to explore functions of an image that are invariant to these photometric events. In this paper we describe a class of such invariants that result from exploiting color information in images of dichromatic surfaces. These invariants are derived from illuminant-dependent ‘subspaces’ of RGB color space, and they enable the application of Lambertian-based vision techniques to a broad class of specular, non-Lambertian scenes. Using implementations of recent algorithms taken from the literature, we demonstrate the practical utility of these invariants for a wide variety of applications, including stereo, shape from shading, photometric stereo, material-based segmentation, and motion estimation.

Robust photometric invariant features from the color tensor

Luminance-based features are widely used as low-level input for computer vision applications, even when color data is available. The extension of feature detection to the color domain prevents information loss due to isoluminance and allows us to exploit the photometric information. To fully exploit the extra information in the color data, the vector nature of color data has to be taken into account and a sound framework is needed to combine feature and photometric invariance theory. In this paper, we focus on the structure tensor, or color tensor, which adequately handles the vector nature of color images. Further, we combine the features based on the color tensor with photometric invariant derivatives to arrive at photometric invariant features. We circumvent the drawback of unstable photometric invariants by deriving an uncertainty measure to accompany the photometric invariant derivatives. The uncertainty is incorporated in the color tensor, hereby allowing the computation of robust photometric invariant features. The combination of the photometric invariance theory and tensor-based features allows for detection of a variety of features such as photometric invariant edges, corners, optical flow, and curvature. The proposed features are tested for noise characteristics and robustness to photometric changes. Experiments show that the proposed features are robust to scene incidental events and that the proposed uncertainty measure improves the applicability of full invariants.