Scene Irradiance Research Articles

Source camera attribution via PRNU emphasis: Towards a generalized multiplicative model

The photoresponse non-uniformity (PRNU) is a camera-specific pattern, which acts as unique fingerprint of any imaging sensor and thus is widely adopted to solve multimedia forensics problems such as device identification or forgery detection. Customarily, the theoretical analysis of this fingerprint relies on a multiplicative model for the denoising residuals. This setup assumes that the nonlinear mapping from scene irradiance to preprocessed luminance, that is, the composition of the Camera Response Function (CRF) with the digital preprocessing pipeline, is a gamma correction. However, this assumption seldom holds in practice. In this paper, we improve the multiplicative model by including the influence of this nonlinear mapping, termed PRNU emphasis, on the denoising residuals. On the theoretical side, we conduct first an exploratory analysis to show that the response of typical cameras deviates from a gamma correction. We also propose a regularized least squares estimator to measure this effect. On the practical side, we argue that the PRNU emphasis is especially beneficial for a source camera attribution problem with cropped images. We back our argument with an extensive empirical evaluation using different denoisers and both compressed and uncompressed images. This new model will pave the way to future PRNU estimators and detectors.

A lightweight exposure bracketing strategy for HDR imaging without access to camera raw

A lightweight learning-based exposure bracketing strategy is proposed in this paper for high dynamic range (HDR) imaging without access to camera RAW. Some low-cost, power-efficient cameras, such as webcams, video surveillance cameras, sport cameras, mid-tier cellphone cameras, and navigation cameras on robots, can only provide access to 8-bit low dynamic range (LDR) images. Exposure fusion is a classical approach to capture HDR scenes by fusing images taken with different exposures into a 8-bit tone-mapped HDR image. A key question is what the optimal set of exposure settings are to cover the scene dynamic range and achieve a desirable tone. The proposed lightweight neural network predicts these exposure settings for a 3-shot exposure bracketing, given the input irradiance information from 1) the histograms of an auto-exposure LDR preview image, and 2) the maximum and minimum levels of the scene irradiance. Without the processing of the preview image streams, and the circuitous route of first estimating the scene HDR irradiance and then tone-mapping to 8-bit images, the proposed method gives a more practical HDR enhancement for real-time and on-device applications. Experiments on a number of challenging images reveal the advantages of our method in comparison with other state-of-the-art methods qualitatively and quantitatively.

Representing Camera Response Function by a Single Latent Variable and Fully Connected Neural Network

Modelling the mapping from scene irradiance to image intensity is essential for many computer vision tasks. Such mapping is known as the camera response. Most digital cameras use a nonlinear function to map irradiance, as measured by the sensor to an image intensity used to record the photograph. Modelling of the response is necessary for the nonlinear calibration. In this paper, a new high-performance camera response model that uses a single latent variable and fully connected neural network is proposed. The model is produced using unsupervised learning with an autoencoder on real-world (example) camera responses. Neural architecture searching is then used to find the optimal neural network architecture. A latent distribution learning approach was introduced to constrain the latent distribution. The proposed model achieved state-of-the-art CRF representation accuracy in a number of benchmark tests, but is over twice as fast as the best current models when performing the maximum likelihood estimation during camera response calibration due to the simple yet efficient model representation.

Free-viewpoint Indoor Neural Relighting from Multi-view Stereo

We introduce a neural relighting algorithm for captured indoors scenes, that allows interactive free-viewpoint navigation. Our method allows illumination to be changed synthetically, while coherently rendering cast shadows and complex glossy materials. We start with multiple images of the scene and a three-dimensional mesh obtained by multi-view stereo (MVS) reconstruction. We assume that lighting is well explained as the sum of a view-independent diffuse component and a view-dependent glossy term concentrated around the mirror reflection direction. We design a convolutional network around input feature maps that facilitate learning of an implicit representation of scene materials and illumination, enabling both relighting and free-viewpoint navigation. We generate these input maps by exploiting the best elements of both image-based and physically based rendering. We sample the input views to estimate diffuse scene irradiance, and compute the new illumination caused by user-specified light sources using path tracing. To facilitate the network's understanding of materials and synthesize plausible glossy reflections, we reproject the views and compute mirror images . We train the network on a synthetic dataset where each scene is also reconstructed with MVS. We show results of our algorithm relighting real indoor scenes and performing free-viewpoint navigation with complex and realistic glossy reflections, which so far remained out of reach for view-synthesis techniques.

A high-dynamic-range visual sensing method for feature extraction of welding pool based on adaptive image fusion

The high dynamic range existing in arc welding with high energy density challenges most of the industrial cameras, causing badly exposed pixels in the captured images and bringing difficulty to the feature detection from internal weld pool. This paper proposes a novel monitoring method called adaptive image fusion, which increases the amount of information contained in the welding images and can be realized on the common industrial camera with low cost. It combines original images captured rapidly by the camera into one fused image, and the setting of these images is based on the real-time analysis of realistic scene irradiance during the welding process. Experiments are carried out to find out the operating window for the adaptive image fusion method, providing the rules for getting a fused image with as much as information as possible. The comparison between the imaging with or without the proposed method proves that the fused image has a wider dynamic range and includes more useful features from the weld pool. The improvement is also verified by extracting both the internal and external features of weld pool within a same fused image with proposed method. The results show that the proposed method can adaptively expand the dynamic range of visual monitoring system with low cost, which benefits the feature extraction from the internal weld pool.

Comparison of Absorbed and Intercepted Fractions of PAR for Individual Trees Based on Radiative Transfer Model Simulations

The fraction of absorbed photosynthetically active radiation (fAPAR) is a key parameter for estimating the gross primary production (GPP) of trees. For continuous, dense forest canopies, fAPAR, is often equated with the intercepted fraction, fIPAR. This assumption is not valid for individual trees in urban environments or parkland settings where the canopy is sparse and there are well-defined tree crown boundaries. Here, the distinction between fAPAR and fIPAR can be strongly influenced by the background and large illumination variations due to multi-scattering and shadows of buildings. This study investigates the radiative budget of PAR bands using a coupled leaf-canopy radiative transfer model (PROSPECT-DART), considering a suite of tropical tree species over a wide range of assumed leaf chlorophyll contents. The analyses simulate hyperspectral images (5 nm bandwidth) of individual tree crowns for the selected background (concrete vs. grass) and illumination conditions. We then use an artificial neural network-based method to partition sunlit vs. shaded leaves within each crown, as the latter have lower fAPAR and fIPAR values. Our results show fAPAR of sunlit leaves decreases with the ratio of diffuse to direct scene irradiance (SKYL), while SKYL has minimal influence for shaded leaves. Both fAPAR and fIPAR decrease at more oblique solar zenith angles (SZA). Higher values of fAPAR and fIPAR occur with concrete backgrounds and the influence of the background is larger at higher diffuse ratio and solar zenith angles. The results show that fIPAR is typically 6–9% higher than fAPAR, and up to 14% higher for sunlit leaves with a concrete background at SKYL = 0. The differences between the fIPAR and fAPAR also depend on the health condition of the leaves, such as chlorophyll content. This study can improve the understanding of urban individual trees fAPAR/fIPAR and facilitate the development of protocols for fAPAR field measurements.

DART radiative transfer modelling for sloping landscapes

Topography is one of the key factors that impact remotely sensed data and their interpretation. Indeed, combined with the viewing geometry and neighbour effects, it strongly affects the direct, diffuse and multi-scattered scene irradiance, which in turn impacts the radiative budget and remote sensing signals of the landscapes. The increased availability of digital elevation models (DEM) and the advancement of 3D radiative transfer (RT) models allow us to better address these topographic effects. DART (Discrete Anisotropic Radiative Transfer) is one of the most accurate and comprehensive 3D RT models that simulate remote sensing observations of natural and urban landscapes with topography and atmosphere. It simulates environmental effects (i.e., impact of adjacent landscape on the observed landscape) using a so-called infinite slope mode that infinitely duplicates the observed landscape while ensuring the continuity of slope and altitude at the DEM edges. Up to DART version 5.7.4, this mode was slightly inaccurate and computer intensive, depending on the topography. This paper presents an innovative modelling strategy that greatly improves it in terms of accuracy, image quality and computer efficiency. For that, a fictive auxiliary oblique plane, adapted to the landscape topography, is introduced for managing the scene illumination, the Earth-Atmosphere coupling and the storage of the radiation that exits the scene before being projected onto the sensor plane. Improvements and validations are illustrated both visually and quantitatively by DART images, radiometric products and radiative budget. For example, the observed reflectance of a Lambertian slope is equal to the expected analytical value. In addition, the solar plane reflectance of a forest on a mountain slope (experimental scene) has an average error of about 0.01% relative to the reflectance of the same forest stand in the reference scene (i.e., nine duplications of the experimental scene). This new modelling is already integrated in the official DART version (https://dart.omp.eu).

Turning a Digital Camera into an Absolute 2D Tele‐Colorimeter

AbstractWe present a simple and effective technique for absolute colorimetric camera characterization, invariant to changes in exposure/aperture and scene irradiance, suitable in a wide range of applications including image‐based reflectance measurements, spectral pre‐filtering and spectral upsampling for rendering, to improve colour accuracy in high dynamic range imaging. Our method requires a limited number of acquisitions, an off‐the‐shelf target and a commonly available projector, used as a controllable light source, other than the reflected radiance to be known. The characterized camera can be effectively used as a 2D tele‐colorimeter, providing the user with an accurate estimate of the distribution of luminance and chromaticity in a scene, without requiring explicit knowledge of the incident lighting power spectra. We validate the approach by comparing our estimated absolute tristimulus values (XYZ data in ) with the measurements of a professional 2D tele‐colorimeter, for a set of scenes with complex geometry, spatially varying reflectance and light sources with very different spectral power distribution.

Demonstration of the CDMA-mode CAOS smart camera.

Demonstrated is the code division multiple access (CDMA)-mode coded access optical sensor (CAOS) smart camera suited for bright target scenarios. Deploying a silicon CMOS sensor and a silicon point detector within a digital micro-mirror device (DMD)-based spatially isolating hybrid camera design, this smart imager first engages the DMD starring mode with a controlled factor of 200 high optical attenuation of the scene irradiance to provide a classic unsaturated CMOS sensor-based image for target intelligence gathering. Next, this CMOS sensor provided image data is used to acquire a focused zone more robust un-attenuated true target image using the time-modulated CDMA-mode of the CAOS camera. Using four different bright light test target scenes, successfully demonstrated is a proof-of-concept visible band CAOS smart camera operating in the CDMA-mode using up-to 4096 bits length Walsh design CAOS pixel codes with a maximum 10 KHz code bit rate giving a 0.4096 seconds CAOS frame acquisition time. A 16-bit analog-to-digital converter (ADC) with time domain correlation digital signal processing (DSP) generates the CDMA-mode images with a 3600 CAOS pixel count and a best spatial resolution of one micro-mirror square pixel size of 13.68 μm side. The CDMA-mode of the CAOS smart camera is suited for applications where robust high dynamic range (DR) imaging is needed for un-attenuated un-spoiled bright light spectrally diverse targets.

An Image Defogging Approach Based on a Constrained Energy Functional under Bayesian and Variation Theories

Hazy images produce negative influences on visual applications in the open air since they are in poor visibility with low contrast and whitening color. Numerous existing methods tend to derive a totally rough estimate of scene depth. Unlike previous work, we focus on the probability distribution of depth that is considered as a scene prior. Inspired by the denoising work of multiplicative noises, the inverse problem for hazy removal is recast as deriving the optimal difference between scene irradiance and the airlight from a constrained energy functional under Bayesian and variation theories. Logarithmic maximum a posteriori estimator and a mixed regularization term are introduced to formulate the energy functional framework where the regularization parameter is adaptively selected. The airlight, another unknown quantity, is inferred precisely under a geometric constraint and dark channel prior. With these two estimates, scene irradiance can be recovered. The experimental results on a series of hazy images reveal that, in comparison with several relevant and most state-of-the-art approaches, the proposed method outperforms in terms of vivid color and appropriate contrast.

Regularization approach to scene-based nonuniformity correction

Various scene-based nonuniformity correction (SBNUC) methods have been proposed to diminish the residual nonuniformity (RNU) of the infrared focal plane array (IRFPA) sensors. Most existing SBNUC techniques require a relatively large number of image frames to reduce the RNU. In some applications, however, there is not enough time for capturing a large number of image frames prior to the camera operation, or only several image frames are available to users. A new scene-based approach that can correct the RNU using only several image frames is proposed. The proposed method formulates the SBNUC process as an energy minimization problem. In the proposed energy function, we introduce regularization terms for the parameter regarding the responsivity of the IRFPA as well as for the true scene irradiance. Correction results are obtained by minimizing the energy function using a numerical technique. Experimental results demonstrate the effectiveness of the proposed method.

Simultaneous High Dynamic Range and Superresolution Imaging without Regularization

Modern digital camera sensors capture only a limited amplitude and frequency range of the irradiance of a scene. A recent trend is to acquire and combine multiple images to raise the quality of the final image. Multi-image techniques are used in high dynamic range processing, where multiple exposure times are used to reconstruct the full range of irradiance. With multi-image superresolution, the difference between sampling grids caused by the motion of the camera is used to generate a high resolution image. It is possible to combine these two processes into one to generate a superresolved image with a full dynamic range. In this paper, we study the reconstruction error of high dynamic range superresolution imaging without regularization, under an affine motion hypothesis. From this study, we deduce a strategy for choosing the number of images and the exposure times, which makes the unregularized problem well conditioned. With these acquisition parameters, if the affine motion hypothesis holds and sufficiently long exposure time is available, the recovery of all the amplitude and frequency content of the scene irradiance is guaranteed.

Linearisation of RGB Camera Responses for Quantitative Image Analysis of Visible and UV Photography: A Comparison of Two Techniques

Linear camera responses are required for recovering the total amount of incident irradiance, quantitative image analysis, spectral reconstruction from camera responses and characterisation of spectral sensitivity curves. Two commercially-available digital cameras equipped with Bayer filter arrays and sensitive to visible and near-UV radiation were characterised using biexponential and Bézier curves. Both methods successfully fitted the entire characteristic curve of the tested devices, allowing for an accurate recovery of linear camera responses, particularly those corresponding to the middle of the exposure range. Nevertheless the two methods differ in the nature of the required input parameters and the uncertainty associated with the recovered linear camera responses obtained at the extreme ends of the exposure range. Here we demonstrate the use of both methods for retrieving information about scene irradiance, describing and quantifying the uncertainty involved in the estimation of linear camera responses.

A two-frame approach for scene-based nonuniformity correction in array sensors

This paper introduces a new scene-based technique to correct the fixed-pattern noise (FPN) in array sensors. This method registers a pair of image frames exhibiting small relative scene translation and then the noise pattern can be reconstructed using the constrained least-squares estimation. The key advantage of this technique is that the accurate estimates of the bias nonuniformity can be obtained with only two images, without imposing any assumptions on the structure of the FPN. Besides, the method works on almost static scene, and therefore does not require larger scale global motion and statistical assumptions on the scene irradiance. We test our method on synthetically generated FPN as well as with real infrared data, and experimental results demonstrate the significant reduction in FPN, validating the effectiveness of our approach. Finally, we validate the feasibility and validity of using the proposed method as a first step fostering the success of more sophisticated registration-based time-evolving correction algorithms.

Non-Uniform Deblurring in HDR Image Reconstruction

Hand-held cameras inevitably result in blurred images caused by camera-shake, and even more so in high dynamic range imaging applications where multiple images are captured over a wide range of exposure settings. The degree of blurring depends on many factors such as exposure time, stability of the platform, and user experience. Camera shake involves not only translations but also rotations resulting in nonuniform blurring. In this paper, we develop a method that takes input non-uniformly blurred and differently exposed images to extract the deblurred, latent irradiance image. We use transformation spread function (TSF) to effectively model the blur caused by camera motion. We first estimate the TSFs of the blurred images from locally derived point spread functions by exploiting their linear relationship. The scene irradiance is then estimated by minimizing a suitably derived cost functional. Two important cases are investigated wherein 1) only the higher exposures are blurred and 2) all the captured frames are blurred.

Turbid Scene Enhancement Using Multi-Directional Illumination Fusion

Ambient light is strongly attenuated in turbid media. Moreover, natural light is often more highly attenuated in some spectral bands, relative to others. Hence, imaging in turbid media often relies heavily on artificial sources for illumination. Scenes irradiated by an off-axis single point source have enhanced local object shadow edges, which may increase object visibility. However, the images may suffer from severe nonuniformity, regions of low signal (being distant from the source), and regions of strong backscatter. On the other hand, simultaneously illuminating the scene from multiple directions increases the backscatter and fills-in shadows, both of which degrade local contrast. Some previous methods tackle backscatter by scanning the scene, either temporally or spatially, requiring a large number of frames. We suggest using a few frames, in each of which wide field scene irradiance originates from a different direction. This way, shadow contrast can be maintained and backscatter can be minimized in each frame, while the sequence at large has a wider, more spatially uniform illumination. The frames are then fused by post processing to a single, clearer image. We demonstrate significant visibility enhancement underwater using as little as two frames.

Active Polarization Descattering

Vision in scattering media is important but challenging. Images suffer from poor visibility due to backscattering and attenuation. Most prior methods for scene recovery use active illumination scanners (structured and gated), which can be slow and cumbersome, while natural illumination is inapplicable to dark environments. The current paper addresses the need for a non-scanning recovery method, that uses active scene irradiance. We study the formation of images under widefield artificial illumination. Based on the formation model, the paper presents an approach for recovering the object signal. It also yields rough information about the 3D scene structure. The approach can work with compact, simple hardware, having active widefield, polychromatic polarized illumination. The camera is fitted with a polarization analyzer. Two frames of the scene are taken, with different states of the analyzer or polarizer. A recovery algorithm follows the acquisition. It allows both the backscatter and the object reflection to be partially polarized. It thus unifies and generalizes prior polarization-based methods, which had assumed exclusive polarization of either of these components. The approach is limited to an effective range, due to image noise and illumination falloff. Thus, the limits and noise sensitivity are analyzed. We demonstrate the approach in underwater field experiments.

Brightness calibration of charge-coupled device camera systems

Brightness calibration of vision sensors is an important step in developing machine vision applications, yet it is often ignored and overlooked. It enables the accurate representation of image intensity readings with respect to a common linear brightness scale that is proportional to the scene irradiance. We present a brightness calibration procedure for CCD camera systems. We also describe the characteristics of today's CCD cameras, which are key to their effective use as well as the visual brightness sensory information in industrial applications. Aspects of brightness measurement are discussed. Results from calibrating a stereo CCD camera system and testing the calibrated system are also presented and discussed.