Color Filter Array Research Articles

Spatio-spectral binary patterns based on multispectral filter arrays for texture classification.

To discriminate gray-level texture images, spatial texture descriptors can be extracted using the local binary pattern (LBP) operator. This operator has been extended to color images at the expense of increased memory and computation requirements. Some authors propose to compute texture descriptors directly from raw images provided through a Bayer color filter array, which both avoids the demosaicking step and reduces the descriptor size. Recently, multispectral snapshot cameras have emerged to sample more than three wavelength bands using a multispectral filter array. Such cameras provide a raw image in which a single spectral channel value is available at each pixel. In this paper we design a local binary pattern operator that jointly extracts the spatial and spectral texture information directly from a raw image. Extensive experiments on a large dataset show that the proposed descriptor has both reduced computation cost and high discriminative power with regard to classical LBP descriptors applied to demosaicked images.

Colour filter array demosaicking over compression through modified grey wolf optimization technique

ABSTRACTThe three colour intensity values red, green and blue are acquired in a single pixel on using a multi-sensor camera. The single sensor camera uses the colour filter array to grab only one intensity value in a pixel location. The reconstruction of the so-called raw image into a full colour image is called demosaicking. This paper proposes a novel framework for effective demosaicking combined with the compression over the input image using a modified grey wolf optimization algorithm. The demosaicking process is efficient in applying the Modified Grey Wolf Optimized wavelet compression to optimally select the coefficients and Modified Huffman Coding technique to the image before demosaicking. The proposed technique is evaluated by the Peak Signal Noise Ratio, Structural Similarity Index Measure and Feature Similarity Index Measure values and has proved both subjectively and objectively to have better demosaicking results.

Lossless and reversible colour space transformation for Bayer colour filter array images

We present two variants of a colour space transformation algorithm to encode Bayer colour filter array images that are based on integer coefficients; as a result, the algorithms are fully lossless and reversible in nature. These transformation algorithms are derived using an optimisation model that reduces the spectral redundancy of Bayer colour components, which results in lower prediction error variance and inter-colour correlation. These methods, known as optimum reversible colour space transform (ORCT-1 and ORCT-2), improve the lossless bitrate of low complexity prediction model without using high complexity interpolation and inter-colour prediction scheme. Extensive experimentation is performed using five sets of test images for different lossless compression algorithms: JPEG-LS, JPEG-2000 and JPEG-XR. Experimental results show that, in all cases, the proposed schemes perform competitively with other methods with lower computational complexity, which makes them suitable for low-cost imaging applications.

Hyperspectral demosaicking and crosstalk correction using deep learning

Precision agriculture using unmanned aerial vehicles (UAVs) is gaining popularity. These UAVs provide a unique aerial perspective suitable for inspecting agricultural fields. With the use of hyperspectral cameras, complex inspection tasks are being automated. Payload constraints of UAVs require low weight and small hyperspectral cameras; however, such cameras with a multispectral color filter array suffer from crosstalk and a low spatial resolution. The research described in this paper aims to reduce crosstalk and to increase spatial resolution using convolutional neural networks. We propose a similarity maximization framework which is trained to perform end-to-end demosaicking and crosstalk-correction of a $$4 \times 4$$ raw mosaic image. The proposed method produces a hyperspectral image cube with 16 times the spatial resolution of the original cube while retaining a median structural similarity (SSIM) index of 0.85 (compared to an SSIM of 0.55 when using bilinear interpolation). Furthermore, this paper provides insight into the beneficial effects of crosstalk for hyperspectral demosaicking and gives best practices for several architectural and hyperparameter variations as well as a theoretical reasoning behind certain observations.

Hyperspectral Image Super-Resolution with a Mosaic RGB Image.

Recently, many hyperspectral (HS) image superresolution methods that merge a low spatial resolution HS image and a high spatial resolution three-channel RGB image have been proposed in spectral imaging. A largely ignored fact is that most existing commercial RGB cameras capture high resolution images by a single CCD/CMOS sensor equipped with a color filter array (CFA). In this paper, we account for the common imaging mechanism of commercial RGB cameras, and propose to use a mosaic RGB image for HS image super-resolution, which prevents demosaicing error and thus its propagation into the HS image super-resolution results. We design a proper nonlocal low-rank regularization to exploit the intrinsic properties - rich self-repeating patterns and high correlation across spectra - within HS images of natural scenes, and formulate the HS image super-resolution task into a variational optimization problem, which can be efficiently solved via the alternating direction method of multipliers (ADMM). The effectiveness of the proposed method has been evaluated on two benchmark datasets, demonstrating that the proposed method can provide substantial improvement over the current state-of-the-art HS image superresolution methods without considering the mosaicing effect. Finally, we show that our method can also perform well in the real capture system.

A hybrid technique to discriminate Natural Images, Computer Generated Graphics Images, Spliced, Copy Move tampered images and Authentic images by using features and ELM classifier

The paper presents a novel technique in the field of image forensics. The proposed algorithm is able to differentiate between computer generated images (CGIs), natural images (NIs), spliced images, copy moved tampered images and authentic images. The algorithm is also able to localize the tampered region in spliced and copy moved tampered images. At present, most of the detection techniques are able to encounter only one type of tampering at a time but the proposed algorithm can encounter all the three types of tampering simultaneously. The algorithm has two phases, in the first phase the algorithm has used laplacian of gaussian and autocorrelation to differentiate CGIs from NIs. In the second phase, the algorithm has used estimation of color filter array pattern and sensor noise to detect spliced region. The Grey level co-occurrence matrix (GLCM) has used to detect copy moved tampered area. Additionally, the algorithm can detect such tampering in presence of Gaussian blur, JPEG compression and scaling. The algorithm is also able to localize rotated and scaled portions of copy move tampered area. At present, none of the research papers encounter all these three problems simultaneously. The efficacy of the proposed algorithm has been verified through extensive simulation.

Low Cost Edge Sensing for High Quality Demosaicking.

Digital cameras that use Color Filter Arrays (CFA) entail a demosaicking procedure to form full RGB images. To digital camera industry, demosaicking speed is as important as demosaicking accuracy, because camera users have been accustomed to viewing captured photos instantly. Moreover, the cost associated with demosaicking should not go beyond the cost saved by using CFA. For this purpose, we revisit the classical Hamilton-Adams (HA) algorithm, which outperforms many sophisticated techniques in both speed and accuracy. Our analysis shows that the HA pipeline is highly efficient to exploit the originally captured data, but its oversimplified inter- and intra-channel smoothness formulation hinders its accuracy. We therefore propose a very low cost edge sensing scheme, which guides demosaicking by a logistic functional of the difference between directional variations. We extensively compare our algorithm with 27 demosaicking algorithms by running their open source codes on benchmark datasets. Compared to methods of similar computational cost, our method achieves substantially higher accuracy; Whereas compared to methods of similar accuracy, our method has significantly lower cost. On test images of currently popular resolution, the quality of our algorithm is comparable to top performers, yet our speed is tens of times faster. Source code for this work will be released with paper publication.

Camera-Aware Multi-Resolution Analysis for Raw Image Sensor Data Compression.

We propose novel lossless and lossy compression schemes for color filter array (CFA) sampled images based on the Camera-A ware Multi-Resolution Analysis, or CAMRA. Specifically, by CAMRA we refer to modifications that we make to wavelet transform of CFA sampled images in order to achieve a very high degree of decorrelation at the finest scale wavelet coefficients; and a series of color processing steps applied to the coarse scale wavelet coefficients, aimed at limiting the propagation of lossy compression errors through the subsequent camera processing pipeline. We validated our theoretical analysis and the performance of the proposed compression schemes using the images of natural scenes captured in a raw format. The experimental results verify that our proposed methods improve coding efficiency relative to the standard and the state-of-the-art compression schemes for CFA sampled images.

Intercolor-Filter Crosstalk Model for Image Sensors With Color Filter Array

A color crosstalk model for an area image sensor with a color filter (CF) array is proposed, which quantifies the crosstalk in a physically intuitive manner by introducing “inter-CF optical crosstalk (inter-CF crosstalk).” The model separates the structural design originated crosstalk from the CF material itself, revealing the sole characteristics of microstructures on a photodiode. A constrained nonlinear estimation is used to solve the model equation of system matrices with an objective function focusing on matching between the modeled quantum efficiency spectra and measured ones at both the peak and crosstalk regions. The model is applied to 1.1- $\mu \text{m}$ backside-illuminated CMOS image sensors having four different kinds of optical structures on the photodiode of the same design, and successfully provides the ground of comparison with excellent matching between the models and the measurement results.

Sensitivity and Resolution Improvement in RGBW Color Filter Array Sensor.

Recently, several red-green-blue-white (RGBW) color filter arrays (CFAs), which include highly sensitive W pixels, have been proposed. However, RGBW CFA patterns suffer from spatial resolution degradation owing to the sensor composition having more color components than the Bayer CFA pattern. RGBW CFA demosaicing methods reconstruct resolution using the correlation between white (W) pixels and pixels of other colors, which does not improve the red-green-blue (RGB) channel sensitivity to the W channel level. In this paper, we thus propose a demosaiced image post-processing method to improve the RGBW CFA sensitivity and resolution. The proposed method decomposes texture components containing image noise and resolution information. The RGB channel sensitivity and resolution are improved through updating the W channel texture component with those of RGB channels. For this process, a cross multilateral filter (CMF) is proposed. It decomposes the smoothness component from the texture component using color difference information and distinguishes color components through that information. Moreover, it decomposes texture components, luminance noise, color noise, and color aliasing artifacts from the demosaiced images. Finally, by updating the texture of the RGB channels with the W channel texture components, the proposed algorithm improves the sensitivity and resolution. Results show that the proposed method is effective, while maintaining W pixel resolution characteristics and improving sensitivity from the signal-to-noise ratio value by approximately 4.5 dB.

A Markov based image forgery detection approach by analyzing CFA artifacts

The image acquisition device, the light is filtered through a Color Filter Array (CFA), where each pixel captures only one color (from Red, Green, and Blue), while others are calibrated. This process is known as interpolation process, and the artifacts introduced are called CFA or interpolation artifacts. The structure of these artifacts in the image is disturbed while a forgery is introduced in an image. In this paper, a high-order statistical approach is proposed to detect the inconsistencies in the artifacts of different parts of the image to expose any forgery present. The Markov Transition Probability Matrix (MTPM) is employed to develop various features that will detect the presence or absence of CFA artifacts in a particular region of the image. The Markov random process is applied because it provides an enhanced efficiency and reduced computational complexity for the forgery detection model. The algorithm is tested on 2 × 2 pixel block of the image which provides the results of a fine quality. There is no prior information of the location of the forged region of the image. The algorithm is tested on various images, taken from various social networking websites. The proposed forgery detection technique outperforms the existing state-of-the-art techniques for the different forgery scenarios by providing an average accuracy of 90.58%.

Multi-pulse shadowgraphic RGB illumination and detection for flow tracking

This work demonstrates the application of a multi-color LED and a consumer color camera for visualizing phase boundaries in two-phase flows, in particular for particle tracking velocimetry. The LED emits a sequence of short light pulses, red, green, then blue (RGB), and through its color-filter array, the camera captures all three pulses on a single RGB frame. In a backlit configuration, liquid droplets appear as shadows in each color channel. Color reversal and color cross-talk correction yield a series of three frozen-flow images that can be used for further analysis, e.g., determining the droplet velocity by particle tracking. Three example flows are presented, solid particles suspended in water, the penetrating front of a gasoline direct-injection spray, and the liquid break-up region of an “air-assisted” nozzle. Because of the shadowgraphic arrangement, long path lengths through scattering media lower image contrast, while visualization of phase boundaries with high resolution is a strength of this method. Apart from a pulse-and-delay generator, the overall system cost is very low.

Multiwavelength Absolute Phase Retrieval from Noisy Diffractive Patterns: Wavelength Multiplexing Algorithm

We study the problem of multiwavelength absolute phase retrieval from noisy diffraction patterns. The system is lensless with multiwavelength coherent input light beams and random phase masks applied for wavefront modulation. The light beams are formed by light sources radiating all wavelengths simultaneously. A sensor equipped by a Color Filter Array (CFA) is used for spectral measurement registration. The developed algorithm targeted on optimal phase retrieval from noisy observations is based on maximum likelihood technique. The algorithm is specified for Poissonian and Gaussian noise distributions. One of the key elements of the algorithm is an original sparse modeling of the multiwavelength complex-valued wavefronts based on the complex-domain block-matching 3D filtering. Presented numerical experiments are restricted to noisy Poissonian observations. They demonstrate that the developed algorithm leads to effective solutions explicitly using the sparsity for noise suppression and enabling accurate reconstruction of absolute phase of high-dynamic range.

Deep Joint Design of Color Filter Arrays and Demosaicing

AbstractWe present a convolutional neural network architecture for performing joint design of color filter array (CFA) patterns and demosaicing. Our generic model allows the training of CFAs of arbitrary sizes, optimizing each color filter over the entire RGB color space. The patterns and algorithms produced by our method provide high‐quality color reconstructions. We demonstrate the effectiveness of our approach by showing that its results achieve higher PSNR than the ones obtained with state‐of‐the‐art techniques on all standard demosaicing datasets, both for noise‐free and noisy scenarios. Our method can also be used to obtain demosaicing strategies for pre‐defined CFAs, such as the Bayer pattern, for which our results also surpass even the demosaicing algorithms specifically designed for such a pattern.

P‐153: High Contrast Ratio LCD with an In‐cell Polarizer

We develop a rigorous model to simulate an LCD's contrast ratio (CR) and viewing angle by considering the depolarization effect. To improve CR, an in‐cell polarizer is added between LC layer and color filter array. The obtained maximum CR of a multi‐domain vertical alignment LCD is boosted to > 20,000:1.

An Effective Directional Residual Interpolation Algorithm for Color Image Demosaicking

In this paper, we propose an effective directional Bayer color filter array (CFA) demosaicking algorithm based on residual interpolation (RI). The proposed directional interpolation algorithm aims to reduce computational complexity and get more accurate interpolated pixel values in the complex edge areas. We use the horizontal and vertical weights to combine and smooth color difference estimations. Compared with four directional weights in minimized Laplacian residual interpolation, the proposed algorithm not only guarantees the quality of color images but also reduces the computational complexity. Generally, the directional estimations may be inaccurately calculated because of the false edge information in irregular edges. We alleviate it by using a new method to calculate the directional color difference estimations. Experimental results show that the proposed algorithm provides outstanding performance compared with some previous algorithms, especially in the complex edge areas. In addition, it has lower computational complexity and better visual effect.

Optimized Multi-Spectral Filter Array Based Imaging of Natural Scenes.

Multi-spectral imaging using a camera with more than three channels is an efficient method to acquire and reconstruct spectral data and is used extensively in tasks like object recognition, relighted rendering, and color constancy. Recently developed methods are used to only guide content-dependent filter selection where the set of spectral reflectances to be recovered are known a priori. We present the first content-independent spectral imaging pipeline that allows optimal selection of multiple channels. We also present algorithms for optimal placement of the channels in the color filter array yielding an efficient demosaicing order resulting in accurate spectral recovery of natural reflectance functions. These reflectance functions have the property that their power spectrum statistically exhibits a power-law behavior. Using this property, we propose power-law based error descriptors that are minimized to optimize the imaging pipeline. We extensively verify our models and optimizations using large sets of commercially available wide-band filters to demonstrate the greater accuracy and efficiency of our multi-spectral imaging pipeline over existing methods.

Convolutional Sparse Coding for RGB+NIR Imaging.

Emerging sensor designs increasingly rely on novel color filter arrays (CFAs) to sample the incident spectrum in unconventional ways. In particular, capturing a near-infrared (NIR) channel along with conventional RGB color is an exciting new imaging modality. RGB+NIR sensing has broad applications in computational photography, such as low-light denoising, it has applications in computer vision, such as facial recognition and tracking, and it paves the way toward low-cost single-sensor RGB and depth imaging using structured illumination. However, cost-effective commercial CFAs suffer from severe spectral cross talk. This cross talk represents a major challenge in high-quality RGB+NIR imaging, rendering existing spatially multiplexed sensor designs impractical. In this work, we introduce a new approach to RGB+NIR image reconstruction using learned convolutional sparse priors. We demonstrate high-quality color and NIR imaging for challenging scenes, even including high-frequency structured NIR illumination. The effectiveness of the proposed method is validated on a large data set of experimental captures, and simulated benchmark results which demonstrate that this work achieves unprecedented reconstruction quality.

Deep generative video prediction

Video prediction plays a fundamental role in video analysis and pattern recognition. However, the generated future frames are often blurred, which are not sufficient for further research. To overcome this obstacle, this paper proposes a new deep generative video prediction network under the framework of generative adversarial nets. The network consists of three components: a motion encoder, a frame generator and a frame discriminator. The motion encoder receives multiple frame differences (also known as Eulerian motion) as input and outputs a global video motion representation. The frame generator is a pseudo-reverse two-stream network to generate the future frame. The frame discriminator is a discriminative 3D convolution network to determine whether the given frame is derived from the true future frame distribution or not. The frame generator and frame discriminator train jointly in an adversarial manner until a Nash equilibrium. Motivated by theories on color filter array, this paper also designs a novel cross channel color gradient (3CG) loss as a guidance of deblurring. Experiments on two state-of-the-art data sets demonstrate that the proposed network is promising.

Polyscale gradients residual demosaicking

Image acquisition through a single-sensor camera captures only a fraction of the image data renowned as the raw image. It uses the colour filter array to acquire only a single colour channel in each pixel location. The reconstruction of the so called raw image into a full colour image is known as demosaicking or interpolation. Demosaicking is a significant process in the imaging pipeline. It implicates the reconstruction of the missing two colour channels from an incomplete image obtained from a single-sensor digital camera. This paper proposes a new approach of combining polyscale gradients algorithm and residual demosaicking. The first phase utilizes the gradients at different scales and residual demosaicking (RD) involves subtraction of the pragmatic and the provisional pixel values in the residual province takes place in the second phase. The experimental results show that the proposed demosaicking algorithm outperforms the conventional demosaicking algorithms for the Kodak and McMaster (McM) dataset.