Color Filter Array Research Articles

Seeing Photons in Color

Megapixel single-photon avalanche diode (SPAD) arrays have been developed recently, opening up the possibility of deploying SPADs as generalpurpose passive cameras for photography and computer vision. However, most previous work on SPADs has been limited to monochrome imaging. We propose a computational photography technique that reconstructs high-quality color images from mosaicked binary frames captured by a SPAD array, even for high-dyanamic-range (HDR) scenes with complex and rapid motion. Inspired by conventional burst photography approaches, we design algorithms that jointly denoise and demosaick single-photon image sequences. Based on the observation that motion effectively increases the color sample rate, we design a blue-noise pseudorandom RGBW color filter array for SPADs, which is tailored for imaging dark, dynamic scenes. Results on simulated data, as well as real data captured with a fabricated color SPAD hardware prototype shows that the proposed method can reconstruct high-quality images with minimal color artifacts even for challenging low-light, HDR and fast-moving scenes. We hope that this paper, by adding color to computational single-photon imaging, spurs rapid adoption of SPADs for real-world passive imaging applications.

86‐4: High‐Performance Perovskite Nanocrystals and Photoresists for In‐Pixel Color Conversion

Perovskite nanocrystals are outstanding materials for color conversion in all wide‐color gamut display architectures. Due to their unrivalled color purity, conversion efficiency and optical density, they are particularly attractive for in‐pixel color conversion. In this contribution we demonstrate that perovskite nanocrystals can achieve exceptional optical performance in non‐patterned acrylate films. To make these materials available to display manufacturers, we have developed photoresist formulations compatible with production lines used to make color filter arrays. The patterned films show excellent homogeneity and resolution which, when combined with good stability, make perovskites the materials of choice for emerging display architecture.

72‐2: Invited Paper: Color‐Conversion Liquid Crystal Display with In‐Cell Polarizer

An approach to developing a thin‐film polarizer with high polarization efficiency in liquid crystal cells is presented. The in‐cell polarizer facilitates the realization of novel color‐conversion liquid crystal displays, where advanced quantum color‐converters are used as photoluminescent color filter arrays. We experimentally demonstrated using azo‐dye‐based embedded polarizers for color‐conversion liquid crystal displays for the first time. We systematically designed the device structure and fabrication process. The display is color parallax free and can be driven at regular conditions with good contrast.

21‐3: Quantum Rod Color Filters with High Ambient Contrast Ratio

Applying quantum dot (QDs)‐based photoluminescent color filter arrays to mono‐color OLED displays and liquid crystal displays is considered a promising method to broaden the color gamut and boost the brightness and efficiency of the display. However, the ambient light could excite the semiconductor QDs and QRs, decreasing the display ambient contrast ratio. Adding a traditional absorptive CF array has limited functions in solving this problem due to the broadband transmission band of pigments. With a larger Stokes shift, QRs are more suitable for suppressing photoluminescent reflection. In this article, we simulate the ambient contrast ratio of a QD/QR‐OLED display, considering the parameters of backlight, the Stokes shift, the absorption strength, and spectral properties of QRs, QDs, and absorptive CFs, our discussion provides insight for optimization of the display ambient contrast ratio.

Joint demosaicking and denoising benefits from a two-stage training strategy

Image demosaicking and denoising are the first two key steps of the color image production pipeline. The classical processing sequence has for a long time consisted of applying denoising first, and then demosaicking. Applying the operations in this order leads to oversmoothing and checkerboard effects. Yet, it was difficult to change this order, because once the image is demosaicked, the statistical properties of the noise are dramatically changed and hard to handle by traditional denoising models. In this paper, we address this problem by a hybrid machine learning method. We invert the traditional color filter array (CFA) processing pipeline by first demosaicking and then denoising. Our demosaicking algorithm, trained on noiseless images, combines a traditional method and a residual convolutional neural network (CNN). This first stage retains all known information, which is the key point to obtain faithful final results. The noisy demosaicked image is then passed through a second CNN restoring a noiseless full-color image. This pipeline order completely avoids checkerboard effects and restores fine image detail. Although CNNs can be trained to solve jointly demosaicking–denoising end-to-end, we find that this two-stage training performs better and is less prone to failure. It is shown experimentally to improve on the state of the art, both quantitatively and in terms of visual quality.

A multi-channel approach for detecting tampering in colour filter images

Most devices capable of capturing photographs are provided with a Colour Filter Array (CFA). Images generated by these devices can be analysed to detect the so known CFA artefacts left by demosaicing methods. These artefacts are important in digital image forensics since they prove useful for determining the authenticity of an image. Previous works have focused exclusively on the analysis of the green band of the Bayer filter, neglecting the information in the remaining bands. In this work a process to estimate the pattern left by CFA artefacts regardless of their configuration is proposed, which can be used regardless of the size or colours used by the filter, or even in different colour spaces, obtaining in this way new sources of information for forensic analysis. A review and comparisons against state-of-the-art projects addressing the detection of manipulations using CFA artefacts are performed. The sensitivity of these proposals to post-processing methods such as compression and filtering will also be reviewed. We will also observe experimentally that the proposed technique can detect manipulations even in images affected by high levels of JPEG compression.

Proposed Extensions to the Third Edition of JPEG XS (ISO/IEC 21122) Standard

At its 94th meeting (17–21 January 2022), International Organization for Standardization (ISO)/IEC SC29WG1 (also known as JPEG Committee) decided to create a third edition of the JPEG XS standard (ISO/IEC 21122) for lightweight, low-latency image coding. While existing coding tools in the standard already support the compression of natural content and color filter array (CFA) RAW Bayer pattern data well, this third edition will extend the previous editions with coding tools that improve its performance on the compression of screen content and image sequences with static background. The first experiments demonstrate that the proposed coding tools can improve the average peak signal-to-noise ratio (PSNR) performance of JPEG XS on such sequences of more than 10 dB, and approximately 3 dB on computer-generated content. In this work, the authors will present the proposed coding tools and will evaluate their performance objectively by PSNR.

Grayscale-patterned integrated multilayer-metal-dielectric microcavities for on-chip multi/hyperspectral imaging in the extended visible bandwidth.

Pixelated filter arrays of Fabry-Perot (FP) cavities are widely integrated with photodetectors to achieve a WYSIWYG ("what you see is what you get") on-chip spectral measurements. However, FP-filter-based spectral sensors typically have a trade-off between their spectral resolution and working bandwidth due to design limitations of conventional metal or dielectric multilayer microcavities. Here, we propose a new idea of integrated color filter arrays (CFAs) consisting of multilayer metal-dielectric-mirror FP microcavities that, enable a hyperspectral resolution over an extended visible bandwidth (∼300 nm). By introducing another two dielectric layers on the metallic film, the broadband reflectance of the FP-cavity mirror was greatly enhanced, accompanied by as-flat-as-possible reflection-phase dispersion. This resulted in balanced spectral resolution (∼10 nm) and spectral bandwidth from 450 nm to 750 nm. In the experiment, we used a one-step rapid manufacturing process by using grayscale e-beam lithography. A 16-channel (4 × 4) CFA was fabricated and demonstrated on-chip spectral imaging with a CMOS sensor and an impressive identification capability. Our results provide an attractive method for developing high-performance spectral sensors and have potential commercial applications by extending the utility of low-cost manufacturing process.

OPTIMIZATION ASSISTED AUTOREGRESSIVE METHOD WITH DEEP CONVOLUTIONAL NEURAL NETWORK-BASED ENTROPY FILTER FOR IMAGE DEMOSAICING

The natural scenes are captured by digital cameras that adopt a single charged-coupled device (CCD) or Complementary Metal Oxide Semiconductor (CMOS) sensor with a Color Filter Array (CFA). To reconstruct a full color image from the mosaiced CFA image, the missing components of color are recovered by a technique called color demosaicing. This research work introduces Deep Convolution Neural Network (DCNN)-based optimization assisted autoregressive method for image demosaicing. Here, the proposed optimization algorithm, termed Adaptive Autoregressive Water Wave Optimization algorithm (Adaptive Autoregressive-WWO), that is formed by combining Conditional autoregressive value at risk (CAViaR) model and Water Wave Optimization algorithm (WWO). Fusion process is carried out for residual images to generate the final demosaiced image based on entropy measure. Here, the output generated from DCNN is the first residual image. The LPA-ICI filter utilizes the optimization algorithm that generated second order polynomial coefficients to produce the second residual output image. Moreover, this proposed method is evaluated for its performance using metrics, such as Peak signal-to-noise ratio (PSNR) and Second Derivative like Measurement (SDME) and attained the highest PSNR values of 40.379dB and highest SDME values of 50.675dB.

Nonlinear demosaicking method and apparatus for nonlinear CMOS image sensors exhibiting low-density salt-and-pepper noise

Simple demosaicking methods designed for linear CMOS image sensors, such as MATLAB’s demosaic function, may be used with a monotonic nonlinear sensor having a Bayer color filter array (CFA). However, such methods may be inadequate at handling dynamic salt-and-pepper noise (SPN), i.e., outlier pixels, which is expected in images taken with a nonlinear sensor. Although SPN is present with linear sensors, nonlinear sensors express low-density light-dependent SPN that requires filtering. Extending a recent work on dynamic SPN filtering of a nonlinear sensor, we propose, evaluate, and verify a nonlinear method and apparatus to demosaic images, taken with a Bayer CFA, while simultaneously filtering the SPN. The approach relies on the use of weighted medians to filter the SPN, especially at densities that imply isolated outliers in small neighborhoods, while determining an accurate red, green, and blue (RGB) color at every pixel location. For explanatory purposes, three variants of the proposed method are presented and evaluated. A ground-truth image set, in which RGB channels were not obtained by demosaicking, is subsampled in a Bayer CFA pattern to produce mosaicked images for testing. In varying densities, SPN is introduced to these Kodak images for method and apparatus evaluation. Results of the proposed method and its variants are compared with those obtained with MATLAB’s demosaic function. Considering the alternatives and also apparatus complexity, the proposed nonlinear demosaicking method proves superior in visual quality, for smooth textures and along edges, and in peak signal-to-noise ratio, when there is low-density SPN.

D$^\text{2}$UF: Deep Coded Aperture Design and Unrolling Algorithm for Compressive Spectral Image Fusion

Compressive spectral imaging (CSI) has attracted significant attention since it employs synthetic apertures to codify spatial and spectral information, sensing only 2D projections of the 3D spectral image. However, these optical architectures suffer from a trade-off between the spatial and spectral resolution of the reconstructed image due to technology limitations. To overcome this issue, compressive spectral image fusion (CSIF) employs the projected measurements of two CSI architectures with different resolutions to estimate a high-spatial high-spectral resolution. This work presents the fusion of the compressive measurements of a low-spatial high-spectral resolution coded aperture snapshot spectral imager (CASSI) architecture and a high-spatial low-spectral resolution multispectral color filter array (MCFA) system. Unlike previous CSIF works, this paper proposes joint optimization of the sensing architectures and a reconstruction network in an end-to-end (E2E) manner. The trainable optical parameters are the coded aperture (CA) in the CASSI and the colored coded aperture in the MCFA system, employing a sigmoid activation function and regularization function to encourage binary values on the trainable variables for an implementation purpose. Additionally, an unrolling-based network inspired by the alternating direction method of multipliers (ADMM) optimization is formulated to address the reconstruction step and the acquisition systems design jointly. Finally, a spatial-spectral inspired loss function is employed at the end of each unrolling layer to increase the convergence of the unrolling network. The proposed method outperforms previous CSIF methods, and experimental results validate the method with real measurements.

Customized Structural Color Filters by Pixel‐Level Electrothermal Regulation

AbstractReflective structural colors have backlight‐free supremacy and can provide energy‐efficient and environmental‐friendly color production for many applications. However, they are hindered by the complex process and incorrect colors after fabrication. Here, a pixelated electrothermal oxidation technique is proposed that can precisely tailor and produce reflective structural colors covering the entire visible range, providing the possibility for customized structural color filter arrays from a titanium canvas. Furthermore, as a demonstration, a photonic‐firework‐like color filter array is achieved in a single step via thermal engineering. Based on this one‐step‐forming color array, a computational spectrometer is also realized featuring a resolvability of 10 nm and operating bandwidth covering the whole visible range. Considering its compactness and mass production capability, this method has numerous potential applications in, e.g., imaging, anti‐counterfeiting, printing, color display, and spectroscopy.

Color performance of 0.8 um CMOS image sensor with CMY color filters

Modern digital cameras capture images with a subsampling method via mosaic color filter array (CFA). Of particular interest is the CFA with Cyan-Magenta-Yellow (CMY) color filters. Despite the improvement of the sensitivity, images reconstructed from CMY CFA usually suffer from lower color fidelity compared to the conventional Red-Green-Blue (RGB) color filters. In this paper, we proposed a CMY CFA with novel spectral sensitivities (CMY2.0), which were carefully designed to overcome the shortcomings of previous CMY CFA (CMY1.0). A CMY CMOS image sensor (CIS) with such optimized spectral sensitivities is then realized in order to evaluate the color performance and signal-to-noise ratio (SNR). As a result, the camera equipped with the CMY CFA with proposed spectral sensitivities (CMY2.0) features both an improved sensitivity and a high color fidelity, which is suitable for a wide range of applications, such as low-light photography, under-screen cameras and automotive cameras.

Image demosaicing: Subjective analysis and evaluation of image quality metrics

Most cameras use a single-sensor arrangement with Color Filter Array (CFA). Color interpolation techniques performed during image demosaicing are normally the reason behind visual artifacts generated in a captured image. While the severity of the artifacts depends on the demosaicing methods used, the artifacts themselves are mainly zipper artifacts (block artifacts across the edges) and false-color distortions. In this study and to evaluate the performance of demosaicing methods, a subjective pair-comparison method with 15 observers was performed on six different methods (namely Nearest Neighbours, Bilinear interpolation, Laplacian, Adaptive Laplacian, Smooth hue transition, and Gradient-Based image interpolation) and nine different scenes. The subjective scores and scene images are then collected as a dataset and used to evaluate a set of no-reference image quality metrics. Assessment of the performance of these image quality metrics in terms of correlation with the subjective scores show that many of the evaluated no-reference metrics cannot predict perceived image quality.

Machine learning estimation of camera spectral sensitivity functions with non-RGB color filters

The spectral sensitivity functions of a digital image sensor determine the sensor’s color response to scene-radiated light. Knowing these spectral sensitivity functions is very important for applications that require accurate color, such as computer vision. Traditional measurements of these functions are time consuming, and require expensive lab equipment to generate narrow-band monochromatic light. Previous works have shown that sensitivity curves can be estimated using images of a color checker chart with known spectral reflectances, using either numerical optimization or machine learning. However, previous works in the literature have not considered sensitivity functions for CFAs (color filter arrays) other than RGB, such as RCCB (Red Clear Blue) or RYYCy (Red Yellow Cyan). Non-RGB CFAs have been shown to be useful for automotive and security camera applications, especially in low light situations. We propose a machine learning method to estimate the sensitivity curves of sensors with non-RGB filters, in addition to the RGB filters addressed previously in the literature, using a single image of a color chart under unknown illumination. Including non-RGB filters makes the estimation problem much more challenging, since the resulting space of color filters is no longer modelled by simple Gaussian shapes.

The case for an LMS camera

Consumer color cameras employ sensors that do not mimic human cone spectral sensitivities, and more generally do not meet the Luther condition since the accompanying color correction substantially amplifies noise in the red channel. This begs the question: if cone spectral sensitivities yield low SNR, why has the Human Visual System so evolved? We answer the above question by noting that since modern ISPs - and the ancient HVS - remove virtually all chrominance noise, chrominance denoising artifacts rather than the chrominance noise itself should be considered. While sensor green, blue are reasonable analogs of the human M, S cones, the spectral sensitivity of red is much narrower than that of L and does not overlap much with green. An imager employing L instead of red suffers from increased red noise but is also more sensitive. This allows a high SNR (L + M)/2 luminance image to be reconstructed and used for denoising. Modeling the color filter array on the human retina, with a higher density of L pixels at the expense of S pixels, further improves the red SNR without the accompanying loss of blue quality being perceptible. The resulting LMS camera outperforms conventional RGB cameras in color accuracy and luminance SNR while being competitive in chrominance quality.

Universal Demosaicking for Interpolation-Friendly RGBW Color Filter Arrays

Interpolation-friendly RGBW color filter arrays (CFAs) and the popular sequential demosaicking contain the idea of computational photography, where the CFA and the demosaicking method are co-designed. Due to the advantages, interpolation-friendly RGBW CFAs have been extensively used in commercial color cameras. However, most associated demosaicking methods rely on strict assumptions or are limited to a few specific CFAs with a given camera. In this paper, we propose a universal demosaicking method for interpolation-friendly RGBW CFAs, which enables the comparison of different CFAs. Our new method belongs to sequential demosaicking, i.e., W channel is interpolated first and then RGB channels are reconstructed with guidance from the interpolated W channel. Specifically, it first interpolates the W channel using only available W pixels followed by an aliasing reduction technique to remove aliasing artifacts. Then it employs an image decomposition model to built relations between W channel and each of RGB channels with known RGB values, which can be easily generalized to the full-size demosaicked image. We apply the linearized alternating direction method (LADM) to solve it with convergence guarantee. Our demosaicking method can be applied to all interpolation-friendly RGBW CFAs with varying color cameras and lighting conditions. Extensive experiments confirm the universal property and advantage of our proposed method with both simulated and real raw images.

Методика расчета параметров антиалайсинговых фильтров для коррекции спектральных характеристик в зависимости от используемых структур дискретизации массива светофильтров

When developing algorithms and methods for generating and processing image signals (IS) used in devices based on single-matrix light-to-signal converters (LSC), the effects of not only the elements of the system included in the optical pathway (OP) located directly between the exposed image and device sensor, but also the specifics of the structure of color filter arrays (CFA) used in such cameras, applied to the surface of the matrix. In this article analyzed the most popular SSF subsampling structures and their impact on the spatial characteristics of the generated images. Based on the model for constructing these characteristics, a method is given for calculating the optimal parameters of anti-aliasing filters that match the properties of the OP and the sensor for a given characteristic of the optical system. Specific examples of calculating and optimizing such filters and evaluating their impact on the amplitude characteristics of signals are given. Based on the results of applying the developed method, formulated recommendations for the design of the CFA.

Guided Colorization Using Mono-Color Image Pairs.

Compared to color images captured by conventional RGB cameras, monochrome (mono) images usually have higher signal-to-noise ratios (SNR) and richer textures due to the lack of color filter arrays in mono cameras. Therefore, using a mono-color stereo dual-camera system, we can integrate the lightness information of target monochrome images with the color information of guidance RGB images to accomplish image enhancement in a colorization manner. In this work, based on two assumptions, we introduce a novel probabilistic-concept guided colorization framework. First, adjacent contents with similar luminance are likely to have similar colors. By lightness matching, we can utilize colors of the matched pixels to estimate the target color value. Second, by matching multiple pixels from the guidance image, if more of these matched pixels have similar luminance values to the target one, we can estimate colors with more confidence. Based on the statistical distribution of multiple matching results, we retain the reliable color estimates as initial dense scribbles and then propagate them to the rest of the mono image. However, for a target pixel, the color information provided by its matching results is quite redundant. Hence, we introduce a patch sampling strategy to accelerate the colorization process. Based on the analysis of the posteriori probability distribution of the sampling results, we can use much fewer matches for color estimation and reliability assessment. To alleviate incorrect color propagation in the sparsely scribbled regions, we generate extra color seeds according to the existed scribbles to guide the propagation process. Experimental results show that, our algorithm can efficiently and effectively restore color images with higher SNR and richer details from the mono-color image pairs, and achieves good performance in solving the color bleeding problem.

Multispectral Filter Array Design by Optimal Sphere Packing.

Multispectral imaging (MSI) collects a datacube of spatio-spectral information of a scene. Many acquisition methods for spectral imaging use scanning, preventing its widespread usage for dynamic scenes. On the other hand, the conventional color filter array (CFA) method often used to sample color images has also been extended to snapshot MSI using a Multispectral Filter Array (MSFA), which is a mosaic of selective spectral filters placed over the Focal Plane Array (FPA). However, even state-of-the-art MSFAs coding patterns produce artifacts and distortions in the reconstructed spectral images, which might be due to the nonoptimal distribution of the spectral filters. To reduce the appearance of artifacts and provide tools for the optimal design of MSFAs, this paper proposes a novel mathematical framework to design MSFAs using a Sphere Packing (SP) approach. By assuming that each sampled filter can be represented by a sphere within the discrete datacube, SP organizes the position of the equal-size and disjoint spheres's centers in a cubic container. Our method is denoted Multispectral Filter Array by Optimal Sphere Packing (MSFA-OSP), which seeks filter positions that maximize the minimum distance between the spheres's centers. Simulation results show an image quality improvement of up to 2 dB and a remarkable boost in spectral similarity when using our proposed MSFA design approach for a variety of reconstruction algorithms. Moreover, MSFA-OSP notably reduces the appearance of false colors and zipper effect artifacts, often seen when using state-of-the-art demosaicking algorithms. Experiments using synthetic and real data prove that the proposed MSFA-OSP outperforms state-of-the-art MSFAs in terms of spatial and spectral fidelity. The code that reproduces the figures of this article is available at https://github.com/nelson10/DemosaickingMultispectral3DSpherePacking.git.