Color Filter Array Research Articles

Image distortion inference based on correlation between line pattern and character

Samsung introduced pixel-merging technologies such as Tetrapixel, and these enabled the mobile image sensors to reproduce colors properly depend on light conditions. High resolution image can be acquired when in well-lit area, by reorganize colors on the color filter array to RGB Bayer pattern. The aforementioned process is called remosaic algorithm, based on estimating direction information. It causes some artifacts on edges with various or unclear direction, for example in text-image, especially at high spatial frequencies. We focused on such artifacts caused by remosaicing, and proposed suitable image quality metric that can measure a degree of the artifacts.

Potentials of combined visible light and near infrared imaging for driving automation

An extension of automotive imaging from the visible (VIS) to the near infrared (NIR) spectrum is promising for driving automation applications because the technology is readily available and offers potential benefits in low visibility conditions, in low light conditions with active illumination, and by collection of complementary data. We propose the evaluation of VIS-NIR imaging in simulation using an extended version of our camera simulation and optimization framework. Our extended framework generates realistic spectral irradiance data of synthetic scenes in the VIS and NIR spectral range and includes physically based camera models with characteristic increased NIR sensitivity of VIS-NIR CMOS imagers, modified automotive VIS-NIR color filter arrays and adapted image processing. We evaluate the reproduction of potential benefits of VIS-NIR imaging in our simulated camera images using exemplary night time and daylight traffic scenes, and discuss further extensions for creation of a well-balanced VIS-NIR dataset for quantitative evaluation.

DePhaseNet: A deep convolutional network using phase differentiated layers and frequency based custom loss for RGBW image sensor demosaicing

Panchromatic Color Filter Arrays with white signal were introduced a while ago, such as RGBW Kodak (CFA2.0) array, assuming to have better resolution in lowlight due to panchromatic signal. However, there is no successful RGBW image sensor in the industry targeting mobile cameras until now. In this work, we introduce a novel Samsung RGBW image sensor and we study its performance in a popular remosaic scenario. We propose a DePhaseNet - a deep fully convolutional network to solve RGBW remosaicing or demosaicing problem. We propose to have 3 layers of phase differentiated inputs and custom frequency based loss function for each layer. Proposed method successfully suppress False Colors inherent to RGBW sensor due to heavily under-sampled colors. By using this method, we were able not only to increase details preservation, but also increased color reproduction by 2% over conventional method. We found that RGBW sensor is beneficial not only in low light scenarios, but also in widely spread remosaic scenarios. Experiments show improvement in image quality, yielding CPSNR of 42dB for Kodak dataset, reaching the bar of Bayer CFA demosaicing result. Proposed method advances state-of-the-art in RGBW demosaic, by 6dB in CPSNR.

Patterning All‐Inorganic Halide Perovskite with Adjustable Phase for High‐Resolution Color Filter and Photodetector Arrays

AbstractPerovskite has been actively studied for optoelectronic applications, such as photodetectors and light‐emitting diodes (LEDs), because of its excellent optoelectronic properties. However, ionic bonds of the perovskite structure are vulnerable to chemicals, which makes perovskite incompatible with photolithography processes that use polar solvents. Such incompatibility with photolithography hinders perovskite patterning and device integration. Here, an all‐solution based cesium lead halide perovskite (CsxPbyBrz) patterning method is introduced in which PbBr2 is patterned and then synthesized into CsxPbyBrz. Each step of the top‐down patterning process (e.g., developing, etching, and rinsing) is designed to be compatible with existing photolithography equipment. Structural, chemical, and optical analyses show that the PbBr2 pattern of (10 µm)2 squares is successfully transformed into CsPbBr3 and Cs4PbBr6 with excellent absorption and emission properties. High‐resolution photoconductor arrays and luminescent pattern arrays are fabricated with CsPbBr3 and Cs4PbBr6 on various substrates, including flexible plastic films, to demonstrate their potential applications in image sensors or displays. The research provides a fundamental understanding of the properties and growth of perovskite and promotes technological advancement by preventing degradation during the photolithography process, enabling the integration of perovskite arrays into image sensors and displays.

Adaptive and Progressive Multispectral Image Demosaicking

Common consumer RGB cameras use a single image sensor to capture RGB color images with a color filter array (CFA) placed in front of the image sensor. This system captures particularly one color sample ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R$</tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$G$</tex-math></inline-formula> , or <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$B$</tex-math></inline-formula> ) at each pixel location. This captured raw image is called a CFA image. The CFA demosaicking is used then to construct the complete color image. A similar approach of working with a single image sensor and extension of CFA to multispectral filter array (MSFA) enables us to develop a low-cost multispectral camera with the help of MSFA demosaicking. However, due to many spectral bands and their very sparse sampling, extending CFA demosaicking to MSFA demosaicking is not easy. This paper proposes an adaptive and progressive MSFA demosaicking method for various MSFA patterns for 5 to 15 bands multispectral images. Binary tree-based MSFA patterns are used to create MSFA images, and these MSFA patterns can be designed for any N-band MSFA image. The spectral bands are arranged specifically so that the middle band has the highest probability of appearance (PoA) in the MSFA pattern. The middle band is interpolated first using PoA based progressive adaptive interpolation, and the modified bilinear spectral difference method is used to estimate all other spectral bands with the interpolated middle band’s help. Experimental results and comparative analysis on two different datasets reveal that our proposed MSFA demosaicking method outperforms the existing generic MSFA demosaicking methods in terms of computational time and various image quality metrics considered.

A Cubic Convolution Interpolation-Based Chroma Subsampling Method for Bayer and RGBW CFA Raw Images

The Bayer and RGBW color filter array (CFA) raw images, denoted by <inline-formula> <tex-math notation="LaTeX">$I^{Bayer}$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$I^{RGB}$ </tex-math></inline-formula>, respectively, have been widely used in consumer markets. In the demosaicking-first compression scheme, chroma subsampling is necessary prior to compressing <inline-formula> <tex-math notation="LaTeX">$I^{RGB}$ </tex-math></inline-formula> and <inline-formula> <tex-math notation="LaTeX">$I^{RGBW}$ </tex-math></inline-formula>. Several linear interpolation-based chroma subsampling methods have been developed for <inline-formula> <tex-math notation="LaTeX">$I^{Bayer}$ </tex-math></inline-formula>, but no nonlinear interpolation-based chroma subsampling methods have targeted the above two CFA image types simultaneously. In this paper, we first propose a nonlinear interpolation-based, namely the cubic convolution interpolation-based (CCI-based), <inline-formula> <tex-math notation="LaTeX">$2\times 2$ </tex-math></inline-formula> block-distortion function for each <inline-formula> <tex-math notation="LaTeX">$2\times 2$ </tex-math></inline-formula> CFA block <inline-formula> <tex-math notation="LaTeX">$B^{CbCr}$ </tex-math></inline-formula>. Next, using the Cauchy-Schwarz inequality, we prove that the proposed block-distortion function is a convex function in the real domain, which further serves as the base of the initial subsampled chroma solution. Then, a CCI-based iterative method is proposed to improve the initial subsampled chroma solution. The results of comprehensive experimental tests using Bayer and RGBW CFA images created from the three RGB full-color datasets, namely IMAX, SCI (screen content images), and CI (classical images), demonstrate that on the Versatile Video Coding (VVC) platform VTM-11.0, the proposed method achieves substantial quality enhancement and quality-bitrate tradeoff merits of the reconstructed Bayer and RGBW CFA images compared with existing chroma subsampling methods.

Hybrid Successive CFA Image Encryption-Watermarking Algorithm Based on the Quaternionic Wavelet Transform (QWT)

In this paper, we present a new robust hybrid algorithm combining successively chaotic encryption and blind watermarking of images based on the quaternionic wavelet transform (QWT) to ensure the secure transfer of digital data. The calculations of the different evaluation parameters have been performed in order to determine the robustness of our algorithm to certain attacks. The application of this hybrid algorithm on CFA (Color Filter Array) images, allowed us to guarantee the integrity of the digital data and to propose an autonomous transmission system. The results obtained after simulation of this successive hybrid algorithm of chaotic encryption and then blind watermarking are appreciated through the values of the evaluation parameters which are the peak signal-to-noise ratio (PSNR) and the correlation coefficient (CC), and by the visual observation of the extracted watermarks before and after attacks. The values of these parameters show that this successive hybrid algorithm is robust against some conventional attacks.

Edge-Aware Extended Star-Tetrix Transforms for CFA-Sampled Raw Camera Image Compression.

Codecs using spectral-spatial transforms efficiently compress raw camera images captured with a color filter array (CFA-sampled raw images) by changing their RGB color space into a decorrelated color space. This study describes two types of spectral-spatial transform, called extended Star-Tetrix transforms (XSTTs), and their edge-aware versions, called edge-aware XSTTs (EXSTTs), with no extra bits (side information) and little extra complexity. They are obtained by (i) extending the Star-Tetrix transform (STT), which is one of the latest spectral-spatial transforms, to a new version of our previously proposed wavelet-based spectral-spatial transform and a simpler version; (ii) considering that each 2D predict step of the wavelet transform is a combination of two 1D diagonal or horizontal-vertical transforms; (iii) weighting the transforms along the edge directions in the images. Compared with XSTTs, the EXSTTs can decorrelate CFA-sampled raw images well: they reduce the difference in energy between the two green components by about 3.38-30.08 % for high-quality camera images and 8.97-14.47 % for mobile phone images. The experiments on JPEG 2000-based lossless and lossy compression of CFA-sampled raw images show better performance than conventional methods. For high-quality camera images, the XSTTs/EXSTTs produce results equal to or better than the conventional methods: especially for images with many edges, the type-I EXSTT improves them by about 0.03-0.19 bpp in average lossless bitrate and the XSTTs improve them by about 0.16-0.96 dB in average Bjøntegaard delta peak signal-to-noise ratio. For mobile phone images, our previous work perform the best, whereas the XSTTs/EXSTTs show similar trends to the case of high-quality camera images.

Lossless White Balance for Improved Lossless CFA Image and Video Compression.

Color filter array is a spatial multiplexing of pixel-sized filters fabricated over pixel sensors in most color image sensors. The state-of-the-art lossless coding techniques of raw sensor data captured by such sensors leverage spatial or cross-color correlation using lifting schemes. In this paper, we propose a lifting-based lossless white balance algorithm. When applied to the raw sensor data, the spatial bandwidth of the implied chrominance signals decreases. We propose to use this white balance as a pre-processing step to lossless CFA subsampled image/video compression, improving the overall coding efficiency of the raw sensor data.

Deep Unrolling for Light Field Compressed Acquisition Using Coded Masks

Compressed sensing using color-coded masks has been recently considered for capturing light fields using a small number of measurements. Such an acquisition scheme is very practical, since any consumer-level camera can be turned into a light field acquisition camera by simply adding a coded mask in front of the sensor. We present an efficient and mathematically grounded deep learning model to reconstruct a light field from a set of measurements obtained using a color-coded mask and a color filter array (CFA). Following the promising trend of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">unrolling optimization algorithms</i> with learned priors, we formulate our task of light field reconstruction as an inverse problem and derive a principled deep network architecture from this formulation. We also introduce a closed-form extraction of information from the acquisition, while similar methods found in the recent literature systematically use an approximation. Compared to similar deep learning methods, we show that our approach allows for a better reconstruction quality. We further show that our approach is robust to noise using realistic simulations of the sensing acquisition process.

Image Demoireing via U-Net for Detection of Display Defects

Mura defects, which occur during display manufacturing, degrade the quality of the display. Therefore, Mura detection is critical. When the camera is focused on the display for accurately detecting Mura defects, a moire pattern occurs in a captured image because of the frequency difference between the subpixels of the display and the color filter array of the camera. Typical image data handled with existing demoireing methods do not have Mura defects and include synthetic moire images. Therefore, we created a dataset to detect Mura defects that include real moire patterns, classified into two categories: weak and strong. We propose a new demoireing framework to remove the moire patterns in the captured image, thereby accurately detecting Mura defects. We also propose inserting ArUco markers for accurate alignment and automation, conducting multiple experiments with U-Net. Based on the captured data, the proposed U-Net, which combines a frequency loss and data augmentation, improves the performance by 6.41 dB higher for the weak moire pattern and 4.14dB higher for the strong moire pattern than state-of-the-art networks in terms of peak signal-to-noise ratio.

Estimation of optimized window size for hybridized kNN-random forest algorithm based image demosaicing

Image Demosaicing is gaining popularity in the field of image processing as it helps in identifying the missing elements by using already known value of surrounding pixels obtained from color filter array overlaid on bayer pattern. The proposed work aims at calculating the image quality metrics (Signal to Noise Ratio and Peak Signal to Noise Ratio) for finding the optimized window size by performing image demosaicing based on two machine learning algorithms i.e. hybrid kNN and random forest algorithm with 20 learning cycles. To pick out the best out of many possible window sizes (3×3, 5x3, 5x5, 7x5), a set of standard images from Kodak database have been taken into consideration. The proposed work of hybrid algorithm has been implemented in MATLAB tool. The findings of the proposed work show that 7x5 window size outperforms the other window sizes for almost all the images as well as from its counterpart kNN only.

The review of demosaicing methods for Bayer color filter array image

The review of demosaicing methods for Bayer color filter array image

LAGRANGE INTERPOLATION FOR NATURAL COLOUR IMAGE DEMOSAICING

The quality of the digital image of the camera is determined by the colour demosaicing. The size and cost of the camera is reduced by a single sensor to capture the image which uses a Bayer Colour Filter Array (CFA). The captured image by the camera is the CFA image and the missing components of Red (R), Green (G), and Blue (B). The proposed system uses Lagrange interpolation in both directions to find the missing components of G. To get the whole G image, Inter Channel Correlation (ICC) is utilized. Then Colour different interpolation is employed for the reconstruction of R and B image. A fine-tuning step is designed to improve the quality. Natural images which are obtained from Kodak and IMAX are used for performance analysis. The qualities of the reconstructed images are validated in terms of Peak Signal to Noise Ratio (PSNR).&nbsp; The comparative analysis with the state-of-the-art approaches in the literature shows that the Lagrange interpolation provides better quality than others.

Plasmonic color filter array based visible light spectroscopy

Compared with traditional Fabry–Perot optical filters, plasmonic color filters could greatly remedy the complexity and reduce the cost of manufacturing. In this paper we present end-to-end demonstration of visible light spectroscopy based on highly selective plasmonic color filter array based on resonant grating structure. The spectra of 6 assorted samples were measured using an array of 20 narrowband color filters and detected signals were used to reconstruct original spectra by using new unmixing algorithm and by solving least squares problem with smoothing regularization. The original spectra were reconstructed with less than 0.137 root mean squared error. This works shows promise towards fully integrating plasmonic color filter array in imagers used in hyperspectral cameras.

Joint demosaicing of colour and polarisation from filter arrays

This article considers the joint demosaicing of colour and polarisation image content captured with a Colour and Polarisation Filter Array imaging system. The Linear Minimum Mean Square Error algorithm is applied to this case, and its performance is compared to the state-of-theart Edge-Aware Residual Interpolation algorithm. Results show that the LMMSE demosaicing method gives statistically higher scores on the largest tested database, in term of peak signal-to-noise ratio relatively to a CPFA-dedicated algorithm.

Data-Driven Convolutional Model for Digital Color Image Demosaicing

Modern digital cameras use specific arrangement of Color Filter Array to sample light wavelength corresponding to visible colors. The most common Color Filter Array is the Bayer filter that samples only one color per pixel. To recover the full resolution image, an interpolation algorithm can be used. This process is called demosaicing and it is one of the first processing stages of a digital imaging pipeline. We introduce a novel data-driven model for demosaicing that takes into account the different requirements for reconstruction of the image Luma and Chrominance channels. The final model is a parallel composition of two reconstruction networks with individual architecture and trained with distinct loss functions. In order to solve the overfitting problem, we prepared a dataset that contains groups of patches that share common chromatic and spectral characteristics. We reported the reconstruction error on noise-free images and measured the effect of random noise and quantization noise in the demosaicing reconstruction. To test our model performance, we implemented the network on NVIDIA Jetson Nano, obtaining an end-to-end running time of less than one second for a full frame 12 MPixel image.

On-sky demonstration of precision photometry with Bayer color filter arrays

Consumer-level digital single-lens reflex (DSLR) cameras are typically not used in professional astronomy because of the systematic errors present in the data as a result of the strong intra- and interpixel variations associated with each of the three different colors (RGB) of the Bayer color filter array. Nevertheless, because the cost of DSLRs compared with traditional astronomical CCDs is so much lower, they represent a potentially underexplored area of scientific quality astronomical imaging, especially in the area of wide-field transient surveys. We demonstrate an algorithm that can achieve ≈ 1 % level photometry in each of the RGB color channels from a stellar source and discuss the application of this algorithm to a ground-based transiting exoplanet survey. The algorithm primarily takes advantage of the large number of stellar sources available for statistical averaging within a single image, using a “lucky point-spread function” approach to identify sources in the image that exhibit systematic errors consistent with a chosen target from the same image. The selection of the appropriate“lucky” reference stars is accomplished through a comparison of the stellar image morphology as it appears on the Bayer array and the reference stars. These references are linearly combined to form a synthetic comparison star that can be used for differential photometry with the target. One key to the algorithm is that all data are retained at the individual pixel-level until the final differential comparison, which helps to alleviate systematic effects that might otherwise cancel each other out during the flux-summing process. We demonstrate the algorithm on HD 339461, a mV = 8.93 G0-type star on which we achieve single-percent level photometry that approaches the fundamental noise floor possible from a single camera.

Demosaicking enhancement method for color image based on low-rank constraint

Demosaicking is a method to restore the mosaicked image generated by the color filter array to a full-color image. The performance of the demosaicking algorithm is crucial to the reconstruction of high-quality color images. However, current demosaicking algorithms inevitably cause zipper effects or false color artifacts in the process of demosaicking. We combine low-rank constraint based on spectral domain with image non-local similarity to correct the interpolation errors. Our method is applicable to various existing demosaicking algorithms and can further improve the quality of full color image reconstruction. Experimental results demonstrate that the proposed algorithm can improve the color peak signal to noise ratio of the reconstructed images.

Coarse-to-fine-grained method for image splicing region detection

In this study, we aim to improve the accuracy of image splicing detection. We propose a progressive image splicing detection method that can detect the position and shape of spliced region. Because image splicing is likely to destroy or change the consistent correlation pattern introduced by color filter array (CFA) interpolation process, we first used a covariance matrix to reconstruct the R, G and B channels of image and utilized the inconsistencies of the CFA interpolation pattern to extract forensics feature. Then, these forensics features were used to perform coarse-grained detection, and texture strength features were used to perform fine-grained detection. Finally, an edge smoothing method was applied to realize precise localization. As compared to the state-of-the-art CFA-based image splicing detection methods, the proposed method has a high-level detection accuracy and strong robustness against content-preserving manipulations and JPEG compression.