Bayer Pattern Research Articles

Image tampering detection by estimating interpolation patterns

Several works have addressed the problem of detecting manipulations in images acquired from devices that use colour filter arrays, typical in the market due to low production costs. These devices use chromatic interpolation algorithms during the image formation process, allowing them to perform statistical analyses of inconsistencies generated from this process for authentication purposes. Most of the works focus on analysing the green band of the Bayer filter since it contains more information than blue and red bands. The lack of methods for effectively analysing other bands or different colour filters reduces the detection capability of known tools. The main purpose of this work is to provide a general methodology for detecting manipulations in this type of devices, in addition to providing new techniques that allow generalising the analysis in a great diversity of sensors.

Comparison of Denoising Algorithms for Demosacing Low Lighting Images Using CFA 2.0

In modern digital cameras, the Bayer color filter array (CFA) has been widely used. It is also widely known as CFA 1.0. However, Bayer pattern is inferior to the red-green-blue-white (RGBW) pattern, which is also known as CFA 2.0, in low lighting conditions in which Poisson noise is present. It is well known that demosaicing algorithms cannot effectively deal with Poisson noise and additional denoising is needed in order to improve the image quality. In this paper, we propose to evaluate various conventional and deep learning based denoising algorithms for CFA 2.0 in low lighting conditions. We will also investigate the impact of the location of denoising, which refers to whether the denoising is done before or after a critical step of demosaicing. Extensive experiments show that some denoising algorithms can indeed improve the image quality in low lighting conditions. We also noticed that the location of denoising plays an important role in the overall demosaicing performance.

Variable Macropixel Spectral-Spatial Transforms With Intra- and Inter-Color Decorrelations for Arbitrary RGB CFA-Sampled Raw Images

A raw image captured by a color filter array (CFA), such as a Bayer pattern, is usually compressed after demosaicing with some processings (denoising, deblurring, tone-mapping, and so on). However, since photographers, designers, and high-end users prefer to work with the raw image sampled by CFA (referred to as “raw image”) directly, a raw image should be compressed before demosaicing. For effective raw image compression, this study introduces variable macropixel spectral-spatial transforms (VMSSTs), that can successfully decorrelate not only Bayer raw images but any other pure-color (RGB) ones. The proposed VMSSTs are designed by the following two steps: 1) intra-color decorrelation and 2) inter-color decorrelation. In lossless compression with JPEG 2000, compared with methods which do not use transforms, the VMSSTs reduced the average bitrates of three types of CFAs: from approximately 0.09 to 0.12 bpp for the modified Bayer CFA, from 0.25 to 0.65 bpp for the diagonal stripe CFA, and from 0.33 to 0.70 bpp for the Fujifilm X-Trans CFA due to their high color decorrelation efficiency. In addition, in lossy compression with JPEG 2000, compared with a rearranged method, the VMSSTs improved the average bitrates of the Bjontegaard delta by around 3.97%, 14.95%, and 18.65% for each CFA model, respectively. Although a data-dependent adaptive transformation, the Karhunen-Loeve transform (KLT), showed the best performance in lossy compression, the introduced VMSSTs have shown performances comparable to those of the KLT in lossless compression, despite their simple structures.

Joint Demosaicing and Super-Resolution (JDSR): Network Design and Perceptual Optimization

Image demosaicing and super-resolution are two important tasks in color imaging pipeline. So far they have been mostly independently studied in the open literature of deep learning; little is known about the potential benefit of formulating a joint demosaicing and super-resolution (JDSR) problem. In this article, we propose an end-to-end optimization solution to the JDSR problem and demonstrate its practical significance in computational imaging. Our technical contributions are mainly two-fold. On network design, we have developed a Residual-Dense Squeeze-and-Excitation Networks (RDSEN) supported by a pre-demosaicing network (PDNet) as the pre-processing step. We address the issue of spatio-spectral attention for color-filter-array (CFA) data and discuss how to achieve better information flow by concatenating Residue-Dense Squeeze-and-Excitation Blocks (RDSEBs) for JDSR. Experimental results have shown that significant PSNR/SSIM gain can be achieved by RDSEN over previous network architectures including state-of-the-art RCAN. On perceptual optimization, we propose to leverage the latest ideas including relativistic discriminator and pre-excitation perceptual loss function to further improve the visual quality of textured regions in reconstructed images. Our extensive experiment results have shown that Texture-enhanced Relativistic average Generative Adversarial Network (TRaGAN) can produce both subjectively more pleasant images and objectively lower perceptual distortion scores than standard GAN for JDSR. Finally, we have verified the benefit of JDSR to high-quality image reconstruction from real-world Bayer pattern data collected by NASA Mars Curiosity.

Demosaicing of Bayer and CFA 2.0 Patterns for Low Lighting Images

It is commonly believed that having more white pixels in a color filter array (CFA) will help the demosaicing performance for images collected in low lighting conditions. However, to the best of our knowledge, a systematic study to demonstrate the above statement does not exist. We present a comparative study to systematically and thoroughly evaluate the performance of demosaicing for low lighting images using two CFAs: the standard Bayer pattern (aka CFA 1.0) and the Kodak CFA 2.0 (RGBW pattern with 50% white pixels). Using the clean Kodak dataset containing 12 images, we first emulated low lighting images by injecting Poisson noise at two signal-to-noise (SNR) levels: 10 dBs and 20 dBs. We then created CFA 1.0 and CFA 2.0 images for the noisy images. After that, we applied more than 15 conventional and deep learning based demosaicing algorithms to demosaic the CFA patterns. Using both objectives with five performance metrics and subjective visualization, we observe that having more white pixels indeed helps the demosaicing performance in low lighting conditions. This thorough comparative study is our first contribution. With denoising, we observed that the demosaicing performance of both CFAs has been improved by several dBs. This can be considered as our second contribution. Moreover, we noticed that denoising before demosaicing is more effective than denoising after demosaicing. Answering the question of where denoising should be applied is our third contribution. We also noticed that denoising plays a slightly more important role in 10 dBs signal-to-noise ratio (SNR) as compared to 20 dBs SNR. Some discussions on the following phenomena are also included: (1) why CFA 2.0 performed better than CFA 1.0; (2) why denoising was more effective before demosaicing than after demosaicing; and (3) why denoising helped more at low SNRs than at high SNRs.

Convolutional demosaicing network for joint chromatic and polarimetric imagery.

Due to the latest progress in image sensor manufacturing technology, the emergence of a sensor equipped with an RGGB Bayer filter and a directional polarizing filter has brought significant advantages to computer vision tasks where RGB and polarization information is required. In this regard, joint chromatic and polarimetric image demosaicing is indispensable. However, as a new type of array pattern, there is no dedicated method for this challenging task. In this Letter, we collect, to the best of our knowledge, the first chromatic-polarization dataset and propose a chromatic-polarization demosaicing network (CPDNet) to address this joint chromatic and polarimetric image demosaicing issue. The proposed CPDNet is composed of the residual block and the multi-task structure with the costumed loss function. The experimental results show that our proposed methods are capable of faithfully recovering full 12-channel chromatic and polarimetric information for each pixel from a single mosaic image in terms of quantitative measures and visual quality.

Optical distortion evaluation based on the Integrated Colour CCD Moiré Method.

Camera calibration is an important part of high-precision optical measurement, which is especially difficult in the micro-nano field. Based on the integrated correlation calculation and CCD moiré method, this paper describes the development of a lens calibration technique called the Integrated Colour CCD Moiré Method (ICCM). The CCD moiré fringes, formed by superimposing a periodic optical signal of a specimen grating with a CCD target array or a Bayer filter array, significantly enlarges the deformation modulated by lens distortion and the calibration plate attitude (i.e. the rotation angle relative to the camera coordinate system). To measure lens distortion using CCD moiré, the deformation pattern that is governed by the lens distortion, specimen grating attitude and carrier was used to construct a CCD fringe image. If the constructed CCD fringe image based on the trial lens distortion and rotation angles have a maximum similarity to the captured CCD moiré image, the lens distortion and rotation angles are correctly inversed. Particle swarm optimisation algorithm was selected to search for the true value so that the accuracy and robustness could be improved. Numerical experiments verified that the ICCM method developed in this work can simultaneously inverse the lens distortion, rotation angle and the grating pitch with high precision. The lens distortion of the metallographic microscope has been successfully characterised by the developed method with an 833 nm pitch grating. Simulations and experiments showed that ICCM is an intuitive, accurate, anti-noise and robust distortion calibration method.

Effective Gradient Descent-Based Chroma Subsampling Method for Bayer CFA Images in HEVC

The most widely used color filter array (CFA) pattern in commercial digital color cameras is the Bayer pattern, and the captured image is called the Bayer CFA image, in which each pixel contains only one color value and each image consists of 25% red, 50% green, and 25% blue color values. The chroma 4:2:2 or 4:2:0 subsampling of Bayer CFA images is a necessary process prior to compression. According to the block-distortion minimization principle, in this paper, we propose an effective gradient descent-based chroma subsampling (GDCS) method for Bayer CFA images. Based on the test Bayer CFA images collected from the Kodak and IMAX datasets, experimental results demonstrated that in high efficiency video coding, our GDCS method has better quality and quality-bitrate tradeoff performance of the reconstructed images when compared with the existing chroma subsampling methods.

Soft computing based color image demosaicing for medical Image processing

As digital cameras become more enhanced and small, CCD sensors can relate to only one color of a pixel. This color mosaic pattern is called as Bayer Pattern(BP) which requires processing to obtain a color image with a higher resolution. Each image pixel that undergoes interpolation has a full color spectrum based on surrounding pixel colors. Here we introduce Adaptive CFA(ACFA) interpolation model. For normal image regions hue technique is used while edge regions adapt the new technique. It is proposed to apply fuzzy logic and fuzzy rule which is based on Genetic Algorithm that uses random local search to enhance the PSNR. Medical image reconstruction by this proposed fuzzy based method outperforms the other medical image reconstruction methods.

Реализация интерполяции байерского шаблона на ПЛИС в задачах обработки видеопотока

Реализация интерполяции байерского шаблона на ПЛИС в задачах обработки видеопотока

Plasmon-coupled hybrid Fabry–Pérot cavity modes in submicron metal–dielectric–metal arrays for enhanced color filtering

We design and experimentally demonstrate micron- and submicron-sized color filters in the visible region using metal–dielectric–metal (MDM) Fabry–Perot cavity arrays. Large-area MDM filters (150 μm × 150 μm) of varying dielectric thicknesses show polarization-independent bandpass filtering of primary and secondary colors. The lateral dimensions of each cavity and the spacing between adjacent cavities are systematically reduced to study the possibility of using these color filters as Bayer filter arrays for color cameras. Up to micron-sized dimensions with micron-sized spacing between the adjacent MDM cavities, it is observed that the dominant mechanism for color filtering is Fabry–Perot effect. However, as the size and the pitch of the arrays of MDM are reduced further to the submicron length scale, polarization-dependent transmission is observed, and the dominant effect for color filtering is observed to be the leaked radiation from the grating-coupled plasmons. By keeping the pitch and lateral dimensions of MDM arrays fixed for plasmon resonance, the dielectric spacer thickness is systematically varied. Our electromagnetic simulations and microscopy-based imaging reveal that there exists a critical dielectric thickness for a specific cavity period, where the Fabry–Perot resonance and the Plasmons couple to each other and exhibit a high transmittivity. However, at other dielectric thicknesses, two weaker transmission peaks corresponding to decoupled modes; one corresponding to Fabry–Perot effect and the other corresponding to the signature of grating-coupled plasmons are present. Such MDM filter arrays can act as polarization-insensitive Bayer filters, as well as current-drawing contacts of multispectral imaging sensors with pixel sizes and separations, each up to a micron. With the pixel sizes in submicron regime, the integrated color filters with photodiodes may be useful for polarization-sensitive color imaging and surface sensors.

Weights-Based Image Demosaicking Using Posteriori Gradients and the Correlation of R–B Channels in High Frequency

In this paper, we propose a weights-based image demosaicking algorithm which is based on the Bayer pattern color filter array (CFA). When reconstructing the missing G components, the proposed algorithm uses weights based on posteriori gradients to mitigate color artifacts and distortions. Furthermore, the proposed algorithm makes full use of the correlation of R–B channels in high frequency when interpolating R/B values at B/R positions. Experimental results show that the proposed algorithm is superior to previous similar algorithms in composite peak signal-to-noise ratio (CPSNR) and subjective visual effect. The biggest advantage of the proposed algorithm is the use of posteriori gradients and the correlation of R–B channels in high frequency.

High ISO JPEG Image Denoising by Deep Fusion of Collaborative and Convolutional Filtering.

Capturing images at high ISO modes will introduce much realistic noise, which is difficult to be removed by traditional denoising methods. In this paper, we propose a novel denoising method for high ISO JPEG images via deep fusion of collaborative and convolutional filtering. Collaborative filtering explores the non-local similarity of natural images, while convolutional filtering takes advantage of the large capacity of convolutional neural networks (CNNs) to infer noise from noisy images. We observe that the noise variance map of a high ISO JPEG image is spatial-dependent and has a Bayer-like pattern. Therefore, we introduce the Bayer pattern prior in our noise estimation and collaborative filtering stages. Since collaborative filtering is good at recovering repeatable structures and convolutional filtering is good at recovering irregular patterns and removing noise in flat regions, we propose to fuse the strengths of the two methods via deep CNN. The experimental results demonstrate that our method outperforms the state-of-the-art realistic noise removal methods for a wide variety of testing images in both subjective and objective measurements. In addition, we construct a dataset with noisy and clean image pairs for high ISO JPEG images to facilitate research on this topic.

Comparison of Deep Learning and Conventional Demosaicing Algorithms for Mastcam Images

Bayer pattern filters have been used in many commercial digital cameras. In National Aeronautics and Space Administration’s (NASA) mast camera (Mastcam) imaging system, onboard the Mars Science Laboratory (MSL) rover Curiosity, a Bayer pattern filter is being used to capture the RGB (red, green, and blue) color of scenes on Mars. The Mastcam has two cameras: left and right. The right camera has three times better resolution than that of the left. It is well known that demosaicing introduces color and zipper artifacts. Here, we present a comparative study of demosaicing results using conventional and deep learning algorithms. Sixteen left and 15 right Mastcam images were used in our experiments. Due to a lack of ground truth images for Mastcam data from Mars, we compared the various algorithms using a blind image quality assessment model. It was observed that no one algorithm can work the best for all images. In particular, a deep learning-based algorithm worked the best for the right Mastcam images and a conventional algorithm achieved the best results for the left Mastcam images. Moreover, subjective evaluation of five demosaiced Mastcam images was also used to compare the various algorithms.

Hyperspectral imaging fluorescence excitation scanning (HIFEX) microscopy for live cell imaging.

In the past two decades, spectral imaging technologies have expanded the capacity of fluorescence microscopy for accurate detection of multiple labels, separation of labels from cellular and tissue autofluorescence, and analysis of autofluorescence signatures. These technologies have been implemented using a range of optical techniques, such as tunable filters, diffraction gratings, prisms, interferometry, and custom Bayer filters. Each of these techniques has associated strengths and weaknesses with regard to spectral resolution, spatial resolution, temporal resolution, and signal-to-noise characteristics. We have previously shown that spectral scanning of the fluorescence excitation spectrum can provide greatly increased signal strength compared to traditional emission-scanning approaches. Here, we present results from utilizing a Hyperspectral Imaging Fluorescence Excitation Scanning (HIFEX) microscope system for live cell imaging. Live cell signaling studies were performed using HEK 293 and rat pulmonary microvascular endothelial cells (PMVECs), transfected with either a cAMP FRET reporter or a Ca2+ reporter. Cells were further labeled to visualize subcellular structures (nuclei, membrane, mitochondria, etc.). Spectral images were acquired using a custom inverted microscope (TE2000, Nikon Instruments) equipped with a 300W Xe arc lamp and tunable excitation filter (VF-5, Sutter Instrument Co., equipped with VersaChrome filters, Semrock), and run through MicroManager. Timelapse spectral images were acquired from 350-550 nm, in 5 nm increments. Spectral image data were linearly unmixed using custom MATLAB scripts. Results indicate that the HIFEX microscope system can acquire live cell image data at acquisition speeds of 8 ms/wavelength band with minimal photobleaching, sufficient for studying moderate speed cAMP and Ca2+ events.

A Bayer motion estimation for motion-compensated frame interpolation

We propose a Bayer ME algorithm which is used to improve the performance of Motion-Compensated Frame Interpolation (MCFI). The core of the proposed algorithm is a predictive model designed from the alternate arrangement of Bayer pattern. According to the predictive model, the Motion Vector Field (MVF) of the interpolated frame is first split into basic blocks and absent blocks, and then an improved Bilateral Motion Estimation (BME) is proposed to compute the MVs of basic blocks. Finally, with the local stationary statistics of MVF, the MV of an absent block is predicted from the MVs of its neighboring basic blocks. Experimental results show that the proposed Bayer ME algorithm can improve both objective and subjective quality of the interpolated frame with a low computational complexity.

Demosaicking with adaptive reference range selection

Image demosaicking is a method of reconstructing an RGB image from a Bayer pattern, which is required when color information is lacking because a single charge-coupled device is used during the image extraction process of a digital camera. There are many restrictions on the reconstruction of a Bayer image to an RGB image. Given that each pixel contains only one-color information, artifacts, such as false color or the zipper effect, may occur at the edges, which can arise as a result of significant differences in brightness and color change. We propose a demosaicking method for adaptively selecting the reference range of color difference to obtain reliable information from texture regions and reconstructing into the RGB image. In particular, we determine the adaptive weight and reference range for four directions, east (E), west (W), south (S), and north (N), to improve the reliability of the color pixel obtained by a color difference estimation using guided filtering applied on residuals. In our experiment, we compare the results of the proposed method for the Kodak and IMAX datasets with those of nine demosaicking methods. The proposed method shows similar or improved results in terms of the color peak signal-to-noise ratio. In addition, compared to other methods, the visual quality improved by reducing residual artifacts.

FOVEON VS BAYER: COMPARISON OF 3D RECONSTRUCTION PERFORMANCES

Abstract. The main idea of this particular study was to validate if the new FOVEON technology implemented by sigma cameras can provide better overall results and outperform the traditional Bayer pattern sensor cameras regarding the radiometric information that records as well as the photogrammetric point cloud quality that can provide. Based on that, the scope of this paper is separated into two evaluations. First task is to evaluate the quality of information reconstructed during de-mosaicking step for Bayer pattern cameras by detecting potential additional colour distortion added during the de-mosaicking step, and second task is the geometric comparisons of point clouds generated by the photos by Bayer and FOVEON sensors against a reference point cloud. The first phase of the study is done using various de-mosaicking algorithms to process various artificial Bayern pattern images and then compare them with reference FOVEON images. The second phase of the study is carried on by reconstructing 3D point clouds of the same objects captured by a Bayer and a FOVEON sensor respectively and then comparing the various point clouds with a reference one, generated by a structured light hand-held scanner. The comparison is separated into two parts, where initially we evaluate five separate point clouds (RGB, Gray, Red, Green, Blue) for each camera sensor per site and then a second comparison is evaluated on colour classified RGB point cloud segments.

A Fast Algorithm for the Demosaicing Problem Concerning the Bayer Pattern

Introduction: In this paper, we deal with the demosaicing problem when the Bayer pattern is used. We propose a fast heuristic algorithm, consisting of three parts. Methods: In the first one, we initialize the green channel by means of an edge-directed and weighted average technique. In the second part, the red and blue channels are updated, thanks to an equality constraint on the second derivatives. The third part consists of a constant-hue-based interpolation. Results: We show experimentally how the proposed algorithm gives in mean better reconstructions than more computationally expensive algorithms.

Simple method for volumetric thickness measurement using a color camera.

A simple volumetric thickness measurement method for in-line high-speed inspection is proposed. With a color camera alone, spectrally resolved reflectance in a spatial domain is obtained: a Bayer filter spectrally resolves the reflected signal, and a CMOS sensor acquires three multi-spectral reflectances from RGB data at a single shot. To determine an accurate thickness, a modified reflectance is derived to convert a conventional spectral reflectance throughout a wavelength domain into an adequate form in an RGB domain by considering the characteristics of wide-band multi-spectral acquisition. The proposed method is validated by the measurement of a uniformly deposited SiO2 film and a tapered SiNx film.