Bayer Pattern Research Articles

Simultaneous bright-field and fluorescence lensless imaging with high excitation light extinction for microfluidics applications

Lensless imaging fluorescence applications are limited by the unavoidable trade-off between the object-sensor distance and the excitation light attenuation at the detection plane. Indeed, a shorter object-sensor distance reduces the spreading of fluorescent signals but at the cost of losing excitation light attenuation provided by filter thickness. Accordingly, the combination of four synergistic pathways to attain further attenuation of excitation light is proposed: the optimization of total internal reflection by microfluidic chip geometry modification, cross-polarization extinction, interchangeable absorption longpass filters, and the CMOS sensor built-in bandpass Bayer filters. The advantages of such a combined setup are demonstrated with a microfluidic application, as we managed to keep the object-sensor distance as short as 600 µm and still distinguish the fluorescence of individual droplets, flowing as close as 650 µm to each other, while attenuating excitation light to an OD 5.9 level. This approach also allowed simultaneous fluorescence and bright-field observation of on-chip droplet fluorescence during flow at a rate of 30 fps. The enhanced capabilities in our scheme pave the way to further lensless fluorescence applications requiring parallelism, lower detection thresholds, and real-time acquisition, such as fluorescence biosensing.

Learned Multi-aperture Color-coded Optics for Snapshot Hyperspectral Imaging

Learned optics, which incorporate lightweight diffractive optics, coded-aperture modulation, and specialized image-processing neural networks, have recently garnered attention in the field of snapshot hyperspectral imaging (HSI). While conventional methods typically rely on a single lens element paired with an off-the-shelf color sensor, these setups, despite their widespread availability, present inherent limitations. First, the Bayer sensor's spectral response curves are not optimized for HSI applications, limiting spectral fidelity of the reconstruction. Second, single lens designs rely on a single diffractive optical element (DOE) to simultaneously encode spectral information and maintain spatial resolution across all wavelengths, which constrains spectral encoding capabilities. This work investigates a multi-channel lens array combined with aperture-wise color filters, all co-optimized alongside an image reconstruction network. This configuration enables independent spatial encoding and spectral response for each channel, improving optical encoding across both spatial and spectral dimensions. Specifically, we validate that the method achieves over a 5dB improvement in PSNR for spectral reconstruction compared to existing single-diffractive lens and coded-aperture techniques. Experimental validation further confirmed that the method is capable of recovering up to 31 spectral bands within the 429--700 nm range in diverse indoor and outdoor environments.

An authentication scheme for color images with grayscale invariance and recoverability using image demosaicing

This paper proposes a novel recoverable authentication scheme for color images that ensures the consistency of grayscale values after embedment. In the proposed scheme, the color image is sub-sampled based on the Bayer pattern, and the most significant bits (MSBs) of the sub-sampled image are used as the recovery codes. Image blocks are divided into two categories, feasible and infeasible. Feasible blocks are utilized for embedding both authentication and recovery codes, whereas infeasible ones are only used for embedding authentication codes. In case of tampering, the authentication codes are employed to detect the tampered regions, and the undamaged recovery codes can be utilized to reconstruct the sub-sampled image. The demosaicing algorithm is applied to the sub-sampled image to obtain the demosaicked image, which is then used to recover the tampered regions. Experiments demonstrate that the proposed scheme yields an acceptable image quality even when the tampering rate is as high as 40 %.

The AI-generated Images Authenticity Determination with Bayer Pattern Artifacts

The AI-generated Images Authenticity Determination with Bayer Pattern Artifacts

A light-efficient and versatile multiplexing method for snapshot spectral imaging

The study of rapid and stochastic events that involve multiple species, such as chemical reactions and plasma dynamics, requires means to capture multispectral information in two dimensions at both high temporal- and spatial resolution. Commercially available cameras that provide high temporal resolution are based on either signal intensification or rapid data acquisition. Intensified cameras provide extremely short acquisition times using intensification by means of micro channel plates, but the conversion between electrons and photons makes these cameras inherently monochrome. In contrast, high-speed cameras can achieve color-sensitivity through integrated Bayer filters but suffer from a reduced light collection efficiency and a fixed spectral composition. In this article we present a non-integrated optical arrangement for instantaneous multispectral imaging based on FRAME image multiplexing. By spectrally separating the signal using lossless dichroic mirrors, a 16-fold increase in light-collection efficiency is gained (compared to past solutions), resulting in an equivalent increase in temporal resolution. This improvement provides new avenues for multispectral imaging of rapid events. We demonstrate the system’s versatility and suitability for studies of such processes by applying it for (i) temperature mapping using a high-resolution CCD camera, (ii) high-speed videography up to 10 kHz at four spectral channels and (iii) dual-species visualization in a plasma discharge using an intensified sCMOS camera.

Identification of exfoliated monolayer hexagonal boron nitride films with a digital color camera under white light illumination

Optical microscopy with white light illumination has been employed when obtaining exfoliated monolayer hexagonal boron nitride (1L hBN) films from a large number of randomly placed films on a substrate. However, real-time observation of 1L hBN using a color camera under white light illumination remains challenging since hBN is transparent in the visible wavelength range. The poor optical constant of 1L hBN films in microphotographs is significantly improved using a Si substrate coated with a SiN x thin-film (SiN x /Si). When observing hBN thin films on SiN x /Si using a color digital camera in an optical microscope under white light illumination, the clarity of the captured color images depends on the thickness of the SiN x film (d). For real-time direct observation, the d was optimized based on quantitative chromatic studies tailored to Bayer filters of a color image sensor. Through image simulation, it was determined that the color difference between 1L hBN and the bare substrate is maximized at d = 59 or 70 nm, which was experimentally verified. The SiN x /Si with optimized d values visualized 1L hBN films without requiring significant contrast enhancement via image processing under white light illumination in real-time. Furthermore, the captured color photographs facilitate the reliable determination of the number of layers in few-layer hBN films using the contrast of the green channel of the images.

P‐117: A Demura Method For OLED Under White Image With Monochrome Camera

A demura method using high resolution monochrome camera is proposed for OLED panels, enabling single data capturing for white images under different graylevels (GLs). Our method minimizes TACT for low GLs due to its high efficiency of light utilization, compared to demura with color camera of Bayer pattern.

Freeform metasurface color router for deep submicron pixel image sensors.

Advances in imaging technologies have led to a high demand for ultracompact, high-resolution image sensors. However, color filter-based image sensors, now miniaturized to deep submicron pixel sizes, face challenges such as low signal-to-noise ratio due to fewer photons per pixel and inherent efficiency limitations from color filter arrays. Here, we demonstrate a freeform metasurface color router that achieves ultracompact pixel sizes while overcoming the efficiency limitations of conventional architectures by splitting and focusing visible light instead of filtering. This development is enabled by a fully differentiable topology optimization framework to maximize the use of the design space while ensuring fabrication feasibility and robustness to fabrication errors. The metasurface can distribute an average of 85% of incident visible light according to the Bayer pattern with a pixel size of 0.6 μm. The device and design methodology enable the compact, high-sensitivity, and high-resolution image sensors for various modern technologies and pave the way for the advanced photonic device design.

High-efficiency broadband pixelated polarization routers based on metalenses

Polarization imaging is of particular significance in numerous fields ranging from underwater depth measurement, anti-counterfeiting to medical diagnosis. Metasurface-based polarizers have been efficiently demonstrated in various polarization measurements, which have great application prospects compared with traditional optical elements. However, Polarizers based on metalenses (i.e. metasurface lenses) are often designed to work in a single wavelength and are hard to work in a broadband wavelength region, which hinders practical imaging applications. Here, we designed a metalenses-based pixelated polarization router (PPR) that can be used as a full-Stokes polarizer for the whole visible wavelength range. As the PPRs are designed using polarization multiplexing to encode two different phase profiles and utilizing spatial multiplexing by the construction of a supercell, the transmittance and focusing efficiencies are calculated to be as high as ∼60% over the whole visible regime. The visible PPRs can be directly inserted between the Bayer color filter and the intensity-based photodetectors in a conventional imaging system, so that both the color and polarization information can be collected simultaneously.

Quad Bayer Joint Demosaicing and Denoising Based on Dual Encoder Network with Joint Residual Learning

The recent imaging technology Quad Bayer CFA brings better imaging PSNR and higher visual quality compared to traditional Bayer CFA, but also serious challenges for demosaicing and denoising during the ISP pipeline. In this paper, we propose a novel dual encoder network, namely DRNet, to achieve joint demosaicing and denoising for Quad Bayer CFA. The dual encoders are carefully designed in that one is mainly constructed by a joint residual block to jointly estimate the residuals for demosaicing and denoising separately. In contrast, the other one is started with a pixel modulation block which is specially designed to match the characteristics of Quad Bayer pattern for better feature extraction. We demonstrate the effectiveness of each proposed component through detailed ablation investigations. The comparison results on public benchmarks illustrate that our DRNet achieves an apparent performance gain~(0.38dB to the 2nd best) from the state-of-the-art method and balances performance and efficiency well. The experiments on real-world images show that the proposed method could enhance the reconstruction quality from the native ISP algorithm.

Enhancing the Performance of Optical Hardware through Software Applications: A Study

The process of image demosaicing, essential for reconstructing fullcolour images from incomplete colour samples obtained from a colourfilter array (CFA), has garnered renewed interest, particularly with the prevalent use of the Bayer pattern. This resurgence can be attributed to the growing availability of source codes and executables, facilitating reproducible research in the field. In this article, a thorough survey encompassing over fifty published works on demosaicing since 1999. The objective is to complement prior reviews and provide insights into the evolution and current landscape of demosaicing techniques. By addressing key issues and delineating fundamental differences among various approaches, a thorough overview of the field is made. A notable finding from the survey is the popularity of spatial demosaicing methods, where the initial step involves interpolating the luminance channel, followed by reconstructing the chrominance channels based on the recovered luminance information. The article highlights three crucial areas requiring additional research. Firstly, addressing the difficulty of demosaicking images with low spectral correlation is paramount, along with enhancing our comprehension of the trade-off between spatial correlation and spectral correlation. It is also found that there is a necessity for more comprehensive investigation into evaluating the performance of demosaicking algorithms, particularly concerning the utilization of reference images, and understanding supplementary components within the imaging pipeline.

COLOR-INDEX DETERMINATION OF LEO SATELLITES USING COLOR IP-CAMERAS

The study is devoted to determining the color characteristics of the structural surfaces of artificial satellites. Observations of satellites were carried out in automatic mode using software for detection fast moving object, developed at RI “MAO”. The satellite observations equipment consists of Canon EF 85mm f/1.8 USM photographic lens and professional VIVOTEK IP816A-HP network camera. The camera is directed to the zenith and has a field of view (4.9°х2.8°). The camera has progressive CMOS sensor with Bayer color filter⁰₀ array for producing RGB color image. Coordinate and photometric processing of saved images is carried out using SExtractor and Astrometry.net software along with additional Python scripts. In this paper, we present first results of our LEO satellites observations using network IP-cameras with RGB Bayer filter. To convert the instrumental magnitudes from RGB Bayer system into standard bp/rp magnitudes of the Gaia EDR3 photometric system, a system of equations was solved, where the calculated magnitudes in the bp/rp photometric system were represented as functions of magnitudes and colors in the instrumental RGB system. The mean differences between the calculated magnitudes and the catalog magnitudes are (0.03±0.15)mag and (0.01±0.16)mag for bp and rp band, respectively. The calculated Brjb_bp and Rrjb_rp magnitudes using obtained transformation coefficients show a good linear correlation with the bp and rp Gaia EDR3 magnitudes. Only main sequence’s stars in the range of (5-13)mag were used as reference stars for determining the transformation coefficients. The results of determining the color indexes (Brjb_bp - Rrjb_rp) for 8 LEO satellites with the obtained transformation coefficients were received. The average accuracy of the obtained values of color indexes is about 0.2mag.

Low-cost, high-speed multispectral imager via spatiotemporal modulation based on a color camera

Spectral imaging is a powerful tool in industrial processes, medical imaging, and fundamental scientific research. However, for the commonly used spatial/spectral-scanning spectral imager, the slow response time has posed a big challenge for its employment in dynamic scenes. In this paper, we propose a spatiotemporal modulation concept and build a simple, low-cost spectral imager by combining a liquid crystal (LC) cell with a commercial color camera. By the synergic effect of temporal modulation of the LC materials and spatial modulation of the Bayer filter in a color camera, high-quality multispectral imaging is successfully demonstrated with a high rate of 8 Hz, far beyond the counterparts. Experimental results show that even with three tuning states of the LC material, optical signals with a 10-nm band can be resolved in the range between 410 and 700 nm by this method, overcoming the tradeoff between spectral resolution and time resolution. As a proof of demonstration, we present its potential usage for metamerism recognition, showing superiority over traditional color cameras with more spectral details. Considering its low cost, miniaturization and monolithic-integration ability on color sensors, this simple approach may bring the spectral imaging technology closer to the consumer market and even to ubiquitous smartphones for health care, food inspection and other applications.

Inverse design of a light nanorouter for a spatially multiplexed optical filter.

It is attractive to use an optical nanorouter by artificial nanostructures to substitute the traditional Bayer filter for an image array sensor, which, however, poses great challenges in balancing the design strategy and the ease of fabrication. Here, we implement and compare two inverse design schemes for rapid optimization of RGGB Bayer-type optical nanorouter. One is based on the multiple Mie scattering theory and the adjoint gradient that is applicable to arrays of nanospheres with varying sizes, and the other is based on the rigorous coupled wave analysis and the genetic algorithm. In both cases, we study layered nanostructures that can be efficiently modeled respectively which greatly accelerates the inverse design. It is shown that the color-dependent peak collection efficiencies of nanorouters designed in the two methods for red, green, and blue wavelengths reach 37%, 44%, and 45% and 52%, 50%, and 66%, respectively. We further demonstrate color nanorouters that provide light focusing to four quadrants working in both the visible and infrared bands, which promises multispectral imaging applications.

Imaging low-energy positron beams in real-time with unprecedented resolution

Particle beams focused to micrometer-sized spots play a crucial role in forefront research using low-energy positrons. Their expedient and wide application, however, requires highly-resolved, fast beam diagnostics. We have developed two different methods to modify a commercial imaging sensor to make it sensitive to low-energy positrons. The first method consists in removing the micro-lens array and Bayer filter from the sensor surface and depositing a phosphor layer in their place. This procedure results in a detector capable of imaging positron beams with energies down to a few tens of eV, or an intensity as low as 35e+/s/mm2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${35}\\,{\\hbox {e}^+/\\hbox {s/mm}^{2}}$$\\end{document} when the beam energy exceeds 10 keV\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${\\hbox {keV}}$$\\end{document}. The second approach omits the phosphor deposition; with the resulting device we succeeded in detecting single positrons with energies upwards of 6keV\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${6}\\,{\\hbox {keV}}$$\\end{document} and efficiency up to 93%. The achieved spatial resolution of 0.97 μm\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\upmu \\hbox {m}$$\\end{document} is unprecedented for real-time positron detectors.

Direct feature extraction and image co-registration of morphological structure from Bayer pattern raw planetary images

Feature extraction from planetary remote sensing images is a primary image processing task for object recognition, crater counter, morphological structure dimension and unpaired image co-registration. The paper presents a novel methodology to extract features directly from Bayer Pattern raw planetary images. Gradient information is extracted from Bayer Pattern image using standard edge operator that follows Color Difference Constancy (CDC) assumption. The proposed method's advantage is that it can skip the computationally intensive image processing pipeline and can have the Bayer Pattern raw planetary image flow directly for useful information extraction. Sobel Edge Detector is applied on Indian Mars Color Camera (MCC) Bayer Pattern raw images, and Gradient Magnitude Map (GMM) is generated at different Martian terrains. In addition, we have developed a direct image co-registration approach for MCC Bayer intensity raw image with respect to Mars Digital Image Model (MDIM) 2.1 reference using Mode-Mean Combo Patch Filler and Gradient Intensity induced Scale Invariant Feature Transform (GI-SIFT) based feature matching. The outlier matched points are removed by Feature Similarity Score guided Random Sample Consensus (FSS-RANSAC) estimation technique. The visual evaluation and quantitative metrics indicate that GMM from MCC Bayer Pattern raw image has negligible degradation with respect to GMM extracted from MCC demosaic image. The Root Mean Square Error (RMSE) is computed at different Mars regions, and it is found that the average image co-registration accuracy is less than 0.5 pixel.

Polarization image demosaicing and RGB image enhancement for a color polarization sparse focal plane array.

The color division of focal plane (DoFP) polarization sensor structure mostly uses Bayer filter and polarization filter superimposed on each other, which makes the polarization imaging unsatisfactory in terms of photon transmission rate and information fidelity. In order to obtain high-resolution polarization images and high-quality RGB images simultaneously, we simulate a sparse division of focal plane polarization sensor structure, and seek a sweet spot of the simultaneous distribution of the Bayer filter and the polarization filters to obtain both high-resolution polarization images and high-quality RGB images. In addition, From the perspective of sparse polarization sensor imaging, leaving aside the traditional idea of polarization intensity interpolation, we propose a new sparse Stokes vector completion method, in which the network structure avoids the introduction and amplification of noise during polarization information acquisition by mapping the S1 and S2 components directly. The sparsely polarimetric image demosaicing (Sparse-PDM) model is a progressive combined structure of RGB image artifact removal enhancement network and sparsely polarimetric image completion network, which aims to compensate sparsely polarimetric Stokes parameter images with the de-artifacts RGB image as a guide, thus achieving high-quality polarization information and RGB image acquisition. Qualitative and quantitative experimental results on both self-constructed and publicly available datasets prove the superiority of our method over state-of-the-art methods.

A multi-channel approach for detecting tampering in colour filter images

Most devices capable of capturing photographs are provided with a Colour Filter Array (CFA). Images generated by these devices can be analysed to detect the so known CFA artefacts left by demosaicing methods. These artefacts are important in digital image forensics since they prove useful for determining the authenticity of an image. Previous works have focused exclusively on the analysis of the green band of the Bayer filter, neglecting the information in the remaining bands. In this work a process to estimate the pattern left by CFA artefacts regardless of their configuration is proposed, which can be used regardless of the size or colours used by the filter, or even in different colour spaces, obtaining in this way new sources of information for forensic analysis. A review and comparisons against state-of-the-art projects addressing the detection of manipulations using CFA artefacts are performed. The sensitivity of these proposals to post-processing methods such as compression and filtering will also be reviewed. We will also observe experimentally that the proposed technique can detect manipulations even in images affected by high levels of JPEG compression.

Research on the Modulation Transfer Function Detection Method of a Bayer Filter Color Camera.

Bayer filter color cameras are more and more widely used in the field of aerospace remote sensing, but the Bayer filter causes great degradation in image quality; therefore, obtaining a means of achieving the high-precision measurement of the modulation transfer function (MTF) of Bayer filter color cameras is an urgent problem. In order to solve this problem, this paper develops a slanted-edge method via three steps: the detection of the slanted edge, the acquisition and processing of the edge spread function (ESF), and the acquisition and processing of the line spread function (LSF). A combination of the Canny operator and Hough transform is proposed for the detection of the slanted edge, which improves the fitting accuracy and anti-interference ability of the algorithm. Further, the Canny operator is improved by constructing an adaptive filter function and introducing the Otsu method, which can more effectively smooth the image and remove its false edges. A method of processing ESF data by combining cubic spline interpolation and Savitzky-Golay (SG) filtering is proposed, which reduces the effects of noise and the non-uniform sampling of ESF on MTF. A method of LSF processing using Gaussian function fitting is proposed to further reduce the effect of noise on MTF. The improved algorithm is verified by the MTF measurement test applied to a specific type of Bayer filter color space camera. The simulation and test results show that the improved slanted-edge method discussed in this paper has greater precision and a better anti-interference ability, and it can effectively solve the difficult problem associated with MTF detection in Bayer filter color space cameras.

Proposed Extensions to the Third Edition of JPEG XS (ISO/IEC 21122) Standard

At its 94th meeting (17–21 January 2022), International Organization for Standardization (ISO)/IEC SC29WG1 (also known as JPEG Committee) decided to create a third edition of the JPEG XS standard (ISO/IEC 21122) for lightweight, low-latency image coding. While existing coding tools in the standard already support the compression of natural content and color filter array (CFA) RAW Bayer pattern data well, this third edition will extend the previous editions with coding tools that improve its performance on the compression of screen content and image sequences with static background. The first experiments demonstrate that the proposed coding tools can improve the average peak signal-to-noise ratio (PSNR) performance of JPEG XS on such sequences of more than 10 dB, and approximately 3 dB on computer-generated content. In this work, the authors will present the proposed coding tools and will evaluate their performance objectively by PSNR.