Bayer Pattern Research Articles

Bionic Birdlike Imaging Using a Multi-Hyperuniform LED Array.

Digital cameras obtain color information of the scene using a chromatic filter, usually a Bayer filter, overlaid on a pixelated detector. However, the periodic arrangement of both the filter array and the detector array introduces frequency aliasing in sampling and color misregistration during demosaicking process which causes degradation of image quality. Inspired by the biological structure of the avian retinas, we developed a chromatic LED array which has a geometric arrangement of multi-hyperuniformity, which exhibits an irregularity on small-length scales but a quasi-uniformity on large scales, to suppress frequency aliasing and color misregistration in full color image retrieval. Experiments were performed with a single-pixel imaging system using the multi-hyperuniform chromatic LED array to provide structured illumination, and 208 fps frame rate was achieved at 32 × 32 pixel resolution. By comparing the experimental results with the images captured with a conventional digital camera, it has been demonstrated that the proposed imaging system forms images with less chromatic moiré patterns and color misregistration artifacts. The concept proposed verified here could provide insights for the design and the manufacturing of future bionic imaging sensors.

Analysis and Comparison of Three Classical Color Image Interpolation Algorithms

With the development of society, the progress of science and technology, the level of computer has been improved. Digital imaging technology has also achieved unprecedented development. The most commonly used digital cameras are Charge-coupled device (CCD) and Complementary Metal Oxide Semiconductor (CMOS) image sensors. Bayer filters are often used to help sensors capture color information. However, only one component can be achieved per pixel. Full-color images must be obtained by color interpolation. Therefore, the color interpolation algorithm has attracted the attention of the academic community. In this paper, three classical color interpolation algorithms- Bilinear interpolation algorithm, COK Algorithm, Gradient Interpolation Algorithm- are compared and studied. The Optimal Algorithm is selected by comparing false color, noise and color moire fringe. After comparison, it is found that the three basic algorithms will have color moire fringes appear. But the gradient interpolation algorithm has the least moire fringe and noise. COK Algorithm, second of all. But the COK Algorithm has color overflows.

Effect of Temperature on Channel Compensation in Optical Camera Communication

General-purpose Complementary Metal Oxide Semiconductor (CMOS) sensors perform the image desegregation in three channels (red, green, and blue) as a result of a band-pass wavelength filtering carried out using Foveon or Bayer filters. This characteristic can be used in Optical Camera Communication (OCC) systems for increasing the links’ data rate by introducing Wavelength Division Multiplexing (WDM) or Color Shift Keying (CSK) modulation schemes. However, these techniques need a compensation stage to mitigate the cross-talk between channels introduced by the filters. This compensation is performed by a Channel State Information (CSI) estimation and a zero-forcing compensation scheme. The impact of the temperature effects of light-emitting diode (LED) emissions on the zero-forcing compensation scheme’s performance has not been analyzed in depth. This work presents a comprehensive methodology and experimental characterization of this impact for Foveon and Bayer-based image sensors, assuming that the CSI is estimated under temperature conditions different from the LED’s stationary temperature regime. Besides, Signal-to-Interference-plus-Noise Ratio (SINR) and Bit Error Rate (BER) performance metrics are presented in order to estimate the repercussion in an OCC link. The results reveal that the Foveon sensor obtains more unsatisfactory performance than the Bayer-based sensor. On the other hand, the blue band is the most penalized by the thermal effect.

Gradient-Based Feature Extraction From Raw Bayer Pattern Images.

In this paper, the impact of demosaicing on gradient extraction is studied and a gradient-based feature extraction pipeline based on raw Bayer pattern images is proposed. It is shown both theoretically and experimentally that the Bayer pattern images are applicable to the central difference gradient-based feature extraction algorithms with negligible performance degradation, as long as the arrangement of color filter array (CFA) patterns matches the gradient operators. The color difference constancy assumption, which is widely used in various demosaicing algorithms, is applied in the proposed Bayer pattern image-based gradient extraction pipeline. Experimental results show that the gradients extracted from Bayer pattern images are robust enough to be used in histogram of oriented gradients (HOG)-based pedestrian detection algorithms and shift-invariant feature transform (SIFT)-based matching algorithms. By skipping most of the steps in the image signal processing (ISP) pipeline, the computational complexity and power consumption of a computer vision system can be reduced significantly.

A Low-Complexity End-to-End Stereo Matching Pipeline From Raw Bayer Pattern Images to Disparity Maps

Conventional computer vision algorithms, including stereo matching algorithms, take finely rendered color images as input. However, existing image signal processing (ISP) pipelines for color image generation are designed for photography with a goal of generating pleasing images for human eyes. This paper describes a new end-to-end pipeline for stereo matching from raw Bayer pattern images to disparity maps with customized ISP. Unlike conventional stereo matching systems which need a complete ISP module to render full-size standard RGB (sRGB) images, a subsampling-based demosaicing-downsampling (SDD) operation is introduced in the proposed pipeline to demosaic and downsample the Bayer pattern images. The resultant half-size color image pairs are processed with simple denoising and tone mapping algorithms to generate the final input images of stereo matching algorithms. It is found that the simple nearest neighbor upsampling method is good enough to generate the final full-size disparity maps. Experimental results show that the proposed pipeline is capable of generating comparable or even better stereo matching results than the conventional pipeline. By skipping most of the unnecessary ISP steps and reducing the size of input images, the computational complexity of the end-to-end stereo matching pipeline is significantly reduced.

Image Tampering Localization Using Demosaicing Patterns and Singular Value Based Prediction Residue

Almost all image sensors measure only one color per pixel through the color filter array. Missing pixels are estimated using a demosaicing process. For this reason, a demosaiced image leaves a particular trace. When an image is manipulated or tampered, the demosaicing trace can be changed. This change can serve as a basic clue for detecting or localizing image tampering. Demosaicing pattern-based tampering localization algorithms require a re-interpolation process, and the prediction residue between the given image and the re-interpolated image is commonly used to localize tampered regions. However, the prediction residue is not always valid because the demosaicing interpolation kernel cannot be known, which deteriorates the localization performance. This paper presents an effective re-interpolation process using singular value decomposition for an unknown demosaicing method. First, the green channel of the given image is decomposed into four sub-images according to the Bayer pattern. For a small block of each sub-image, the singular value decomposition is performed. The prediction residue is obtained by reconstructing the image block after removing the largest singular value. The feature to localize the forged regions is extracted by the logarithm ratio of the prediction residue variance. The proposed method does not require any statistical model for the extracted feature, because the prediction residue is more accurate than that of conventional methods. We perform intensive experiments for three test datasets and compare the proposed method with state-of-the-art tampering localization methods, the results of which indicate that the proposed scheme outperforms existing approaches.

Low-Light Demosaicking and Denoising for Small Pixels Using Learned Frequency Selection

Low-light imaging is a challenging task because of the excessive photon shot noise. Color imaging in low-light is even more difficult because one needs to demosaick and denoise simultaneously. Existing demosaicking algorithms are mostly designed for well-illuminated scenarios, which fail to work with low-light. Recognizing the recent development of small pixels and low read noise image sensors, we propose a learning-based joint demosaicking and denoising algorithm for low-light color imaging. Our method combines the classical theory of color filter arrays and modern deep learning. We use an explicit carrier to demodulate the color from the input Bayer pattern image. We integrate trainable filters into the demodulation scheme to improve flexibility. We introduce a guided filtering module to transfer knowledge from the luma channel to the chroma channels, thus offering substantially more reliable denoising. Extensive experiments are performed to evaluate the performance of the proposed method, using both synthetic datasets and real data. Results indicate that the proposed method offers consistently better performance over the current state-of-the-art, across several standard evaluation metrics.

Detailed characterization of a mosaic based hyperspectral snapshot imager

Some widely used optical measurement systems require a scan in wavelength or in one spatial dimension to measure the topography in all three dimensions. Novel hyperspectral sensors based on an extended Bayer pattern have a high potential to solve this issue as they can measure three dimensions in a single shot. This paper presents a detailed examination of a hyperspectral sensor including a description of the measurement setup. The evaluated sensor (Ximea MQ022HG-IM-SM5X5-NIR) offers 25 channels based on Fabry–Perot filters. The setup illuminates the sensor with discrete wavelengths under a specified angle of incidence. This allows characterization of the spatial and angular response of every channel of each macropixel of the tested sensor on the illumination. The results of the characterization form the basis for a spectral reconstruction of the signal, which is essential to obtain an accurate spectral image. It turned out that irregularities of the signal response for the individual filters are present across the whole sensor.

Laser wavelength metrology with low-finesse etalons and Bayer filters

We present a wavelength meter with picometer-scale resolution based on etaloning effects of inexpensive glass slides and the built-in color filters of a consumer grade CMOS camera. After calibrating the device to a commercial meter, we tested the device's calibration stability using two tunable visible lasers for a period of over 16 days. The wavelength error over that entire period has a standard deviation of 5.29 parts per million (ppm) about a most probable error of 0.90 ppm. Within 24 hours of calibration, this improves to 0.04 ppm with a standard deviation of 3.94 ppm.

Comparison of Denoising Algorithms for Demosacing Low Lighting Images Using CFA 2.0

In modern digital cameras, the Bayer color filter array (CFA) has been widely used. It is also widely known as CFA 1.0. However, Bayer pattern is inferior to the red-green-blue-white (RGBW) pattern, which is also known as CFA 2.0, in low lighting conditions in which Poisson noise is present. It is well known that demosaicing algorithms cannot effectively deal with Poisson noise and additional denoising is needed in order to improve the image quality. In this paper, we propose to evaluate various conventional and deep learning based denoising algorithms for CFA 2.0 in low lighting conditions. We will also investigate the impact of the location of denoising, which refers to whether the denoising is done before or after a critical step of demosaicing. Extensive experiments show that some denoising algorithms can indeed improve the image quality in low lighting conditions. We also noticed that the location of denoising plays an important role in the overall demosaicing performance.

High-throughput measurement of coefficient of thermal expansion using a high-resolution digital single-lens reflex camera and digital image correlation.

High-throughput measurement of thermal deformation and determination of coefficient of thermal expansion (CTE) using a high-resolution digital single lens reflex (DSLR) camera and digital image correlation (DIC) is described. To mitigate the mosaic effect caused by the Bayer filter of DSLR cameras, a color image pre-processing method, which adjusts the brightness and equalizes the color channels of the raw image, is carried out. In addition, a Gaussian pre-filtering step is adopted for denoising the images captured with DSLR cameras to enhance the subpixel registration accuracy. Then, by processing the recorded images using the state-of-the-art DIC algorithm, full-field displacements and strains can be determined. Compared with conventional industrial cameras, a DSLR camera offers not only portability, compactness, and economy but also much higher resolution of recorded images, allowing CTE characterization with higher throughput. Real experiments, including a verification experiment of the color image pre-processing technique, a benchmark CTE determination of Al alloy, and a high-throughput CTE determination of 15 samples of three different metals, validated the feasibility and accuracy of the proposed technique. The proposed method is cost-effective and time-saving, showing great potential in the high-throughput CTE measurement and other high-throughput strain measurement scenarios.

Bias-selected full Red/Green/Blue color sensing and imaging based on inversely stacked radial PINIP junctions

Seeking an ultra-compact new architecture or strategy to achieve filter-less Red/Green/Blue (RGB) color sensing is critical for developing a wide range of high-resolution, low-cost and flexible color discrimination and imaging applications. In this work, a new inversely stacked biomimetic radial junction (RJ) PINIP photodetector, comprising of coaxial multilayers of intrinsic and doped hydrogenated amorphous Si (a-Si:H) thin films coated over silicon nanowires (SiNWs), is demonstrated, which can accomplish bias-selected and tunable spectrum responses to the R, G and B color bands, mimicking the response mechanism in human eyes. Compared to the Bayer matrix sensors, the width of a fundamental RJ-PINIP sensing unit is reduced from ∼3 μm to <500 nm, with an absorber i-layer thinned substantially from ∼1 μm to only ∼50 nm, which is particularly important to achieve a faster photo response and suppress the instability issues in a-Si:H absorber. Notably, the 3D RJ-PINIP sensors can now be easily constructed upon flexible aluminum foils (AF) and operate with a convenient two-terminal connection, which is a prerequisite for scalable matrix device implementation. As a prototype demo, the RGB color sensing and reconstruction functionalities are verified in a 125-RGB color imaging with 520*700 pixels, highlighting their unique potential for developing ultra-compact and flexible RGB imaging applications.

Demosaicing of CFA 3.0 with Applications to Low Lighting Images.

Low lighting images usually contain Poisson noise, which is pixel amplitude-dependent. More panchromatic or white pixels in a color filter array (CFA) are believed to help the demosaicing performance in dark environments. In this paper, we first introduce a CFA pattern known as CFA 3.0 that has 75% white pixels, 12.5% green pixels, and 6.25% of red and blue pixels. We then present algorithms to demosaic this CFA, and demonstrate its performance for normal and low lighting images. In addition, a comparative study was performed to evaluate the demosaicing performance of three CFAs, namely the Bayer pattern (CFA 1.0), the Kodak CFA 2.0, and the proposed CFA 3.0. Using a clean Kodak dataset with 12 images, we emulated low lighting conditions by introducing Poisson noise into the clean images. In our experiments, normal and low lighting images were used. For the low lighting conditions, images with signal-to-noise (SNR) of 10 dBs and 20 dBs were studied. We observed that the demosaicing performance in low lighting conditions was improved when there are more white pixels. Moreover, denoising can further enhance the demosaicing performance for all CFAs. The most important finding is that CFA 3.0 performs better than CFA 1.0, but is slightly inferior to CFA 2.0, in low lighting images.

Color contamination correction based on light intensity correlation in two-color, double-exposure particle tracking velocimetry

An algorithm for correcting color contamination is proposed for two-color, double-exposure particle tracking velocimetry (PTV). A two-color PTV system used in this study consists of blue and green laser diodes and a consumer digital camera, where a laser pulser with an avalanche transistor is developed for achieving optical pulses of 50 ns for application in airflows. In the PTV, the camera captures images of tracer particles illuminated by a sequence of green and blue light pulses with a certain time interval. Because of spectral characteristics of a Bayer filter, camera sensors in a blue channel respond to green light scattered by the particles. This color contamination results in pseudo-particles in the blue channel. Pixels occupied by the pseudo-particles have very high correlation of light intensity between green and blue channels. In the proposed method, the pseudo-particles caused by the color contamination are detected and removed based on the high correlation. The present color contamination correction hardly affects real particles illuminated by the blue laser diodes. Measurements of an airflow induced by DC fans confirm that the proposed system with the color contamination correction works well for PTV.

Histogram of Oriented Gradients Feature Extraction From Raw Bayer Pattern Images

This brief studies the redundancy in the image processing pipeline for histogram of oriented gradients (HOG) feature extraction. The impact of demosaicing on the extracted HOG features is analyzed and experimented. It is shown that by taking advantage of the inter-channel correlation of natural images, the HOG features can be directly extracted from the Bayer pattern images with proper gamma compression. Due to the elimination of the image processing pipeline, the power consumption and computational complexity of the detection system can be significantly reduced. Experimental results show that the Bayer pattern image-based HOG features can be used in pedestrian detection systems with little performance degradation.

FPGA Acceleration of Color Interpolation Algorithm Based on Gradient and Color Difference

To quickly and efficiently achieve a high-quality color image on the embedded platform, designed and used FPGA to complete Bayer pattern image Color interpolation. In this paper, we proposed a color interpolation algorithm based on image gradient direction and color difference. The algorithm calculates the G component according to the gradient value and color difference at the pixel to be interpolated, and then calculates the missing R component and B component based on the G component interpolation. Finally, the pixels of the image boundary area are interpolated. Tested on the Kodak dataset, the experimental results show that the subjective visual quality and objective image quality evaluation of the algorithm is superior to the conventional algorithm. On the other hand, and reduce problems such as image aliasing and edge blurring. The proposed algorithm has been applied to the actual Xilinx’s FPGA platform, a real-time speed of 120 MHz, enabling real-time image color Interpolation.

ANÁLISE DE DISTORÇÕES RADIOMÉTRICAS DE IMAGENS DA CÂMERA TETRACAM ADC LITE

Muitos trabalhos executados com Aeronaves Remotamente Pilotadas (ARP) tem se valido da utilização da câmera multiespectral ADC Lite da fabricante Tetracam. As imagens produzidas pela câmera são analisadas e interpretadas qualitativamente e quantitativamente sem que se tenha procedido à correção de distorções radiométricas das mesmas, tarefa que o rigor científico exige para que qualquer conclusão resultante tenha validade prática e acadêmica. Neste trabalho ficou demonstrado que distorções tais como read noise, não-uniformidade de resposta dos pixels à radiância proveniente dos alvos e vignetting estão presentes nas imagens. Para além destas, os pesos do arquivo de interpolação do filtro de Bayer fornecidos pela Tetracam produzem resultados incorretos em muitos casos, ocasionando a perda de informação, particularmente na banda do Vermelho, com frequências significativas (~105) de pixels com valores saturados para preto. A correção de todos estes efeitos e de artefatos remanescentes, nas bordas das imagens, do processo de interpolação dos valores de brilho, é imprescindível para que se possa trabalhar com as imagens, tal qual demonstra-se neste artigo, por meio da comparação de índices obtidos da razão de bandas a partir das imagens com e sem distorções.

A High‐Resolution Multifocal‐Plane Virtual Microscope for Teaching Histology

Virtual microscopy (using a computer to view digitally‐scanned microscope slides) is widely employed in teaching histology labs. The increase in the memory capacity of computer servers now makes it possible for virtual microscopy to display virtual slides whose resolution equals that obtained using “real” microscope slides and microscopes with high numerical aperture (NA) oil‐immersion objectives. Because high NA objectives have a very narrow depth‐of‐field, the ability to focus up‐and‐down through the specimen becomes very useful when examining high‐resolution images. We describe here a process for creating multifocal‐plane virtual‐microscope slides whose resolution approaches the theoretical diffraction limit for a light microscope (~250nm) and the creation of a virtual microscope, intended for use in histology teaching labs, that allows the user to focus up‐and‐down through these high‐resolution, multifocal‐plane virtual slides.Our first step in building a high‐resolution, multifocal‐plane virtual microscopy system was to scan histology slides using a Zeiss AxioImager.M2 microscope equipped with a 63x NA 1.4 PlanApo oil‐immersion objective. The final digital magnification of these specimens is 14.4 pixels/μm, giving a theoretical “digital resolution” for these specimens of: 2 pixels ÷ 14.4 pixels/μm = ~140nm. The “digital resolution” is decreased somewhat by the RGB camera’s Bayer filter, but the overall resolution obtained for virtual‐slides scanned by this system is close to the theoretical “optical resolution” of ~250nm.The second step in building our virtual microscopy system was to convert the scanned digital image into a format suitable for our microscope viewer; we chose to use jpeg images in an image‐tile architecture similar to that used by Google Maps. Each focal plane of a virtual‐slide is exported from the Zeiss (*.czi) format as a separate bigTIFF file, and the resulting TIFF file is then converted into a Google‐Maps‐compatible image pyramid using the libvips image processing library. Thus, a multifocal plane virtual slide is converted into a series of separate, but aligned, image pyramids.The final step was to create a virtual microscope that is capable of displaying our multifocal plane digital specimens. HTML/javascript programming was used to create a client‐side viewer that runs within the user’s internet browser. This viewer allows the user to zoom‐in/out to view the specimen at varying magnifications, and to focus up‐and‐down through the specimen. Most of the computing needed to display the specimen is done on the client‐side by the viewer, with the viewer only calling on the server for information about the slide and for image tiles. To provide a responsive interface, the viewer maintains a buffer of images from the focal planes and zoom‐levels adjacent to the image currently being viewed.One of the keys to understanding the microscopic anatomy of an organ is being able to visualize the 3‐dimensional relationships of the cells and tissues comprising the organ. The virtual microscopy system described here clearly demonstrates the utility of being able to focus through a histological specimen at high magnification when attempting to visualize these 3‐dimensional relationships.

A single sensor based multispectral imaging camera using a narrow spectral band color mosaic integrated on the monochrome CMOS image sensor

A multispectral image camera captures image data within specific wavelength ranges in narrow wavelength bands across the electromagnetic spectrum. Images from a multispectral camera can extract a additional information that the human eye or a normal camera fails to capture and thus may have important applications in precision agriculture, forestry, medicine, and object identification. Conventional multispectral cameras are made up of multiple image sensors each fitted with a narrow passband wavelength filter and optics, which makes them heavy, bulky, power hungry, and very expensive. The multiple optics also create an image co-registration problem. Here, we demonstrate a single sensor based three band multispectral camera using a narrow spectral band red–green–blue color mosaic in a Bayer pattern integrated on a monochrome CMOS sensor. The narrow band color mosaic is made of a hybrid combination of plasmonic color filters and a heterostructured dielectric multilayer. The demonstrated camera technology has reduced cost, weight, size, and power by almost n times (where n is the number of bands) compared to a conventional multispectral camera.

Estimation of available epinephrine dose in expired and discolored autoinjectors via quantitative smartphone imaging.

Epinephrine autoinjectors (EAIs) are important first aid medications for treating anaphylaxis. A 10-fold price increase over the past 12years and evidence that expired EAIs may still contain significant doses of available epinephrine have motivated interest in the efficacy of expired EAIs as treatments of last resort. Degradation of expired EAIs, which can be caused by improper storage conditions, results in various degrees of discoloration of the epinephrine solution. Previous studies have determined that significant epinephrine remains available in expired EAIs, but these have only considered EAIs that show no discoloration. Here, we investigate the potential for colorimetric estimation of available epinephrine dose based on the degree of discoloration in expired EAIs. The correlation of available epinephrine dose and time since expiration date was poor (r = - 0.37), as determined by an industry standard UHPLC protocol. Visible absorbance of the samples integrated across the range 430-475nm correlated well with available epinephrine dose (r= - 0.71). This wavelength corresponds to the blue channel of a typical smartphone camera Bayer filter. Smartphone camera images of the EAI solutions in various illumination conditions were analyzed to assign color indices representing the degree of discoloration. Color index of the samples showed similar correlation (|r| > 0.7) with available epinephrine dose as that of visible spectrophotometry. Smartphone imaging colorimetry is proposed as a potential point-of-use epinephrine dose estimator for expired and degraded EAIs. Graphical abstract.