Demosaicking Algorithm Research Articles

Fourier-domain filtering analysis for color-polarization camera demosaicking.

We review Fourier-domain methods for demosaicking Bayer-filter color cameras and monochrome polarization cameras, and then generalize the approach for the quad-Bayer-filter mosaic and for color-polarization cameras. For each of these four mosaic filter types, we provide theoretical expressions for the sampling functions, the Fourier-domain channels, and the linear combination of reconstructed channels (the demosaicking algorithm) needed to estimate the input (presampled) image. A useful advantage of the Fourier-domain approach is that it provides a direct means of visualizing and quantifying when aliasing is likely or unlikely to be present. For the Bayer and quad-Bayer-filter types, we provide simulated images, while for the polarization camera types we provide experimental images and videos to illustrate the algorithm and analyze crosstalk error.

Joint constraints of guided filtering based confidence and nonlocal sparse tensor for color polarization super-resolution imaging.

This paper introduces a camera-array-based super-resolution color polarization imaging system designed to simultaneously capture color and polarization information of a scene in a single shot. Existing snapshot color polarization imaging has a complex structure and limited generalizability, which are overcome by the proposed system. In addition, a novel reconstruction algorithm is designed to exploit the complementarity and correlation between the twelve channels in acquired color polarization images for simultaneous super-resolution (SR) imaging and denoising. We propose a confidence-guided SR reconstruction algorithm based on guided filtering to enhance the constraint capability of the observed data. Additionally, by introducing adaptive parameters, we effectively balance the data fidelity constraint and the regularization constraint of nonlocal sparse tensor. Simulations were conducted to compare the proposed system with a color polarization camera. The results show that color polarization images generated by the proposed system and algorithm outperform those obtained from the color polarization camera and the state-of-the-art color polarization demosaicking algorithms. Moreover, the proposed algorithm also outperforms state-of-the-art SR algorithms based on deep learning. To evaluate the applicability of the proposed imaging system and reconstruction algorithm in practice, a prototype was constructed for color polarization image acquisition. Compared with conventional acquisition, the proposed solution demonstrates a significant improvement in the reconstructed color polarization images.

Enhancing the Performance of Optical Hardware through Software Applications: A Study

The process of image demosaicing, essential for reconstructing fullcolour images from incomplete colour samples obtained from a colourfilter array (CFA), has garnered renewed interest, particularly with the prevalent use of the Bayer pattern. This resurgence can be attributed to the growing availability of source codes and executables, facilitating reproducible research in the field. In this article, a thorough survey encompassing over fifty published works on demosaicing since 1999. The objective is to complement prior reviews and provide insights into the evolution and current landscape of demosaicing techniques. By addressing key issues and delineating fundamental differences among various approaches, a thorough overview of the field is made. A notable finding from the survey is the popularity of spatial demosaicing methods, where the initial step involves interpolating the luminance channel, followed by reconstructing the chrominance channels based on the recovered luminance information. The article highlights three crucial areas requiring additional research. Firstly, addressing the difficulty of demosaicking images with low spectral correlation is paramount, along with enhancing our comprehension of the trade-off between spatial correlation and spectral correlation. It is also found that there is a necessity for more comprehensive investigation into evaluating the performance of demosaicking algorithms, particularly concerning the utilization of reference images, and understanding supplementary components within the imaging pipeline.

Unsupervised design for broadband multispectral and polarization filter array patterns.

Imaging multiple wavelength and polarization components is problematic due to the complexity of equipment and the increase in the number of imaging shots, so imaging using filter arrays with various patterns has been widely reported from elemental research to practical applications. Most of them use bandpass filters with different center wavelengths for each pixel. Recently, however, filter arrays with multimodal transmission characteristics have been proposed using photonic crystals or Fabry-Perot filters. In any of these methods, the design of the filter array arrangement pattern is important to improve the quality of the captured image, as well as the improvement of the demosaicking algorithm. One way to design a filter array pattern is to minimize the mean squared error (MSE) between the ideal image and the demosaicked image. However, the more multidimensional the imaging components, the more difficult it becomes to collect training data. In such cases, it is necessary to empirically determine candidate transmission characteristics and patterns of filter arrays. In this study, we propose a method for evaluating filter array patterns without using any training data in the design of filter arrays for multispectral and polarization imaging. The proposed method estimates the MSE by approximating the autocorrelation matrix without using image data by expressing the imaging model as a linear forward problem and the demosaicking as a linear inverse problem. Since this method can be applied not only to ideal bandpass filter arrangements, but also to multispectral filter arrays with multimodal spectral transmission characteristics and even multispectral polarization filter arrays with different extinction ratios at different wavelengths, we will show that image quality can be improved over empirical arrangements by evaluating these patterns and by testing examples of optimal designs using genetic algorithms.

Joint demosaicking and denoising benefits from a two-stage training strategy

Image demosaicking and denoising are the first two key steps of the color image production pipeline. The classical processing sequence has for a long time consisted of applying denoising first, and then demosaicking. Applying the operations in this order leads to oversmoothing and checkerboard effects. Yet, it was difficult to change this order, because once the image is demosaicked, the statistical properties of the noise are dramatically changed and hard to handle by traditional denoising models. In this paper, we address this problem by a hybrid machine learning method. We invert the traditional color filter array (CFA) processing pipeline by first demosaicking and then denoising. Our demosaicking algorithm, trained on noiseless images, combines a traditional method and a residual convolutional neural network (CNN). This first stage retains all known information, which is the key point to obtain faithful final results. The noisy demosaicked image is then passed through a second CNN restoring a noiseless full-color image. This pipeline order completely avoids checkerboard effects and restores fine image detail. Although CNNs can be trained to solve jointly demosaicking–denoising end-to-end, we find that this two-stage training performs better and is less prone to failure. It is shown experimentally to improve on the state of the art, both quantitatively and in terms of visual quality.

Spatial gradient consistency for unsupervised learning of hyperspectral demosaicking: application to surgical imaging

PurposeHyperspectral imaging has the potential to improve intraoperative decision making if tissue characterisation is performed in real-time and with high-resolution. Hyperspectral snapshot mosaic sensors offer a promising approach due to their fast acquisition speed and compact size. However, a demosaicking algorithm is required to fully recover the spatial and spectral information of the snapshot images. Most state-of-the-art demosaicking algorithms require ground-truth training data with paired snapshot and high-resolution hyperspectral images, but such imagery pairs with the exact same scene are physically impossible to acquire in intraoperative settings. In this work, we present a fully unsupervised hyperspectral image demosaicking algorithm which only requires exemplar snapshot images for training purposes.MethodsWe regard hyperspectral demosaicking as an ill-posed linear inverse problem which we solve using a deep neural network. We take advantage of the spectral correlation occurring in natural scenes to design a novel inter spectral band regularisation term based on spatial gradient consistency. By combining our proposed term with standard regularisation techniques and exploiting a standard data fidelity term, we obtain an unsupervised loss function for training deep neural networks, which allows us to achieve real-time hyperspectral image demosaicking.ResultsQuantitative results on hyperspetral image datasets show that our unsupervised demosaicking approach can achieve similar performance to its supervised counter-part, and significantly outperform linear demosaicking. A qualitative user study on real snapshot hyperspectral surgical images confirms the results from the quantitative analysis.ConclusionOur results suggest that the proposed unsupervised algorithm can achieve promising hyperspectral demosaicking in real-time thus advancing the suitability of the modality for intraoperative use.

A new adaptive window‐based guided filtering and interpolation for polarization image demosaicing

AbstractThis paper presents a new adaptive window‐based guided filtering and interpolation for polarization image demosaicing. Considering the correlation between the polarization channels, the authors use the total intensity S0 as a guide image to guide the interpolation of all channels. The guided filtering is performed with a new adaptive window size selection technique. A novel iterative residual interpolation is proposed to construct accurate full resolution polarization. Experiments are conducted with the sample images from the demosaicking algorithms dataset. Objective evaluations demonstrate that the proposed algorithm can reduce root mean squared error by at least 27% over other methods. Subjective visual inspection can also verify the improvements in the clarity of objects in angle and degree of the linear polarization images.

Multispectral Filter Array Design by Optimal Sphere Packing.

Multispectral imaging (MSI) collects a datacube of spatio-spectral information of a scene. Many acquisition methods for spectral imaging use scanning, preventing its widespread usage for dynamic scenes. On the other hand, the conventional color filter array (CFA) method often used to sample color images has also been extended to snapshot MSI using a Multispectral Filter Array (MSFA), which is a mosaic of selective spectral filters placed over the Focal Plane Array (FPA). However, even state-of-the-art MSFAs coding patterns produce artifacts and distortions in the reconstructed spectral images, which might be due to the nonoptimal distribution of the spectral filters. To reduce the appearance of artifacts and provide tools for the optimal design of MSFAs, this paper proposes a novel mathematical framework to design MSFAs using a Sphere Packing (SP) approach. By assuming that each sampled filter can be represented by a sphere within the discrete datacube, SP organizes the position of the equal-size and disjoint spheres's centers in a cubic container. Our method is denoted Multispectral Filter Array by Optimal Sphere Packing (MSFA-OSP), which seeks filter positions that maximize the minimum distance between the spheres's centers. Simulation results show an image quality improvement of up to 2 dB and a remarkable boost in spectral similarity when using our proposed MSFA design approach for a variety of reconstruction algorithms. Moreover, MSFA-OSP notably reduces the appearance of false colors and zipper effect artifacts, often seen when using state-of-the-art demosaicking algorithms. Experiments using synthetic and real data prove that the proposed MSFA-OSP outperforms state-of-the-art MSFAs in terms of spatial and spectral fidelity. The code that reproduces the figures of this article is available at https://github.com/nelson10/DemosaickingMultispectral3DSpherePacking.git.

4-Band Multispectral Images Demosaicking Combining LMMSE and Adaptive Kernel Regression Methods

In recent years, multispectral imaging systems are considerably expanding with a variety of multispectral demosaicking algorithms. The most crucial task is setting up an optimal multispectral demosaicking algorithm in order to reconstruct the image with less error from the raw image of a single sensor. In this paper, we presented a four-band multispectral filter array (MSFA) with the dominant blue band and a multispectral demosaicking algorithm that combines the linear minimum mean square error (LMMSE) and the adaptive kernel regression methods. To estimate the missing blue bands, we used the LMMSE algorithm and for the other spectral bands, the directional gradient method, which relies on the estimated blue bands. The adaptive kernel regression is then applied to each spectral band for their update without persistent artifacts. The experiment results demonstrate that our proposed method outperforms other existing approaches both visually and quantitatively in terms of peak signal-to-noise-ratio (PSNR), structural similarity index (SSIM) and root mean square error (RMSE).

Pushing the colorimetry camera-based fluorescence microscopy to low light imaging by denoising and dye combination.

Colorimetry camera-based fluorescence microscopy (CCFM) is a single-frame imaging method for observing multiple biological events simultaneously. Compared with the traditional multi-color fluorescence microscopy methods based on sequential excitation or spectral splitting, the CCFM method simplifies multi-color fluorescence imaging experiments, while keeping a high spatial resolution. However, when the level of the detected fluorescence signal decreases, the image quality, the demosaicking algorithm precision, and the discrimination of fluorescence channels on the colorimetry camera will also decrease. Thus, CCFM has a poor color resolution under a low signal level. For example, the crosstalk will be higher than 10% when the signal is less than 100 photons/pixel. To solve this problem, we developed a new algorithm that combines sCMOS noise correction with demosaicking, and a dye selection method based on the spectral response characteristics of the colorimetry camera. By combining the above two strategies, low crosstalk can be obtained with 4 ∼ 6 fold fewer fluorescence photons, and low light single-frame four-color fluorescence imaging was successfully performed on fixed cos-7 cells. This study expands the power of the CCFM method, and provides a simple and efficient way for various bioimaging applications in low-light conditions.

Modified Newton-residual interpolation for division of focal plane polarization image demosaicking.

With the improvement of semiconductor processing technology, polarization sensors using division of focal plane have gradually become the mainstream method of polarization imaging. Similar to the color restoration method of the Bayer array sensor, the spatial information of polarized image is also recovered through the polarization demosaicking algorithm. In this paper, we propose a new modified Newton-residual interpolation polarization image demosaicking algorithm based on residual interpolation, which is suitable for a monochrome or color polarization filter array. First, we use the modified Newton interpolation method to generate edge-sensitive guiding images. Then, we carry out the improvement of the guide process during the residual interpolation by performing variance statistics on the local window image in the guiding process, so that the edges and flat image blocks have different guiding weights. Finally, we obtain edge-preserving results by applying these two improvements, which reduces the zipper effect and edge confusion. We compare the results of various algorithms on experimental data, demonstrating that our algorithm has impactful improvements in the evaluation metrics based on the ground-truth images.

Searching for Fast Demosaicking Algorithms

We present a method to automatically synthesize efficient, high-quality demosaicking algorithms, across a range of computational budgets, given a loss function and training data. It performs a multi-objective, discrete-continuous optimization which simultaneously solves for the program structure and parameters that best tradeoff computational cost and image quality. We design the method to exploit domain-specific structure for search efficiency. We apply it to several tasks, including demosaicking both Bayer and Fuji X-Trans color filter patterns, as well as joint demosaicking and super-resolution. In a few days on 8 GPUs, it produces a family of algorithms that significantly improves image quality relative to the prior state-of-the-art across a range of computational budgets from 10 s to 1000 s of operations per pixel (1 dB–3 dB higher quality at the same cost, or 8.5–200× higher throughput at same or better quality). The resulting programs combine features of both classical and deep learning-based demosaicking algorithms into more efficient hybrid combinations, which are bandwidth-efficient and vectorizable by construction. Finally, our method automatically schedules and compiles all generated programs into optimized SIMD code for modern processors.

FPGA-based real-time Bayer demosaicking algorithm and its implementation

Bayer阵列被广泛地应用于CMOS/CCD (complementary metal oxide semiconductor/charge-coupled device)等前端传感器中，用以对彩色图像进行压缩编码。通过解马赛克算法将Bayer阵列还原为红、绿、蓝彩色阵列，算法性能影响着成像有效分辨率和纹理细节。随着半导体工艺的发展和目标识别等新型应用需求的提出，图像设备向着高分辨率、低延迟的方向发展，原有的解马塞克算法遇到性能瓶颈。提出了一种基于FPGA (field programmable gate array)的实时解马赛克算法，能够准确地提取图像局部梯度方向并以引导实现色彩的插值复原，整体算法仅需7行数据延迟。算法设计充分考虑FPGA的硬件特点，设计了行缓存、梯度算子、梯度方向插值等模块，降低硬件开销。实验表明，本文算法在实现微米级延迟的同时，保持了对图像纹理细节区域的复原效果，在柯达数据集上的平均峰值信噪比达到38.26 dB。

Extension of luminance component based demosaicking algorithm to 4- and 5-band multispectral images

Multispectral imaging systems are currently expanding with a variety of multispectral demosaicking algorithms. But these algorithms have limitations due to the remarkable presence of artifacts in the reconstructed image. In this paper, we propose a powerful multispectral image demosaicking method that focuses on the G band and luminance component. We've first identified a relevant 4-and 5-band multispectral filter array (MSFA) with the dominant G band and then proposed an algorithm that consistently estimates the missing G values and other missing components using a convolution operator and a weighted bilinear interpolation algorithm based on the luminance component. Using the considered MSFA patterns, we've also demonstrated that our algorithm outperforms existing approaches both visually and quantitatively in terms of the PSNR and SSIM.

Demosaicking enhancement method for color image based on low-rank constraint

Demosaicking is a method to restore the mosaicked image generated by the color filter array to a full-color image. The performance of the demosaicking algorithm is crucial to the reconstruction of high-quality color images. However, current demosaicking algorithms inevitably cause zipper effects or false color artifacts in the process of demosaicking. We combine low-rank constraint based on spectral domain with image non-local similarity to correct the interpolation errors. Our method is applicable to various existing demosaicking algorithms and can further improve the quality of full color image reconstruction. Experimental results demonstrate that the proposed algorithm can improve the color peak signal to noise ratio of the reconstructed images.

A Mathematical Analysis and Implementation of Residual Interpolation Demosaicking Algorithms

Demosaicking is the process of reconstructing the full color image from its mosaic version on a Bayer pattern. It is an integral part of the image processing pipeline for single sensor digital color cameras. Demosaicking algorithms based on residual interpolation are interesting because they produce competitive results with a low computational complexity. In this article, we provide an analysis and careful implementation of the most relevant residual based demosaicking algorithms. Our contribution is twofold. First, we present an analysis of the mathematical principles of demosaicking algorithms from the Hamilton-Adams interpolation to the recent 'adaptive residual interpolation'. Our analysis untangles the relations of these algorithms and how each is improving on the preceding ones. Lastly, we provide a comparison between most recent state of the art methods on several image data sets and discuss their performances.

Linear Polarization Demosaicking for Monochrome and Colour Polarization Focal Plane Arrays

AbstractDivision‐of‐focal‐plane (DoFP) polarization image sensors allow for snapshot imaging of linear polarization effects with inexpensive and straightforward setups. However, conventional interpolation based image reconstruction methods for such sensors produce unreliable and noisy estimates of quantities such as Degree of Linear Polarization (DoLP) or Angle of Linear Polarization (AoLP). In this paper, we propose a polarization demosaicking algorithm by inverting the polarization image formation model for both monochrome and colour DoFP cameras. Compared to previous interpolation methods, our approach can significantly reduce noise induced artefacts and drastically increase the accuracy in estimating polarization states. We evaluate and demonstrate the performance of the methods on a new high‐resolution colour polarization dataset. Simulation and experimental results show that the proposed reconstruction and analysis tools offer an effective solution to polarization imaging.

Low-Light Demosaicking and Denoising for Small Pixels Using Learned Frequency Selection

Low-light imaging is a challenging task because of the excessive photon shot noise. Color imaging in low-light is even more difficult because one needs to demosaick and denoise simultaneously. Existing demosaicking algorithms are mostly designed for well-illuminated scenarios, which fail to work with low-light. Recognizing the recent development of small pixels and low read noise image sensors, we propose a learning-based joint demosaicking and denoising algorithm for low-light color imaging. Our method combines the classical theory of color filter arrays and modern deep learning. We use an explicit carrier to demodulate the color from the input Bayer pattern image. We integrate trainable filters into the demodulation scheme to improve flexibility. We introduce a guided filtering module to transfer knowledge from the luma channel to the chroma channels, thus offering substantially more reliable denoising. Extensive experiments are performed to evaluate the performance of the proposed method, using both synthetic datasets and real data. Results indicate that the proposed method offers consistently better performance over the current state-of-the-art, across several standard evaluation metrics.

A NOVEL ITERATIVE DEMOSAICING ALGORITHM USING FUZZY BASED DUAL TREE WAVELET TRANSFORM

In this research paper, an efficient demosaicking algorithm for Color Filter Array (CFA) is proposed to overcome the problem of reconstructing the correlated signals from the down sampled samples. The aim of this work is reconstruction of a full color image from the spatially under sampled color channels. The demosaicing is done by an iterative methodology for finding the values of missing pixels. The Dual Tree Complex Wavelet Transform (DTCWT) is used with the combination of fuzzy ‘if-then’ rule to refine the color plane values. The DTCWT has shift invariance and directionality property that achieves the perfect reconstruction of images and improves quality of the image. The new iterative interpolation algorithm uses color difference domain model and reduces false coloring by using an effective stopping criteria. The research work shows result as, the explicated interpolation method yields better psnr values than bilinear interpolation technique, edge based method, subband based method, and other iterative based demosaicing methods.

A Hybrid Demosaicking Algorithm for Area Scan Industrial Camera Based on Fuzzy Edge Strength and Residual Interpolation

Area scan cameras are widely used in industrial manufacturing for quality controlling, production monitoring, or defect inspection. The area scan cameras are normally integrated in a single sensor covered by the Bayer color filter array for RGB signal capturing, and the raw image with one channel for each pixel is obtained in one exposure. Therefore, it is significant to reconstruct the other two channels at each pixel which is called demosaicking. In this article, one hybrid demosaicking algorithm is proposed based on fuzzy edge strength and residual interpolation (RI) method. We first demosaic the G channel with one edge-directed filter and refine the high textured details by using fuzzy edge strength interpolation. Then, the RI is employed to calculate R and B channels, where G is used as the guide image for computing residuals. In the experiment, we evaluate our proposed method with nine prevalent interpolation-based algorithms on the IMAX and Kodak image datasets. The experimental results demonstrate that the proposed algorithm outperform the others by synthesizing the performance in peak signal-to-noise ratio, structural similarity, visual comparison, and computational time.