Multispectral Filter Array Research Articles

Adaptive and Progressive Multispectral Image Demosaicking

Common consumer RGB cameras use a single image sensor to capture RGB color images with a color filter array (CFA) placed in front of the image sensor. This system captures particularly one color sample ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R$</tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$G$</tex-math></inline-formula> , or <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$B$</tex-math></inline-formula> ) at each pixel location. This captured raw image is called a CFA image. The CFA demosaicking is used then to construct the complete color image. A similar approach of working with a single image sensor and extension of CFA to multispectral filter array (MSFA) enables us to develop a low-cost multispectral camera with the help of MSFA demosaicking. However, due to many spectral bands and their very sparse sampling, extending CFA demosaicking to MSFA demosaicking is not easy. This paper proposes an adaptive and progressive MSFA demosaicking method for various MSFA patterns for 5 to 15 bands multispectral images. Binary tree-based MSFA patterns are used to create MSFA images, and these MSFA patterns can be designed for any N-band MSFA image. The spectral bands are arranged specifically so that the middle band has the highest probability of appearance (PoA) in the MSFA pattern. The middle band is interpolated first using PoA based progressive adaptive interpolation, and the modified bilinear spectral difference method is used to estimate all other spectral bands with the interpolated middle band’s help. Experimental results and comparative analysis on two different datasets reveal that our proposed MSFA demosaicking method outperforms the existing generic MSFA demosaicking methods in terms of computational time and various image quality metrics considered.

Robust Facial Recognition System using One Shot Multispectral Filter Array Acquisition System

Face recognition in the visible and Near Infrared range has received a lot of attention in recent years. The current Multispectral (MS) imaging systems used for facial recognition are based on multiple cameras having multiple sensors. These acquisition systems are normally slow because they take one MS image in several shots, which makes them unable to acquire images in real time and to capture moving scenes. On the other hand, currently there are snapshot multispectral imaging systems which integrate a single sensor with Multispectral Filter Arrays (MSFA) allow having at each acquisition an image on several spectra. These systems drastically reduce image acquisition time and are able to capture moving scenes in real time. This paper proposes a study of robust facial recognition using Multispectral Filter Array acquisition system. For this goal, a MSFA one-shot camera was used to collect the images and a robust facial recognition method based on Fast Discrete Curvelet Transform and Convolutional Neural Network is proposed. This camera covers the spectral range from 650 nm to 950 nm. A comparison of the facial recognition system using Multispectral Filter Arrays camera is made with those that using multiple cameras. Experimental results proved that face recognition systems whose acquisition systems are designed using MSFA perform more efficiently with an accuracy of 100%.

Generic Multispectral Demosaicking Based on Directional Interpolation

The low-cost snapshot multispectral spectral imaging systems with multispectral filter array (MSFA) require generic MSFA demosaicking methods to generate the multispectral image (MSI) with the variable number of spectral bands depending on the applications. Most of the existing MSFA demosaicking methods are either non-generic or perform inadequately. This paper presents a new generic MSFA demosaicking method based on the directional weighted interpolation. Our proposed method calculates the four directional estimates around the location of the unknown pixel and combines them in a weighted manner using the local edge magnitude in the corresponding directions to estimate the missing pixel values. Experimental results confirm that the proposed demosaicking method provides improvement, compared to the state-of-the-art generic demosaicking methods in terms of both subjective and objective evaluations on the two benchmark MSI datasets.

Extension of luminance component based demosaicking algorithm to 4- and 5-band multispectral images

Multispectral imaging systems are currently expanding with a variety of multispectral demosaicking algorithms. But these algorithms have limitations due to the remarkable presence of artifacts in the reconstructed image. In this paper, we propose a powerful multispectral image demosaicking method that focuses on the G band and luminance component. We've first identified a relevant 4-and 5-band multispectral filter array (MSFA) with the dominant G band and then proposed an algorithm that consistently estimates the missing G values and other missing components using a convolution operator and a weighted bilinear interpolation algorithm based on the luminance component. Using the considered MSFA patterns, we've also demonstrated that our algorithm outperforms existing approaches both visually and quantitatively in terms of the PSNR and SSIM.

Batch fabrication and compact integration of customized multispectral filter arrays towards snapshot imaging.

Snapshot multispectral imaging (MSI) has been widely employed in the rapid visual inspection by virtues of the non-invasive detection mode and short integration time. As the critical functional elements of snapshot MSI, narrowband, customizable, and pixel-level multispectral filter arrays (MSFAs) that are compatible with imaging sensors are difficult to be efficiently manufactured. Meanwhile, monolithically integrating MSFAs into snapshot multispectral imagers still remains challenging considering the strict alignment precision. Here, we propose a cost-efficient, wafer-level, and customized approach for fabricating transmissive MSFAs based on Fabry-Perot structures, both in the pixel-level and window-tiled configuration, by utilizing the conventional lithography combined with the deposition method. The MSFA chips own a total dimension covering the area of 4.8 mm × 3.6 mm with 4 × 4 bands, possessing the capability to maintain narrow line widths (∼25 nm) across the whole visible frequencies. After the compact integration with the imaging sensor, the MSFAs are validated to be effective in filtering and target identification. Our proposed fabrication method and imaging mode show great potentials to be an alternative to MSFAs production and MSI, by reducing both complexity and cost of manufacturing, while increasing flexibility and customization of imaging system.

Data-driven framework for high-accuracy color restoration of RGBN multispectral filter array sensors under extremely low-light conditions.

RGBN multispectral filter array provides a cost-effective and one-shot acquisition solution to capture well-aligned RGB and near-infrared (NIR) images which are useful for various optical applications. However, signal responses of the R, G, B channels are inevitably distorted by the undesirable spectral crosstalk of the NIR bands, thus the captured RGB images are adversely desaturated. In this paper, we present a data-driven framework for effective spectral crosstalk compensation of RGBN multispectral filter array sensors. We set up a multispectral image acquisition system to capture RGB and NIR image pairs under various illuminations which are subsequently utilized to train a multi-task convolutional neural network (CNN) architecture to perform simultaneous noise reduction and color restoration. Moreover, we present a technique for generating high-quality reference images and a task-specific joint loss function to facilitate the training of the proposed CNN model. Experimental results demonstrate the effectiveness of the proposed method, outperforming the state-of-the-art color restoration solutions and achieving more accurate color restoration results for desaturated and noisy RGB images captured under extremely low-light conditions.

Design of mid-infrared filter array based on plasmonic metal nanodiscs array and its application to on-chip spectrometer

Mid-infrared wavelengths are called the molecular fingerprint region because it contains the fundamental vibrational modes inherent to the substances of interest. Since the mid-infrared spectrum can provide non-destructive identification and quantitative analysis of unknown substances, miniaturized mid-infrared spectrometers for on-site diagnosis have attained great concern. Filter-array based on-chip spectrometer has been regarded as a promising alternative. In this study, we explore a way of applying a pillar-type plasmonic nanodiscs array, which is advantageous not only for excellent tunability of resonance wavelength but also for 2-dimensional integration through a single layer process, to the multispectral filter array for the on-chip spectrometer. We theoretically and experimentally investigated the optical properties of multi-periodic triangular lattices of metal nanodiscs array that act as stopband filters in the mid-infrared region. Soft-mold reverse nanoimprint lithography with a subsequent lift-off process was employed to fabricate the multispectral filter array and its filter function was successfully extracted using a Fourier transform infrared microscope. With the measured filter function, we tested the feasibility of target spectrum reconstruction using a Tikhonov regularization method for an ill-posed linear problem and evaluated its applicability to the infrared spectroscopic sensor that monitors an oil condition. These results not only verify that the multispectral filter array composed of stopband filters based on the metal nanodiscs array when combined with the spectrum reconstruction technique, has great potential for use to a miniaturized mid-infrared on-chip spectrometer, but also provide effective guidance for the filter design.

Demosaicing Method for Multispectral Images Using Derivative Operations

Multispectral images have been found useful for various applications such as remote sensing, medical imaging, military surveillance, vision inspection for food quality control, etc. but the high costs of multispectral cameras limit their usage. Low cost multispectral cameras can be developed using a single sensor multispectral filter array (MSFA) and a demosaicing method to reconstruct the complete image from under sampled multispectral image data acquired using a single sensor MSFA imaging system. In this paper, we present a new demosaicing method based on the derivative operations for the multi-spectral images. To design MSFA patterns, binary tree method is often used and the band sequence is chosen such that the middle band has a higher probability of appearance in MSFA pattern. In the proposed method, first the middle spectral band pixel values are estimated and then it is used to compute derivatives that help estimate other spectral band pixel values. Unlike many recently developed demosaicing methods that are applicable to only specific band size multispectral images, the proposed method is generic and can be applied to obtain multispectral images for any number of spectral bands. The TokyoTech dataset and CAVE dataset of multispectral images are used for the evaluation purpose, and the experimental results show that the proposed method outperforms currently best known generic multispectral demosaicing method, namely binary tree edge sensing (BTES) method on both datasets and for different band-size multispectral images.

Mosaic Convolution-Attention Network for Demosaicing Multispectral Filter Array Images

This paper presents a mosaic convolution-attention network (MCAN) for demosaicing spectral mosaic images captured using multispectral filter array (MSFA) imaging sensors. MSFA-based multispectral imaging systems acquire multispectral information of a scene in a single snap-shot operation. A complete multispectral image is reconstructed by demosaicing an MSFA-based spectral mosaic image. To avoid aliasing and artifacts in demosaicing, we utilize joint spatial-spectral correlation in a raw mosaic image. The proposed MCAN includes a mosaic convolution module (MCM) and a mosaic attention module (MAM). The MCM extracts features via a learning approach with a margin between splitting the periodic spectral mosaic and keeping the underlying spatial information of the raw image. Based on the strategy of position-sensitive weight sharing, MCM assigns the same weight to pixels with the same relative position in an MSFA. The MAM uses a position-sensitive feature aggregation strategy to describe the loading of mosaic patterns within the feature maps, which gradually reduces mosaic distortion through the attention mechanism. The experimental results on synthetic as well as real-world data show that the proposed scheme outperforms state-of-the-art methods in terms of spatial details and spectral fidelity.

Generalized MSFA Engineering With Structural and Adaptive Nonlocal Demosaicing.

The emerging multispectral-filter-array (MSFA) cameras require generalized demosaicing for MSFA engineering. The existing interpolation, compressive sensing and deep learning based methods suffer from either limited reconstruction accuracy or poor generalization. In this work, we report a generalized demosaicing method with structural and adaptive nonlocal optimization, enabling boosted reconstruction accuracy for different MSFAs. The advantages lie in the following three aspects. First, the nonlocal low-rank optimization is applied and extended to the multiple spatial-spectral-temporal dimensions to exploit more crucial details. Second, the block matching accuracy is promoted by employing a novel structural similarity metric instead of the conventional Euclidean distance. Third, the running efficiency is boosted by an adaptive iteration strategy. We built a prototype system to capture raw mosaic images under different MSFAs, and used the technique as an off-the-shelf tool to demonstrate MSFA engineering. The experiments show that the binary tree (BT) based filter array produces higher accuracy than the random and regular ones for different number of channels.

Direct estimation of NIR reflection spectra utilizing a snapshot-type spectrometer with photonic crystal multi-spectral filter arrays.

We fabricated 16-, 25-, 36-, and 64-channel distributed passband-type multi-spectral filter arrays by utilizing a multilayer-type photonic crystal and integrated them onto a CCD to form a snapshot-type spectroscopic sensor. Reflection spectra from target objects (fruits) under broadband light illumination were estimated directly using the Wiener estimation method. A root mean square error of the reflectivity on the order of 2∼5% was obtained under optical shot noise with 6×6 pixel binning. A number of constituent filters of 36 was sufficient for this type of fruit spectral measurement. We also visualized reflection images at specified wavelengths by applying the estimation method to a multiple filter region on the sensor.

Multispectral imaging via nanostructured random broadband filtering.

It is a challenge to acquire a snapshot image of very high resolutions in both spectral and spatial domain via a single short exposure. In this setting one cannot trade time for spectral resolution, such as via spectral bands scanning. Cameras of color filter arrays (CFA) (e.g., the Bayer mosaic) cannot obtain high spectral resolution. To overcome these difficulties, we propose a new multispectral imaging system that makes random linear broadband measurements of the spectrum via a nanostructured multispectral filter array (MSFA). These MSFA random measurements can be used by sparsity-based recovery algorithms to achieve much higher spectral resolution than conventional CFA cameras, without sacrificing spatial resolution. The key innovation is to jointly exploit both spatial and spectral sparsity properties that are inherent to spectral irradiance of natural objects. Experimental results establish the superior performance of the proposed multispectral imaging system over existing ones.

Sparse spectral signal reconstruction for one proposed nine-band multispectral imaging system

Multispectral filter array (MSFA) imaging with one single sensor is a portable and inexpensive means of acquiring spectral image which is widely used for object detection, material analysis and mechanical system diagnosis. The most challenging task for MSFA imaging is the multispectral demosaicking with the aim of reconstructing the captured raw/mosaic image, especially for the systems with many bands which result in higher sparseness of the raw data. In this paper, we present a 9-band MSFA imaging system in a repetitive 4 × 4 filter array on a single sensor, and propose a demosaicking algorithm for reconstructing the raw spectral image. Within the 4 × 4 MSFA pattern, the fifth spectral band takes up half of the total spatial position while the remaining eight bands occupy 1/16 respectively. To reconstruct the sparse raw data, we first recover the fifth band by propagating the neighboring sampled pixels to the unsampled position using the image gradients, and then employ the reconstructed fifth band as a guided image to demosaick the other bands with the guided filter and residual interpolation. Finally, we estimate the spectral reflectance values from the multispectral image and the characterization matrix. In the experiment, we evaluate the performance of the 9-band imaging system with the binary tree-based edge-sensing (BTES) algorithm, compressed sensing (CS) algorithm, and our proposed demosaicking algorithm. The experiment results demonstrate that our demosaicking algorithm not only outperforms BTES and CS algorithms in terms of objective image quality, e.g., PSNR values and spectral errors, but also reduces the demosaicking artifacts in terms of subjective evaluations.

Camera Array for Multi-Spectral Imaging.

Recently, many new applications arose for multispectral and hyper-spectral imaging. Besides modern biometric systems for identity verification, also agricultural and medical applications came up, which measure the health condition of plants and humans. Despite the growing demand, the acquisition of multi-spectral data is up to the present complicated. Often, expensive, inflexible, or low resolution acquisition setups are only obtainable for specific professional applications. To overcome these limitations, a novel camera array for multi-spectral imaging is presented in this article for generating consistent multispectral videos. As differing spectral images are acquired at various viewpoints, a geometrically constrained multi-camera sensor layout is introduced, which enables the formulation of novel registration and reconstruction algorithms to globally set up robust models. On average, the novel acquisition approach achieves a gain of 2.5 dB PSNR compared to recently published multi-spectral filter array imaging systems. At the same time, the proposed acquisition system ensures not only a superior spatial, but also a high spectral, and temporal resolution, while filters are flexibly exchangeable by the user depending on the application. Moreover, depth information is generated, so that 3D imaging applications, e.g., for augmented or virtual reality, become possible. The proposed camera array for multi-spectral imaging can be set up using off-the-shelf hardware, which allows for a compact design and employment in, e.g., mobile devices or drones, while being cost-effective.

Grayscale-to-Color: Scalable Fabrication of Custom Multispectral Filter Arrays

Snapshot multispectral image (MSI) sensors have been proposed as a key enabler for a plethora of multispectral imaging applications, from diagnostic medical imaging to remote sensing. With each application requiring a different set, and number, of spectral bands, the absence of a scalable, cost-effective manufacturing solution for custom multispectral filter arrays (MSFAs) has prevented widespread MSI adoption. Despite recent nanophotonic-based efforts, such as plasmonic or high-index metasurface arrays, large-area MSFA manufacturing still consists of many-layer dielectric (Fabry–Perot) stacks, requiring separate complex lithography steps for each spectral band and multiple material compositions for each. It is an expensive, cumbersome, and inflexible undertaking, but yields optimal optical performance. Here, we demonstrate a manufacturing process that enables cost-effective wafer-level fabrication of custom MSFAs in a single lithographic step, maintaining high efficiencies (∼75%) and narrow line widths (∼25 nm) across the visible to near-infrared. By merging grayscale (analog) lithography with metal–insulator–metal (MIM) Fabry–Perot cavities, whereby exposure dose controls cavity thickness, we demonstrate simplified fabrication of MSFAs up to N-wavelength bands. The concept is first proven using low-volume electron beam lithography, followed by the demonstration of large-volume UV mask-based photolithography with MSFAs produced at the wafer level. Our framework provides an attractive alternative to conventional MSFA manufacture and metasurface-based spectral filters by reducing both fabrication complexity and cost of these intricate optical devices, while increasing customizability.

Optimized Multi-Spectral Filter Arrays for Spectral Reconstruction.

Multispectral filter array (MSFA)-based imaging is a compact, practical technique for snapshot spectral image capturing and reconstruction. The imaging and reconstruction quality is highly influenced by the spectral sensitivities and spatial arrangement of channels on MSFAs, and the used reconstruction method. In order to design a MSFA with high imaging capacity, we propose a sparse representation based approach to optimize spectral sensitivities and spatial arrangement of MSFAs. The proposed approach first overall models the various errors associated with spectral reconstruction, and then uses a global heuristic searching method to optimize MSFAs via minimizing the estimated error of MSFAs. Our MSFA optimization method can select filters from off-the-shelf candidate filter sets while assigning the selected filters to the designed MSFA. Experimental results on three datasets show that the proposed method is more efficient, flexible, and can design MSFAs with lower spectral construction errors when compared with existing state-of-the-art methods. The MSFAs designed by our method show better performance than others even using different spectral reconstruction methods.

Demosaicking Using a Spatial Reference Image for an Anti-Aliasing Multispectral Filter Array.

Multispectral imaging with a multispectral filter array (MSFA) facilitates snapshot imaging; however, a demosaicking process is required to estimate a fully defined multispectral image based on undersampled sensor data. Undersampling induces aliasing and adverse artifacts in the reconstructed image. To solve this problem, Jia et al. proposed the Fourier spectral filter array (FSFA), which can reduce aliasing. In this paper, we analyze the FSFA and a more generalized anti-aliasing MSFA, and we identify the property that makes MSFAs anti-aliasing. Furthermore, we propose a novel demosaicking method that is a hybrid of frequency-decomposition-based and compressive-sensing-based demosaicking. Anti-aliasing MSFAs enable demosaicking to comprehend the precise spatial structures of an image. The image assists our proposed method in precisely reconstructing images using compressive sensing. Our experimental results demonstrated that the proposed method performs better than the existing demosaicking methods, especially in terms of spatial reconstruction.

Sensitivity Improvement of Extremely Low Light Scenes with RGB-NIR Multispectral Filter Array Sensor.

Recently, several red-green-blue near-infrared (RGB-NIR) multispectral filter arrays (MFAs), which include near infrared (NIR) pixels, have been proposed. For extremely low light scenes, the RGB-NIR MFA sensor has been extended to receive NIR light, by adding NIR pixels to supplement for the insufficient visible band light energy. However, the resolution reconstruction of the RGB-NIR MFA, using demosaicing and color restoration methods, is based on the correlation between the NIR pixels and the pixels of other colors; this does not improve the RGB channel sensitivity with respect to the NIR channel sensitivity. In this paper, we propose a color restored image post-processing method to improve the sensitivity and resolution of an RGB-NIR MFA. Although several linear regression based color channel reconstruction methods have taken advantage of the high sensitivity NIR channel, it is difficult to accurately estimate the linear coefficients because of the high level of noise in the color channels under extremely low light conditions. The proposed method solves this problem in three steps: guided filtering, based on the linear similarity between the NIR and color channels, edge preserving smoothing to improve the accuracy of linear coefficient estimation, and residual compensation for lost spatial resolution information. The results show that the proposed method is effective, while maintaining the NIR pixel resolution characteristics, and improving the sensitivity in terms of the signal-to-noise ratio by approximately 13 dB.

Spatio-spectral binary patterns based on multispectral filter arrays for texture classification.

To discriminate gray-level texture images, spatial texture descriptors can be extracted using the local binary pattern (LBP) operator. This operator has been extended to color images at the expense of increased memory and computation requirements. Some authors propose to compute texture descriptors directly from raw images provided through a Bayer color filter array, which both avoids the demosaicking step and reduces the descriptor size. Recently, multispectral snapshot cameras have emerged to sample more than three wavelength bands using a multispectral filter array. Such cameras provide a raw image in which a single spectral channel value is available at each pixel. In this paper we design a local binary pattern operator that jointly extracts the spatial and spectral texture information directly from a raw image. Extensive experiments on a large dataset show that the proposed descriptor has both reduced computation cost and high discriminative power with regard to classical LBP descriptors applied to demosaicked images.

Joint Optimization of Multispectral Filter Arrays and Demosaicking for Pathological Images

A capturing system with multispectral filter array (MSFA) technology is proposed for shortening the capture time and reducing costs. Therein, a mosaicked image captured using an MSFA is demosaicked to reconstruct multispectral images (MSIs). Joint optimization of the spectral sensitivity of the MSFAs and demosaicking is considered, and pathology-specific multispectral imaging is proposed. This optimizes the MSFA and the demosaicking matrix by minimizing the reconstruction error between the training data of a hematoxylin and eosin-stained pathological tissue and a demosaicked MSI using a cost function. Initially, the spectral sensitivity of the filter array is set randomly and the mosaicked image is obtained from the training data. Subsequently, a reconstructed image is obtained using Wiener estimation. To minimize the reconstruction error, the spectral sensitivity of the filter array and the Wiener estimation matrix are optimized iteratively through an interior-point approach. The effectiveness of the proposed MSFA and demosaicking is demonstrated by comparing the recovered spectrum and RGB image with those obtained using a conventional method.