Filter Array Research Articles

Longwave infrared multispectral image sensor system using aluminum-germanium plasmonic filter arrays

A multispectral camera records image data in various wavelengths across the electromagnetic spectrum to acquire additional information that a conventional camera fails to capture. With the advent of high-resolution image sensors and color filter technologies, multispectral imagers in the visible wavelengths have become popular with increasing commercial viability in the last decade. However, multispectral imaging in longwave infrared (LWIR, 8–14 μm) is still an emerging area due to the limited availability of optical materials, filter technologies, and high-resolution sensors. Images from LWIR multispectral cameras can capture emission spectra of objects to extract additional information that a human eye fails to capture and thus have important applications in precision agriculture, forestry, medicine, and object identification. In this work, we experimentally demonstrate an LWIR multispectral image sensor with three wavelength bands using optical elements made of an aluminum (Al)-based plasmonic filter array sandwiched in germanium (Ge). To realize the multispectral sensor, the filter arrays are then integrated into a three-dimensional (3D) printed wheel stacked on a low-resolution monochrome thermal sensor. Our prototype device is calibrated using a blackbody and its thermal output has been enhanced with computer vision methods. By applying a state-of-the-art deep learning method, we have also reconstructed multispectral images to a better spatial resolution. Scientifically, our work demonstrates a versatile spectral thermography technique for detecting target signatures in the LWIR range and other advanced spectral analyses.

Design of a Narrow Band Filter Based on a Photonic Crystal Cavity for CO2 Sensing Application

This paper investigates the use of a miniaturized filter based on a triangular lattice of holes in a photonic crystal (PhC) slab. The plane wave expansion method (PWE) and finite-difference time-domain (FDTD) techniques were utilized to analyze the dispersion and transmission spectrum, as well as the quality factor and free spectral range (FSR) of the filter. A 3D simulation has demonstrated that for the designed filter, an FSR of more than 550 nm and a quality factor of 873 can be attained by adiabatically coupling light from a slab waveguide into a PhC waveguide. This work designs a filter structure that is implemented into the waveguide and is suitable for a fully integrated sensor. The small size of the device provides a strong potential for the realization of large arrays of independent filters on a single chip. The fully integrated character of this filter has further advantages such as reducing power loss in coupling light from sources to filters and also from filters to waveguides. The ease of fabrication is another benefit of completely integrating the filter.

The performance of speckle filters on Copernicus Sentinel-1 SAR images containing natural oil slicks

Synthetic aperture radar (SAR) is traditionally used in the identification, mapping and analysis of petroleum slicks, regardless of their origin. On SAR images, oil slicks appear as dark patches that contrast with the brightness of the surrounding sea surface. This distinction allows for automated detection algorithms to be designed using computer vision methods for objective oil slick identification. Nevertheless, efficient interpretation of the SAR imagery by statistical analysis can be diminished due to the speckle effect present on SAR images, a granular artefact associated with the coherent nature of SAR that visually degrades the image quality. In this study, a quantitative and qualitative assessment of common SAR image despeckling methods is presented, analysing their performance when applied to images containing natural oil slicks. The assessment is performed on Copernicus Sentinel-1 images acquired with various temporal and environmental conditions. The assessment covers a diverse array of filters that employ Bayesian and non-linear statistics in the spatial, transform and wavelet domains, focussing on their demonstrated performance and capabilities for edge and texture retention. In summary, the results reveal that filters using local statistics in the spatial domain produce consistent desired effects. The novel SAR-BM3D algorithm can be used effectively, albeit with a higher computational demand. Supplementary material: Implementations of the speckle filters used in this paper are made available at https://github.com/cavrinceanu/specklefilters under an MIT license. Image statistics data are available in the supplementary table at https://doi.org/10.6084/m9.figshare.13010405 Thematic collection: This article is part of the Remote sensing for site investigations on Earth and other planets collection available at: https://www.lyellcollection.org/cc/remote-sensing-for-site-investigations-on-earth-and-other-planets

A multi-channel approach for detecting tampering in colour filter images

Most devices capable of capturing photographs are provided with a Colour Filter Array (CFA). Images generated by these devices can be analysed to detect the so known CFA artefacts left by demosaicing methods. These artefacts are important in digital image forensics since they prove useful for determining the authenticity of an image. Previous works have focused exclusively on the analysis of the green band of the Bayer filter, neglecting the information in the remaining bands. In this work a process to estimate the pattern left by CFA artefacts regardless of their configuration is proposed, which can be used regardless of the size or colours used by the filter, or even in different colour spaces, obtaining in this way new sources of information for forensic analysis. A review and comparisons against state-of-the-art projects addressing the detection of manipulations using CFA artefacts are performed. The sensitivity of these proposals to post-processing methods such as compression and filtering will also be reviewed. We will also observe experimentally that the proposed technique can detect manipulations even in images affected by high levels of JPEG compression.

3D-patterned inverse-designed mid-infrared metaoptics

Modern imaging systems can be enhanced in efficiency, compactness, and application through the introduction of multilayer nanopatterned structures for manipulation of light based on its fundamental properties. High transmission multispectral imaging is elusive due to the commonplace use of filter arrays which discard most of the incident light. Further, given the challenges of miniaturizing optical systems, most cameras do not leverage the wealth of information in polarization and spatial degrees of freedom. Optical metamaterials can respond to these electromagnetic properties but have been explored primarily in single-layer geometries, limiting their performance and multifunctional capacity. Here we use advanced two-photon lithography to realize multilayer scattering structures that achieve highly nontrivial optical transformations intended to process light just before it reaches a focal plane array. Computationally optimized multispectral and polarimetric sorting devices are fabricated with submicron feature sizes and experimentally validated in the mid-infrared. A final structure shown in simulation redirects light based on its angular momentum. These devices demonstrate that with precise 3-dimensional nanopatterning, one can directly modify the scattering properties of a sensor array to create advanced imaging systems.

Dual potential- and waveband-resolved electrochemiluminescence platform for simultaneous determination of three lung cancer biomarkers

The sensitive simultaneous detection of multiplex biomarkers is critical for the early cancer diagnosis. Herein, a dual potential- and waveband-resolved electrochemiluminescence (ECL) platform is reported for multiplexing immunoassay, using a rotating band-pass filter array for ECL spectrum discrimination. Three lung cancer biomarkers, neuron-specific enolase (NSE), carcinoembryonic antigen (CEA) and cytokeratin 19 fragment antigen (CYFRA21–1) were adopted as model analytes, with nanocomposites of gold nanoparticles/graphene oxide/N, N′- dicaproate sodium-3,4,9,10-perylenedicarboximide (AuNPs/GO/PDI), CdSe nanocrystals (NCs) and CdTe NCs as ECL tags, respectively. After forming corresponding sandwich-type immune-complexes on the surface of modified electrode via an one-pot immune-reaction, the intensities from the ECL tags of AuNPs/GO/PDI (Ep = −0.41∼ −0.46 V), CdSe NCs (λmax = 550 nm, Ep = −1.44∼ −1.46 V), and CdTe NCs (λmax=750 nm, Ep = –1.49 ∼ −1.51 V) in the signal antibody bio-conjugates were measured in a single cathodic potential scan run with K2S2O8 as the coreactant. The interference from the ECL of O2/S2O82- enhanced by the three ECL tags was corrected by the intensity at 649 nm in the waveband-resolved ECL mode. Under optimized conditions, NSE, CYFRA21–1 and CEA were simultaneously quantified with the detection limit values of 0.86, 2.6 and 0.53 fg mL−1, respectively.

Double-Band Metasurface Infrared Optics for Integrated Multichannel Spectral Sensors

Simultaneously identifying the composition of mixtures and determining their concentration levels has aroused much attentions recent years to meet the large demands of compact and low-cost infrared spectral sensors to replace their conventional bulky counterparts with costly separated infrared components. In this work, we propose a design method for multi-channel metasurface infrared array optics covering two bands (MWIR/LWIR) for construction of integrated infrared gas sensors. The use of metasurface microstructures with same height realizes the effective filtering and focusing of electromagnetic waves at multiple wavelengths in the both infrared bands, which is beneficial to decrease the manufacturing costs. Four-channel metasurface lens arrays and filter arrays with operating wavelengths of 3.5 μm, 9.7 μm, 10.6 μm and 3.9 μm corresponding to the infrared absorption peaks of three gases of ethanol, ammonia, ethylene and one reference wavelength respectively, were designed and produced by photolithographic patterning and dry etching processes on silicon substrates. The multi-channel metasurface infrared optics have the advantages of easy fabrication and integration with wide-band infrared detector array to form a compact infrared spectral detector, which meets the requirements of small volume and low cost to advanced infrared spectral sensors for applications in portable smart sensing systems and internet of things.

Proposed Extensions to the Third Edition of JPEG XS (ISO/IEC 21122) Standard

At its 94th meeting (17–21 January 2022), International Organization for Standardization (ISO)/IEC SC29WG1 (also known as JPEG Committee) decided to create a third edition of the JPEG XS standard (ISO/IEC 21122) for lightweight, low-latency image coding. While existing coding tools in the standard already support the compression of natural content and color filter array (CFA) RAW Bayer pattern data well, this third edition will extend the previous editions with coding tools that improve its performance on the compression of screen content and image sequences with static background. The first experiments demonstrate that the proposed coding tools can improve the average peak signal-to-noise ratio (PSNR) performance of JPEG XS on such sequences of more than 10 dB, and approximately 3 dB on computer-generated content. In this work, the authors will present the proposed coding tools and will evaluate their performance objectively by PSNR.

Retina-inspired narrowband perovskite sensor array for panchromatic imaging.

The retina is the essential part of the human visual system that receives light, converts it to neural signal, and transmits to brain for visual recognition. The red, green, and blue (R/G/B) cone retina cells are natural narrowband photodetectors (PDs) sensitive to R/G/B lights. Connecting with these cone cells, a multilayer neuro-network in the retina provides neuromorphic preprocessing before transmitting to brain. Inspired by this sophistication, we develop the narrowband (NB) imaging sensor combining R/G/B perovskite NB sensor array (mimicking the R/G/B photoreceptors) with a neuromorphic algorithm (mimicking the intermediate neural network) for high-fidelity panchromatic imaging. Compared to commercial sensors, we use perovskite "intrinsic" NB PD to exempt the complex optical filter array. In addition, we use an asymmetric device configuration to collect photocurrent without external bias, enabling a power-free photodetection feature. These results display a promising design for efficient and intelligent panchromatic imaging.

Grayscale-patterned integrated multilayer-metal-dielectric microcavities for on-chip multi/hyperspectral imaging in the extended visible bandwidth.

Pixelated filter arrays of Fabry-Perot (FP) cavities are widely integrated with photodetectors to achieve a WYSIWYG ("what you see is what you get") on-chip spectral measurements. However, FP-filter-based spectral sensors typically have a trade-off between their spectral resolution and working bandwidth due to design limitations of conventional metal or dielectric multilayer microcavities. Here, we propose a new idea of integrated color filter arrays (CFAs) consisting of multilayer metal-dielectric-mirror FP microcavities that, enable a hyperspectral resolution over an extended visible bandwidth (∼300 nm). By introducing another two dielectric layers on the metallic film, the broadband reflectance of the FP-cavity mirror was greatly enhanced, accompanied by as-flat-as-possible reflection-phase dispersion. This resulted in balanced spectral resolution (∼10 nm) and spectral bandwidth from 450 nm to 750 nm. In the experiment, we used a one-step rapid manufacturing process by using grayscale e-beam lithography. A 16-channel (4 × 4) CFA was fabricated and demonstrated on-chip spectral imaging with a CMOS sensor and an impressive identification capability. Our results provide an attractive method for developing high-performance spectral sensors and have potential commercial applications by extending the utility of low-cost manufacturing process.

Snapshot multispectral imaging using a diffractive optical network

Multispectral imaging has been used for numerous applications in e.g., environmental monitoring, aerospace, defense, and biomedicine. Here, we present a diffractive optical network-based multispectral imaging system trained using deep learning to create a virtual spectral filter array at the output image field-of-view. This diffractive multispectral imager performs spatially-coherent imaging over a large spectrum, and at the same time, routes a pre-determined set of spectral channels onto an array of pixels at the output plane, converting a monochrome focal-plane array or image sensor into a multispectral imaging device without any spectral filters or image recovery algorithms. Furthermore, the spectral responsivity of this diffractive multispectral imager is not sensitive to input polarization states. Through numerical simulations, we present different diffractive network designs that achieve snapshot multispectral imaging with 4, 9 and 16 unique spectral bands within the visible spectrum, based on passive spatially-structured diffractive surfaces, with a compact design that axially spans ~72λm, where λm is the mean wavelength of the spectral band of interest. Moreover, we experimentally demonstrate a diffractive multispectral imager based on a 3D-printed diffractive network that creates at its output image plane a spatially repeating virtual spectral filter array with 2 × 2 = 4 unique bands at terahertz spectrum. Due to their compact form factor and computation-free, power-efficient and polarization-insensitive forward operation, diffractive multispectral imagers can be transformative for various imaging and sensing applications and be used at different parts of the electromagnetic spectrum where high-density and wide-area multispectral pixel arrays are not widely available.

MAPKAP Kinase-2 phosphorylation of PABPC1 controls its interaction with 14-3-3 proteins after DNA damage: A combined kinase and protein array approach.

14-3-3 proteins play critical roles in controlling multiple aspects of the cellular response to stress and DNA damage including regulation of metabolism, cell cycle progression, cell migration, and apoptotic cell death by binding to protein substrates of basophilic protein kinases following their phosphorylation on specific serine/threonine residues. Although over 200 mammalian proteins that bind to 14-3-3 have been identified, largely through proteomic studies, in many cases the relevant protein kinase responsible for conferring 14-3-3-binding to these proteins is not known. To facilitate the identification of kinase-specific 14-3-3 clients, we developed a biochemical approach using high-density protein filter arrays and identified the translational regulatory molecule PABPC1 as a substrate for Chk1 and MAPKAP Kinase-2 (MK2) in vitro, and for MK2 in vivo, whose phosphorylation results in 14-3-3-binding. We identify Ser-470 on PABPC1 within the linker region connecting the RRM domains to the PABC domain as the critical 14-3-3-binding site, and demonstrate that loss of PABPC1 binding to 14-3-3 results in increased cell proliferation and decreased cell death in response to UV-induced DNA damage.

Research on the Spectral Reconstruction of a Low-Dimensional Filter Array Micro-Spectrometer Based on a Truncated Singular Value Decomposition-Convex Optimization Algorithm

Currently, the engineering of miniature spectrometers mainly faces three problems: the mismatch between the number of filters at the front end of the detector and the spectral reconstruction accuracy; the lack of a stable spectral reconstruction algorithm; and the lack of a spectral reconstruction evaluation method suitable for engineering. Therefore, based on 20 sets of filters, this paper classifies and optimizes the filter array by the K-means algorithm and particle swarm algorithm, and obtains the optimal filter combination under different matrix dimensions. Then, the truncated singular value decomposition-convex optimization algorithm is used for high-precision spectral reconstruction.In terms of spectral evaluation, due to the strong randomness of the target detected during the working process of the spectrometer, the standard value of the target spectrum cannot be obtained. Therefore, we adopt the method of joint cross-validation of multiple sets of data for spectral evaluation. The results show that when the random error of +/− 2 code values is applied multiple times for reconstruction, the spectral angle cosine value between the reconstructed curves becomes more than 0.995, which proves that the spectral reconstruction under this algorithm has high stability. At the same time, the spectral angle cosine value of the spectral reconstruction curve and the standard curve can reach above 0.99, meaning that it realizes a high-precision spectral reconstruction effect. A high-precision spectral reconstruction algorithm based on truncated singular value-convex optimization, is established in this paper, providing important scientific research value for the engineering application of micro-spectrometers.

OPTIMIZATION ASSISTED AUTOREGRESSIVE METHOD WITH DEEP CONVOLUTIONAL NEURAL NETWORK-BASED ENTROPY FILTER FOR IMAGE DEMOSAICING

The natural scenes are captured by digital cameras that adopt a single charged-coupled device (CCD) or Complementary Metal Oxide Semiconductor (CMOS) sensor with a Color Filter Array (CFA). To reconstruct a full color image from the mosaiced CFA image, the missing components of color are recovered by a technique called color demosaicing. This research work introduces Deep Convolution Neural Network (DCNN)-based optimization assisted autoregressive method for image demosaicing. Here, the proposed optimization algorithm, termed Adaptive Autoregressive Water Wave Optimization algorithm (Adaptive Autoregressive-WWO), that is formed by combining Conditional autoregressive value at risk (CAViaR) model and Water Wave Optimization algorithm (WWO). Fusion process is carried out for residual images to generate the final demosaiced image based on entropy measure. Here, the output generated from DCNN is the first residual image. The LPA-ICI filter utilizes the optimization algorithm that generated second order polynomial coefficients to produce the second residual output image. Moreover, this proposed method is evaluated for its performance using metrics, such as Peak signal-to-noise ratio (PSNR) and Second Derivative like Measurement (SDME) and attained the highest PSNR values of 40.379dB and highest SDME values of 50.675dB.

Nonlinear demosaicking method and apparatus for nonlinear CMOS image sensors exhibiting low-density salt-and-pepper noise

Simple demosaicking methods designed for linear CMOS image sensors, such as MATLAB’s demosaic function, may be used with a monotonic nonlinear sensor having a Bayer color filter array (CFA). However, such methods may be inadequate at handling dynamic salt-and-pepper noise (SPN), i.e., outlier pixels, which is expected in images taken with a nonlinear sensor. Although SPN is present with linear sensors, nonlinear sensors express low-density light-dependent SPN that requires filtering. Extending a recent work on dynamic SPN filtering of a nonlinear sensor, we propose, evaluate, and verify a nonlinear method and apparatus to demosaic images, taken with a Bayer CFA, while simultaneously filtering the SPN. The approach relies on the use of weighted medians to filter the SPN, especially at densities that imply isolated outliers in small neighborhoods, while determining an accurate red, green, and blue (RGB) color at every pixel location. For explanatory purposes, three variants of the proposed method are presented and evaluated. A ground-truth image set, in which RGB channels were not obtained by demosaicking, is subsampled in a Bayer CFA pattern to produce mosaicked images for testing. In varying densities, SPN is introduced to these Kodak images for method and apparatus evaluation. Results of the proposed method and its variants are compared with those obtained with MATLAB’s demosaic function. Considering the alternatives and also apparatus complexity, the proposed nonlinear demosaicking method proves superior in visual quality, for smooth textures and along edges, and in peak signal-to-noise ratio, when there is low-density SPN.

MSFA-Net: A convolutional neural network based on multispectral filter arrays for texture feature extraction

Multispectral snapshot cameras fitted with a multispectral filter array (MSFA) acquire several spectral bands in one shot and provide a raw mosaic image in which a single channel value is available at each pixel. Texture features are classically extracted from fully-defined images that are estimated by demosaicing. Such an estimation may however cause spatio-spectral artifacts. Moreover, texture feature extraction becomes computationally inefficient and yields to high-dimensional features as the number of bands increases. In this paper, we propose an original approach based on a convolutional neural network called MSFA-Net to capture spatio-spectral interactions in raw images at reduced computation costs. Experiments of multispectral image classification and outdoor image segmentation show that the proposed approach outperforms several hand-crafted and deep learning-based feature extractors.

D$^\text{2}$UF: Deep Coded Aperture Design and Unrolling Algorithm for Compressive Spectral Image Fusion

Compressive spectral imaging (CSI) has attracted significant attention since it employs synthetic apertures to codify spatial and spectral information, sensing only 2D projections of the 3D spectral image. However, these optical architectures suffer from a trade-off between the spatial and spectral resolution of the reconstructed image due to technology limitations. To overcome this issue, compressive spectral image fusion (CSIF) employs the projected measurements of two CSI architectures with different resolutions to estimate a high-spatial high-spectral resolution. This work presents the fusion of the compressive measurements of a low-spatial high-spectral resolution coded aperture snapshot spectral imager (CASSI) architecture and a high-spatial low-spectral resolution multispectral color filter array (MCFA) system. Unlike previous CSIF works, this paper proposes joint optimization of the sensing architectures and a reconstruction network in an end-to-end (E2E) manner. The trainable optical parameters are the coded aperture (CA) in the CASSI and the colored coded aperture in the MCFA system, employing a sigmoid activation function and regularization function to encourage binary values on the trainable variables for an implementation purpose. Additionally, an unrolling-based network inspired by the alternating direction method of multipliers (ADMM) optimization is formulated to address the reconstruction step and the acquisition systems design jointly. Finally, a spatial-spectral inspired loss function is employed at the end of each unrolling layer to increase the convergence of the unrolling network. The proposed method outperforms previous CSIF methods, and experimental results validate the method with real measurements.

Open planar acoustic notch filter using a film-integrated Helmholtz resonator array

Acoustic filters are used to transmit or attenuate noise in a specific frequency band. However, previous acoustic filters consisting of rigid structures generally required a large housing of a size equivalent to the wavelength, limiting their downsizing. This study presents an open planar acoustic notch filter using a film-integrated Helmholtz resonator (HR) array. A film divides the HR chamber and resonates with the HR. The film-integrated HR array filter achieved smaller housing dimensions than a no-film HR array filter with the same notch frequency. The transmittance of the film-integrated HR array was theoretically calculated using the transfer matrix method. The calculated spectra were very similar to the simulation results. The notch frequency of the film-integrated HR array filter was 4.04, 0.74 kHz lower than a no-film HR array filter. The design and analysis of the acoustic filter presented should motivate further development of advanced acoustic filters.

Inverse design of photonic crystal filters with arbitrary correlation and size for accurate spectrum reconstruction.

Spectroscopic technique based on nanophotonic filters can recover spectral information through compressive sensing theory. The spectral information is encoded by nanophotonic response functions and decoded by computational algorithms. They are generally ultracompact, low in cost, and offer single-shot operation with spectral resolution better than 1nm. Thus, they could be ideally suited for emerging wearable and portable sensing and imaging applications. Previous work has revealed that successful spectral reconstruction relies on well-designed filter response functions with sufficient randomness and low mutual correlation, but no thorough discussion has been performed on the filter array design. Here, instead of blind selection of filter structures, inverse design algorithms are proposed to obtain a photonic crystal filter array with predefined correlation coefficients and array size. Such rational spectrometer design can perform accurate reconstruction for a complex spectrum and maintain the performance under noise perturbation. We also discuss the impact of correlation coefficient and array size on the spectrum reconstruction accuracy. Our filter design method can be extended to different filter structures and suggests a better encoding component for reconstructive spectrometer applications.

Superposition Fabry-Perot filter array for a computational hyperspectral camera.

Computational hyperspectral cameras with broadband encoded filter arrays enable high precision spectrum reconstruction with only a few filters. However, these types of hyperspectral cameras have limited application, because it is difficult for conventional encoded filter arrays to balance among the spectrum regulation capacity, angle insensitivity, and processibility. This Letter presents a new, to the best of our knowledge, encoded filter composed of superposition Fabry-Perot resonance cavity (SFP) that can simultaneously take all three aspects into consideration. By learning the parameters of an SFP encoder and a neural network decoder in an end-to-end manner, a computational hyperspectral camera based on an SFP filter array presents up to 2.24 times higher spectral reconstruction accuracy, 10 times wider working angle, and can be produced with a low-cost manufacturing process.