Filter Array Research Articles

Estimating the Spectral Response of Eight-Band MSFA One-Shot Cameras Using Deep Learning

Eight-band one-shot MSFA (multispectral filter array) cameras are innovative technologies used to capture multispectral images by capturing multiple spectral bands simultaneously. They thus make it possible to collect detailed information on the spectral properties of the observed scenes economically. These cameras are widely used for object detection, material analysis, and agronomy. The evolution of one-shot MSFA cameras from 8 to 32 bands makes obtaining much more detailed spectral data possible, which is crucial for applications requiring delicate and precise analysis of the spectral properties of the observed scenes. Our study aims to develop models based on deep learning to estimate the spectral response of this type of camera and provide images close to the spectral properties of objects. First, we prepare our experiment data by projecting them to reflect the characteristics of our camera. Next, we harness the power of deep super-resolution neural networks, such as very deep super-resolution (VDSR), Laplacian pyramid super-resolution networks (LapSRN), and deeply recursive convolutional networks (DRCN), which we adapt to approximate the spectral response. These models learn the complex relationship between 8-band multispectral data from the camera and 31-band multispectral data from the multi-object database, enabling accurate and efficient conversion. Finally, we evaluate the images’ quality using metrics such as loss function, PSNR, and SSIM. The model evaluation revealed that DRCN outperforms others in crucial performance. DRCN achieved the lowest loss with 0.0047 and stood out in image quality metrics, with a PSNR of 25.5059, SSIM of 0.8355, and SAM of 0.13215, indicating better preservation of details and textures. Additionally, DRCN showed the lowest RMSE 0.05849 and MAE 0.0415 values, confirming its ability to minimize reconstruction errors more effectively than VDSR and LapSRN.

Parallel Lossless Compression of Raw Bayer Images on FPGA-Based High-Speed Camera

Digital image compression is applied to reduce camera bandwidth and storage requirements, but real-time lossless compression on a high-speed high-resolution camera is a challenging task. The article presents hardware implementation of a Bayer colour filter array lossless image compression algorithm on an FPGA-based camera. The compression algorithm reduces colour and spatial redundancy and employs Golomb–Rice entropy coding. A rule limiting the maximum code length is introduced for the edge cases. The proposed algorithm is based on integer operators for efficient hardware implementation. The algorithm is first verified as a C++ model and later implemented on AMD-Xilinx Zynq UltraScale+ device using VHDL. An effective tree-like pipeline structure is proposed to concatenate codes of compressed pixel data to generate a bitstream representing data of 16 parallel pixels. The proposed parallel compression achieves up to 56% reduction in image size for high-resolution images. Pipelined implementation without any state machine ensures operating frequencies up to 320 MHz. Parallelised operation on 16 pixels effectively increases data throughput to 40 Gbit/s while keeping the total memory requirements low due to real-time processing.

IntAct Database for Accessing IMEx's Contextual Metadata of Molecular Interactions.

The International Molecular Exchange Consortium (IMEx) has evolved into a vital partnership of open resources dedicated to curating molecular interaction data from the scientific literature. This consortium, which includes IntAct, MINT, MatrixDB, and DIP, is a collaborative effort with a central mission of aggregating detailed molecular interaction experimental evidence in a machine-readable format, supported by controlled vocabularies and standard ontologies. The IntAct molecular interaction database (www.ebi.ac.uk/intact), as an IMEx partner, serves as a valuable portal for accessing IMEx data through user-friendly search options and an array of interactive filters. The resource currently hosts an extensive repository of 1,293,508 binary interactions meticulously captured from 75,098 experiments documented in 23,366 publications (as of the February 2024 release), with this corpora being added to by regular data releases. IMEx curation policy has consistently prioritized a fine-grained data and curation model, with a focus on capturing the relevant experimental details essential for interpreting molecular interaction data effectively. Our curation process is designed to support the generation of interactomes tailored to contexts such as disease-specific or tissue-/cell-type-specific interactomes. These interactions are ranked according to a scoring system based on the Proteomics Standard Initiative Molecular Interaction (PSI MI) standards. This scoring system allows users to assess the degree of confidence in binary interactions, enhancing the value of the data. The resource provides insights into the nature of relationships among interacting partners as defined by the experimental setup and the associated biological context. Interactive filters enable users to navigate these rich, multilayered data, promoting a deeper understanding of biological complexity. Additionally, the IntAct website fosters the creation of networks for collaborative analyses by the scientific community. The recent transformation of the IntAct website, supported by a graph-type database, empowers users to execute custom queries tailored to their specific research interests. This article illustrates the diverse levels of annotations available for interactions and the multiple search options at users' disposal to access data of interest. © 2024 European Molecular Biology Laboratory, European Bioinformatics Institute. Current Protocols published by Wiley Periodicals LLC. Basic Protocol 1: Using Quick Search, network visualization, and filters Support Protocol: Accessing fine annotations from intact: Unlocking the molecular details Alternate Protocol: Using batch search: Querying multiple interactors Basic Protocol 2: Using advanced search: Precision and customization.

High-sensitivity miniaturized spectrometers using photonic crystal slab filters.

Miniaturized spectrometers have emerged as pivotal tools in numerous scientific and industrial applications, offering advantages such as portability, cost-effectiveness, and the capability for onsite analysis. Despite these significant benefits, miniaturized spectrometers face critical challenges, particularly in sensitivity. Reduced dimensions often lead to compromises in optical path length and component quality, which can diminish detection limits and limit their applications in areas such as low-light-level measurements. Here we developed a compact spectrometer that integrates an array of photonic crystal slab filters with band-stop spectral transmission characteristics into an image sensor. Compared to traditional gratings or bandpass filter strategies, where each detector can only read light of a single wavelength component, our band-stop strategy allows each detector to read the light of all wavelengths except the band-stop wavelength. This maximizes energy extraction from incident signals, significantly improving the sensitivity of the spectrometer. Spectral reconstruction is achieved mathematically using pre-calibrated band-stop responses combined with a single coded image. Our spectrometer delivers a spectral resolution of 1.9 nm and demonstrates sensitivity more than ten times greater than that of conventional grating spectrometers during fluorescence spectroscopy of Ascaris lumbricoides. The design is fully compatible with complementary metal-oxide-semiconductor (CMOS) technology, allowing for mass production at low costs and thus promising broad deployment in sensitive applications.

High-accuracy direction measurement and high-resolution computational spectral reconstruction based on photonic crystal array

Portable and wearable miniaturized spectrometers play a crucial role in various fields. In this paper, we present a method for simultaneously realizing high-accuracy direction measurement and high-resolution computational spectral reconstruction based on the angle sensitivity of conventional photonic crystals (PCs), wherein an optical filter array is composed of multiple one-dimensional PCs. The high-angle sensitivity of PCs results in angle-dependent optical spectra. When these spectra with different angles are used to reconstruct the target spectra in an unknown direction and the interval between adjacent angles is sufficiently small, the accurate direction of the target can be automatically identified. Moreover, the computational spectra still have high resolution over a wide range of incidences. The computational spectra under arbitrary polarizations can also be recognized based on the polarization dependence of the PCs at an oblique incidence. Our research results are significant for engineering a new miniaturized comprehensive computational spectrometer with target-direction perception and omnidirectional spectral reconstruction abilities.

PbS-based SWIR micro-spectrometer with on-chip Fabry-Perot filter array.

Miniaturized and portable on-chip spectrometers have been widely explored to facilitate many applications including chemical analysis, environmental monitoring, medical diagnostics, and astronomical observations. However, the optical spectra of micro-spectrometers are mostly within the visible range. Here, we develop high-performance short-wave infrared (SWIR) micro-spectrometers through the integration of wafer-scale uniform lead sulfide (PbS) thin films with an on-chip Fabry-Perot filter array. The optoelectronic performance of PbS-based detectors could be markedly improved through the optimization of chemical bath deposition (CBD) conditions. The high-sensitivity PbS detectors based on the Fabry-Perot filter array demonstrate chemical analysis application.

Miniaturized Hyperspectral Imager Utilizing a Reconfigurable Filter Array for Both High Spatial and Spectral Resolutions.

Miniaturized hyperspectral imaging based on filter arrays has attracted much attention in consumer applications, such as food safety and biomedical applications. In this Letter, we demonstrate a miniaturized hyperspectral imager using a reconfigurable filter array to tackle the existing trade-off issue between the spectral and spatial resolutions. Utilizing tens of intermediate states of a vanadium dioxide cavity, we increase the total number of physical spectral channels by tens of times from a 2 × 2 mosaic filter unit, providing both high spatial and spectral resolutions for spectral imaging. The reconfigurable filter has a good spectral resolvability of 10 nm in the visible range with a wavelength inaccuracy of less than 2.1 nm. Hyperspectral imaging is demonstrated with a frame rate of 4.5 Hz.

A Sensitive Fluorescence Analysis Method of Pathogenic Microorganisms Based on Silicon Photomultiplier.

The monitoring of pathogenic microorganisms in water is important for public health and disease outbreaks prediction. Recently, optical detection techniques have drawn much attention due to the advantages of rapid response, security and high sensitivity. In this paper, a fluorescence spectrometer based on 375nm exciting laser and the microchannel liquid sample flow technology is proposed. The 4 × 4 narrowband filter array was coupled to a Silicon Photomultiplier (SiPM) array with single-photon sensitivity. B500 fluorescent microspheres and Escherichia coli were used for performance evaluation of the spectrometer. As a result, it is feasible to use random particle counting method to detect the bacteria concentration level in water even low to several CFU/mL. In addition, based on Python tools and neural network algorithm models, the fluorescence spectra of different kinds of substances (biotic and abiotic) can be classified with an accuracy of more than 97%. The method was successfully applied to tap water samples. The results suggest that the proposed method is applicable for on-site bacteria detection.

Joint learning of RGBW color filter arrays and demosaicking

RGBW color filter arrays (CFAs) have gained widespread attention for their superior performance in low-light conditions. Most existing demosaicking methods are tailored for specific RGBW CFAs or involve manual joint design with CFAs. The close relationship between sampling and reconstruction means that restricting the search space through predefined CFAs and demosaicking methods severely limits the ability to achieve optimal performance. In this paper, we propose a new approach for joint learning of RGBW CFA and demosaicking. This approach can simultaneously learn optimal CFAs and demosaicking methods of any size, while also being capable of reconstructing mosaicked images of any size. We use a surrogate function and arbitrary CFA interpolation to ensure end-to-end learning of the RGBW CFA. We also propose a dual-branch fusion reconstruction network that utilizes the W channel to guide the reconstruction of R, G, and B channels, reducing color errors while preserving more image details. Extensive experiments demonstrate the superiority of our proposed method. Our code is available at: https://github.com/fql0528/learncfa.

A Long Wave-Infrared Miniatured Quantum Dot Spectrometer.

In this work, a long-wave infrared (LWIR) quantum dot (QD) spectrometer was constructed for the first time by integrating a 255-element HgSe QD filter array with an LWIR array detector. The filter array was fabricated using a combination inkjet printing strategy with eight types of T-dodecyl mercaptan-terminated HgSe QD inks. The stability and morphology of the QDs were improved by optimizing the purification methodology and ligand modification. Combined with the compressive-sensing-based least-squares linear regression (CS-LS) algorithm, the LWIR QD spectrometer achieved a spectral resolution of 5.4 cm-1 over a wide spectral range of 8 to 14 μm, enabling the detection of the chemical warfare agent simulant dimethyl methylphosphonate. This technology is expected to facilitate the development of smaller volumes and more accurate identification of various targets in the future. This paper offers an approach to fabricating low-cost LWIR spectrometers and promoting the scale-up applications of QD devices.

An impartial framework to investigate demosaicking input embedding options

Convolutional Neural Networks (CNNs) have proven highly effective for demosaicking, transforming raw Color Filter Array (CFA) sensor samples into standard RGB images. Directly applying convolution to the CFA tensor can lead to misinterpretation of the color context, so existing demosaicking networks typically embed the CFA tensor into the Euclidean space before convolution. The most prevalent embedding options are Reordering and Pre-interpolation. However, it remains unclear which option is more advantageous for demosaicking. Moreover, no existing demosaicking network is suitable for conducting a fair comparison. As a result, in practice, the selection of these two embedding options is often based on intuition and heuristic approaches. This paper addresses the non-comparability between the two options and investigates whether pre-interpolation contributes additional knowledge to the demosaicking network. Based on rigorous mathematical derivation, we design pairs of end-to-end fully convolutional evaluation networks, ensuring that the performance difference between each pair of networks can be solely attributed to their differing CFA embedding strategies. Under strictly fair comparison conditions, we measure the performance contrast between the two embedding options across various scenarios. Our comprehensive evaluation reveals that the prior knowledge introduced by pre-interpolation benefits lightweight models. Additionally, pre-interpolation enhances the robustness to imaging artifacts for larger models. Our findings offer practical guidelines for designing imaging software or Image Signal Processors (ISPs) for RGB cameras.

An Edge-Preserving Regularization Model for the Demosaicing of Noisy Color Images

This paper proposes an edge-preserving regularization technique to solve the color image demosaicing problem in the realistic case of noisy data. We enforce intra-channel local smoothness of the intensity (low-frequency components) and inter-channel local similarity of the depth of object borders and textures (high-frequency components). Discontinuities of both the low-frequency and high-frequency components are accounted for implicitly, i.e., through suitable functions of the proper derivatives. For the treatment of even the finest image details, derivatives of first, second, and third orders are considered. The solution to the demosaicing problem is defined as the minimizer of an energy function, accounting for all these constraints plus a data fidelity term. This non-convex energy is minimized via an iterative deterministic algorithm, applied to a family of approximating functions, each implicitly referring to geometrically consistent image edges. Our method is general because it does not refer to any specific color filter array. However, to allow quantitative comparisons with other published results, we tested it in the case of the Bayer CFA and on the Kodak 24-image dataset, the McMaster (IMAX) 18-image dataset, the Microsoft Demosaicing Canon 57-image dataset, and the Microsoft Demosaicing Panasonic 500-image dataset. The comparisons with some of the most recent demosaicing algorithms show the good performance of our method in both the noiseless and noisy cases.

Cr-Coot: Chronological Coot Algorithm-Based Deep Learning Model for Video Demosaicing

Usually, digital cameras comprise sensor arrays enclosed by Color Filter Arrays (CFAs), mosaics of minute color filters. Thus, every pixel sensor usually records limited spectral data regarding relevant pixels. Demosaicing is defined as the procedure of deducing the misplaced data for every pixel, which plays a vital role in recreating high-quality full-color images. Denoising and demosaicing are the major processes in the camera imaging chain for both videos and images. Here, reconstruction errors occur in these points and have undesirable effects on the final outcome, when it is not appropriately managed. The demosaicing process provokes color and spatial correlation of noises, and it is improved by means of imagining a pipeline. This organized noise usually destroys the quality of the image as well as fails to prevent accurate interpretation of an image. During the mitigation of this structured noise on processed data, denoising techniques diminish the texture and information. Therefore, an effectual demosaicing technique is essential for recreating the full-color image from the defective color samples. Thus, in this paper, an effectual video demosaicing model is proposed using an optimized deep learning system. The designed video demosaicing system achieved better performance with a Peak Signal-to-Noise Ratio (PSNR) of 59.74 dB, Second Derivative, like Measure of Enhancement (SDME) of 63.51, and Root Mean Squared Error (RMSE) of 0.3660.

The Effect of Varying the Light Spectrum of a Scene on the Localisation of Photogrammetric Features

In modern digital photogrammetry, an image is usually registered via a digital matrix with an array of colour filters. From the registration of the image until feature points are detected on the image, the image is subjected to a series of calculations, i.e., demosaicing and conversion to greyscale, among others. These algorithms respond differently to the varying light spectrum of the scene, which consequently results in the feature location changing. In this study, the effect of scene illumination on the localisation of a feature in an image is presented. The demosaicing and greyscale conversion algorithms that produce the largest and smallest deviation of the feature from the reference point were assessed. Twelve different illumination settings from polychromatic light to monochromatic light were developed and performed, and five different demosaicing algorithms and five different methods of converting a colour image to greyscale were analysed. A total of 300 different cases were examined. As the study shows, the lowest deviation in the polychromatic light domain was achieved for light with a colour temperature of 5600 K and 5000 K, while in the monochromatic light domain, it was achieved for light with a green colour. Demosaicing methods have a significant effect on the localisation of a feature, and so the smallest feature deviation was achieved for smooth hue-type demosaicing, while for greyscale conversion, it was achieved for the mean type. Demosaicing and greyscale conversion methods for monochrome light had no effect. The article discusses the problem and concludes with recommendations and suggestions in the area of illuminating the scene with artificial light and the application of the algorithms, in order to achieve the highest accuracy using photogrammetric methods.

Deep demosaicking convolution neural network and quantum wavelet transform-based image denoising

ABSTRACT Demosaicking is a popular scientific area that is being explored by a vast number of scientists. Current digital imaging technologies capture colour images with a single monochrome sensor. In addition, the colour images were captured using a sensor coupled with a Colour Filter Array (CFA). Furthermore, the demosaicking procedure is required to obtain a full-colour image. Image denoising and image demosaicking are the two important image restoration techniques, which have increased popularity in recent years. Finding a suitable strategy for multiple image restoration is critical for researchers. Hence, a deep learning (DL) based image denoising and image demosaicking is developed in this research. Moreover, the Autoregressive Circle Wave Optimization (ACWO) based Demosaicking Convolutional Neural Network (DMCNN) is designed for image demosaicking. The Quantum Wavelet Transform (QWT) is used in the image denoising process. Similarly, Quantum Wavelet Transform (QWT) is used to analyse the abrupt changes in the input image with noise. The transformed image is then subjected to a thresholding technique, which determines an appropriate threshold range. Once the threshold range has been determined, soft thresholding is applied to the resulting wavelet coefficients. After that, the extraction and reconstruction of the original image is carried out using the Inverse Quantum Wavelet Transform (IQWT). Finally, the fused image is created by combining the results of both processes using a weighted average. The denoised and demosaicked images are combined using the weighted average technique. Furthermore, the proposed QWT+DMCNN-ACWO model provided the ideal values of Peak signal-to-noise ratio (PSNR), Second derivative like measure of enhancement (SDME), Structural Similarity Index (SSIM), Figure of Merit (FOM) of 0.890, and computational time of 49.549 dB, 59.53 dB, 0.963, 0.890, and 0.571, respectively.

Compact multispectral light field camera based on an inkjet-printed microlens array and color filter array

With emerging advanced optical sensing technologies and their wide-ranging applications, gathering comprehensive optical data from real-world scenes is becoming increasingly crucial for their accurate reconstruction and analysis. In order to capture both three-dimensional (3D) spatial and spectral information from a scene, multiple devices or time-intensive scanning processes are often involved. Here, we demonstrate a multispectral light field camera that allows for the simultaneous acquisition of 3D information and spectral data in a single snapshot. By utilizing inkjet printing as the fabrication technology, the miniaturized optical components in the camera were manufactured with high precision and can be integrated into a standard camera housing. Furthermore, the microlens arrays and the color filter arrays were fabricated on the same substrate, and a precise alignment between the two arrays was achieved. The compact multispectral camera opens the door to a multitude of possibilities for mobile applications, ranging from autonomous driving and consumer electronics such as smartphones to medical technology such as endoscopes.

A Multi-Shot Approach for Spatial Resolution Improvement of Multispectral Images from an MSFA Sensor.

Multispectral imaging technology has advanced significantly in recent years, allowing single-sensor cameras with multispectral filter arrays to be used in new scene acquisition applications. Our camera, developed as part of the European CAVIAR project, uses an eight-band MSFA to produce mosaic images that can be decomposed into eight sparse images. These sparse images contain only pixels with similar spectral properties and null pixels. A demosaicing process is then applied to obtain fully defined images. However, this process faces several challenges in rendering fine details, abrupt transitions, and textured regions due to the large number of null pixels in the sparse images. Therefore, we propose a sparse image composition method to overcome these challenges by reducing the number of null pixels in the sparse images. To achieve this, we increase the number of snapshots by simultaneously introducing a spatial displacement of the sensor by one to three pixels on the horizontal and/or vertical axes. The set of snapshots acquired provides a multitude of mosaics representing the same scene with a redistribution of pixels. The sparse images from the different mosaics are added together to get new composite sparse images in which the number of null pixels is reduced. A bilinear demosaicing approach is applied to the composite sparse images to obtain fully defined images. Experimental results on images projected onto the response of our MSFA filter show that our composition method significantly improves image spatial resolution and minimizes reconstruction errors while preserving spectral fidelity.

Smart mid-infrared metasurface microspectrometer gas sensing system

Smart, low-cost and portable gas sensors are highly desired due to the importance of air quality monitoring for environmental and defense-related applications. Traditionally, electrochemical and nondispersive infrared (IR) gas sensors are designed to detect a single specific analyte. Although IR spectroscopy-based sensors provide superior performance, their deployment is limited due to their large size and high cost. In this study, a smart, low-cost, multigas sensing system is demonstrated consisting of a mid-infrared microspectrometer and a machine learning algorithm. The microspectrometer is a metasurface filter array integrated with a commercial IR camera that is consumable-free, compact ( ~ 1 cm3) and lightweight ( ~ 1 g). The machine learning algorithm is trained to analyze the data from the microspectrometer and predict the gases present. The system detects the greenhouse gases carbon dioxide and methane at concentrations ranging from 10 to 100% with 100% accuracy. It also detects hazardous gases at low concentrations with an accuracy of 98.4%. Ammonia can be detected at a concentration of 100 ppm. Additionally, methyl-ethyl-ketone can be detected at its permissible exposure limit (200 ppm); this concentration is considered low and nonhazardous. This study demonstrates the viability of using machine learning with IR spectroscopy to provide a smart and low-cost multigas sensing platform.

On-Chip Reconstructive Spectrometer Based on Parallel Cascaded Micro-Ring Resonators

In contrast to cumbersome benchtop spectrometers, integrated on-chip spectrometers are well-suited for portable applications in health monitoring and environmental sensing. In this paper, we have developed an on-chip spectrometer with a programmable silicon photonic filter by simply using parallel cascaded micro-ring resonators (MRs). By altering the transmission spectrum of the filter, multiple and diverse sampling of the input spectrum is achieved. Then, combined with an artificial neural network (ANN) model, the incident spectrum is reconstructed from the sampled signals. Each MR is coupled to adjacent ones, and the phase shifts within each MR can be independently tuned. Through dynamic programming of the phases of these MRs, sampling functions featuring diverse characteristics are obtained based on a single programmable filter with an adjustable number of sampling channels. This eliminates the need for a filter array, significantly reducing the area of the on-chip reconstructive spectrometer. The simulation results demonstrate that the proposed design can achieve the reconstruction of continuous and sparse spectra within the wavelength range of 1450 nm to 1650 nm, with a tunable resolution ranging from 2 nm to 0.2 nm, depending on the number of sampling states employed. This benefit arises from the programmable nature of the device. The device holds tremendous potential for applications in wearable optical sensing, portable spectrometry, and other related scenarios.

Design and Implementation of Digital Down Converter for WiFi Network

This paper introduces a field-programmable gate array (FPGA)-based digital down converter (DDC) processing a sampling frequency of about 3.64 GHz to a down-converted frequency of 28.4375 MHz to match the IEEE 802.11ah Wi-Fi HaLow standard. The proposed DDC uses a polyphase mixer(PM) and a resampling filter. The PM adopts parallel coordinate rotation digital computer (CORDIC) processors and lowpass filter arrays to reduce high-speed data rates with minimum resource utilization. Again, the resampling filter employs a cascaded integrator comb (CIC) filter associated with a parallel prefixed adder (PPA) and a multi-channel systolic finite impulse response (FIR) filter implemented with canonical expression, attaining optimum hardware cost. Converting floating-point to fixed-point data types provides significant resource savings. Finally, the improved design is coded in the Xilinx Vivado synthesis tool and successfully tested on the FPGA Kintex-7 device. In contrast to other recent architectures, the proposed design substantially reduces area requirements and power utilization. The Matlab tool verifies the design to achieve an acceptable spurious-free dynamic range (SFDR) of 115 dB.