Monochromatic Sensor Research Articles

MONOCHROME AUGMENTED LOW-LIGHT IMAGE ENHANCEMENT

Low light short exposure photography is challenging, but an important factor in capturing images in temporarily dynamic scenes avoiding unwanted effects such as ghosting, motion blur, camera shakes, image artifacts, etc. Monochrome augmented low-light image enhancement aims to get improved low-light short-exposure images by using an additional monochrome sensor and its data. Monochrome images typically possess a higher SNR (Signal-to-Noise Ratio) and better luma information, since it avoids the attenuation by the Bayer Filter. The objective here is to develop a deep learning based approach to enhance low-light short exposure images from the main sensor by using an additional low resolution monochrome sensor.

Deep demosaicking convolution neural network and quantum wavelet transform-based image denoising

ABSTRACT Demosaicking is a popular scientific area that is being explored by a vast number of scientists. Current digital imaging technologies capture colour images with a single monochrome sensor. In addition, the colour images were captured using a sensor coupled with a Colour Filter Array (CFA). Furthermore, the demosaicking procedure is required to obtain a full-colour image. Image denoising and image demosaicking are the two important image restoration techniques, which have increased popularity in recent years. Finding a suitable strategy for multiple image restoration is critical for researchers. Hence, a deep learning (DL) based image denoising and image demosaicking is developed in this research. Moreover, the Autoregressive Circle Wave Optimization (ACWO) based Demosaicking Convolutional Neural Network (DMCNN) is designed for image demosaicking. The Quantum Wavelet Transform (QWT) is used in the image denoising process. Similarly, Quantum Wavelet Transform (QWT) is used to analyse the abrupt changes in the input image with noise. The transformed image is then subjected to a thresholding technique, which determines an appropriate threshold range. Once the threshold range has been determined, soft thresholding is applied to the resulting wavelet coefficients. After that, the extraction and reconstruction of the original image is carried out using the Inverse Quantum Wavelet Transform (IQWT). Finally, the fused image is created by combining the results of both processes using a weighted average. The denoised and demosaicked images are combined using the weighted average technique. Furthermore, the proposed QWT+DMCNN-ACWO model provided the ideal values of Peak signal-to-noise ratio (PSNR), Second derivative like measure of enhancement (SDME), Structural Similarity Index (SSIM), Figure of Merit (FOM) of 0.890, and computational time of 49.549 dB, 59.53 dB, 0.963, 0.890, and 0.571, respectively.

Wavelength-scanning pixel-super-resolved lens-free on-chip quantitative phase microscopy with a color image sensor

We report a wavelength-scanning-based lens-free on-chip microscope using a color CMOS sensor and a matching modified phase retrieval algorithm for pixel super-resolution. Compared to traditional monochrome industrial cameras, color sensors favored by the consumer electronics industry have smaller pixel sizes, higher performance, and lower costs. However, the color filtering array (CFA) introduces inherent modulation to the holograms acquired under quasi-monochromatic illumination, which complicates the data processing in lens-free on-chip microscopy. Without physically removing the CFA positioned on the sensor chip, we demonstrate quantitative phase imaging (QPI) with a lateral half-width resolution of 615 nm over a wide field-of-view of 51.88 mm2 by exploiting the green-channel data from Bayer-masked holograms. The resulting spatial bandwidth product is 137.2 megapixels, over 10 times that of a conventional optical microscope. The rationale for using only green-channel data is that the information from each sampling point is not lost during propagation but rather distributed to all pixels in the image. Therefore, the missing data in other channels can be recovered by exploiting the sufficient differences among the raw images captured at different wavelengths. Compared to the scheme with monochrome sensors, this method requires the acquisition of several more images to guarantee the convergence of the algorithm. Experimental results show that we can achieve high-quality QPI performance, thus demonstrating the applicability of cost-effective color sensors in the field of lens-free holographic microscopy.

Multi-View Acoustic Field Imaging With Digital Color Holography

This paper proposes an approach for acoustic field imaging using simultaneous multi-view digital holography based on three-color digital off-axis holography. Considering spatio-chromatic multiplexing and the recording with a monochrome sensor, the numerical processing of time-sequences of holograms yields both the amplitude and phase of the acoustic field along three different directions of observation. Distortion analysis is presented and the acousto-optic interaction along the optical beam is discussed using a theoretical modelling. Experimental results with an emitter at 40 kHz establish the proof-of-concept of the proposed multi-view imaging for acoustic fields.

Deep learning-based hyperspectral image reconstruction from emulated and real computed tomography imaging spectrometer data

The computed tomography imaging spectrometer (CTIS) is a hyperspectral imaging (HSI) approach where spectral and spatial information of a scene is mixed during the imaging process onto a monochromatic sensor. This mixing is due to a diffractive optical element integrated into the underlying optics and creates a set of diffraction orders. To reconstruct a three-dimensional hyperspectral cube from the CTIS sensor image, iterative algorithms were applied. Unfortunately, such methods are highly sensitive to noise and require high computational time for reconstruction thus hindering their applicability in real-time and high frame-rate applications. To overcome such limitations, we propose a lightweight and efficient deep convolutional neural network for hyperspectral image reconstruction from CTIS sensor images. Compared with classical approaches our model delivers considerably better reconstruction results on synthetic as well as real CTIS images in under 0.17 s, which is over 60 times faster compared with the standard iterative approach. In addition, the reshaping method we have developed enables a lightweight network architecture with over 100 times fewer parameters than previously reported.

X-ray scintillator lens-coupled with CMOS camera for pre-clinical cardiac vascular imaging-A feasibility study.

We present the design and characterization of an X-ray imaging system consisting of an off-the-shelf CMOS sensor optically coupled to a CsI scintillator. The camera can perform both high-resolution and functional cardiac imaging. High-resolution 3D imaging requires microfocus X-ray tubes and expensive detectors, while pre-clinical functional cardiac imaging requires high flux pulsed (clinical) X-ray tubes and high-end cameras. Our work describes an X-ray camera, namely an “optically coupled X-ray(OCX) detector,” used for both the aforementioned applications with no change in the specifications. We constructed the imaging detector with two different CMOS optical imaging cameras called CMOS sensors, 1.A monochrome CMOS sensor coupled with an f1.4 lens and 2.an RGB CMOS sensor coupled with an f0.95 prime lens. The imaging system consisted of our X-ray camera, micro-focus X-ray source (50kVp and 1mA), and a rotary stage controlled from a personal computer (PC) and LabVIEW interface. The detective quantum efficiency (DQE) of the imaging system(monochrome) estimated using a cascaded linear model was 17% at 10 lp/mm. The system modulation transfer function (MTF) and the noise power spectrum (NPS) were inputs to the DQE estimation. Because of the RGB camera’s low quantum efficiency (QE), the OCX detector DQE was 19% at 5 lp/mm. The contrast to noise ratio (CNR) at different frame rates was studied using the capillary tubes filled with various dilutions of iodinated contrast agents. In-vivo cardiac angiography demonstrated that blood vessels of the order of 100 microns or above were visible at 40 frames per second despite the low X-ray flux. For high-resolution 3D imaging, the system was characterized by imaging a cylindrical micro-CT contrast phantom and comparing it against images from a commercial scanner.

Single-shot multiwavelength digital holography using Bragg diffraction of light by several ultrasound waves [Invited

We report on wavelength-multiplexed digital holographic imaging based on simultaneous Bragg diffraction of wideband light by several ultrasound waves of different frequencies in crystalline media. This technique is easy to implement, avoids spectral scanning, and is applicable in various digital holography schemes. It also enables single-shot acquisition of a few spectral fringe patterns by a single monochrome sensor and wavelength demultiplexing of the resulting interferogram. We have assembled a Mach-Zehnder interferometer with an acousto-optical tunable filter operating in the multifrequency mode and have validated the proposed technique by dual-, three-, and four-wavelength quantitative phase imaging of the test samples.

Capturing detailed deformations of moving human bodies

We present a new method to capture detailed human motion, sampling more than 1000 unique points on the body. Our method outputs highly accurate 4D (spatio-temporal) point coordinates and, crucially, automatically assigns a unique label to each of the points. The locations and unique labels of the points are inferred from individual 2D input images only, without relying on temporal tracking or any human body shape or skeletal kinematics models. Therefore, our captured point trajectories contain all of the details from the input images, including motion due to breathing, muscle contractions and flesh deformation, and are well suited to be used as training data to fit advanced models of the human body and its motion. The key idea behind our system is a new type of motion capture suit which contains a special pattern with checkerboard-like corners and two-letter codes. The images from our multi-camera system are processed by a sequence of neural networks which are trained to localize the corners and recognize the codes, while being robust to suit stretching and self-occlusions of the body. Our system relies only on standard RGB or monochrome sensors and fully passive lighting and the passive suit, making our method easy to replicate, deploy and use. Our experiments demonstrate highly accurate captures of a wide variety of human poses, including challenging motions such as yoga, gymnastics, or rolling on the ground.

Capturing detailed deformations of moving human bodies

We present a new method to capture detailed human motion, sampling more than 1000 unique points on the body. Our method outputs highly accurate 4D (spatio-temporal) point coordinates and, crucially, automatically assigns a unique label to each of the points. The locations and unique labels of the points are inferred from individual 2D input images only, without relying on temporal tracking or any human body shape or skeletal kinematics models. Therefore, our captured point trajectories contain all of the details from the input images, including motion due to breathing, muscle contractions and flesh deformation, and are well suited to be used as training data to fit advanced models of the human body and its motion. The key idea behind our system is a new type of motion capture suit which contains a special pattern with checkerboard-like corners and two-letter codes. The images from our multi-camera system are processed by a sequence of neural networks which are trained to localize the corners and recognize the codes, while being robust to suit stretching and self-occlusions of the body. Our system relies only on standard RGB or monochrome sensors and fully passive lighting and the passive suit, making our method easy to replicate, deploy and use. Our experiments demonstrate highly accurate captures of a wide variety of human poses, including challenging motions such as yoga, gymnastics, or rolling on the ground.

Snapshot Imaging Spectrometer Based on Pixel-Level Filter Array (PFA).

Snapshot spectral imaging technology plays an important role in many fields. However, most existing snapshot imaging spectrometers have the shortcomings of a large volume or heavy computational burden. In this paper, we present a novel snapshot imaging spectrometer based on the pixel-level filter array (PFA), which can simultaneously obtain both spectral and spatial information. The system is composed of a fore-optics, a PFA, a relay lens, and a monochromatic sensor. The incoming light first forms an intermediate image on the PFA through the fore-optics. Then, the relay lens reimages the spectral images on the PFA onto the monochromatic sensor. Through the use of the PFA, we can capture a three-dimensional (spatial coordinates and wavelength) datacube in a single exposure. Compared with existing technologies, our system possesses the advantages of a simple implementation, low cost, compact structure, and high energy efficiency by removing stacked dispersive or interferometric elements. Moreover, the characteristic of the direct imaging mode ensures the low computational burden of the system, thus shortening the imaging time. The principle and design of the system are described in detail. An experimental prototype is built and field experiments are carried out to verify the feasibility of the proposed scheme.

Deep Light Field Acquisition Using Learned Coded Mask Distributions for Color Filter Array Sensors

Compressive light field photography enables light field acquisition using a single sensor by utilizing a color coded mask. This approach is very cost effective since consumer-level digital cameras can be turned into light field cameras by simply placing a coded mask between the sensor and the aperture plane. This paper describes a deep learning architecture for compressive light field acquisition using a color coded mask and a sensor with Color Filter Array (CFA). Unlike previous methods where a fixed mask pattern is used, our deep network learns the optimal distribution of the color coded mask pixels. The proposed solution enables end-to-end learning of the color-coded mask distribution and the reconstruction network, taking into account the sensor CFA. Consequently, the resulting network can efficiently perform joint demosaicing and light field reconstruction of images acquired with color-coded mask and a CFA sensor. Compared to previous methods based on deep learning with monochrome sensors, as well as traditional compressive sensing approaches using CFA sensors, we obtain superior color reconstruction of the light fields.

Multi-Mask Camera Model for Compressed Acquisition of Light Fields

We present an all-in-one camera model that encompasses the architectures of most existing compressive-sensing light-field cameras, equipped with a single lens and multiple amplitude coded masks that can be placed at different positions between the lens and the sensor. The proposed model, named the equivalent multi-mask camera (EMMC) model, enables the comparison between different camera designs, e.g using monochrome or CFA-based sensors, single or multiple acquisitions, or varying pixel sizes, via a simple adaptation of the sampling operator. In particular, in the case of a camera equipped with a CFA-based sensor and a coded mask, this model allows us to jointly perform color demosaicing and light field spatio-angular reconstruction. In the case of variable pixel size, it allows to perform spatial super-resolution in addition to angular reconstruction. While the EMMC model is generic and can be used with any reconstruction algorithm, we validate the proposed model with a dictionary-based reconstruction algorithm and a regularization-based reconstruction algorithm using a 4D Total-Variation-based regularizer for light field data. Experimental results with different reconstruction algorithms show that the proposed model can flexibly adapt to various sensing schemes. They also show the advantage of using an in-built CFA sensor with respect to monochrome sensors classically used in the literature.

Microfluidic Fabrication Techniques for High-Pressure Testing of Microscale Supercritical CO2 Foam Transport in Fractured Unconventional Reservoirs.

Pressure limitations of many microfluidic platforms have been a significant challenge in microfluidic experimental studies of fractured media. As a result, these platforms have not been fully exploited for direct observation of high-pressure transport in fractures. This work introduces microfluidic platforms that enable direct observation of multiphase flow in devices featuring surrogate permeable media and fractured systems. Such platforms provide a pathway to address important and timely questions such as those related to CO2 capture, utilization and storage. This work provides a detailed description of the fabrication techniques and an experimental setup that may serve to analyze the behavior of supercritical CO2 (scCO2) foam, its structure and stability. Such studies provide important insights regarding enhanced oil recovery processes and the role of hydraulic fractures in resource recovery from unconventional reservoirs. This work presents a comparative study of microfluidic devices developed using two different techniques: photolithography/wet-etching/thermal-bonding versus Selective Laser-induced Etching. Both techniques result in devices that are chemically and physically resistant and tolerant of high pressure and temperature conditions that correspond to subsurface systems of interest. Both techniques provide pathways to high-precision etched microchannels and capable lab-on-chip devices. Photolithography/wet-etching, however, enables fabrication of complex channel networks with complex geometries, which would be a challenging task for laser etching techniques. This work summarizes a step-by-step photolithography, wet-etching and glass thermal-bonding protocol and, presents representative observations of foam transport with relevance to oil recovery from unconventional tight and shale formations. Finally, this work describes the use of a high-resolution monochromatic sensor to observe scCO2 foam behavior where the entirety of the permeable medium is observed simultaneously while preserving the resolution needed to resolve features as small as 10 µm.

Single Sensor Compressive Light Field Video Camera

AbstractThis paper presents a novel compressed sensing (CS) algorithm and camera design for light field video capture using a single sensor consumer camera module. Unlike microlens light field cameras which sacrifice spatial resolution to obtain angular information, our CS approach is designed for capturing light field videos with high angular, spatial, and temporal resolution. The compressive measurements required by CS are obtained using a random color‐coded mask placed between the sensor and aperture planes. The convolution of the incoming light rays from different angles with the mask results in a single image on the sensor; hence, achieving a significant reduction on the required bandwidth for capturing light field videos. We propose to change the random pattern on the spectral mask between each consecutive frame in a video sequence and extracting spatio‐angular‐spectral‐temporal 6D patches. Our CS reconstruction algorithm for light field videos recovers each frame while taking into account the neighboring frames to achieve significantly higher reconstruction quality with reduced temporal incoherencies, as compared with previous methods. Moreover, a thorough analysis of various sensing models for compressive light field video acquisition is conducted to highlight the advantages of our method. The results show a clear advantage of our method for monochrome sensors, as well as sensors with color filter arrays.

A single sensor based multispectral imaging camera using a narrow spectral band color mosaic integrated on the monochrome CMOS image sensor

A multispectral image camera captures image data within specific wavelength ranges in narrow wavelength bands across the electromagnetic spectrum. Images from a multispectral camera can extract a additional information that the human eye or a normal camera fails to capture and thus may have important applications in precision agriculture, forestry, medicine, and object identification. Conventional multispectral cameras are made up of multiple image sensors each fitted with a narrow passband wavelength filter and optics, which makes them heavy, bulky, power hungry, and very expensive. The multiple optics also create an image co-registration problem. Here, we demonstrate a single sensor based three band multispectral camera using a narrow spectral band red–green–blue color mosaic in a Bayer pattern integrated on a monochrome CMOS sensor. The narrow band color mosaic is made of a hybrid combination of plasmonic color filters and a heterostructured dielectric multilayer. The demonstrated camera technology has reduced cost, weight, size, and power by almost n times (where n is the number of bands) compared to a conventional multispectral camera.

Development of a Low-Cost Multi-Waveband LED Illumination Imaging Technique for Rapid Evaluation of Fresh Meat Quality

Determining the quality of meat has always been essential for the food industry because consumers prefer superior quality meat. Therefore, the food industry requires the development of a rapid and non-destructive method for meat-quality determination. Over the past few years, a number of techniques have been presented for monitoring meat–chemical attributes. However, most previous techniques are quite expensive, destructive, and require complex hardware to operate. Thus, in this work, we demonstrate a low-cost sensing technique (eliminating the expensive equipment and complicated design) for meat–chemical quality detection. The newly developed system was integrated with a low-cost monochrome camera and ordinary light-emitting diode (LED) light sources, with fifteen different wavebands ranging from 458 to 950 nm. The monochrome camera captures images of the meat sample across a spectral range from 458 to 950 nm using a single snapshot method. The chemical values (e.g., moisture, fat, and protein) were also determined using conventional methods. The collected images were combined to produce a multispectral data cube and to extract spectral data. Partial least squares (PLS) and support vector regression (SVR) modeling were used on the extracted spectra and chemical values. The developed models for meat samples displayed accurate chemical-component prediction ( R 2 > 0.80). Our model, based on a monochrome sensor using only fifteen wavebands, provided reasonable results compared with the previously developed expensive spectroscopic techniques. Therefore, this complementary metal-oxide semiconductor (CMOS) based multispectral sensing technique may have the potential to detect meat quality, thereby facilitating a simple, fast, and cost-effective method applicable to small-scale meat-processing industries.

A Natural Steganography Embedding Scheme Dedicated to Color Sensors in the JPEG Domain

Using Natural Steganography (NS), a cover raw image acquired at sensitivity ISO 1 is transformed into a stego image whose statistical distribution is similar to a cover image acquired at sensitivity ISO 2 > ISO 1. This paper proposes such an embedding scheme for color sensors in the JPEG domain, extending thus the prior art proposed for the pixel domain and the JPEG domain for monochrome sensors. We first show that color sensors generate strong intra-block and inter-block dependencies between DCT coefficients and that theses dependencies are due to the demosaicking step in the development process. Capturing theses dependencies using an empirical covariance matrix, we propose a pseudo-embedding algorithm on greyscale JPEG images which uses up to four sub-lattices and 64 lattices to embed information while preserving the estimated correlations among DCT coefficients. We then compute an approximation of the average embedding rate w.r.t. the JPEG quality factor and evaluate the empirical security of the proposed scheme for linear and non-linear demosaicing schemes. Our experiments show that we can achieve high capacity (around 2 bit per nzAC) with a high empirical security (P E 30% using DCTR at QF 95).

Benchmarking Smartphone Fluorescence-Based Microscopy with DNA Origami Nanobeads: Reducing the Gap toward Single-Molecule Sensitivity.

Smartphone-based fluorescence microscopy has been rapidly developing over the last few years, enabling point-of-need detection of cells, bacteria, viruses, and biomarkers. These mobile microscopy devices are cost-effective, field-portable, and easy to use, and benefit from economies of scale. Recent developments in smartphone camera technology have improved their performance, getting closer to that of lab microscopes. Here, we report the use of DNA origami nanobeads with predefined numbers of fluorophores to quantify the sensitivity of a smartphone-based fluorescence microscope in terms of the minimum number of detectable molecules per diffraction-limited spot. With the brightness of a single dye molecule as a reference, we compare the performance of color and monochrome sensors embedded in state-of-the-art smartphones. Our results show that the monochrome sensor of a smartphone can achieve better sensitivity, with a detection limit of ∼10 fluorophores per spot. The use of DNA origami nanobeads to quantify the minimum number of detectable molecules of a sensor is broadly applicable to evaluate the sensitivity of various optical instruments.

Thermometry of combustion gas measuring two-band near-infrared emissions less than 1.1 μm from water molecules

Thermometry using two-band near-infrared (NIR) emissions less than 1.1 μm has been implemented for combustion gases formed on a flat flame burner. The gas temperature was calibrated using a SiC thin filament pyrometry (TFP) above 1500 K for H2/air and CH4/air mixtures. The emission spectra were measured with a spectrometer and a monochrome CMOS sensor with an image doubler and two bandpass filters. Two pairs of bandpass filters (850 nm/925 nm and 975 nm/925 nm) were tested to measure temperature. Results indicated that strong H2O emissions existed in the NIR region, and they became stronger with an increase in the temperature. The CO2 emission spectra in the NIR region were negligible compared to that of H2O. Calibration temperature lines were obtained as a function of the intensity ratio of the two-band emissions in the NIR region. Using a CMOS camera, the uncertainties of temperature were 59 K for 850 nm/925 nm and 145 K for 975 nm/925 nm. The resolutions were 0.7 K for 850 nm/925 nm and 1 K for 975 nm/925 nm. Using the calibration line, two-dimensional temperature measurements were performed for H2/air flames, CH4/air flames and Bunsen flames of 13A city gas. For the high temperature region above 1500 K, a quantitative two-dimensional temperature field was measured for lean and rich combustion. In addition, temperature measurement at high pressure was conducted for combustion gas of propane/air mixture with the equivalence ratio of 0.7 using a rapid compression machine. Temperature could be estimated using the calibration line obtained for the flat flame burner. We have demonstrated the performance and the possibility of thermometry two-band H2O emissions in the NIR region.

Natural Steganography in JPEG Compressed Images

In natural steganography, the secret message is hidden by adding to the cover image a noise signal that mimics the heteroscedastic noise introduced naturally during acquisition. The method requires the cover image to be available in its RAW form (the sensor capture). To bring this idea closer to a practical embedding method, in this paper we embed the message in quantized DCT coefficients of a JPEG file by adding independent realizations of the heteroscedastic noise to pixels to make the embedding resemble the same cover image acquired at a larger sensor ISO setting (the so-called cover source switch). To demonstrate the feasibility and practicality of the proposed method and to validate our simplifying assumptions, we work with two digital cameras, one using a monochrome sensor and a second one equipped with a color sensor. We then explore several versions of the embedding algorithm depending on the model of the added noise in the DCT domain and the possible use of demosaicking to convert the raw image values. These experiments indicate that the demosaicking step has a significant impact on statistical detectability for high JPEG quality factors when making independent embedding changes to DCT coefficients. Additionally, for monochrome sensors or low JPEG quality factors very large payload can be embedded with high empirical security.