Compact Image Sensor Research Articles

Perspectives on 2D materials for hybrid and beyond-Si image sensor applications

Abstract The complementary metal–oxide–semiconductor (CMOS) image sensor has become essential and ubiquitous in our daily lives as it is present in almost every pocket. As demand for compact, multifunction, and high-efficiency Internet of Things applications continues to rise, novel configuration designs and manufacturing methods, such as neural network integration and 3D stacking have been implemented to enhance the CMOS image sensor’s (CIS) performance. However, the progress of image sensors based on silicon CMOS technology would eventually be limited by the intrinsic optical, electrical, and mechanical properties of silicon material. This has led to the exploration of two-dimensional materials (2DMs) and the emergence of 2DMs as promising candidates for the next generation of optoelectronic devices. In this article, we discuss the current advancements and challenges associated with silicon CISs and the potential benefits of incorporating 2DMs in the image sensor. We highlight three critical opportunities for 2DMs, including Si CMOS/2DMs hybrid structure and direct growth techniques of 2DMs on Si for back-end-of-line integration, 2DMs-based neuromorphic photodetectors (PDs) and optical neural networks for in-image-sensor-processing, and curved image sensor based on 2DMs PDs for bionic detection. With the growing maturity of 2DM technologies, we anticipate that the device scaling and the increase of integration density of 2DM electronics in the image sensor will continue, leading to the development of highly efficient, compact, intelligent, and versatile 2DM image sensors in the near future.

Freeform metasurface color router for deep submicron pixel image sensors.

Advances in imaging technologies have led to a high demand for ultracompact, high-resolution image sensors. However, color filter-based image sensors, now miniaturized to deep submicron pixel sizes, face challenges such as low signal-to-noise ratio due to fewer photons per pixel and inherent efficiency limitations from color filter arrays. Here, we demonstrate a freeform metasurface color router that achieves ultracompact pixel sizes while overcoming the efficiency limitations of conventional architectures by splitting and focusing visible light instead of filtering. This development is enabled by a fully differentiable topology optimization framework to maximize the use of the design space while ensuring fabrication feasibility and robustness to fabrication errors. The metasurface can distribute an average of 85% of incident visible light according to the Bayer pattern with a pixel size of 0.6 μm. The device and design methodology enable the compact, high-sensitivity, and high-resolution image sensors for various modern technologies and pave the way for the advanced photonic device design.

Charge Management Enables Efficient Spontaneous Chromatic Adaptation Bipolar Photodetector.

Solution-processed photodetectors have emerged as promising candidates for next-generation of visible-near infrared (vis-NIR) photodetectors. This is attributed to their ease of processing, compatibility with flexible substrates, and the ability to tune their detection properties by integrating complementary photoresponsive semiconductors. However, the limited performance continues to hinder their further development, primarily influenced by the difference of charge transport properties between perovskite and organic semiconductors. In this work, a perovskite-organic bipolar photodetectors (PDs) is introduced with multispectral responsivity, achieved by effectively managing charges in perovskite and a ternary organic heterojunction. The ternary heterojunction, incorporating a designed NIR guest acceptor, exhibits a faster charge transfer rate and longer carrier diffusion length than the binary heterojunction. By achieving a more balanced carrier dynamic between the perovskite and organic components, the PD achieves a low dark current of 3.74nAcm-2 at -0.2V, a fast response speed of <10 µs, and a detectivity of exceeding 1012 Jones. Furthermore, a bioinspired retinotopic system for spontaneous chromatic adaptation is achieved without any optical filter. This charge management strategy opens up possibilities for surpassing the limitations of photodetection and enables the realization of high-purity, compact image sensors with exceptional spatial resolution and accurate color reproduction.

A 0.5 MP, 3D-Stacked, Voltage-Domain Global Shutter Image Sensor with NIR QE Enhancement, Event Detection Modes, and 90 dB Dynamic Range

We introduce a compact voltage-domain global shutter CMOS image sensor for a wide range of applications including consumer, IoT, and industrial applications. With 0.5 MP and 2.79 µm pixels packed in a die size of only 2.3 mm × 2.8 mm, the sensor achieves more than 92% quantum efficiency (QE) in the visible wavelength and more than 36% at near-infrared (940 nm) all while drawing a mere 20 mW at a 10-bit, 30-frame-per-second operational mode. In this article, we focus on the architecture of the sensor and the design challenges encountered to fit all the necessary circuitry in such a limited footprint. Moreover, we detail the new solutions we have developed to meet the demanding specifications of low-power operation and high dynamic range (HDR).

Software-defined meta-optics

Rapid advancements in autonomous systems and the Internet of Things have necessitated the development of compact and low-power image sensors to bridge the gap between the digital and physical world. To that end, sub-wavelength diffractive optics, commonly known as meta-optics, have garnered significant interest from the optics and photonics community due to their ability to achieve multiple functionalities within a small form factor. Despite years of research, however, the performance of meta-optics has often remained inferior compared to that of traditional refractive optics. In parallel, computational imaging techniques have emerged as a promising path to miniaturize optical systems, albeit often at the expense of higher power and latency. The lack of desired performance from either meta-optical or computational solutions has motivated researchers to look into a jointly optimized meta-optical–digital solution. While the meta-optical front end can preprocess the scene to reduce the computational load on the digital back end, the computational back end can in turn relax requirements on the meta-optics. In this Perspective, we provide an overview of this up-and-coming field, termed here as “software-defined meta-optics.” We highlight recent contributions that have advanced the current state of the art and point out directions toward which future research efforts should be directed to leverage the full potential of subwavelength photonic platforms in imaging and sensing applications. Synergistic technology transfer and commercialization of meta-optic technologies will pave the way for highly efficient, compact, and low-power imaging systems of the future.

Improvement of measurement method for compact range image sensor using disparity and blur of multi-slit laser

Improvement of measurement method for compact range image sensor using disparity and blur of multi-slit laser

Hetero‐Integrated InGaAs Photodiode and Oxide Memristor‐Based Artificial Optical Nerve for In‐Sensor NIR Image Processing

AbstractIn‐sensor computing is an emerging architectural paradigm that fuses data acquisition and processing within a sensory domain. The integration of multiple functions into a single domain reduces the system footprint while it minimizes the energy and time for data transfer between sensory and computing units. However, it is challenging for a simple and compact image sensor array to achieve both sensing and computing in each pixel. Here, this work demonstrates a focal plane array with a heterogeneously integrated one‐photodiode one‐resistor (1P‐1R)‐based artificial optical neuron that emulates the sensing, computing, and memorization of a biological retina system. This work employs an InGaAs photodiode featuring a high responsivity and a broad spectrum that covers near‐infrared (NIR) signals and employs an HfO2 memristor as the artificial synapse to achieve the computing/memorization in an analog domain. Using the fabricated focal plane array integrated with an artificial neural network, this work performs in‐sensor image identification of finger veins driven by NIR light illumination (≈84 % accuracy). The proposed in‐sensor image computing architecture that broadly covers the NIR spectrum offers widespread application of focal plane array for computer vision, neuromorphic computing, biomedical engineering, etc.

Compact meta-spectral image sensor for mobile applications

Abstract We have demonstrated a compact and efficient metasurface-based spectral imager for use in the near-infrared range. The spectral imager was created by fabricating dielectric multilayer filters directly on top of the CMOS image sensor. The transmission wavelength for each spectral channel was selected by embedding a Si nanopost array of appropriate dimensions within the multilayers on the corresponding pixels, and this greatly simplified the fabrication process by avoiding the variation of the multilayer-film thicknesses. The meta-spectral imager shows high efficiency and excellent spectral resolution up to 2.0 nm in the near-infrared region. Using the spectral imager, we were able to measure the broad spectra of LED emission and obtain hyperspectral images from wavelength-mixed images. This approach provides ease of fabrication, miniaturization, low crosstalk, high spectral resolution, and high transmission. Our findings can potentially be used in integrating a compact spectral imager in smartphones for diverse applications.

Yield prediction and validation of onion (Allium cepa L.) using key variables in narrowband hyperspectral imagery and effective accumulated temperature

In this study, hyperspectral imagery was used to develop models for predicting onion yields in two years (2017 and 2018). First, the bands for the full width half maximum (FWHM) measurement of 5 nm in the canopied areas were merged as FWHM 10, 25, and 50 nm. This was based on a commercialized band-pass filter that considered the development of the compact multispectral image sensors. Then, band rationing was performed to correct the unstable reflectance through incomplete radiating normalization. Stepwise and variable importance in projection in partial least squares (PLS_VIP) approaches were applied to select the optimal FWHM by evaluating both models’ performance and the number of overlapped band ratios. The optimal FWHM measurement was 10 nm, with both high model performances and the highest number of overlapped band ratios. The overlapped ratios of 440/450, and 730/760 in variable importance in projection (VIP) 1 were fixed in the onion-yield prediction model. Conversely, the non-fixed, non-overlapped ratios of 420/430, 490/500, 500/510, 590/600, 620/630, 660/670, 670/680, 710/720, 810/820, and 870/880 were reduced one by one; this was dependent on their removal ranking in descending order of the mean ratio of reduction (MROR) based on the RMSE value, using the leave-one-out method. These combinations in both the fixed and non-fixed band ratios were used to develop prediction models with and without effective accumulated temperature (EAT) values. In all combinations, the models’ performance developed with EAT were increased by preventing slight or sharp decrease in performance, compared to those without EAT. The models, in each year, developed by seven band ratios (420/430, 440/450, 500/510, 590/600, 620/630, 670/680, and 730/760) among the combinations were maintained. The prediction models with EAT were cross-validated (by predicting the 2017 yields using the 2018 model and the 2018 yields using the 2017 model) to evaluate the reproducibility in other years. The reproducibility of the model developed by the seven band ratios was optimal with errors of RMSE = 172 g/m2, RE = 30.3% in 2017 using the 2018 model and RMSE = 215 g/m2 and RE = 33.2% in 2018 using the 2017 model. Consequently, the key band ratios for predicting the onion yields were identified as the seven band ratios with EAT at 420/430, 440/450, 500/510, 590/600, 620/630, 670/680, and 730/760. Ultimately, these findings will provide compact multispectral image sensors specialized in onion-yield prediction that can monitor wide agriculture fields mounted on various platforms.

THREE-DIMENSIONAL PATH PLANNING OF UAVS IMAGING FOR COMPLETE PHOTOGRAMMETRIC RECONSTRUCTION

Abstract. Lightweight unmanned aerial vehicles (UAVs) have been widely used in image acquisition for 3D reconstruction. With the availability of compact and high-end imaging sensors, UAVs can be the platform for precise photogrammetric reconstruction. However, the completeness and precision of complex environment or targets highly rely on the flight planning due to the self-occlusion of structures. Flight paths with back-and-forth pattern and nadir views will result in incompleteness and precision loss of the 3D reconstruction. Therefore, multiple views from different directions are preferred in order to eliminate the occlusion. We propose a 3D path planning method for multirotor UAVs aiming at capturing images for complete and precise photogrammetric 3D reconstructions. This method takes the coarse model from an initial flight as prior knowledge and estimates its completeness and precision. New imaging positions are then planned taking photogrammetric constraints into account. The real-world experiment on a ship lock shows that the proposed method can acquire a more complete result with similar precision compared with an existing 3D planning method.

Hardware-Compliant Compressive Image Sensor Architecture Based on Random Modulations and Permutations for Embedded Inference

This work presents a compact CMOS Image Sensor (CIS) architecture enabling embedded object recognition facilitated by a dedicated end-of-column Compressive Sensing (CS), reducing on-chip memory needs. Our sensing scheme is based on a combination of random modulations and permutations leading to an implementation with very limited hardware impacts. It is designed to meet both theoretical ( i.e. , stable embedding, measurements incoherence) and practical requirements ( i.e. , silicon footprint, power consumption). The only additional hardware compared to a standard CIS architecture using first order incremental Sigma-Delta ( $\Sigma \Delta $ ) Analog to Digital Converter (ADC) are a pseudo-random data mixing circuit, an in- $\Sigma \Delta ~\pm 1$ modulator and a small Digital Signal Processor (DSP). On the algorithmic side, three variants are presented to perform the inference on compressed measurements with a tunable complexity ( i.e. , one-vs.-all SVM, hierarchical SVM and small ANN with 1-D max-pooling). An object recognition accuracy of $\simeq 98.8$ % is reached on the COIL database (COIL, 100 classes) using our dedicated Neural Network classifier. We stress that the signal-independent dimensionality reduction performed by our dedicated CS scheme (1/480 in $480 \times 640$ VGA resolution case) allows to dramatically reduce memory requirements mainly related to the remotely learned coefficients used for the inference stage.

Focus on photonics, spectrometry, and spectroscopy

Focus on photonics, spectrometry, and spectroscopy

Photo-Rheological Fluid-Based Colorimetric Ultraviolet Light Intensity Sensor.

This study presents an introduction to a new type of ultraviolet (UV) light intensity sensor using photo-rheological (PR) fluids whose properties, such as color, can be changed by UV light. When the PR fluids were irradiated by UV light, colorimetric transitions were observed. Effectively, this means that their color changed gradually from yellow to red. The degree of the color change depended on the UV light intensity and was characterized by the hue value of the images acquired with a compact image sensor. We demonstrated that UV light-responsive capabilities can be readily imparted to PR fluids, and that the colorimetric responses to different UV light intensities can be used to measure the UV light intensities.

Complex amplitude reconstruction by iterative amplitude-phase retrieval algorithm with reference

Multi-image iterative phase retrieval methods have been successfully applied in plenty of research fields due to their simple but efficient implementation. However, there is a mismatch between the measurement of the first long imaging distance and the sequential interval. In this paper, an amplitude-phase retrieval algorithm with reference is put forward without additional measurements or priori knowledge. It gets rid of measuring the first imaging distance. With a designed update formula, it significantly raises the convergence speed and the reconstruction fidelity, especially in phase retrieval. Its superiority over the original amplitude-phase retrieval (APR) method is validated by numerical analysis and experiments. Furthermore, it provides a conceptual design of a compact holographic image sensor, which can achieve numerical refocusing easily.

A proposal of low-cost and low-power embedded wireless image sensor node for IoT applications

This paper proposes the design of a low-cost, low-power and small size image acquisition device to be used in Internet of Things (IoT) applications which collects, processes and sends image data through a 2.4GHz Wi-Fi connection. The proposed processing unit is a low-cost high-performance STM32F7 microcontroller which is able to acquire and transfer images at full rate keeping the core idle to do image processing. The device is supported by a 1.3 Megapixel CMOS camera sensor, and a low consumption Wi-Fi module. This setup allows to create a very compact image sensor device with a built-in processing algorithm for a specific functionality in IoT. In this work, the device hardware architecture is described and discussed, and a comparison with alternative hardware solutions are presented.

GaN Metalens for Pixel-Level Full-Color Routing at Visible Light.

Metasurface-based components are known to be one of the promising candidates for developing flat optical systems. However, their low working efficiency highly limits the use of such flat components for feasible applications. Although the introduction of the metallic mirror has been demonstrated to successfully enhance the efficiency, it is still somehow limited for imaging and sensing applications because they are only available for devices operating in a reflection fashion. Here, we demonstrate three individual GaN-based metalenses working in a transmission window with extremely high operation efficiency at visible light (87%, 91.6%, and 50.6% for blue, green, and red light, respectively). For the proof of concept, a multiplex color router with dielectric metalens, which is capable of guiding individual primary colors into different spatial positions, is experimentally verified based on the design of out-of-plane focusing metalens. Our approach with low-cost, semiconductor fabrication compatibility and high working efficiency characteristics offers a way for establishing a complete set of flat optical components for a wide range of applications such as compact imaging sensors, optical spectroscopy, and high-resolution lithography, just named a few.

電界放出陰極アレイを用いた超高感度平面撮像管の開発

A 640×480 pixel field emitter array (FEA) image sensor with a high-gain avalanche rushing amorphous photoconductor (HARP) target was fabricated and tested as a step toward the development of ultrahigh-sensitivity compact image sensors for high-definition TV cameras. Experiments showed that the prototype sensor could obtain clear images with little noise under illumination on a level equivalent to moonlight, with sufficient resolution while consuming far less power than current ultrahigh-sensitivity pickup tubes. Furthermore, various technologies which were suitable for practical image sensor have been developed.

Electrostatic-focusing image sensor with volcano-structured Spindt-type field emitter array

The authors have developed a highly sensitive, compact image sensor comprising a field emitter array (FEA) and a high-gain avalanche rushing amorphous photoconductor (HARP) target with the ultimate aim of developing an ultrahigh sensitivity, compact, high-definition television camera. Double-gated field emitters have the advantage of a compact electron beam focusing system; however, image intensities reproduced by a sensor with the double-gated, Spindt-type field emitter array with the focusing electrode stacked 1.5 μm above the gate electrode were nonuniform owing to low electron beam current. The minimum required electron beam current extracted from the double-gated field emitter array is considered for possible use with the sensor. Furthermore, a suitable field emitter array pitch to balance the electron beam current and the electrostatic-focusing lens strength was simulated. For the sensor's design guidelines, a field emitter array pitch of approximately 3 μm would be reasonable in the case of employing the volcano-structured, double-gated, Spindt-type field emitter array with the focusing electrode placed 0.2 μm below the gate electrode hole. Based on simulation results, an image sensor with the volcano-structured, Spindt-type, 3.1-μm-pitch FEA with the focusing electrode placed 0.2 μm below the gate electrode hole was fabricated. It was confirmed that the minimum electron beam current extracted from the volcano-structured FEA was approximately 2 μA/pix, and the sensor could obtain images with improved image intensity uniformity by utilizing the electrostatic-focusing effect. These results indicate that the volcano-structured, double-gated, Spindt-type FEA is potentially suitable for high-definition television FEA-HARP image sensors.

Double-Gated, Spindt-Type Field Emitter With Improved Electron Beam Extraction

We have developed a highly sensitive, compact image sensor that comprises a field emitter array (FEA) and a high-gain avalanche-rushing amorphous photoconductor target, with the ultimate aim of developing an ultrahighly sensitive, compact, high-definition television camera. Double-gated FEs have an advantage of having a compact electron beam focusing system; however, image intensities reproduced by the sensor were nonuniform due to low electron beam current. Furthermore, the simulated electron beam current disagreed with the measured current. The electron beam current characteristics of two types of double-gated, Spindt-type FEs (both with improved electron beam current extraction) are discussed for possible use within the sensor: convex-structured and volcano-structured. A highly accurate simulation model of the image sensor using a double-gated, Spindt-type FE has been examined; the simulated electron beam currents extracted from the double-gated, Spindt-type FE are in agreement with the measured electron beam currents when the initial electron velocity is assumed, thus suggesting that the simulated anode current–anode voltage characteristic conforms to the measured one. For example, the electron beam currents extracted from the convex-structured and volcano-structured FEs when the focusing electrode is placed 0.2 $\mu \text{m}$ below the gate electrode opening at a focusing electrode voltage of 15 V are, respectively, 1.8 and 1.9 times larger than that extracted from the previously used FEs when the focusing electrode is stacked 1.5 $\mu \text{m}$ above the gate electrode. The results show potential for reducing the degradation of the uniformity of the reproduced image’s intensity, and show that the highly accurate simulation model of the sensor is valid to design the double-gated FEAs for the sensor.

Gamma‐ray tolerance of CdS/CdTe photodiodes for radiation tolerant compact image sensor with field emitter array

AbstractThe effect of gamma irradiation on two types (superstrate‐ and substrate‐type) of CdS/CdTe photodiodes, which we have proposed as potential photoconducting films for compact image sensors with a field emitter array (FEA), was investigated. A 60Co gamma‐ray source was employed, and the total gamma irradiation dose was more than 500 kGy. For the superstrate‐type structure, the irradiation caused the short‐circuit current density (JSC) to decrease, probably due to the resulting decreased transmittance of the glass substrate. On the other hand, little degradation of the I ‐V characteristics or spectral response was observed after the gamma irradiation for the substrate‐type structure. These results suggest that, taking into account the decrease in transmittance of the glass substrate, the CdS/CdTe photodiodes have sufficient tolerance to high gamma‐ray exposure. (© 2016 WILEY‐VCH Verlag GmbH &amp; Co. KGaA, Weinheim)