CMOS Image Research Articles

Global shutter CMOS vision sensors and event cameras for on‐chip dynamic information

AbstractThe on‐chip extraction of dynamic information from a scene can be addressed with either frame‐based CMOS vision, also called smart image sensors, or dynamic vision sensors, also known as event cameras. When implemented with a pinned photodiode (PPD) as a 4‐transistor active pixel sensor (4T‐APS), the former brings about the benefits of low temporal noise and dark current but without high dynamic range (HDR). The latter comes with the benefits of HDR and a fast event detection rate with low power consumption. The drawback is the background activity noise, which leads to additional hardware or algorithms to keep it low. This paperanalyses the mismatch and noise of a global shutter 4T‐APS implementation with local HDR through an overflow capacitor and correlated double sampling (CDS) to provide low noise events through frame differencing. The aim is to narrow the gap with dynamic vision sensors in terms of event rate and dynamic range. We show that our solution would be competitive with event cameras in scenarios with slow moving objects and a relatively wide dynamic range (85 dB).

Ultrahigh-speed imaging for high-impact concrete deformation analysis in pre- and post-cracking stages.

The evaluation of high-speed camera image sequence analysis results in concrete material testing under high-impact loading necessitates the consideration of the effect of the image quality on the measurement accuracy and thus on the potential of the geometric measurements derived from the image sequences. In this contribution, we evaluate the application potential of three ultrahigh-speed cameras with frame rates up to 10Mfps to analyze the deformation of concrete specimens before and after main crack formation in bending and compression tests. Specifically, we evaluate the Kirana 7M and Shimadzu HPV-X2 cameras with ISIS sensor architecture, and the Phantom TMX 7510 camera with BSI CMOS sensor technology. Three-point bending tests and split-Hopkinson pressure bar tests are performed on 160×40×40m m 3 cuboids and on 80mm long, 50mm diameter cylinders. Prior to main crack formation, the displacement vector field represents the specimen deformation, with higher values indicating the position where main cracks will initiate and propagate. Deformations of 80µm in 54µs for a bending test and of 154µm in 36.67µs for a compression test could be measured. The main cracks are then detected using displacement vector field discontinuity analysis techniques, and their evolution is followed to estimate the crack propagation velocity. Average velocities in bending tests between 603 and 854m/s have been determined over a time interval up to 40µs. An investigation of the camera sensor operation of the three optical devices is presented to assess their suitability for deformation analysis. Laboratory tests and real experimental results show that the quality of the propagation vector field, the crack detection, and the crack tip tracking are obviously affected by the image quality, but more significantly by the spatial and temporal resolution due to the small relative step deformations.

Short-wave Infrared Photoluminescence Lifetime Mapping of Rare-Earth Doped Nanoparticles Using All-Optical Streak Imaging.

The short-wave infrared (SWIR) photoluminescence lifetimes of rare-earth doped nanoparticles (RENPs) have found diverse applications in fundamental and applied research. Despite dazzling progress in the novel design and synthesis of RENPs with attractive optical properties, existing optical systems for SWIR photoluminescence lifetime imaging are still considerably restricted by inefficient photon detection, limited imaging speed, and low sensitivity. To overcome these challenges, SWIR photoluminescence lifetime imaging microscopy using an all-optical streak camera (PLIMASC) is developed. Synergizing scanning optics and a high-sensitivity InGaAs CMOS camera, SWIR-PLIMASC has a 1D imaging speed of up to 138.9kHz in the spectral range of 900-1700nm, which quantifies the photoluminescence lifetime of RENPs in a single shot. A 2D photoluminescence lifetime map can be acquired by 1D scanning of the sample. To showcase the power of SWIR-PLIMASC, a series of core-shell RENPs with distinct SWIR photoluminescence lifetimes is synthesized. In particular, using Er3+ -doped RENPs, SWIR-PLIMASC enables multiplexed anti-counterfeiting. Leveraging Ho3+ -doped RENPs as temperature indicators, this system is applied to SWIR photoluminescence lifetime-based thermometry. Opening up a new avenue for efficient SWIR photoluminescence lifetime mapping, this work is envisaged to contribute to advanced materials characterization, information science, and biomedicine.

Influence of the vapor shield potential control on the post-arc sheath development in vacuum interrupters

In tank-type multi-break vacuum circuit breakers (VCBs) and gas-insulated switchgear (GIS), the non-uniform distribution of vapor shield potential in the vacuum interrupters impacts the post-arc process significantly. To explore the effect of the vapor shield potential on the development of the post-arc sheath, a particle-in-cell model with shield potential control is proposed to obtain two-dimensional distributions of post-arc particles. Based on the synthetic test circuit and probe array, the test platform of the post-arc characteristic diagnostic is set up to verify the influence of vapor shield potential distribution on the post-arc sheath. The extinguishing process of the vacuum arc is also observed by the high-speed CMOS camera. Simulation and experimental results show that at shield potential of 0.5 transient recovery voltage, the post-arc sheath develops most rapidly and the distribution of cathode spots is the most uniform. These findings can provide valuable insights for the design of vacuum interrupters and the control of shield potential in tank-type GIS.

Dynamic light scattering and laser speckle contrast imaging of the brain: theory of the spatial and temporal statistics of speckle pattern evolution.

Dynamic light scattering (DLS) and laser speckle contrast imaging (LSCI) are closely related techniques that exploit the statistics of speckle patterns, which can be utilized to measure cerebral blood flow (CBF). Conventionally, the temporal speckle intensity auto-correlation function is calculated in DLS, while the spatial speckle contrast Ks is calculated in LSCI measurements. Due to the rapid development of CMOS detection technology with increased camera frame rates while still maintaining a large number of pixels, the ensemble or spatial average of as well as the temporal contrast Kt can be easily calculated and utilized to quantify CBF. Although many models have been established, a proper summary is still lacking to fully characterize DLS and LSCI measurements for spatial and temporal statistics, laser coherence properties, various motion types, etc. As a result, there are many instances where theoretical models are misused. For instance, mathematical formulas derived in the diffusive regime or for ergodic systems are sometimes applied to small animal brain measurements, e.g., mice brains, where the assumptions are not valid. Therefore, we aim to provide a review of the speckle theory for both DLS and LSCI measurements with detailed derivations from first principles, taking into account non-ergodicity, spatial and temporal statistics of speckles, scatterer motion types, and laser coherence properties. From these calculations, we elaborate on the differences between spatial and temporal averaging for DLS and LSCI measurements that are typically ignored but can result in inaccurate measurements of blood flow, particularly the spatially varying nature of the static component in and Kt. We also obtained maps in in vivo mouse brain measurements using high frame rate CMOS cameras which have not been demonstrated before, and compared with and Ks,t. This work provides a useful guide for choosing the correct model to analyze spatial and temporal speckle statistics in in-vivo DLS and LSCI measurements.

Discussion of Mechanical Shock Test Stress for Ultra-large-scale CMOS Image Sensors

The ultra-large-scale CMOS image sensors are significantly different from the traditional CMOS image sensors in terms of pixel size, chip size and structure. CMOS image sensors generally come with an optical window structure that is sealed to the ceramic housing by means of adhesive. The optical window material is generally sapphire, and the larger the image element size, the larger the required glass optical window area. Ultra-large size CMOS image sensors in the package before the general optical window thickness, parallelism, light window average transmittance assessment. The object of mechanical impact test assessment is the CMOS image sensor optical window structure and capping process. The paper mainly discusses the typical failure cases of ultra-large-scale CMOS image sensors on the basis of the applicability of mechanical shock test standards, and investigates how to define the mechanical shock test stress from simulation, limit test and the requirements by users. This paper provides the qualification assessment basis of mechanical shock for newly developed products.

High-speed readout system of X-ray CMOS image sensor for time domain astronomy

We developed an FPGA-based high-speed readout system for a complementary metal-oxide-semiconductor (CMOS) image sensor to observe soft X-ray transients in future satellite missions, such as HiZ-GUNDAM. Our previous research revealed that the CMOS image sensor has low-energy X-ray detection capability (0.4–4 keV) and strong radiation tolerance, which satisfies the requirements of the HiZ-GUNDAM mission. However, CMOS sensors typically have small pixel sizes (e.g., ∼10 µm), resulting in large volumes of image data. GSENSE400BSI has 2048×2048 pixels, producing 6 Mbyte per frame. These large volumes of observed raw image data cannot be stored in a satellite bus system with a limited storage size. Therefore, only X-ray photon events must be extracted from the raw image data. Furthermore, the readout time of CMOS image sensors is approximately ten times faster than that of typical X-ray CCDs, requiring faster event extraction on a timescale of ∼0.1 s. To address these issues, we have developed an FPGA-based image signal processing system capable of high-speed X-ray event extraction onboard without storing raw image data. The developed compact system enabled mounting on a CubeSat mission, facilitating early in-orbit operation demonstration. Here, we present the design and results of the performance evaluation tests of the proposed FPGA-based readout system. Utilizing X-ray irradiation experiments, the results of the X-ray event extraction with the onboard and offline processing methods were consistent, validating the functionality of the proposed system.

Investigation of proton single-event transient in CMOS image sensor

With the increasingly widespread application of CMOS image sensors (CIS) in radiation environments, such as aerospace, their radiation effects have gained attention. This paper investigates single-event transient (SET) caused by the proton direct ionization on CIS, combining both experimental and simulation methods. The proton beam energy used in the experiment is 12 MeV, with a flux up to 3.5 × 108 p/(cm2 s). Due to the periodicity of the proton beam, the CIS output displays a phenomenon of alternating brightness and darkness. When the proton beam flux is low, numerous SET bright spots with different outputs are observed. To comprehensively analyze these experimental phenomena, a typical three-dimensional 4T pinned photodiode model is constructed in TCAD, and relevant SET simulation is carried out. The results indicate that incident position, incident time, and the number of incident protons significantly affect the output of SET-generated bright spots, which are key factors contributing to the different bright spots observed in the experiment.

Hydrogen Termination Effect on SiO2/Si Interface State Density in CH3O-Molecular-Ion-Implanted Silicon Epitaxial Wafer for CMOS Image Sensors

The reduction in SiO2/Si interface state density (Dit) at the SiO2/Si interface region is important to improve the performance of complementary metal-oxide semiconductor (CMOS) image sensors. The CH3O-ion-implanted region stores hydrogen and releases the stored hydrogen during the subsequent heat treatment. This study demonstrates that a CH3O-ion-implanted epitaxial silicon wafer can reduce the Dit and Pb0 center density in SiO2/Si interface regions, as analyzed by quasi-static capacitance–voltage and electron spin resonance measurements, respectively. Both Dit and Pb0 center density in the CH3O-implanted wafer decreased with increasing heat treatment temperature. Moreover, the activation energy is estimated to be 1.57 eV for the hydrogen termination reactions induced by the CH3O-ion-implanted wafer. The activation energy is close to those of hydrogen molecules and Si dangling bonds at the SiO2/Si interface. This result means that Dit can be reduced by hydrogen from inside the silicon wafer, regardless of the heat treatment atmosphere. It has unique characteristics not found in conventional silicon wafers. The termination effect of the CH3O-molecular-ion-implanted epitaxial silicon wafers can contribute to the high electrical performance of CMOS image sensors.

Explosive Growth of Image Sensors in Smart Government Technology and Economic Scale

CMOS image sensor is a rapidly developing image sensor product in the past decade. Relying on the advantages of process compatibility with standard CMOS process, CMOS image sensor integrates analog photoelectric photosensitive circuit and digital signal processing circuit to become a small and powerful system on chip. This article mainly studies the explosive growth of image sensors in smart government technology and economic scale. The main purpose of this article is to study the technical application of image sensors, reduce the administrative cost of social governance, improve the level of government governance, benefit the public, and realize the requirements of smart government affairs and rapid economic development of the city. This study tested the static performance of the analog-to-digital conversion circuit by inputting a limited number of sampling points, and its integral nonlinearity and differential nonlinearity were both less than 1LSB. The bandgap reference circuit provides a reference level for the entire system. It uses two PNP-type bipolar transistors parasitic in the CMOS process to achieve a temperature and voltage-independent bandgap reference voltage through the feedback of the operational amplifier. It is ideal for use In the case of resistance simulation, its accuracy reaches 16.6ppm. The scanning method of the pixel array uses multi-sampling technology to increase the dynamic range of the sensor by 2N times. The correlated sub-sampling technology eliminates the fixed pattern noise between pixel units. The use of pipelined analog-to-digital converters can achieve good resolution, speed and area compromise.Through the realization and simulation of transistor-level circuits, the performance of each part of the system meets the requirements of design and application.

A Low-Depth-Noise Indirect Time-of-Flight CMOS Image Sensor With Tap-Rotating Technique for Extended Range and Enhanced Imaging Quality

A Low-Depth-Noise Indirect Time-of-Flight CMOS Image Sensor With Tap-Rotating Technique for Extended Range and Enhanced Imaging Quality

First-principles study of effect of impurity compensation on optical properties of Si

Presently, impurity-compensated silicon (Si) has no clear potential applications due to high resistance and few carriers. Thus, it has received little attention from researchers. In this study, we find that impurity compensation can make localized state energy levels form in Si bandgap, which can improve the light absorption of Si in the near infrared region. In this work, in order to comprehensively and deeply understand the photoelectric properties of impurity-compensated Si, the localized state energy levels composed of P<sup>+</sup>/B<sup>–</sup> ions are constructed in Si bandgap through the co-doping of phosphorus (P) and boron (B), thereby forming impurity-compensated Si. The first-principles based on a density functional theory framework is used to study the photoelectric properties of the impurity-compensated Si (n/p-Sic) such as the density of states (DOS), dielectric function and refractive index. The DOS study reveals the following results: after the n- and p-Si with the same concentration of P and B (12.5%) are fully compensated for by impurities, the Fermi energy levels of their compensated counterparts are at the valley bottom formed by the two adjacent DOS peaks, and the DOS is not zero at the valley bottom. In the study of dielectric function and refractive index, it is found that when the doping ratio is <i>C</i><sub>B</sub>/<i>C</i><sub>P0</sub> = 0.25, n-Sic has the largest dielectric function and refractive index in the low energy region. In addition, comparing intrinsic Si with its doped counterparts in the real part (Re) of their dielectric constant, the following regularity is found: in the high energy region of <i>E</i> > 4 eV, the Re values of the intrinsic Si, n/p-Si and p-Sic are negative. In the low energy region of 0.64 eV< <i>E</i> < 1.50 eV, the Re value of n-Sic is negative for the doping ratio of <i>C</i><sub>B</sub>/<i>C</i><sub>P0</sub> = 0.25. The above comparison indicates that the n-Sic with <i>C</i><sub>B</sub>/<i>C</i><sub>P0</sub> = 0.25 can achieve good metallicity in the low energy region, indicating that the electrons in valence band are easily excited by low-energy long-wavelength light. Theoretical studies show that the good photoelectric properties of n-Sic with <i>C</i><sub>B</sub>/<i>C</i><sub>P0</sub> = 0.25 may be related to Si dangling bonds and localized state energy levels in Si bandgap. The Si dangling bonds are caused by the impurity compensation of B dopant for n-Si, leading part of Si-Si bonds to change into Si-B bonds. This study provides theoretical guidance for the application of impurity-compensated Si in the field of photodetectors such as CMOS image sensors and infrared photodetectors.

Experimental study on back-illuminated CMOS image sensor irradiated by CW laser

Experimental study on back-illuminated CMOS image sensor irradiated by CW laser

Event-Enhanced Snapshot Compressive Videography at 10K FPS.

Video snapshot compressive imaging (SCI) encodes the target dynamic scene compactly into a snapshot and reconstructs its high-speed frame sequence afterward, greatly reducing the required data footprint and transmission bandwidth as well as enabling high-speed imaging with a low frame rate intensity camera. In implementation, high-speed dynamics are encoded via temporally varying patterns, and only frames at corresponding temporal intervals can be reconstructed, while the dynamics occurring between consecutive frames are lost. To unlock the potential of conventional snapshot compressive videography, we propose a novel hybrid "intensity + event" imaging scheme by incorporating an event camera into a video SCI setup. Our proposed system consists of a dual-path optical setup to record the coded intensity measurement and intermediate event signals simultaneously, which is compact and photon-efficient by collecting the half photons discarded in conventional video SCI. Correspondingly, we developed a dual-branch Transformer utilizing the reciprocal relationship between two data modes to decode dense video frames. Extensive experiments on both simulated and real-captured data demonstrate our superiority to state-of-the-art video SCI and video frame interpolation (VFI) methods. Benefiting from the new hybrid design leveraging both intrinsic redundancy in videos and the unique feature of event cameras, we achieve high-quality videography at 0.1ms time intervals with a low-cost CMOS image sensor working at 24 FPS.

High Total Ionizing Dose Effects on Backside-Illuminated CMOS Image Sensors

High Total Ionizing Dose Effects on Backside-Illuminated CMOS Image Sensors

Analysis for the Influence of Pixel Structure in Backside Illuminated CMOS Image Sensor

Analysis for the Influence of Pixel Structure in Backside Illuminated CMOS Image Sensor

Toward a Wireless Image Sensor for Real-Time Fluorescence Microscopy in Cancer Therapy.

We present a mm-sized, ultrasonically powered lensless CMOS image sensor as a progress towards wireless fluorescence microscopy. Access to biological information within the tissue has the potential to provide insights guiding diagnosis and treatment across numerous medical conditions including cancer therapy. This information, in conjunction with current clinical imaging techniques that have limitations in obtaining images continuously and lack wireless compatibility, can improve continual detection of multicell clusters deep within tissue. The proposed platform incorporates a 2.4 × 4.7 mm2 integrated circuit (IC) fabricated in TSMC 0.18 µm, a micro laser diode (µLD), a single piezoceramic and off-chip storage capacitors. The IC consists of a 36 × 40 array of capacitive trans-impedance amplifier-based pixels, wireless power management and communication via ultrasound and a laser driver all controlled by a Finite State Machine. The piezoceramic harvests energy from the acoustic waves at a depth of 2 cm to power up the IC and transfer 11.5 kbits/frame via backscattering. During Charge-Up, the off-chip capacitor stores charge to later supply a high-power 78 mW µLD during Imaging. Proof of concept of the imaging front end is shown by imaging distributions of CD8 T-cells, an indicator of the immune response to cancer, ex vivo, in the lymph nodes of a functional immune system (BL6 mice) against colorectal cancer consistent with the results of a fluorescence microscope. The overall system performance is verified by detecting 140 µm features on a USAF resolution target with 32 ms exposure time and 389 ms ultrasound backscattering.

Invited Paper] A High SNR Global Shutter CMOS Image Sensor Technology for High Precision Absorption Imaging Applications

Invited Paper] A High SNR Global Shutter CMOS Image Sensor Technology for High Precision Absorption Imaging Applications

Experiment and Analysis of Damage Effect of Proton Irradiation on Transmission Transistor of CMOS Image Sensor

Experiment and Analysis of Damage Effect of Proton Irradiation on Transmission Transistor of CMOS Image Sensor

A Back-Illuminated SPAD Fabricated With 40 nm CMOS Image Sensor Technology Achieving Near 40% PDP at 940 nm

A Back-Illuminated SPAD Fabricated With 40 nm CMOS Image Sensor Technology Achieving Near 40% PDP at 940 nm