Conventional Image Sensors Research Articles

EventSegNet: Direct Sparse Semantic Segmentation from Event Data

Semantic segmentation tasks encompass various applications, such as autonomous driving, medical imaging, and robotics. Achieving accurate semantic information retrieval under conditions of high dynamic range and rapid scene changes remains a significant challenge for image-based algorithms. This challenge is primarily attributable to the limitations of conventional image sensors, which can experience motion blur or exposure artifacts. In contrast, event-based vision sensors, which asynchronously report changes in pixel intensity, offer a compelling solution by acquiring visual information at the same rate as the scene dynamics, thereby mitigating these limitations. However, we encounter a significant challenge in event-based semantic segmentation tasks: the need to expend time on converting event data into frame images to align with existing image-based semantic segmentation techniques. This approach squanders the inherently high temporal resolution of event data, compromising the accuracy and real-time performance of semantic segmentation tasks. To address these issues, this work explores a sparse semantic segmentation approach that directly addresses event data. We propose a network named EventSegNet that improves the ability to extract geometric features from event data by combining geometric feature enhancement operations and attention mechanisms. Based on this, we propose a large-scale event-based semantic segmentation dataset that provides labels for each event. Our approach achieved a new F1 score of 84.2% on the dataset. In addition, a lightweight and edge-oriented AI inference deployment technique was implemented for the network model. Compared to the baseline model, the optimized network model reduces the F1 score by 1.1% but is more than twice as fast computationally, enabling real-time inference on the NVIDIA AGX Xavier.

(Invited) Bioinspired in-Sensor Computing for Artificial Vision

The visual scene in the physical world integrates multidimensional information (spatial, temporal, polarization, spectrum, etc) and typically displays unstructured characteristics. Conventional image sensors cannot process this multidimensional vision data, creating a need for vision sensors that can efficiently extract features from substantial multidimensional vision data. Vision sensors are able to transform the unstructured visual scene into featured information without relying on sophisticated algorithms and complex hardware. In this talk, I will describe our team’s efforts towards bioinspired in-sensor computing for artificial vision. I will talk about the framework of the in-sensor computing and demonstrate a few vision sensors for different scenarios, including visual adaptation, motion perception, as well as event-driven vision sensors for spiking neural network.

Flexible Soft X-Ray Image Sensors based on Metal Halide Perovskites With High Quantum Efficiency.

Soft X-ray imaging is a powerful tool to explore the structure of cells, probe material with nanometer resolution, and investigate the energetic phenomena in the universe. Conventional soft X-ray image sensors are by and large Si-based charge coupled devices that suffer from low frame rates, complex fabrication processes, mechanical inflexibility, and required cooling below -60°C. Here, a soft X-ray photodiode is reported based on low-cost metal halide perovskite with comparable performance to commercial Si-based device. Nanothrough network electrodeminimized the optical loss due to the shadowing of insensitive layers, while a multidimensional perovskite heterojunctionis generated to reduce the photo-generated carrier loss. This strategy promoted a record quantum efficiency of 8×103% without cooling, several orders of magnitude greater than the previously achieved. Flexible and curved soft X-ray imaging arrays are fabricated based on this high-performance device structure, demonstrating stable soft X-ray response and sharp imaging capabilities. This work highlights the low-cost and efficient perovskite photodiode as a strong candidate for the next-generation soft X-ray image sensors.

Millisecond-scale behaviours of plankton quantified invitro and insitu using the Event-based Vision Sensor.

The Event-based Vision Sensor (EVS) is a bio-inspired sensor that captures detailed motions of objects, aiming to become the 'eyes' of machines like self-driving cars. Compared to conventional frame-based image sensors, the EVS has an extremely fast motion capture equivalent to 10,000-fps even with standard optical settings, plus high dynamic ranges for brightness and also lower consumption of memory and energy. Here, we developed 22 characteristic features for analysing the motions of aquatic particles from the EVS raw data and tested the applicability of the EVS in analysing plankton behaviour. Laboratory cultures of six species of zooplankton and phytoplankton were observed, confirming species-specific motion periodicities up to 41 Hz. We applied machine learning to automatically classify particles into four categories of zooplankton and passive particles, achieving an accuracy up to 86%. At the insitu deployment of the EVS at the bottom of Lake Biwa, several particles exhibiting distinct cumulative trajectory with periodicities in their motion (up to 16 Hz) were identified, suggesting that they were living organisms with rhythmic behaviour. We also used the EVS in the deep sea, observing particles with active motion and periodicities over 40 Hz. Our application of the EVS, especially focusing on its millisecond-scale temporal resolution and wide dynamic range, provides a new avenue to investigate organismal behaviour characterised by rapid and periodical motions. The EVS will likely be applicable in the near future for the automated monitoring of plankton behaviour by edge computing on autonomous floats, as well as quantifying rapid cellular-level activities under microscopy.

Multidimensional vision sensors for information processing.

The visual scene in the physical world integrates multidimensional information (spatial, temporal, polarization, spectrum and so on) and typically shows unstructured characteristics. Conventional image sensors cannot process this multidimensional vision data, creating a need for vision sensors that can efficiently extract features from substantial multidimensional vision data. Vision sensors are able to transform the unstructured visual scene into featured information without relying on sophisticated algorithms and complex hardware. The response characteristics of sensors can be abstracted into operators with specific functionalities, allowing for the efficient processing of perceptual information. In this Review, we delve into the hardware implementation of multidimensional vision sensors, exploring their working mechanisms and design principles. We exemplify multidimensional vision sensors built on emerging devices and silicon-based system integration. We further provide benchmarking metrics for multidimensional vision sensors and conclude with the principle of device-system co-design and co-optimization.

All-optical complex field imaging using diffractive processors.

Complex field imaging, which captures both the amplitude and phase information of input optical fields or objects, can offer rich structural insights into samples, such as their absorption and refractive index distributions. However, conventional image sensors are intensity-based and inherently lack the capability to directly measure the phase distribution of a field. This limitation can be overcome using interferometric or holographic methods, often supplemented by iterative phase retrieval algorithms, leading to a considerable increase in hardware complexity and computational demand. Here, we present a complex field imager design that enables snapshot imaging of both the amplitude and quantitative phase information of input fields using an intensity-based sensor array without any digital processing. Our design utilizes successive deep learning-optimized diffractive surfaces that are structured to collectively modulate the input complex field, forming two independent imaging channels that perform amplitude-to-amplitude and phase-to-intensity transformations between the input and output planes within a compact optical design, axially spanning ~100 wavelengths. The intensity distributions of the output fields at these two channels on the sensor plane directly correspond to the amplitude and quantitative phase profiles of the input complex field, eliminating the need for any digital image reconstruction algorithms. We experimentally validated the efficacy of our complex field diffractive imager designs through 3D-printed prototypes operating at the terahertz spectrum, with the output amplitude and phase channel images closely aligning with our numerical simulations. We envision that this complex field imager will have various applications in security, biomedical imaging, sensing and material science, among others.

An Asynchronous Linear Filter Architecture for Hybrid Event-Frame Cameras.

Event cameras are ideally suited to capture High Dynamic Range (HDR) visual information without blur but provide poor imaging capability for static or slowly varying scenes. Conversely, conventional image sensors measure absolute intensity of slowly changing scenes effectively but do poorly on HDR or quickly changing scenes. In this paper, we present an asynchronous linear filter architecture, fusing event and frame camera data, for HDR video reconstruction and spatial convolution that exploits the advantages of both sensor modalities. The key idea is the introduction of a state that directly encodes the integrated or convolved image information and that is updated asynchronously as each event or each frame arrives from the camera. The state can be read-off as-often-as and whenever required to feed into subsequent vision modules for real-time robotic systems. Our experimental results are evaluated on both publicly available datasets with challenging lighting conditions and fast motions, along with a new dataset with HDR reference that we provide. The proposed AKF pipeline outperforms other state-of-the-art methods in both absolute intensity error (69.4% reduction) and image similarity indexes (average 35.5% improvement). We also demonstrate the integration of image convolution with linear spatial kernels Gaussian, Sobel, and Laplacian as an application of our architecture.

NeuroZoom: Denoising and Super Resolving Neuromorphic Events and Spikes.

Neuromorphic cameras are emerging imaging technology that has advantages over conventional imaging sensors in several aspects including dynamic range, sensing latency, and power consumption. However, the signal-to-noise level and the spatial resolution still fall behind the state of conventional imaging sensors. In this paper, we address the denoising and super-resolution problem for modern neuromorphic cameras. We employ 3D U-Net as the backbone neural architecture for such a task. The networks are trained and tested on two types of neuromorphic cameras: a dynamic vision sensor and a spike camera. Their pixels generate signals asynchronously, the former is based on perceived light changes and the latter is based on accumulated light intensity. To collect the datasets for training such networks, we design a display-camera system to record high frame-rate videos at multiple resolutions, providing supervision for denoising and super-resolution. The networks are trained in a noise-to-noise fashion, where the two ends of the network are unfiltered noisy data. The output of the networks has been tested for downstream applications including event-based visual object tracking and image reconstruction. Experimental results demonstrate the effectiveness of improving the quality of neuromorphic events and spikes, and the corresponding improvement to downstream applications with state-of-the-art performance.

Chip-integrated metasurface full-Stokes polarimetric imaging sensor

Polarimetric imaging has a wide range of applications for uncovering features invisible to human eyes and conventional imaging sensors. Chip-integrated, fast, cost-effective, and accurate full-Stokes polarimetric imaging sensors are highly desirable in many applications, which, however, remain elusive due to fundamental material limitations. Here we present a chip-integrated Metasurface-based Full-Stokes Polarimetric Imaging sensor (MetaPolarIm) realized by integrating an ultrathin (~600 nm) metasurface polarization filter array (MPFA) onto a visible imaging sensor with CMOS compatible fabrication processes. The MPFA is featured with broadband dielectric-metal hybrid chiral metasurfaces and double-layer nanograting polarizers. This chip-integrated polarimetric imaging sensor enables single-shot full-Stokes imaging (speed limited by the CMOS imager) with the most compact form factor, records high measurement accuracy, dual-color operation (green and red) and a field of view up to 40 degrees. MetaPolarIm holds great promise to enable transformative applications in autonomous vision, industry inspection, space exploration, medical imaging and diagnosis.

Multi-depth hologram generation from two-dimensional images by deep learning

Multi-depth hologram generation from two-dimensional images by deep learning

TDI-like multi-slit hyperspectral imaging for enhanced throughput via the Kalman filter.

The time-delay integration (TDI) technique is increasingly used to improve the signal-to-noise ratio (SNR) of remote sensing and imaging by exposing the scene multiple times. Inspired by the principle of TDI, we propose a TDI-like pushbroom multi-slit hyperspectral imaging (MSHSI) approach. In our system, multiple slits are used to significantly improve the throughput of the system, thereby enhancing the sensitivity and SNR through multiple exposures of the same scene during pushbroom scan. Meanwhile, a linear dynamic model for the pushbroom MSHSI is established, where the Kalman filter (KF) is employed to reconstruct the time-varying overlapped spectral images on a single conventional image sensor. Further, we designed and fabricated a customized optical system that can operate in both multi-slit and single slit modes to experimentally verify the feasibility of the proposed method. Experimental results indicate that the developed system improved SNR by a factor of about 7 compared to that of the single slit mode, while demonstrating excellent resolution in both spatial and spectral dimensions.

Pixelated NIR–VIS Spectral Routers Based on 2D Mie‐Type Metagratings

AbstractThe out‐of‐band energy loss caused by in‐built color filters significantly degrades the signal‐to‐noise ratio and the dynamic range of conventional image sensors, which has restricted the attempt to develop ultrahigh‐density imaging devices by merely shrinking the pixel size. This issue will be more serious for security cameras, which need to collect both visible (VIS) and near‐infrared (NIR) light. The existing solutions mostly explore complex photonic nanostructures, which are often too complicated for production. In this work, a pixelated spectral router is demonstrated utilizing a 2D Si3N4 Mie‐type metagrating that can spatially divide NIR (850 nm) and VIS (400–700 nm) light to different pixels at high efficiencies. It has a minimum feature size larger than 360 nm, highly promising for massive production. Compared with the traditional filter design, this router can gain ≈42% and ≈30% signal enhancement for NIR and VIS bands, respectively. It is shown that it also has good polarization insensitivity and incident angle tolerance. The NIR–VIS simultaneous imaging is inspected without any complex reconstruction algorithm. Mode analysis indicates that the periodic interference of multipolar scattered fields caused by this metagrating provides the necessary degrees of freedom to spatially optimize the routing functions for broadband photons.

On the Role of Thermal Imaging in Automotive Applications: A Critical Review

For decades, the number of automobiles in urban areas around the world has been increasing. It causes serious challenges such as traffic congestion, accidents, and pollution, which have a social, economic, and environmental impact on widespread urban cities. To overcome these challenges, we need to explore smart AI-based perception systems for vehicular applications. Such types of systems can provide improved situational awareness to the driver and generate early alarm about upcoming obstacles and road incidents. In this study, we have presented the effective use of uncooled thermal IR sensors for designing smart thermal perception systems as an alternative to CMOS visible imaging by presenting state-of-the-art studies for in-cabin and out-cabin vehicular applications with potential long-term benefits for the automotive industry. The key rationale for selecting thermal IR sensors over conventional image sensors is that visible cameras are highly dependent on lighting conditions and performance is degraded significantly in low-lighting scenarios and harsh weather conditions. Contrary to this, thermal sensors remain largely unaffected by external lighting conditions or most environmental conditions, making it a perfect optical sensor choice for all-weather and harsh environmental conditions. This study presents a review of the current state of the art for automotive thermal imaging with a focus on the contributions and advances achieved by the EU-funded project ‘HELIAUS’ in the domain of AI-based thermal imaging pipelines for safer and reliable road journeys.

Field-programmable encoding for address-event representation.

In conventional frame-based image sensors, every pixel records brightness information and sends this information to a receiver serially in a scanning fashion. This full-frame readout approach suffers from high bandwidth requirements and increased power consumption with the increasing size of the pixel array. Event-based image sensors are gaining popularity for reducing the bandwidth and power requirements by sending only meaningful data in an event-driven approach with the help of address-event representation (AER) communication protocol. However, the event-based readout suffers from increased latency and timing error when the number of pixels with an event increase. In this paper, we introduce a new field-programmable AER (FP-AER) encoding scheme which offers benefits of both frame-based and event-based approaches. The readout design can be configured "in the field" using configuration bits. We also compare the performance of the proposed design against existing AER-based approaches for imaging applications and show that FP-AER performs best in both scanning and event-based readout.

Optical transparency and label-free vessel imaging of zebrafish larvae in shortwave infrared range as a tool for prolonged studying of cardiovascular system development

Optical techniques are utilized for the non-invasive analysis of the zebrafish cardiovascular system at early developmental stages. Being based mainly on conventional optical microscopy components and image sensors, the wavelength range of the collected and analyzed light is not out of the scope of 400–900 nm. In this paper, we compared the non-invasive optical approaches utilizing visible and near infrared range (VISNIR) 400–1000 and the shortwave infrared range (SWIR) 900–1700 nm. The transmittance spectra of zebrafish tissues were measured in these wavelength ranges, then vessel maps, heart rates, and blood flow velocities were calculated from data in VISNIR and SWIR. An increased pigment pattern transparency was registered in SWIR, while the heart and vessel detection quality in this range is not inferior to VISNIR. Obtained results indicate an increased efficiency of SWIR imaging for monitoring heart function and hemodynamic analysis of zebrafish embryos and larvae and suggest a prolonged registration period in this range compared to other optical techniques that are limited by pigment pattern development.

Motion-Based Object Location on a Smart Image Sensor Using On-Pixel Memory.

Object location is a crucial computer vision method often used as a previous stage to object classification. Object-location algorithms require high computational and memory resources, which poses a difficult challenge for portable and low-power devices, even when the algorithm is implemented using dedicated digital hardware. Moving part of the computation to the imager may reduce the memory requirements of the digital post-processor and exploit the parallelism available in the algorithm. This paper presents the architecture of a Smart Imaging Sensor (SIS) that performs object location using pixel-level parallelism. The SIS is based on a custom smart pixel, capable of computing frame differences in the analog domain, and a digital coprocessor that performs morphological operations and connected components to determine the bounding boxes of the detected objects. The smart-pixel array implements on-pixel temporal difference computation using analog memories to detect motion between consecutive frames. Our SIS can operate in two modes: (1) as a conventional image sensor and (2) as a smart sensor which delivers a binary image that highlights the pixels in which movement is detected between consecutive frames and the object bounding boxes. In this paper, we present the design of the smart pixel and evaluate its performance using post-parasitic extraction on a 0.35 µm mixed-signal CMOS process. With a pixel-pitch of 32 µm × 32 µm, we achieved a fill factor of 28%. To evaluate the scalability of the design, we ported the layout to a 0.18 µm process, achieving a fill factor of 74%. On an array of smart pixels, the circuit operates at a maximum frame rate of 3846 frames per second. The digital coprocessor was implemented and validated on a Xilinx Artix-7 XC7A35T field-programmable gate array that runs at 125 MHz, locates objects in a video frame in 0.614 µs, and has a power consumption of 58 mW.

Photodetectors Based on Lead Sulfide Quantum Dot and Organic Absorbers for Multispectral Sensing in the Visible to Short‐Wave Infrared Range

AbstractMultispectral imaging in short‐wave infrared (SWIR) is a powerful analytical technique because of the distinctive spectral properties of many materials in this range. However, conventional SWIR image sensors are beyond the reach of many applications due to their high price. Image sensors based on colloidal quantum dots (CQDs) are expected to deliver affordable infrared image sensors to wider application scope. So far, the demonstrated CQD image sensors do not have a multispectral capability. Here, a dual‐band photodetector based on PbS CQDs is presented. By engineering the surface of CQDs, two oppositely facing pn junctions are fabricated in series, which enable sensing in two spectral channels. Furthermore, an optical cavity is designed that reduces the spectral crosstalk between the two channels and simultaneously enables wavelength‐tunability in one channel. Finally, an organic photodiode (OPD) is integrated with a PbS CQD photodiode in a single device, leveraging a high sensitivity in visible and near‐infrared (NIR) characteristics for OPDs. The presented photodetectors exhibit low dark current below 500 nA cm−2 at 1 V bias, a fast response measured in microseconds, as well as high external quantum efficiency, reaching 70% in NIR and 30% in SWIR.

What Does a One-Bit Quanta Image Sensor Offer?

The one-bit quanta image sensor (QIS) is a photon-counting device that captures image intensities using binary bits. Assuming that the analog voltage generated at the floating diffusion of the photodiode follows a Poisson-Gaussian distribution, the sensor produces either a “1” if the voltage is above a certain threshold or “0” if it is below the threshold. The concept of this binary sensor has been proposed for more than a decade and physical devices have been built to realize the concept. However, what benefits does a one-bit QIS offer compared to a conventional multi-bit CMOS image sensor? Besides the known empirical results, are there theoretical proofs to support these findings? The goal of this paper is to provide new theoretical support from a signal processing perspective. In particular, it is theoretically found that the sensor can offer three benefits: (1) Low-light: One-bit QIS performs better at low-light because it has a low read noise and its one-bit quantization can produce an error-free measurement. However, this requires the exposure time to be appropriately configured. (2) Frame rate: One-bit sensors can operate at a much higher speed because a response is generated as soon as a photon is detected. However, in the presence of read noise, there exists an optimal frame rate beyond which the performance will degrade. A Closed-form expression of the optimal frame rate is derived. (3) Dynamic range: One-bit QIS offers a higher dynamic range. The benefit is brought by two complementary characteristics of the sensor: nonlinearity and exposure bracketing. The decoupling of the two factors is theoretically proved, and closed-form expressions are derived.

Photoelectric Characterization of Intended Source UTBB 1 T FDSOI Based Image Sensor

This paper presents the photoelectrical performance evaluation of proposed intended sourceUltra Thin Body Bias (UTBB) 1 T-FDSOI based image sensor and also a comparative analysis has done with conventional 1 T-UTBB based image sensor [1]. The photoelectrical conversion in proposed image sensor is achieved by utilizing back biasing in intended source UTBBFDSOI based MOSFET. Inthe proposedintended source Ultra Thin Body Bias (UTBB) 1 T-FDSOI based image sensor,the photoelectrical conversion is done within the substrate region (well region) and that signal is read out by MOSFET placed on SOI substrate. The comparative analysis between proposed image sensor and conventional 1 T-UTBB based image sensor has performed on the basis of various parameters such as back-bias voltage, thickness of buried oxide (BOX) layer and doping concentration. In this investigation the proposed image sensor has found better light sensitivity in contrast with conventional image sensor. The performance evaluation of proposed image sensor has done by TCAD simulations.

Event-Based Pedestrian Detection Using Dynamic Vision Sensors

Pedestrian detection has attracted great research attention in video surveillance, traffic statistics, and especially in autonomous driving. To date, almost all pedestrian detection solutions are derived from conventional framed-based image sensors with limited reaction speed and high data redundancy. Dynamic vision sensor (DVS), which is inspired by biological retinas, efficiently captures the visual information with sparse, asynchronous events rather than dense, synchronous frames. It can eliminate redundant data transmission and avoid motion blur or data leakage in high-speed imaging applications. However, it is usually impractical to directly apply the event streams to conventional object detection algorithms. For this issue, we first propose a novel event-to-frame conversion method by integrating the inherent characteristics of events more efficiently. Moreover, we design an improved feature extraction network that can reuse intermediate features to further reduce the computational effort. We evaluate the performance of our proposed method on a custom dataset containing multiple real-world pedestrian scenes. The results indicate that our proposed method raised its pedestrian detection accuracy by about 5.6–10.8%, and its detection speed is nearly 20% faster than previously reported methods. Furthermore, it can achieve a processing speed of about 26 FPS and an AP of 87.43% when implanted on a single CPU so that it fully meets the requirement of real-time detection.