Computational Image Sensor Research Articles

Resolving Multi-Path Interference in Compressive Time-of-Flight Depth Imaging with a Multi-Tap Macro-Pixel Computational CMOS Image Sensor.

Multi-path interference causes depth errors in indirect time-of-flight (ToF) cameras. In this paper, resolving multi-path interference caused by surface reflections using a multi-tap macro-pixel computational CMOS image sensor is demonstrated. The imaging area is implemented by an array of macro-pixels composed of four subpixels embodied by a four-tap lateral electric field charge modulator (LEFM). This sensor can simultaneously acquire 16 images for different temporal shutters. This method can reproduce more than 16 images based on compressive sensing with multi-frequency shutters and sub-clock shifting. In simulations, an object was placed 16 m away from the sensor, and the depth of an interference object was varied from 1 to 32 m in 1 m steps. The two reflections were separated in two stages: coarse estimation based on a compressive sensing solver and refinement by a nonlinear search to investigate the potential of our sensor. Relative standard deviation (precision) and relative mean error (accuracy) were evaluated under the influence of photon shot noise. The proposed method was verified using a prototype multi-tap macro-pixel computational CMOS image sensor in single-path and dual-path situations. In the experiment, an acrylic plate was placed 1 m or 2 m and a mirror 9.3 m from the sensor.

A Dual-Mode 303-Megaframes-per-Second Charge-Domain Time-Compressive Computational CMOS Image Sensor.

An ultra-high-speed computational CMOS image sensor with a burst frame rate of 303 megaframes per second, which is the fastest among the solid-state image sensors, to our knowledge, is demonstrated. This image sensor is compatible with ordinary single-aperture lenses and can operate in dual modes, such as single-event filming mode or multi-exposure imaging mode, by reconfiguring the number of exposure cycles. To realize this frame rate, the charge modulator drivers were adequately designed to suppress the peak driving current taking advantage of the operational constraint of the multi-tap charge modulator. The pixel array is composed of macropixels with 2 × 2 4-tap subpixels. Because temporal compressive sensing is performed in the charge domain without any analog circuit, ultrafast frame rates, small pixel size, low noise, and low power consumption are achieved. In the experiments, single-event imaging of plasma emission in laser processing and multi-exposure transient imaging of light reflections to extend the depth range and to decompose multiple reflections for time-of-flight (TOF) depth imaging with a compression ratio of 8× were demonstrated. Time-resolved images similar to those obtained by the direct-type TOF were reproduced in a single shot, while the charge modulator for the indirect TOF was utilized.

A 0.5-V Real-Time Computational CMOS Image Sensor With Programmable Kernel for Feature Extraction

As the growing demand on artificial intelligence (AI) Internet-of-Things (IoT) devices, smart vision sensors with energy-efficient computing capability are required. This article presents a low-power and low-voltage dual mode 0.5-V computational CMOS image sensor (C <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> IS) with array-parallel computing capability for feature extraction using convolution. In the feature extraction mode, by applying the pulsewidth modulation (PWM) pixel and switch-current integration (SCI) circuit, the in-sensor eight-directional matrix-parallel multiply-accumulate (MAC) operation is realized. Furthermore, the analog-domain convolution-on-readout (COR) operation, the programmable 3×3 kernel with ±3-bit weights, and the tunable-resolution column-parallel analog-to-digital converter (ADC) (1-8 bit) are implemented to achieve the real-time feature extraction without using additional memory and sacrificing frame rate. In the image capturing mode, the sensor provides the linear-response 8-bit raw image data. The C <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> IS prototype has been fabricated in the TSMC 0.18-μm standard process technology and verified to demonstrate the raw and feature images at 480 frames/s with a power consumption of 77/ 117 μW and the resultant FoM of 9.8/14.8 pJ/pixel/frame, respectively. The prototype sensor is used as a real-time edge feature detection frond-end camera and accompanied with a simplified convolutional neural network (CNN) architecture to demonstrate the hand gesture recognition. The prototype system achieves more than 95% validation accuracy.

Dual-Tap Computational Photography Image Sensor With Per-Pixel Pipelined Digital Memory for Intra-Frame Coded Multi-Exposure

A coded-exposure-pixel image sensor for computational imaging applications is presented. Each frame exposure time is divided into ${N}$ subframes. Within each subframe, each pixel sorts photo-generated charge into two charge taps depending on that pixel’s 1-bit binary code. ${N}$ global updates of arbitrary pixel-wise codes are implemented in each frame to enable ${N}$ short global pixel-specific subexposures within one frame. To make these subexposures global, two latches per pixel are utilized in a pipelined fashion. The code for the next subframe is loaded into latch 1 in a row parallel fashion, while the code for the current subframe is being applied by latch 2 globally for photo-generated charge sorting during the current subexposure. A $280^{H}\times 176^{V}$ image sensor prototype with $11.2{\text{-}}\mu \text{m}$ pixel pitch has been fabricated in a $0.11{\text{-}}\mu \text{m}$ CMOS image sensor (CIS) technology. The image sensor has been demonstrated in two computational photography applications, each using only a single frame of a video: 1) computing both albedo (a measure of reflectivity) and 3-D depth maps by means of structured-light imaging and 2) computing surface normals (3-D orientations) map by means of photometric stereo imaging. These demonstrations experimentally validate some of the unique capabilities of this computational image sensor, such as accurate 3-D visual scene reconstruction using only one camera, while maintaining its native specifications: the full spatial resolution and the maximum frame rate.

Guest editorial: Special issue on computational image sensors and smart camera hardware

Guest editorial: Special issue on computational image sensors and smart camera hardware

CMOS Vision Sensors: Embedding Computer Vision at Imaging Front-Ends

CMOS Image Sensors (CIS) are key for imaging technologies. These chips are conceived for capturing optical scenes focused on their surface, and for delivering electrical images, commonly in digital format. CISs may incorporate intelligence; however, their smartness basically concerns calibration, error correction and other similar tasks. The term CVISs (CMOS VIsion Sensors) defines other class of sensor front-ends which are aimed at performing vision tasks right at the focal plane. They have been running under names such as computational image sensors, vision sensors and silicon retinas, among others.

Special issue on computational image sensors and smart camera hardware

Special issue on computational image sensors and smart camera hardware

Low Power Programmable Current Mode Computational Imaging Sensor

A linear current mode computational image sensor is presented in this paper. The sensor monolithically combines current mode imaging elements with current mode digitally programmable analog computational circuitry at the focal plane. The pixel is based on a switchless paradigm and allows for reduced spatial and temporal noise imaging. The imaging array is composed of 128-by-109 pixels. The image sensor provides a noise-corrected image in conjunction with a convolved image in parallel at 50 fps and 50 mW. The maximum measured SNR is 44 dB, maximum FPN of saturated value (after DDS) is 0.22%, and DR is 55 dB.

Compressive Sensing on a CMOS Separable-Transform Image Sensor

This paper demonstrates a computational image sensor capable of implementing compressive sensing operations. Instead of sensing raw pixel data, this image sensor projects the image onto a separable 2-D basis set and measures the corresponding expansion coefficients. The inner products are computed in the analog domain using a computational focal plane and an analog vector-matrix multiplier (VMM). This is more than mere postprocessing, as the processing circuity is integrated as part of the sensing circuity itself. We implement compressive imaging on the sensor by using pseudorandom vectors called noiselets for the measurement basis. This choice allows us to reconstruct the image from only a small percentage of the transform coefficients. This effectively compresses the image without any digital computation and reduces the throughput of the analog-to-digital converter (ADC). The reduction in throughput has the potential to reduce power consumption and increase the frame rate. The general architecture and a detailed circuit implementation of the image sensor are discussed. We also present experimental results that demonstrate the advantages of using the sensor for compressive imaging rather than more traditional coded imaging strategies.

Computational Image Sensors with Motion Detection Function

Computational Image Sensors with Motion Detection Function

Adaptive flat multiresolution multiplexed computational imaging architecture utilizing micromirror arrays to steer subimager fields of view

A thin, agile multiresolution, computational imaging sensor architecture, termed PANOPTES (processing arrays of Nyguist-limited observations to produce a thin electro-optic sensor), which utilizes arrays of microelectromechanical mirrors to adaptively redirect the fields of view of multiple low-resolution subimagers, is described. An information theory-based algorithm adapts the system and restores the image. The modulation transfer function (MTF) effects of utilizing micromirror arrays to steering imaging systems are analyzed, and computational methods for combining data collected from systems with differing MTFs are presented.

Adaptive-integration-time image sensor with real-time reconstruction function

A computational image sensor is proposed in which the pixel controls its integration time to light intensity. The integration time of each pixel is selected from among several lengths of integration time and the integration time is shortened if the pixel intensity becomes saturated. Although the integration time of each pixel varies, the pixel intensity is adjusted on the sensor in real time. The dynamic range of the pixel value output from the proposed sensor is greatly widened. A prototype of 64/spl times/48 pixels has been fabricated by using 2-poly 2-metal 0.8-/spl mu/m CMOS process. The proposed sensor has simple functions for the comparison of intermediate integration value and threshold to control the integration time and nonlinear image reconstruction. Because the maximum number of the comparison-reset operations during a frame is three, one of the four integration times can be selected pixel by pixel. The circuit and layout design of the prototype which has computational elements based on column parallel architecture are described and the fundamental functions have been verified. By the experiments, it has been verified that the sensor can achieve a wide dynamic range by adapting to light.

A computational image sensor with adaptive pixel-based integration time

An image sensor is proposed in which the pixel adapts its integration time to motion and light. The integration time of each pixel is shortened if motion is detected in the pixel or pixel intensity becomes saturated. The adaptivity of motion and light significantly enhances temporal resolution and dynamic range of the sensor. Because the integration time differs pixel-by-pixel, a scene containing both a bright and a dark region will be captured by pixels of shorter and longer integration times. Because the integration time adapts to motion, higher temporal resolution is obtained in a moving area and a higher signal-to-noise ratio in a static area. The control of the integration time is done on the sensor focal plane, with column parallel processing circuits integrated in CMOS image sensor. A prototype of 32/spl times/32 pixels has been fabricated by using 1-poly 2-metal CMOS 1-/spl mu/m process. The fundamental functions have been verified. By the experiments, it has been verified that the sensor can reduce motion blur by adapting the integration time to motion and achieve wide dynamic range by adapting to light when the minimum integration interval is 680 /spl mu/s.

Computational image sensor for on sensor compression

We propose a novel integration of image compression and sensing in order to enhance the performance of an image sensor. By integrating a compression function onto the sensor focal plane, the image signal to be read out from the sensor is significantly reduced and the pixel rate of the sensor ran consequently be increased. The potential applications of the proposed sensor are in high pixel-rate imaging, such as high frame-rate image sensing and high-resolution image sensing. The compression scheme we employ is a conditional replenishment, which detects and encodes moving areas. In this paper, we introduce two architectures for on-sensor compression; one is the pixel parallel approach and the other is the column parallel approach. We prototyped a VLSI chip of the proposed sensor based on the pixel parallel architecture. We show the design and describe the results of the experiments obtained by the prototype chip.