Digital Pixel Research Articles

Energy Efficient Pixel-Parallel Read-Out Circuits for Digital Image Sensors Using Cross-Layer Pixel Depth Control

Digital pixel sensors with pixel-parallel readout provide opportunities for high frame rate and high resolution in 3D integrated camera platforms with in-sensor machine learning. The active energy consumption of the digital pixels at the sensor front end is a key performance parameter for applications involving smart cameras. This work proposes a novel digital pixel design that utilizes cross-layer feedback to improve active energy efficiency of pixels in image sensors. An algorithm-circuit co-simulation of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$336\times256$ </tex-math></inline-formula> image sensor in 65nm CMOS for an object detection task shows 58% active energy saving in digital pixel circuits, while maintaining accuracy of end task.

Design of a multi‐mode digital pixel with conversion data protection

With the development of semiconductor technology, digital pixel has received widespread attention and is applied to various electronic products. However, due to the limitation of area, it forms a challenging task to design a digital pixel with multiple modes. In this paper, a pulse width modulation based digital pixel is proposed, which is compatible with five different modes. By using the multi-purpose capacitors and static random access memory structure, it can realise multi-mode conversion in an equivalent area to that of the single mode digital pixel without performance degradation. Furthermore, a corresponding logic control method is developed, such that the integrity of the frame data is ensured during mode conversion. The simulation result of our proposed digital pixel in Tower Jazz 0.18 μm process shows that in the bright field it achieves a dynamic range of 67 dB. In the dark field, it achieves a conversion gain up to 13.91 μV/e−, with input noise of 37.89 e−per pixel after correlated double sampling.

Extended Stress–Strain Characterization of Automotive Steels at Dynamic Rates

Demanding structural applications require a detailed knowledge of the materials response up to the very late stages before failure. Ductile high-strength steels may undergo pronounced necking over the majority of their straining life; this makes a reliable stress–strain characterization difficult, especially at dynamic rates, because the self-heating from fast adiabatic dissipation may promote thermal effects interplaying with the strain rate effects. Further complications arise in deriving the postnecking flow curves when the material is a metal sheet due to geometrical issues intrinsic in the prismatic flat shape of the specimens. This paper focuses on the experimental derivation of the flow curves of DP1000 and MS1700 steels at strain rates ranging from 1 to 500/s. In addition, the moderately high temperatures achieved due to the self-heating at dynamic rates are imposed at static rates for separately investigating thermal and dynamic effects. Digital Image Correlation (DIC) and pixel counting optical techniques are used together with postprocessing procedures based on standard criteria and on physical considerations proposed by the authors. The resulting hardening curves are compared to each other and the advantages of the proposed method are discussed.

A 4.0 μm Stacked Digital Pixel Sensor Operating in a Dual Quantization Mode for High Dynamic Range

It is anticipated that ubiquitous computer vision (CV) and artificial intelligence (AI) applications used on mobile devices will grow significantly. Such applications require battery-powered, always-on mobile devices to support indoor/outdoor, day/night usages. A global shutter (GS), stacked digital pixel sensor (DPS) is a promising candidate to meet such requirements because of its potential for ultralow-power, ultrahigh dynamic range (HDR), and a small form factor. This article presents a prototype 4.0- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${\mu } \text{m}$ </tex-math></inline-formula> stacked DPS operating in its dual quantization (2Q) to realize HDR. The 4.0- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${\mu }\text{m}$ </tex-math></inline-formula> DPS pixel is formed with two layers, a backside illuminated pinned photodiode (PD) pixel on the top layer and an in-pixel analog-to-digital conversion (ADC) circuit with 9-bit static random access memory (SRAM) on the bottom layer. A Cu-to-Cu hybrid bonding (HB) technology is used to connect the two layers via pixel-level interconnect. In the 2Q scheme, a time-stamp (TS) quantization and a linear ADC are performed sequentially in the same frame, which extends the dynamic range (DR) with a small number of ADC bits of 9. The DPS with a 1024 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times$ </tex-math></inline-formula> 832 pixel array has achieved a single-exposure ultra HDR of 107 dB in a single frame. The nonlinear conversion characteristic of the TS mode provides an equivalent full well capacity (FWC) of 2000ke <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">−</sup> , while the noise floor in the linear ADC mode is 8.3e <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">−</sup> .

A 4.6-μm, 127-dB Dynamic Range, Ultra-Low Power Stacked Digital Pixel Sensor With Overlapped Triple Quantization

This article presents a global shutter (GS), stacked digital pixel sensor (DPS) to meet the ultra-low power, ultra high dynamic range (HDR) requirements for battery-powered, always-on mobile computer vision (CV) applications. The <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$4.6~\mu \text{m}$ </tex-math></inline-formula> DPS pixel is partitioned into two layers; a dual conversion gain (CG) pixel with a backside-illuminated photodiode (PD) on the top layer and an in-pixel ADC circuit with 10 bit SRAM on the bottom layer. A Cu-to-Cu hybrid-bonding (HB) technology is used to connect the two layers via pixel-level interconnect. The sensor features an overlapped triple quantization (3Q) scheme that performs a time-to-saturation (TTS) quantization and the dual-CG linear ADC mode sequentially in the same frame and extends the dynamic range (DR) with a small number of ADC bits. The DPS with a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$512\times512$ </tex-math></inline-formula> pixel array has achieved an ultra-HDR of 127 dB and low power consumption of 5.75 mW at 30 frames per second. The nonlinear conversion characteristics of the TTS mode provide an equivalent full well capacity (FWC) of 9000 ke <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">−</sup> , while the high CG linear ADC mode realizes a low noise floor of 4.2 e <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">−</sup> .

Research and Application of Color of Qiuci Murals Based on Intelligent Digital Image Processing Technology

In order to improve the color research effect of Qiuci murals, this paper combines digital technology, applies intelligent digital image processing technology to the color research of Qiuci murals, and builds an intelligent system. Moreover, this paper analyzes the color mural pixel digitization of murals from the perspective of digitization and conducts research from a microscopic perspective. In addition, in this paper, the expansion operation is performed for particularly small cracks, and the extraction area of the crack is correspondingly increased, which reduces the bad influence on the subsequent repair due to incomplete crack extraction. Finally, this paper constructs an intelligent mural analysis system based on the improved algorithm. The experimental research results show that the color research system of Qiuci murals based on intelligent digital image processing technology proposed in this paper has good effects and can play an important role in the digital protection and digital restoration of art.

An Aesthetics of Disruption: Unsettling the Diasporic Subject

In her writings and video works, Hito Steyerl presents a disruptive tension between the pervading sense of being under constant surveillance and the desire not to be seen or to be invisible. In observing the omnipresence of the camera, Ariella Azoulay also considers photography’s capacity for inscription and surveillance. This withdrawal from representation disrupts our expectations of the photographic process, in which a contract is made between the sitter and the photographer, that their likeness will be captured on the photographic emulsion or digital pixels, a likeness that can be observed, critiqued, printed or shared. Disrupting the political and social ontologies of photography undoes and unsettles what photography is and what it should do; hence Azoulay and others ask, What is a photograph? This essay takes these concepts as a point from which to consider an aesthetics of disruption and the conditions that cause a subject or an image to withdraw, to hide, or to disappear, in the work of three artists from the Arab diaspora: Cherine Fahd, Joana Hadjithomas, and Khalil Joreige.

2.45 e-RMS Low-Random-Noise, 598.5 mW Low-Power, and 1.2 kfps High-Speed 2-Mp Global Shutter CMOS Image Sensor With Pixel-Level ADC and Memory

This article presents a low random noise, a low-power, and a high-speed 2-mega pixels (Mp) global-shutter (GS)-type CMOS image sensor (CIS) using an advanced dynamic random access memory (DRAM) technology. GS CIS is one of the alternatives to solve image distortion issues caused by a conventional rolling-shutter (RS) CIS operation, since a 2-D image data can be simultaneously sampled by the in-pixel analog memory. To achieve a high-performance GS CIS, we proposed a novel architecture for digital pixel sensor (DPS) which is a high-speed GS operation CIS with a pixel-wise analog-to-digital converter (ADC) and an in-pixel digital memory. The major technologies of the proposed DPS can be summarized as follows: 1) two large coupling capacitors with mature DRAM technology; 2) extremely narrow pitch Cu-to-Cu (C2C) bond; and 3) finally low-powered ADC with a near sub-threshold operation. A perfect auto-zero operation for ADC is implemented using two DRAM capacitors, and a large number of transistors have to be integrated in the single pixel for realizing pixel-level ADC. Thus, each pixel has two fine-pitch C2C interconnections. This makes it possible to realize wafer-level stacked unit pixel. The proposed DPS with low-power consuming analog circuits has been successfully designed and developed for extremely fast-readout speed of max. 1200 frames per second (fps) and high sensitivity for low-illumination conditions.

Engineering pupil function for optical adversarial attacks.

Adversarial attacks inject imperceptible noise to images to deteriorate the performance of deep image classification models. However, most of the existing studies consider attacks in the digital (pixel) domain where an image acquired by an image sensor with sampling and quantization is recorded. This paper, for the first time, introduces a scheme for optical adversarial attack, which physically alters the light field information arriving at the image sensor so that the classification model yields misclassification. We modulate the phase of the light in the Fourier domain using a spatial light modulator placed in the photographic system. The operative parameters of the modulator for adversarial attack are obtained by gradient-based optimization to maximize cross-entropy and minimize distortion. Experiments based on both simulation and a real optical system demonstrate the feasibility of the proposed optical attack. We show that our attack can conceal perturbations in the image more effectively than the existing pixel-domain attack. It is also verified that the proposed attack is completely different from common optical aberrations such as spherical aberration, defocus, and astigmatism in terms of both perturbation patterns and classification results.

Design and analysis on low-power and low-noise single slope ADC for digital pixel sensors

A Low-power and low-noise digital pixel sensor (DPS) is presented in this paper. To design and analyze the random noise (RN) of the developed DPS, especially, we utilize a novel simulationmethod called transient-based AC noise simulation (TBAS) which can effectively help to estimate the noise components of the lowpowered single-slope (SS) analog-to-digital converter (ADC). Based on this noise analysis, the high performance DPS has been successfully designed and demonstrated.

A 51-pJ/Pixel 33.7-dB PSNR 4× Compressive CMOS Image Sensor With Column-Parallel Single-Shot Compressive Sensing

This article presents a CMOS image sensor (CIS) with column-parallel single-shot compressive sensing (CS) for always-on Internet-of-Things (IoT) application, which achieves an energy efficiency of 51 pJ/pixel, while maintaining high image quality of PSNR > 33.7 dB and SSIM > 0.89. This is enabled by an energy-efficient encoder, which replaces a densely populated CS encoding matrix with a highly sparse pseudo-diagonal one. Since the proposed column-parallel CS encoder can be implemented directly at pixel outputs with an energy-efficient switched-capacitor matrix multiplier, data compression is achieved prior to the pixel digitization, thereby greatly reducing ADC power, data size, and I/O power. The energy efficiency of the image sensor is further improved by using dynamic single-slope ADCs. A prototype VGA image sensor implemented in a 110-nm CMOS process consumes only 0.7 mW at 45 frames/s. The corresponding energy per pixel (51 pJ/pixel) amounts to more than <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$5\times $ </tex-math></inline-formula> improvement over the previous low-energy benchmark for CS image sensors.

Adaptive Camera Platform Using Deep Learning-Based Early Warning of Task Failures

There is a growing interest in deploying complex deep neural networks (DNN) in smart sensors to extract task-specific information from real-time sensor data. This paper presents an adaptive sensor that uses a novel light-weight deep learning platform, WarningNet, to estimate potential task failures due to perturbations in the sensor data and control its read-out-circuit (ROIC) parameters such as sampling resolution and operating voltage. Simulation results show that WarningNet can provide early warning of the performance degradation of various tasks within a fraction of the time required for the task to complete. Moreover, the early warning guided adaptive control of sampling resolution and operating voltage of the ROIC of a digital pixel sensor shows potential of dynamic trade off between task accuracy and sensor energy/bandwidth.

The Spatio-Temporal Variability of Frost Blisters in a Perennial Frozen Lake along the Antarctic Coast as Indicator of the Groundwater Supply

Remote sensing, and unmanned aerial vehicles (UAVs) in particular, can be a valid tool for assessing the dynamics of cryotic features as frost blisters and to monitor the surface changes and the sublimation rates on perennially frozen lakes that host important ecosystems. In this paper, through the use of these remote sensing techniques, we aim to understand the type of groundwater supply of an Antarctic perennial frozen lake that encompasses two frost blisters (M1 and M2) through the temporal analysis of the features’ elevation changes (frost blisters and lake ice level). The frozen lake is located at Boulder Clay (northern Victoria Land, Antarctica). We relied on several photogrammetric models, past satellite images and ground pictures to conduct differencing of digital elevation models, areal variations and pixel counting. In addition, in situ measurements of the ice sublimation or snow accumulation were carried out. The two frost blisters showed different elevation trends with M1 higher in the past (1996–2004) than recently (2014–2019), while M2 showed an opposite trend, similarly to the ice level. Indeed, the linear regression between M2 elevation changes and the ice level variation was statistically significant, as well as with the annual thawing degree days, while M1 did not show significant results. From these results we can infer that the groundwater supply of M1 can be related to a sublake open talik (hydraulic system) as confirmed also by pressurized brines found below M1, during a drilling in summer 2019. For M2 the groundwater flow is still not completely clear although the hydrostatic system seems the easiest explanation as well as for the uplift of the lake ice.

Implications of Germany’s Electronic Securities Act for Supervisory Technology

Digital transformation requires a holistic approach; every step in a transformation should ultimately contribute to a big picture reflecting an acceptable balance among digital blocks and pixels. Digitalization of financial securities is one of the main building blocks of such transformation. In June 2021, Germany has transcended its conservative securities hinterland by breaking its chains in the paper-based securities realm and strategically anchored in blockchain domains with a new statute, the Electronic Securities Act. Although the act profoundly reforms the German securities industry by introducing a regulated territory for digital securities, its dematerialization rules cause a dual system: digital and paper-based securities can be issued or circulating together, doubling the workloads and challenges of financial supervisors in collecting consolidated (real-time) data and managing (near real-time) market surveillance. Considering the strategic risk management role of supervisory technology (SupTech), we conclude that digital transformation efforts with limited SupTech considerations do not help eliminate asymmetric technology risks and leave markets vulnerable to frauds and failures.

Dynamic target image correction method of digital media based on virtual reality

In order to overcome the problems of low peak signal-to-noise ratio (PSNR), low structure similarity index and resolution coefficient not close to 1, a digital media dynamic target image correction method based on virtual reality is proposed. This method uses virtual reality, image neighbourhood information statistical information, fuzzy information and human visual characteristics to carry out grey value statistics, then combines with the statistical characteristics of grey value to realise image denoising processing; combined with rational function model, obtains the digital media dynamic target image pixel coordinates representing the ground point coordinates, obtains the target image correction model, completes the digital media dynamic target image correction. The experimental results show that the peak signal-to-noise ratio (PSNR) is close to 400 dB, the maximum structure similarity index is 0.6025, and the decision coefficient is close to 1, which can quickly and accurately realise the dynamic target image correction of digital media.

Drawing Ed Ruscha

This project aims to discuss drawing as a method of bridging the void between digital imaging technologies and physical drawing in the fine art domain. It does so by investigating the role of drawing and printing in contemporary portraiture. Drawn and printed silkscreen portraits are made from a synthesis of graphite marks, digital pixels and water-based ink deposited on paper surfaces. The practice-led research described here explores the materiality of the emergent image when drawing is impressed on an electronic media trace. This investigation is timely in the context of the unprecedented impact of digital technologies on contemporary culture that tend to displace the physicality of drawing. By taking an approach to portraiture whereby artist and sitter do not meet in person, the project initiates a portrait of Ed Ruscha using the medium of video images. Digital electronic images held pixel by pixel in smartphone camera and computer hard disks are interpreted into physical drawing environments to make an expressive representation of a human form. Tactile gestural mark-making is contrasted with electronic imaging to create a pensive image where techniques are blended. The process and methodology are described, and the artistic outputs are shared across the globe through digital and analogue communication systems.

Measurement of Water Leaving Reflectance Using a Digital Camera Based on Multiple Reflectance Reference Cards.

With the development of citizen science, digital cameras and smartphones are increasingly utilized in water quality monitoring. The smartphone application HydroColor quantitatively retrieves water quality parameters from digital images. HydroColor assumes a linear relationship between the digital pixel number (DN) and incident radiance and applies a grey reference card to derive water leaving reflectance. However, image DNs change with incident light brightness non-linearly, according to a power function. We developed an improved method for observing and calculating water leaving reflectance from digital images based on multiple reflectance reference cards. The method was applied to acquire water, sky, and reflectance reference card images using a Cannon 50D digital camera at 31 sampling stations; the results were validated using synchronously measured water leaving reflectance using a field spectrometer. The R2 for the red, green, and blue color bands were 0.94, 0.95, 0.94, and the mean relative errors were 27.6%, 29.8%, 31.8%, respectively. The validation results confirm that this method can derive accurate water leaving reflectance, especially when compared with the results derived by HydroColor, which systematically overestimates water leaving reflectance. Our results provide a more accurate theoretical foundation for quantitative water quality monitoring using digital and smartphone cameras.

Cross-Layer Noise Analysis in Smart Digital Pixel Sensors With Integrated Deep Neural Network

Digital pixel based image sensors with embedded deep neural network (DNN) allow many mission critical surveillance applications. However, image noise caused by variations and non-idealities in the sensor aggravates the quality of image and further degrades the performance of a DNN. We propose a digital pixel-DNN cross-layer simulation methodology for accurate training and evaluation of a DNN under image noise induced from sensors. In particular, this paper focuses on the image noise derived from device mismatches in digital pixel circuits with 3D integrated and pixel-parallel readout circuits, and studies the effect of the resulting image noise on the accuracy of a DNN. The simulation results show that the device mismatch in the digital pixel creates distinct noise structure on output image and should be accurately considered while training a DNN. We also present design space explorations using our cross-layer simulation.

On the Memory of Ice: An Interview with Jonathan Marquis by Kathy Kasic

When I discovered Jonathan Marquis’s cameraless photographic work, I found myself entranced. As a filmmaker primarily focused on environmental documentary narratives, I often record digital images of a world I fear we are losing. Perhaps especially because camera electronics transform light and dark shapes into digital pixels, I try to make my images as tactile and sensory as possible so that we may become within a place. Jonathan Marquis’s cyanotype series, Downwaste, created in Glacier National Park, raises questions about photography, time, place, and agency as the climate changes and the glaciers melt. I had the good fortune to spend some time interviewing him about his work. What follows is our discussion.

A reconfigurable CMOS sensor for tracking, pre-shower and digital electromagnetic calorimetry

Recent advances in CMOS imaging sensor technology have led to designs able to withstand much higher radiation levels, up to those required at hadron colliders for all but the innermost layers. Also, with stitching, wafer scale devices have been fabricated on the same process as used for the prototypes described in this submission, for applications such as, for example, direct electron detection in transmission electron microscopy. A small sensor prototype has been designed and fabricated in the TowerJazz 180 nm CMOS imaging process. The prototype has a pixel matrix of 64x64 pixels with a pitch of 55 x 55μm and reads out using fast logic at 40 MHz. Each pixel contains four collection electrodes, trimming logic, pre-amplifier, shaper and discriminator with digital output. It can be reconfigured to function as either a binary short strip sensor, for particle tracking including as a pre-shower, or as a pad sensor, counting the number of pixels above threshold for digital calorimetry. As well as providing a seamless transition from outer tracking to EM calorimetry, for optimal use of particle flow algorithms, digital calorimetry also can give excellent energy resolution. This concept is proposed as a possible option for future digital calorimeters able to read-out at LHC collision rates with good radiation hardness and realisable in low cost commercial technologies for equipping large areas. Results on physics performance simulation in a FCC-hh context and the characterisation of the prototype sensor are presented. The summing logic is demonstrated and the analogue pixel performance is validated by illuminating a test pixel within the matrix with a laser of wavelength of 1064 nm. The absolute laser intensity is calibrated such that the injected charge is similar to that expected for a minimum ionising particle. The measured pre-amplifier and shaper signals are compared to Cadence simulations. Laser illuminations in the digital pixel area and the response measured using a threshold scan confirm successful digital functionality in strip and pad operation modes.