Parallel Pixel Research Articles

PiDiNeXt: Lightweight parallel pixel difference networks for edge detection

PiDiNeXt: Lightweight parallel pixel difference networks for edge detection

Parallel Lossless Compression of Raw Bayer Images on FPGA-Based High-Speed Camera

Digital image compression is applied to reduce camera bandwidth and storage requirements, but real-time lossless compression on a high-speed high-resolution camera is a challenging task. The article presents hardware implementation of a Bayer colour filter array lossless image compression algorithm on an FPGA-based camera. The compression algorithm reduces colour and spatial redundancy and employs Golomb–Rice entropy coding. A rule limiting the maximum code length is introduced for the edge cases. The proposed algorithm is based on integer operators for efficient hardware implementation. The algorithm is first verified as a C++ model and later implemented on AMD-Xilinx Zynq UltraScale+ device using VHDL. An effective tree-like pipeline structure is proposed to concatenate codes of compressed pixel data to generate a bitstream representing data of 16 parallel pixels. The proposed parallel compression achieves up to 56% reduction in image size for high-resolution images. Pipelined implementation without any state machine ensures operating frequencies up to 320 MHz. Parallelised operation on 16 pixels effectively increases data throughput to 40 Gbit/s while keeping the total memory requirements low due to real-time processing.

A novel GM-APD array-based heterodyne imaging detection system supports large array multi-pixel parallel target motion state image detection

Existing laser active imaging detection methods are limited by the detection mechanism, which can only capture the intensity image and distance image of the target, and lack practical detection solutions for acquiring images of the target motion state characteristics (non-macroscopic centroid motion, such as rotation, rolling, and other motion characteristics around the centroid). In this paper, a detection method using Geiger mode avalanche photodiode (GM-APD) array detector for heterodyne imaging to obtain images of the target motion state is proposed. An innovative method based on photon counting time interval calculation (PCTIC) is proposed to obtain the Doppler frequency shift of the target echo for each GM-APD pixel, which effectively eliminates useless data unrelated to the signal. With limited data storage and processing speed, the heterodyne imaging performance has been effectively improved to achieve high-precision heterodyne imaging detection of very weak target echoes and to support heterodyne imaging detection of large arrays with more image pixels in parallel. Experiments show that the method can achieve video-level detection at an imaging resolution of 32×32, and with a paucity of echo photons (detection probability of echo is 0.0032), a millimetre-level velocimetry measurement error can be achieved by accumulating only 6 echo photons per image element. The system also shows good resistance to turbulence interference, with target motion state characteristics detected based on velocity images remaining virtually unaffected under turbulence conditions caused by strong flames. This advancement is crucial for target detection in complex working environments such as smart manufacturing and autonomous driving.

Real-time hierarchical parallel rendering and display of light field images

Light field display technology has attracted considerable interest due to its remarkable 3D display capabilities and compatibility. However, the real-time capture of massive light field image data remains a significant challenge in this field. To improve rendering efficiency and achieve high-quality 3D display, a real-time hierarchical parallel rendering method is proposed. The method parallelizes the rendering pipeline of light field images from multi-levels. Firstly, instancing technology is exploited for parallel rendering of high-dimensional orthogonal parallax information. Secondly, perspective coherence of parallax, lighting, and texture information are used for the parallel rendering of one dimensional parallax effects. Thirdly, the data processing process has been simplified and accelerated by using parallel scanning slicing algorithms and parallel pixel recombination. This multi-level parallel processing architecture significantly reduces the loops and repetitive calculations. An efficient rendering pipeline was developed to achieve real-time rendering of 200 viewpoint light field images with anisotropic lighting effects at a 4 K resolution. The frame rate is 40 frames per second (fps). Compared to traditional methods, this algorithm reduces the time required to render one frame of light image by 17 times. A light field display system was made to validate the effectiveness of the method. High performance 3D images were outputted using the rendered light field images.

High-spatiotemporal resolution microwave-induced thermoacoustic tomography for imaging biological dynamics in deep tissue

Biological systems undergo constant dynamic changes across various spatial and temporal scales. To investigate the intricate biological dynamics in living organisms, there is a strong need for high-speed and high-resolution imaging capabilities with significant imaging depth. In this work, we present high-spatiotemporal resolution microwave-induced thermoacoustic tomography (HR-MTAT) as a method for imaging biological dynamics in deep tissues. HR-MTAT utilizes nanosecond pulsed microwave excitation and ultrasound detection, with appropriate spatial configurations, to achieve high coupling of the sample to the microwaves, to produce images in soft tissue with dielectric contrast and sub-millimeter spatial resolution (230 μm), to a depth of a few centimeters. Notably, by employing a 128-channel parallel signal acquisition and digitization strategy, the field programmable gate array module manages data synthesis, and GPU-based parallel pixel reconstruction facilitates HR-MTAT to accomplish single-frame image reconstruction in an impressive 50 μs. The practical feasibility of HR-MTAT was evaluated in live mice. The results show that HR-MTAT can noninvasively image whole-body small animals (up to 60 mm in depth) with clear resolution of internal organ structures at a frame rate of 100 Hz, without the need for labeling. At this high spatiotemporal resolution, HR-MTAT can capture respiration, heartbeat, and arterial pulse propagation without motion artifacts and track bio-nanoprobes in livers and tumors. These findings demonstrate HR-MTAT's ability to perform dynamic imaging with high contrast and resolution in deep tissues.

Integration of EDWARD readout architecture in full-field fluorescence imaging detector

Data bandwidth, timing resolution and resource utilization in readouts of radiation detectors are a constant challenge. Event driven solutions are pushing against well-trenched framed solutions. The idea for an asynchronous readout architecture called EDWARD (Event-Driven With Access and Reset Decoder) was presented at the TWEPP 2021 conference. Here we show the progress of our work which resulted in two chip prototypes. The first one, named 3FI65P1, is a full device with the analog pixel circuitry suited for full-field fluorescence imaging. It is already manufactured, and preliminary results are presented. The second chip, named EDWARD65P1, contains digital pulse generators with Poisson-exponential distribution in each pixel for extraction of the performance matrix of the EDWARD architecture alone.

First measurements with monolithic active pixel test structures produced in a 65 nm CMOS process

The Inner Tracking System (ITS) of the ALICE experiment at CERN will undergo an upgrade during the LHC long shutdown 3, in which the three innermost tracking layers will be replaced. This upgrade, named the Inner Tracking System 3 (ITS3), employs stitched wafer-scale Monolithic Active Pixel Sensors fabricated in a 65 nm CMOS process. The sensors are 260 mm in length and thinned to less than 50 μm then bent to form truly half-cylindrical half-barrels. The feasibility of this process for the ITS3 was explored with the first test production run (MLR1) in 2021, whose goal was to evaluate the charged particle detection efficiency and the sensor performance under non-ionising and ionising radiation up to the expected levels for ALICE ITS3 of 1013 1 MeV neq cm-2 (NIEL) and 10 kGy (TID). Three sensor flavours were produced to investigate this process: Analog Pixel Test Structure (APTS), Circuit Exploratoire 65 (CE65) and Digital Pixel Test Structure (DPTS).This contribution gives an overview of the MLR1 submission and test results, describing the different sensor flavours and presenting the results of the performance measurements done with particle beams for various chip variants and irradiation levels.

Study of MAPS silicon detector prototypes for the ALICE Inner Tracking System upgrade

The ALICE experiment, which is located at the Large Hadron Collider (LHC) at CERN, has planned an upgrade of the Inner Tracking System (ITS), called ITS3, which will be installed during the LHC Long Shutdown 3, in 2026–2028. The cornerstone of the upgrade is a new 65 nm CMOS pixel chip, produced using Monolithic Active Pixel Sensor (MAPS) technology and the stitching technology to extend the chip length to 26 cm. The produced chips will be bent and they will replace the three innermost layers of the existing detector with new ultra-light, truly cylindrical layers. The ITS3 will improve tracking performance, especially at low transverse momentum, thanks to better track impact-parameter resolution, improved by a factor two with respect to the present ITS in LHC Run 3. In addition, the detector will be closer to the interaction point and will have a much lower material budget, of 0.05% X0/layer. This proceedings paper presents the final configuration and structure of the ITS3, the challenges related to its design and construction, and the results of the current R&D program on the sensor design and characterization. Finally, it also reports the results of beam tests preformed at CERN, where an analog pixel test chip (APTS Source Follower) was investigated in terms of detection efficiency.

Symmetric Color Image Encryption Using a Novel Cross–Plane Joint Scrambling–Diffusion Method

Motivated by the pursuit of both encryption efficiency and security, this paper presents a symmetric color image encryption algorithm using a novel cross–plane joint scrambling–diffusion method. To provide high–strength security, the color planes are firstly spliced horizontally and then vertically to be adequately encrypted by the joint scrambling–diffusion method. Therefore, minor changes in any pixel can cross every color plane that significantly influences the final cipher image. To achieve fast encryption speed, all employed chaotic systems have simple structures but robust properties to rapidly generate high–quality pseudorandom sequences. Additionally, the joint scrambling–diffusion method is vectorized to handle pixels in parallel for satisfactory execution efficiency. Security tests demonstrate the outstanding security performance of the proposed algorithm, with correlation coefficients and entropies of cipher images being close to 0 and greater than 7.99, respectively, and results of NPCR and UACI tests being similar to ideal cipher images, which can resist statistical, differential, exhaustive, and even chosen ciphertext/plaintext attacks. Moreover, because of the O(4W + 4H) time complexity of the novel scrambling–diffusion method, even a color image with a size of 1024 × 1024 only costs 0.26 s to be encrypted. Hence, the proposed algorithm can satisfy the efficiency and security requirements of color image encryption.

Construction of a New 2D Hyperchaotic Map with Application in Efficient Pseudo-Random Number Generator Design and Color Image Encryption

This paper proposes a new two-dimensional discrete hyperchaotic system and utilizes it to design a pseudo-random number generator (PRNG) and an efficient color image encryption algorithm. This hyperchaotic system has very complex dynamic properties and can generate highly random chaotic sequences. The complex hyperchaotic characteristics of the system are confirmed via bifurcation diagram, chaotic attractor, Lyapunov exponents, correlation analysis, approximate entropy and permutation entropy. Compared with some traditional discrete chaotic systems, the new chaotic system has a larger range of chaotic parameters and more complex hyperchaotic characteristics, making it more suitable for application in information encryption. The proposed PRNG can generate highly random bit sequences that can fully pass all NIST testing items. The proposed color image encryption algorithm achieves cross-channel permutation and diffusion of pixels in parallel. These strategies not only greatly improve the encryption speed of color images, but also enhance the security level of cipher images. The simulation experiments and security analysis results show that the algorithm has strong robustness against differential attacks, statistical attacks and interference attacks, and has good application potential in real-time secure communication applications of color images.

Thin monolithic pixel sensors with fast operational amplifier output in a 65 nm imaging technology for ALICE ITS3

This work presents results on the Analog Pixel Test Structure (APTS), a 4 × 4 pixel matrix prototype equipped with an output buffer based on the fast individual operational amplifier (OPAMP) of analogue pixel signals to output pads for exploration of pixel timing performance. This work was part of the ALICE ITS3 upgrade and the CERN-EP R&D on monolithic sensors of the TPSCo 65-nm imaging technology. This upgrade will replace the inner layers of the ALICE Inner Tracking System at CERN with ultra-thin flexible wafer-scale monolithic silicon sensors. They will improve the material budget in this region, the tracking precision and the efficiency at low transverse momentum. This paper will show the gain of the signal chain in the APTS and its speed as a function of the configuration parameters, including calibration results with a 55Fe source. Two different pixel structures will be compared to demonstrate the possibility of enhancing the performance in terms of timing and charge collection efficiency by implant modifications in the epitaxial layer. Finally, test beam results planned at the Super Proton Synchrotron at CERN will provide full spatial and time resolution of the APTS OPAMP structures.

DIET-SNN: A Low-Latency Spiking Neural Network With Direct Input Encoding and Leakage and Threshold Optimization.

Bioinspired spiking neural networks (SNNs), operating with asynchronous binary signals (or spikes) distributed over time, can potentially lead to greater computational efficiency on event-driven hardware. The state-of-the-art SNNs suffer from high inference latency, resulting from inefficient input encoding and suboptimal settings of the neuron parameters (firing threshold and membrane leak). We propose DIET-SNN, a low-latency deep spiking network trained with gradient descent to optimize the membrane leak and the firing threshold along with other network parameters (weights). The membrane leak and threshold of each layer are optimized with end-to-end backpropagation to achieve competitive accuracy at reduced latency. The input layer directly processes the analog pixel values of an image without converting it to spike train. The first convolutional layer converts analog inputs into spikes where leaky-integrate-and-fire (LIF) neurons integrate the weighted inputs and generate an output spike when the membrane potential crosses the trained firing threshold. The trained membrane leak selectively attenuates the membrane potential, which increases activation sparsity in the network. The reduced latency combined with high activation sparsity provides massive improvements in computational efficiency. We evaluate DIET-SNN on image classification tasks from CIFAR and ImageNet datasets on VGG and ResNet architectures. We achieve top-1 accuracy of 69% with five timesteps (inference latency) on the ImageNet dataset with 12x less compute energy than an equivalent standard artificial neural network (ANN). In addition, DIET-SNN performs 20-500x faster inference compared to other state-of-the-art SNN models.

Electrically Wavelength‐Controllable Color Filters with High Optical Transmittance Using Heterogeneous Chiral Liquid Crystals

AbstractOptical signals necessitate using filters that selectively transmit or reflect light with the desired wavelength information. In visual imaging‐related devices, a color filter acts as a wavelength selector to extract certain information from the incident light. To overcome the limitations of commercial pigmented color filters, structural color filters with controllable photonic bandgaps (PBGs) have recently attracted significant interest. Especially, chiral liquid crystal (CLC) has been widely investigated for tunable color filters owing to its selective reflection. However, most of the research has been limited to the control of conventional reflection rather than transmission wavelength. This study proposes an electrically wavelength‐tunable color filter with high optical transmittance using a heterogeneous CLC with extremely small voltages not exceeding 1.4 V. Electrothermal tuning of CLC is approached so that transmission wavelength of the device can be tuned among red, green, and blue. Moreover, analog pixel binning using CLC color filters is explored in a tetra‐layout beyond state‐of‐the‐art pixel binning with post‐capture digital image processing. Dynamic and direct binning can prevent the quality degradation of the images captured under low‐light illumination in image sensors. The wavelength‐tunable transmission color filter proposed in this study is expected to facilitate future electro‐optical dynamic color filter applications.

Pixel Variation Characteristics of a Global Shutter THz Imager and its Calibration Technique

We have developed a Si-CMOS terahertz image sensor to address the paucity of low-cost terahertz detectors. Our imaging pixel directly connects to a VCO-based ADC and achieves pixel parallel ADC architecture for high-speed global shutter THz imaging. In this paper, we propose a digital calibration technique for offset and gain variation of each pixel using global shutter operation. The calibration technique gives reference signal to all pixels simultaneously and takes reference frames as a part of the high-speed image captures. Using this technique, we achieve offset/non-linear gain variation suppression of 85.7% compared to without correction.

Straight-Line Detection Within 1 Millisecond Per Frame for Ultrahigh-Speed Industrial Automation

Detecting straight lines in video plays a fundamental role in camera-based industrial automation. With the increasing demands on production efficiency, detection speed becomes one of the bottlenecks for highly-efficient industrial automation. Because of data dependency and hardware limitation, existing vision systems based on CPU/GPU are unable to detect straight lines at ultra-high speed. This paper addresses this problem and proposes a hardware-friendly Hough transform that can be implemented in fully parallel for ultra-high-speed detection, because of two key features: 1) It processes multiple pixels in parallel and directly calculates line parameters while capturing the current frame; 2) it simultaneously initializes Hough parameter space and votes in Hough parameter space without any delay. Based on the proposed hardware-friendly Hough transform, its chip-level implementation and system-level hardware design are presented. Experimental results show that the main benefits of the proposed architecture are in real-time performances of high frame rate (784 FPS) and ultra-low delay (0.7749 ms/frame).

Reading Pixels Parallel and Processing Pixels Parallel (RPP-PPP): An Approach to Detecting and Tracking Objects in Static and Dynamic camera

Object detection and Tracking are important tasks in both moving and static camera applications like unmanned vehicles, human-assisted vehicles, product parts inspection, video surveillance, and many more. These applications are required to detect moving objects and track static and dynamic cameras. It also needed to process these tasks with rate and lesser time. Many computer vision applications are always used to read the image/frame pixels one by one from row by column or column by row pattern. Reading pixels one by one from the image/frame took more time in real-time applications. In this proposed method we used to read the pixels from the entire image/frame in parallel using Parallel. For utilities and process these pixels in parallel. In the proposed work we used the background subtraction model for object detection and tracking and noted the difference in both sequential and parallel processes and compared the output with the existing background subtraction model. In this work analyzed the time taken to read the entire image using various tools and languages. The work was implemented using C#.NET parallel. For other thread-based functions to minimize the pixel reading time and processing time for object detection and tracking. Finally, the proposed work gave a considerable performance and compared with the existing system.

ACE-SNN: Algorithm-Hardware Co-design of Energy-Efficient & Low-Latency Deep Spiking Neural Networks for 3D Image Recognition.

High-quality 3D image recognition is an important component of many vision and robotics systems. However, the accurate processing of these images requires the use of compute-expensive 3D Convolutional Neural Networks (CNNs). To address this challenge, we propose the use of Spiking Neural Networks (SNNs) that are generated from iso-architecture CNNs and trained with quantization-aware gradient descent to optimize their weights, membrane leak, and firing thresholds. During both training and inference, the analog pixel values of a 3D image are directly applied to the input layer of the SNN without the need to convert to a spike-train. This significantly reduces the training and inference latency and results in high degree of activation sparsity, which yields significant improvements in computational efficiency. However, this introduces energy-hungry digital multiplications in the first layer of our models, which we propose to mitigate using a processing-in-memory (PIM) architecture. To evaluate our proposal, we propose a 3D and a 3D/2D hybrid SNN-compatible convolutional architecture and choose hyperspectral imaging (HSI) as an application for 3D image recognition. We achieve overall test accuracy of 98.68, 99.50, and 97.95% with 5 time steps (inference latency) and 6-bit weight quantization on the Indian Pines, Pavia University, and Salinas Scene datasets, respectively. In particular, our models implemented using standard digital hardware achieved accuracies similar to state-of-the-art (SOTA) with ~560.6× and ~44.8× less average energy than an iso-architecture full-precision and 6-bit quantized CNN, respectively. Adopting the PIM architecture in the first layer, further improves the average energy, delay, and energy-delay-product (EDP) by 30, 7, and 38%, respectively.

Efficiency and time resolution of monolithic silicon pixel detectors in SiGe BiCMOS technology

A monolithic silicon pixel detector prototype has been produced in the SiGe BiCMOS SG13G2 130 nm node technology by IHP. The ASIC contains a matrix of hexagonal pixels with pitch of approximately 100 μm. Three analog pixels were calibrated in laboratory with radioactive sources and tested in a 180 GeV/c pion beamline at the CERN SPS. A detection efficiency of (99.9-0.2 +0.1)% was measured together with a time resolution of (36.4 ± 0.8) ps at the highest preamplifier bias current working point of 150 μA and at a sensor bias voltage of 160 V. The ASIC was also characterized at lower bias voltage and preamplifier current.

Interaction-free imaging of multipixel objects

Quantum imaging, one of the pillars of quantum technologies, is well-suited to study sensitive samples which require low-light conditions, like biological tissues. In this context, interaction-free measurements (IFM) allow us infer the presence of an opaque object without the photon interacting with the sample. Current IFM schemes are designed for single-pixel objects, while real-life samples are structured, multi-pixel objects. Here we extend the IFM imaging schemes to multi-pixel, semi-transparent objects, by encoding the information about the pixels into an internal degree of freedom, namely orbital angular momentum (OAM). This allows us to image the pixels in parallel. Our solution exhibits a better theoretical efficiency than the single-pixel case. Our scheme can be extended to other degrees of freedom, like the photon radial quantum number, in order to image 1D and 2D objects.

Quantification of protein-protein interactions and activation dynamics: A new path to predictive biomarkers

Oncogene dysregulation is a driver of neoplasia development and progression. The use of quantitative molecular imaging to quantify oncogene activation will be crucial in developing companion diagnostics which can identify personalised patient regimens. However, the evaluation of oncogene activation does not necessarily correlate with oncoprotein activation. Post-translational modifications, such as phosphorylation, lipidation and methylation, may enhance oncoprotein functionality. It is this functionality that progresses neoplasia and may correlate with patient outcome. Advanced molecular imaging may be used to directly quantify oncoprotein, as opposed to oncogene, activation. Time-resolved Förster Resonance Energy Transfer (TR-FRET) involves the non-radiative transfer of energy from one chromophore to another over distances of 1-10 nm; allowing FRET to be used as a “chemical ruler”. TR-FRET can be utilised to directly elucidate spatial oncoprotein activation in single cells and patient tissues. In single cells, TR-FRET has uncovered the mechanisms by which PKCβ1 is trafficked to the nucleus and cleaved. Additionally it has revealed the mechanism of activation of Akt/PKB, whereby Akt/PKB undergoes a conformational change, allowing the Thr308 site to be phosphorylated by PDK1. Moreover TR-FRET has been utilised to quantify HER2-HER heterodimerisation and Akt/PKB activation states in patient biopsies, where it is shown to be predictive of outcome/relapse. The role of TR-FRET is not solely limited to intracellular signalling events. A study has used TR-FRET to measure intercellular immune-checkpoint receptor-ligand interactions. Within this study it was seen that PD-L1 expression was not indicative of PD-1/PD-L1 interaction states in a range of solid tumours. Crucially, in melanoma and NSCLC, PD-1/PD-L1 interaction was a predictive of an improved patient outcome. PD-L1 expression did not predict patient outcome. Several groups have worked to improve Fluorescence lifetime imaging microscopy (FLIM) acquisition times, including the use of: window-galvanometers; multifocal multiphoton FLIM and parallel pixel excitation coupled with wide-field time-gated FLIM. The development of novel quantitative molecular imaging will be critical in the development of personalised patient therapies in the future.