Image Sensor System Research Articles

Estimating Power, Performance, and Area for On-Sensor Deployment of AR/VR Workloads Using an Analytical Framework

Augmented Reality and Virtual Reality have emerged as the next frontier of intelligent image sensors and computer systems. In these systems, 3D die stacking stands out as a compelling solution, enabling in situ processing capability of the sensory data for tasks such as image classification and object detection at low power, low latency, and a small form factor. These intelligent 3D CMOS Image Sensor (CIS) systems present a wide design space, encompassing multiple domains (e.g., computer vision algorithms, circuit design, system architecture, and semiconductor technology, including 3D stacking) that have not been explored in-depth so far. This article aims to fill this gap. We first present an analytical evaluation framework, STAR-3DSim, dedicated to rapid pre-RTL evaluation of 3D-CIS systems capturing the entire stack from the pixel layer to the on-sensor processor layer. With STAR-3DSim, we then propose several knobs for PPA (power, performance, area) improvement of the Deep Neural Network (DNN) accelerator that can provide up to 53%, 41%, and 63% reduction in energy, latency, and area, respectively, across a broad set of relevant AR/VR workloads. Last, we present full-system evaluation results by taking image sensing, cross-tier data transfer, and off-sensor communication into consideration.

Hyperspectral Rock Classification Method Based on Spatial-Spectral Multidimensional Feature Fusion

The issues of the same material with different spectra and the same spectra for different materials pose challenges in hyperspectral rock classification. This paper proposes a multidimensional feature network based on 2-D convolutional neural networks (2-D CNNs) and recurrent neural networks (RNNs) for achieving deep combined extraction and fusion of spatial information, such as the rock shape and texture, with spectral information. Experiments are conducted on a hyperspectral rock image dataset obtained by scanning 81 common igneous and metamorphic rock samples using the HySpex hyperspectral sensor imaging system to validate the effectiveness of the proposed network model. The results show that the model achieved an overall classification accuracy of 97.925% and an average classification accuracy of 97.956% on this dataset, surpassing the performances of existing models in the field of rock classification.

Thermally stable photosensing using poly(methyl methacrylate)-coated MoS2 phototransistor for improved imaging reliability

In this paper, we report thermally stable photosensing using MoS2 phototransistor with a poly(methyl methacrylate) (PMMA) coating. The increase in the OFF current of the PMMA-coated MoS2 phototransistor degraded to less than 87.72% of that of the pristine MoS2 phototransistor under harsh temperature conditions (250 °C). PMMA coating on the pristine MoS2 phototransistor improved the photosensitivity and drain current stability as a function of time by 315.71% at 250 °C and 91.26% under intense negative bias temperature illumination stress (NBTIS) test (V gs = −30 V, V ds = 10 V, λ ex = 638 nm, P inc = 1.0 mW, and T = 250 °C), respectively. This simple and useful method provides valuable insight for improving the reliability of photodetectors and image sensor systems under harsh temperature.

Particle Tracking of Orientation and Velocity Of Arbitrary Shape

Dispersions or particulate systems are prevalent in numerous natural and practical applications. Characterisation of such multiphase systems usually involves measuring size, spatial location, velocity and number density, which is usually non-trivial. The 'Depth from Defocus' (DFD) technique provides an imaging-based volumetric approach for estimating the size and position of dispersed spherical particles using the blurring information from a shadowgraph image. The two-image DFD approach involves two cameras to capture simultaneous images at different degrees of blur, which, although simpler, requires a mandatory calibration procedure. This study proposes a single-image DFD technique that provides a calibration-free size estimation using theoretical functions while evaluating position requires a simple calibration. The efficacy of this technique is demonstrated for various sparse spherical dispersions, including target dots, dispersed glass beads, liquid droplets in sprays and surface bubble rupture, and pollen grains. The method further enables precise determination of the measurement volume, which is crucial, allowing for bias-free estimates of the size distribution and volume concentration. The simple optical configuration and semi-automatic calibration procedure make this method easily deployable and accessible for diverse applications. The pixelation effect on the limiting value of the degree of blur is also estimated theoretically and compared with experiments. This effect is of interest concerning any typical discrete sensor imaging system in general.

Systematic analysis of hyperspectral imaging and intelligent sensor systems in multiscale agriculture

This article delves into the systematic analysis of Hyperspectral Imaging Technology (HSI) and Intelligent Sensor Systems in the context of multiscale agriculture, showcasing their critical role in enhancing efficiency and sustainability in agricultural production amid the challenges posed by global population growth. It highlights the pressing need to optimize agricultural processes through advanced technologies to tackle issues such as resource scarcity and environmental pollution. By presenting case studies, the article illustrates the effective integration of HSI and intelligent sensors in key agricultural processessoil analysis, crop monitoring, and pest detectionunderscoring their significance in advancing precision agriculture. The discussion extends to the potentials of data fusion and decision support systems in elevating crop yield and quality. Concluding, the paper emphasizes that despite facing hurdles like technical barriers and maintenance costs, the application of these technologies not only boosts production efficiency and precision but also contributes to agricultural sustainability, underlining the importance of continued research and innovation for a more sustainable and efficient agricultural future.

Characterization of Receptor Binding Affinity for Vascular Endothelial Growth Factor with Interferometric Imaging Sensor.

Wet Age-related macular degeneration (AMD) is the leading cause of vision loss in industrialized nations, often resulting in blindness. Biologics, therapeutic agents derived from biological sources, have been effective in AMD, albeit at a high cost. Due to the high cost of AMD treatment, it is critical to determine the binding affinity of biologics to ensure their efficacy and make quantitative comparisons between different drugs. This study evaluates the in vitro VEGF binding affinity of two drugs used for treating wet AMD, monoclonal antibody-based bevacizumab and fusion protein-based aflibercept, performing quantitative binding measurements on an Interferometric Reflectance Imaging Sensor (IRIS) system. Both biologics can inhibit Vascular Endothelial Growth Factor (VEGF). For comparison, the therapeutic molecules were immobilized on to the same support in a microarray format, and their real-time binding interactions with recombinant human VEGF (rhVEGF) were measured using an IRIS. The results indicated that aflibercept exhibited a higher binding affinity to VEGF than bevacizumab, consistent with previous studies using ELISA and SPR. The IRIS system's innovative and cost-effective features, such as silicon-based semiconductor chips for enhanced signal detection and multiplexed analysis capability, offer new prospects in sensor technologies. These attributes make IRISs a promising tool for future applications in the development of therapeutic agents, specifically biologics.

Simultaneous visualization of multiple wavelengths by filter-free wavelength imaging system

This paper demonstrates the simultaneous imaging of multiple wavelengths through realizing a filter-free wavelength image sensor and imaging system. The sensor can identify the center wavelength of the spectrum based on the current ratio using the optical absorption characteristics of silicon at different wavelengths and the potential peak formed at depth. A wavelength imaging system was fabricated and calibrated by evaluating the light intensity and wavelength characteristics of each pixel. First, an experiment was conducted assuming that fluorescence was emitted in part of the pixel region of the image sensor. The system successfully identified different fluorescence wavelengths only in the region irradiated by light corresponding to the fluorescence using the current ratio. Finally, experiments were conducted using fluorescent beads on actual cells and pathogen clusters. Red and green fluorescent beads were placed separately on the sensor, and we succeeded in simultaneously imaging the wavelengths of the three regions where only the excitation light and different fluorescence signals were emitted. These results suggest that multiplexed fluorescence imaging, which considers the interactions between sites or targets, can provide new insights into the biomedical

Classification of Potential Wetlands using the Random Forest in Google Earth Engine in Geomorphological Units - Rio Grande do Sul, Brazil

Wetlands are important and valuable ecosystems in the landscape of the extreme south of Brazil. However, they are among the ecosystems threatened by human pressures and climate change. Monitoring and managing these environments is a challenge due to their high spatial and temporal dynamics. The use of remote sensing techniques, supervised classification, and machine learning algorithms offers a promising opportunity to map and monitor wetlands. The objective of this work is to develop a method to map Potential Wetlands (PW) from the integration of images from different satellites, sensor systems, spectral, topographic, climatological and hydrological features, made available on the Google Earth Engine (GEE) cloud user platform. Supervised classification was performed on the geomorphological units Coastal Plain (CP) and Central Depression (CD) considering two classes: Potential Wetlands, which encompasses all types of wetlands, and No Wetlands, for the remaining land use and occupation classes. The supervised pixel-by-pixel classification, individual for each geomorphological unit, allowed more accurate results consistent with previous, consolidated mapping. The PW were mapped with a global accuracy higher than 88% and consumer and producer accuracy higher than 81% in the Coastal Plain and Central Depression. These results allow us to affirm that the proposed methodology enabled the identification of 22% to 24% increase in potential wetlands in geomorphological regions, from spectral signatures, and pixel-by-pixel supervised classification of PW using different image collections and sensor systems.

Realization of a filter-free wavelength image sensor and imaging system for visualization of wavelength information

In this study, we fabricated a filter-free wavelength image sensor for the two-dimensional visualization of wavelength information, in addition to light intensity, and realized an imaging system. In this sensor, the electrons generated in silicon in different distributions at each wavelength are divided into two parts by the potential peaks. The wavelength can be identified from the current ratio, and the light intensity can be determined from the total current. In the proposed structure, the pixel current is sequentially output by controlling the rows and columns. The layout was designed based on the proposed structure and was fabricated at the Toyohashi University of Technology LSI factory. An imaging system was constructed using operational amplifiers, A/D converters, and a microcontroller. The ratio and sum of the two signals obtained from the sensor responded linearly to the wavelength and light intensity. The wavelength response was independent of changes in the light intensity. Even when the spot was narrowed down, the response was obtained only at the corresponding pixel and succeeded in the simultaneous imaging of the intensity and wavelength. The above results can be applied to cell analysis, such as multiple fluorescence staining, where multiple fluorescence signals are detected simultaneously.

Microwave imaging sensor system using metamaterial lens for subwavelength resolution

Lenses made of negative refractive index metamaterials can overcome the diffraction-limited resolution of imaging at working distances in the far-field. This paper describes a novel negative index metamaterial (NIM) lens imaging sensor system for subwavelength microwave nondestructive evaluation (NDE) using homodyne detection measurements. The split-ring resonator (SRR) - wire based NIM lens is optimized to work at 6.3 GHz (wavelength λ = 48 mm). Coherent homodyne detection scheme provides a simple, low-cost, and highly sensitive NIM lens imaging sensor system that can be used in the field under practical conditions. Imaging of an omnidirectional source using negative refraction is shown using the proposed system. A focal spot of size 0.62λ was obtained in the image plane of the metamaterial lens with a source to image distance of 1.67λ. The focal spot position was further shifted in the azimuthal plane by simple movement of the source. The study demonstrates unique focus-scanning capability of NIM lenses for target localization without the need for complex tuning mechanisms. The feasibility of the system to image subwavelength defects inside dielectric materials in reflection mode is also investigated for its potential application as an NDE sensor. Defects comprising subwavelength hole of diameter 0.25λ and a groove of dimensions 0.31λ x 0.1λ placed at the focal plane of the lens was imaged using the proposed sensor system.

A Novel Imaging Method for Two-Tap Pulsed-Based Indirect Time-of-Flight Sensor

This article proposes an imaging method for two-tap pulsed-based indirect time-of-flight (I-ToF) sensor, which aims to reduce the distance error and motion artifact. Compared with the conventional imaging method, the proposed imaging method adopts emitted pulse light whose width is halved and height is doubled without changing the CMOS image sensor (CIS) system. Then, by using the distance-solving method that cooperates with this timing, it achieves depth imaging with the same range of distance measurement as the conventional method. The validity of the proposed method is demonstrated by the pixel-array behavior-level model based on the actual I-ToF imaging system. The experimental results show that compared with the conventional method, the distance error and motion artifact are reduced by 29.2% and 67.4%, respectively, under 350-mW/sr light source radiation intensity, 10-klx background illumination, and 500- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mu \text{s}$ </tex-math></inline-formula> integration time. When the above parameters are changed, the proposed method still achieves about 29.3% reduction in distance error and 67.4% reduction in motion artifact. The proposed method is compatible with other methods to further reduce the distance error and motion artifact.

Acid-Base Reaction-Assisted Quantum Dot Patterning via Ligand Engineering and Photolithography.

The integration of quantum dots (QDs) into device arrays for high-resolution display and imaging sensor systems remains a significant challenge in research and industry because of issues associated with the QD patterning process. It is difficult for conventional patterning processes such as stamping, inkjet printing, and photolithography to employ QDs and fabricate high-resolution patterns without degrading the properties of QDs. Here, we introduce a novel strategy for the QD patterning process by treating QDs with a bifunctional ligand for acid-base reaction-assisted photolithography. Bifunctional ligands, such as MPA (mercaptopropionic acid) or TGA (thioglycolic acid), have a carboxyl group on one side that allows the QDs to be etched along with the photoresist (PR) by the base developer, while on the opposite side the ligands have a thiol group that passivates the QD surface. Passivating MPA ligands on QDs facilitates patterning of QD films and makes them compatible with harsh photolithography processes. We successfully achieved the patterning of QDs down to 5 μm. We also fabricated high-resolution patterned QD light-emitting diodes (LEDs) and QD photodetector arrays. Our patterning process provides precise control for the fabrication of highly integrated QD-based optoelectronic devices.

34.3: High Resolution and High Speed Inspection Equipment for Mini‐LED Substrates

A new mini LED substrate inspection equipment using a non‐connected high‐precision proximity capacitance image sensor system named as “EPIS” (Electrical Picture Inspection System) with one‐unit pixel size of 12μm, the number of pixels of 1080 x 1080 and capacitance detection accuracy of 0.1aF has been developed. In this paper, detection results of a mini LED substrate using EPIS and the equipment are discussed.

HIERARCHICAL AERIAL TRIANGULATION OF OBLIQUE IMAGE DATA

Abstract. Oblique imaging sensor systems are becoming a standard for photogrammetric applications such as city modeling, with increasing need for frequently updated geospatial data in urban areas. The ground processing software has to consider and support this develop-ment, providing a highly performant workflow for projects that can consist of tens or hundreds of thousands of images in different view angles: nadir (vertical), left, right, backward, forward. This is a challenge also for the oblique aerial triangulation (AT), both the large number of images to process and the high amount of overlap in-between them, where more than 50 image measurements for a given ground point are common.At the same time, the large overlap even in a single view generally provides stable sub-blocks for each viewing direction and, therefore, allows for separating the oblique AT by view and performing a hierarchical approach: The nadir block is triangulated first, updating the exterior orientation parameters of all nadir images and providing adjusted ground points. This nadir reference can be used to tie the oblique views’ sub-blocks in subsequent AT runs to eventually provide adjusted exterior orientations for all images, with comparable accuracy to an integrated AT – but providing a significant performance gain. The hierarchical AT is a three-step approach: 1. adjust-ment of the nadir imagery, 2. adjustment of each oblique views independently by cardinal direction (East, West, North, South) while tying it to the nadir reference, and 3. computation of combined statistics and report.The paper details our AT approach and evaluates its parameterization for different oblique data sets, comparing results and performance to a fully integrated AT.

A Lightweight Remote Sensing Image Super-Resolution Method and Its Application in Smart Cities

With the growth of urban population, a series of urban problems have emerged, and how to speed up smart city construction has received extensive attention. Remote sensing images have the advantages of wide spatial coverage and rich information, and it is suitable for use as research data for smart cities. However, due to limitations in the imaging sensor conditions and complex weather, remote sensing images face the problems of insufficient resolution and cloud occlusion, which cannot meet the resolution requirements of smart city tasks. The remote sensing image super-resolution (SR) technique can improve the details and texture information without upgrading the imaging sensor system, which becomes a feasible solution for the above problems. In this paper, we propose a novel remote sensing image super-resolution method which leverages the texture features from internal and external references to help with SR reconstruction. We introduce the transformer attention mechanism to select and extract parts of texture features with high reference values to ensure that the network is lightweight, effective, and easier to deploy on edge computing devices. In addition, our network can automatically learn and adjust the alignment angles and scales of texture features for better SR results. Extensive comparison experiments show that our proposed method achieves superior performance compared with several state-of-the-art SR methods. In addition, we also evaluate the application value of our proposed SR method in urban region function recognition in smart cities. The dataset used in this task is low-quality. The comparative experiment between the original dataset and the SR dataset generated by our proposed SR method indicates that our method can effectively improve the recognition accuracy.

A Fast Image Sensor System with an Efficient Multi-Scale Gaussian Filtering Circuit

A Fast Image Sensor System with an Efficient Multi-Scale Gaussian Filtering Circuit

Sensitive and real-time detection of IgG using interferometric reflecting imaging sensor system

Considering the limitations of well-known traditional detection techniques, innovative research studies have focused on the development of new sensors to offer label-free, highly sensitive, real-time, low-cost, and rapid detection for biomolecular interactions. In this study, we demonstrate immunoglobulin G (IgG) detection in aqueous solutions by using real-time and label-free kinetic measurements of the Interferometric Reflectance Imaging Sensor (IRIS) system. By performing kinetic characterization experiments, the sensor's performance is comprehensively evaluated and a high correlation coefficient value (>0.94) is obtained in the IgG concentration range of 1–50 μg/mL with a low detection limit (0.25 μg/mL or 1.67 nM). Moreover, the highly sensitive imaging system ensures accurate quantification and reliable validation of recorded binding events, offering new perspectives in terms of direct biomarker detection for clinical applications.

A Novel, Automated, and Real-Time Method for the Analysis of Non-Human Primate Behavioral Patterns Using a Depth Image Sensor

By virtue of their upright locomotion, similar to that of humans, motion analysis of non-human primates has been widely used in order to better understand musculoskeletal biomechanics and neuroscience problems. Given the difficulty of conducting a marker-based infrared optical tracking system for the behavior analysis of primates, a 2-dimensional (D) video analysis has been applied. Distinct from a conventional marker-based optical tracking system, a depth image sensor system provides 3-D information on movement without any skin markers. The specific aim of this study was to develop a novel algorithm to analyze the behavioral patterns of non-human primates in a home cage using a depth image sensor. The behavioral patterns of nine monkeys in their home cage, including sitting, standing, and pacing, were captured using a depth image sensor. Thereafter, these were analyzed by observers’ manual assessment and the newly written automated program. We confirmed that the measurement results from the observers’ manual assessments and the automated program with depth image analysis were statistically identical.

Color-Ratio Maps Enhanced Optical Filter Design and Its Application in Green Pepper Segmentation.

There is a growing demand for developing image sensor systems to aid fruit and vegetable harvesting, and crop growth prediction in precision agriculture. In this paper, we present an end-to-end optimization approach for the simultaneous design of optical filters and green pepper segmentation neural networks. Our optimization method modeled the optical filter as one learnable neural network layer and attached it to the subsequent camera spectral response (CSR) layer and segmentation neural network for green pepper segmentation. We used not only the standard red–green–blue output from the CSR layer but also the color-ratio maps as additional cues in the visible wavelength and to augment the feature maps as the input for segmentation. We evaluated how well our proposed color-ratio maps enhanced optical filter design methods in our collected dataset. We find that our proposed method can yield a better performance than both an optical filter RGB system without color-ratio maps and a raw RGB camera (without an optical filter) system. The proposed learning-based framework can potentially build better image sensor systems for green pepper segmentation.

Design and Implementation of Artificial Intelligence Image Detection System Based on Noise Monitoring

Image quality detection technology can replace manual inspections to automatically detect video quality, accurately analyze, judge and alarm abnormalities in video images in the surveillance system to ensure the normal operation of the expanding network video surveillance system. The thesis designs an intelligent traffic image sensor system to realize fast motion detection and object recognition of the surveillance scene. This system introduces a background model based on region segmentation and a feature-based object recognition algorithm. Experimental results show that the system can perform automatic movement detection and object classification and recognition efficiently in real time.