Artificial Vision System Research Articles

Artificial Visual Systems Fabricated with Ferroelectric van der Waals Heterostructure for In-Memory Computing Applications.

Rapid developments in artificial neural network techniques and retina-inspired artificial visual systems are required to realize the sensing, processing, and memorization of an optical signal in a single device. Herein, a ferroelectric field-effect transistor fabricated with CuInP2S6 and α-In2Se3 van der Waals heterostructures is proposed and demonstrated for the development of an artificial visual system. The dipole polarizations are coupled and bidirectionally locked inside the ferroelectric α-In2Se3 along the in-plane and out-of-plane directions and are controlled by the gate voltages. Furthermore, light-induced polarization can change the order of polarization of the dipoles inside α-In2Se3. We demonstrate that using the combined control of these electrical and optical signals, the device may function like a retina-inspired vision system. The device can operate across a wide wavelength range (405-850 nm) and detect very low incident light (0.03 mW/cm2). Color recognition, high paired-pulse facilitation (∼170%), and short- to long-term memory transitions through quick learning are observed using this device. Additionally, this device demonstrates different complex processing abilities, including pattern recognition, light adaptation, optical logic operation, and event learning. The proposed ferroelectric heterostructure-based artificial visual system can serve as an essential bridge for fulfilling the future requirements of all-in-one sensing and memory-processing devices.

Neuro-inspired optical sensor array for high-accuracy static image recognition and dynamic trace extraction

Neuro-inspired vision systems hold great promise to address the growing demands of mass data processing for edge computing, a distributed framework that brings computation and data storage closer to the sources of data. In addition to the capability of static image sensing and processing, the hardware implementation of a neuro-inspired vision system also requires the fulfilment of detecting and recognizing moving targets. Here, we demonstrated a neuro-inspired optical sensor based on two-dimensional NbS2/MoS2 hybrid films, which featured remarkable photo-induced conductance plasticity and low electrical energy consumption. A neuro-inspired optical sensor array with 10 × 10 NbS2/MoS2 phototransistors enabled highly integrated functions of sensing, memory, and contrast enhancement capabilities for static images, which benefits convolutional neural network (CNN) with a high image recognition accuracy. More importantly, in-sensor trajectory registration of moving light spots was experimentally implemented such that the post-processing could yield a high restoration accuracy. Our neuro-inspired optical sensor array could provide a fascinating platform for the implementation of high-performance artificial vision systems.

Development of a Low-Cost Artificial Vision System as an Alternative for the Automatic Classification of Persian Lemon: Prototype Test Simulation.

In the present research work, an algorithm of artificial neural network (ANN) has been developed based on the processing of digital images of Persian lemons with the aim of optimizing the quality control of the product. For this purpose, the physical properties (weight, thickness of the peel, diameter, length, and color) of 90 lemons selected from the company Esperanza de San José Ornelas SPR de RL (Jalisco, Mexico) were studied, which were divided into three groups (Category "extra", Category I, and Category II) according to their characteristics. The parameters of weight (26.50 ± 3.00 g), diameter/length (0.92 ± 0.08) and thickness of the peel (1.50 ± 0.29 mm) did not present significant differences between groups. On the other hand, the color (determined by the RGB and HSV models) presents statistically significant changes between groups. Due to the above, the proposed ANN correctly classifies 96.60% of the data obtained for each of the groups studied. Once the ANN was trained, its application was tested in an automatic classification process. For this purpose, a prototype based on the operation of a stepper motor was simulated using Simulink from Matlab, which is connected to three ideal switches powered by three variable pulse generators that receive the information from an ANN and provide the corresponding signal for the motor to turn to a specific position. Manual classification is a process that requires expert personnel and is prone to human error. The scientific development presented shows an alternative for the automation of the process using low-cost computational tools as a potential alternative.

Towards Artificial Visual Sensory System: Organic Optoelectronic Synaptic Materials and Devices

AbstractDeveloping an artificial visual sensory system requires optoelectronic materials and devices that can mimic the behavior of biological synapses. Organic/polymeric semiconductors have emerged as promising candidates for optoelectronic synapses due to their tunable optoelectronic properties, mechanic flexibility, and biological compatibility. In this review, we discuss the recent progress in organic optoelectronic synaptic materials and devices, including their design principles, working mechanisms, and applications. We also highlight the challenges and opportunities in this field and provide insights into potential applications of these materials and devices in next‐generation artificial visual systems. By leveraging the advances in organic optoelectronic materials and devices, we can envision its future development in artificial intelligence.

Towards Artificial Visual Sensory System: Organic Optoelectronic Synaptic Materials and Devices.

Developing an artificial visual sensory system requires optoelectronic materials and devices that can mimic the behavior of biological synapses. Organic/polymeric semiconductors have emerged as promising candidates for optoelectronic synapses due to their tunable optoelectronic properties, mechanic flexibility, and biological compatibility. In this review, we discuss the recent progress in organic optoelectronic synaptic materials and devices, including their design principles, working mechanisms, and applications. We also highlight the challenges and opportunities in this field and provide insights into potential applications of these materials and devices in next-generation artificial visual systems. By leveraging the advances in organic optoelectronic materials and devices, we can envision its future development in artificial intelligence.

Photoelectric biomemristors for artificial visual perception systems

The redundant information generated by traditional image sensors causes a large amount of energy waste, which can no longer meet the needs of the rapid development of current artificial intelligence. Therefore, it is urgent to develop visual bionic systems with perception, memory and processing functions. Photoelectric memristors based on graphene quantum dots (GQDs) and the body fluid of Bombyx mori (BM) were fabricated, which can change its storage state when triggered by optical or electrical signals, and synaptic characteristics such as multilevel data storage, synaptic potentiation and synaptic depression were realized. The Boolean logic "OR" operation, the enhancement/inhibition of the synapse under the light signal and the double pulse dissimilation behavior are realized by UV light-driven resistance variable memory. The results show that photoelectric memristors have attractive biological applications in information perception, storage and computation and provide a potential method for the next generation of artificial visual perception systems.

Visible Light‐Sensitive Artificial Photonic Synapse

AbstractComputing based on artificial neuron networks holds great promise in overcoming the von Neumann bottleneck and significantly enhancing computing efficiency. It offers a wide range of possibilities for artificial visual system applications. Photonic synapses, as essential components, can contribute to improved efficiency in visual information processing by leveraging their low latency, high propagation speed, and broad bandwidth capabilities. Herein, a photonic transistor capable of detecting visible light and exhibiting synaptic characteristics under various visible stimuli is presented. This is achieved by integrating a photo‐sensitive polymer as a floating gate charge trapping medium. The synaptic device successfully emulates fundamental synaptic behaviors such as excitatory postsynaptic current (EPSC), pair‐pulse facilitation, the transition of short‐term memory to long‐term memory, and learning and forgetting processes. Furthermore, leveraging the ESPC responses, the informative synaptic outputs modulated by light signals enable a non‐contact writing method, paving the way for advancements in optical wireless communication. It is believed that this work serves as a promising building block to complete the bio‐inspired photonic computing paradigm.

Inhibitory artificial synapses based on photoelectric co-modulation of graphene/WSe2 van der Waals heterojunctions

Optical artificial synapses possess several advantages, including high bandwidth, strong interference immunity, and ultra-fast signal transmission, overcoming the limitations of electrically stimulated synapses. Among various functional materials, 2D materials exhibit exceptional optical and electrical properties. By utilizing van der Waals heterostructures formed by these materials through rational design, synaptic devices can mimic the information perception ability of biological systems. This lays the foundation for low-energy artificial vision systems and neuromorphic computing. This study introduces an inhibitory artificial synapse based on photoelectric co-modulation of graphene/WSe2 van der Waals heterojunctions. By synergistically applying gate voltage and light pulses, we simulate memory and logic functions observed in the brain’s visual cortex. We achieve the construction of inhibitory synapses, enabling properties such as postsynaptic current response, short-term and long-term plasticity, and paired-pulse facilitation. Additionally, we accomplish the inverse recovery of device conductivity through separate gate voltage stimulation. Through bidirectional modulation of the artificial synaptic conductance, we construct an artificial hardware neural network that achieves 92.5% accuracy in recognizing handwritten digital images from the MNIST dataset. The network also has good recognition accuracy for handwritten digital images with different standard deviation Gaussian noise applied and other datasets. Furthermore, we successfully mimic the neural behavior of aversive learning for alcohol withdrawal in alcoholic patients using the device properties. The promising capabilities of artificial synapses constructed through electrical and optical synergistic modulation make them suitable for wearable electronics and artificial vision systems.

A MAPbBr3/PdSe2 Schottky junction-based optoelectronic sensor with self-powered and switchable photocurrents

Neuromorphic optoelectronic sensors with in-sensor computing architecture hold great promise for applications that require processing large amounts of redundant data, such as the Internet of Things, robotics, and environmental sciences, due to their advantages of low time latency and energy efficiency. Halide perovskites, known for their extraordinary optoelectronic properties and stimuli-responsive characteristics, offer excellent opportunities for developing switchable visual sensors with high sensitivity, fast response, low energy consumption, and wide adaptive range. In this work, we successfully realized a MAPbBr3-PdSe2 heterojunction-based optoelectronic sensor, demonstrating a responsivity of 28 mA/W and a high specific detectivity of 5.2 × 1011 Jones. A fast response time of ∼ 25 μs has also been achieved. Additionally, we investigated the role of voltage-induced ion migration in actively adjusting the device's photoresponse capacity. Under 0 V bias, the device exhibited a wide switchable range of optical responsivity from 137.5% to 27000%, significantly surpassing pure perovskite-based devices in previous reports. Moreover, we demonstrated the device's adaptive learning capabilities and reproducible switching characteristics by simulating Pavlovian classical conditioned reflex experiments using electrical modulation. These findings open up exciting possibilities for next-generation artificial vision systems that are energy-efficient, adaptive, and capable of learning and effectively responding to varying visual conditions.

A flexible PI/graphene heterojunction optoelectronic device modulated by TENG and UV light for neuromorphic vision system

Artificial neural visual electronic devices have attracted great research interest in the fields of low-power and high-efficiency pattern recognition. However, emulating the hyperpolarized behavior of cone and rod cells and further developing image recognition functionalities encounter formidable challenges due to the inherent optoelectronic effects in semiconductors. Here, a flexible optoelectronic synapse device based on polyimide (PI)/Graphene heterojunction is proposed to effectively emulate human visual functionalities. The PI not only serves as a flexible substrate but also exhibits remarkable UV light absorption capability. Furthermore, the electrical conductivity of graphene channel can be bidirectionally regulated through UV light irradiation and triboelectric nanogenerator (TENG) induced negative gas ions adsorption, corresponding to inhibitory and excitatory behaviors, respectively. The negative photoconductance effect, stemming from the recombination between photogenerated electrons in PI and holes in graphene, endows the emulation of hyperpolarized behavior of cone and rod cells. Various synaptic behaviors, such as paired-pulse facilitation, spike-rate dependent plasticity, short-term depression, and long-term depression, can be achieved by modulating light pulses parameters. Additionally, the TENG-induced gas ions adsorbed on the graphene surface act as floating gate, leading to an increase in graphene conductivity. Continuous pulse stimulation through corona discharge enables long-term potentiation behavior in the synaptic device. Ultimately, combining opto-electrically coupled weight updates with artificial neural network algorithms allows for real-time handwritten digit recognition. This work presents a novel pathway for constructing simple flexible artificial visual electronic systems.

Color and luminance processing in V1 complex cells and artificial neural networks.

Object recognition by natural and artificial visual systems benefits from the identification of object boundaries. A useful cue for the detection of object boundaries is the superposition of luminance and color edges. To gain insight into the suitability of this cue for object recognition, we examined convolutional neural network models that had been trained to recognize objects in natural images. We focused specifically on units in the second convolutional layer whose activations are invariant to the spatial phase of a sinusoidal grating. Some of these units were tuned for a nonlinear combination of color and luminance, which is broadly consistent with a role in object boundary detection. Others were tuned for luminance alone, but very few were tuned for color alone. A literature review reveals that V1 complex cells have a similar distribution of tuning. We speculate that this pattern of sensitivity provides an efficient basis for object recognition, perhaps by mitigating the effects of lighting on luminance contrast polarity. The absence of a contrast polarity-invariant representation of chromaticity alone suggests that it is redundant with other representations.

CuInP2S6‐Based Electronic/Optoelectronic Synapse for Artificial Visual System Application

AbstractWith the development of information processing, various neuromorphic synaptic devices have been proposed, including novel devices mimicking multiple sensory systems in the biosome, in which vision is a vital source of information. Due to the pressing issue of high energy consumption and the ever‐increasing complexity of practical application scenarios, there is an urgent need to investigate optoelectronic synapses with simple structure but multifunctional capabilities, thereby broadening their application scope. Here, remarkable performances in both electrical and optical operation modes are achieved in multilayer graphene/CuInP2S6/Au electronic/optoelectronic device. By modulating the electrical and optical pulses, both short‐term and long‐term memory can be emulated in the same device, while visual perception, processing, and memorizing functions are demonstrated in this single cell with relatively low energy consumption. In addition, light adaptive behavior can also be simulated through optical–electrical cooperative modulation in the device, further providing a novel and promising strategy for future applications in artificial visual systems.

Pieces Separation Applied to a RV-2AJ Industrial Manipulator

The objective of this work is to create a system for separating types of pieces for didactic purposes. An artificial vision system applied to a manipulator was used to perform this task of classifying and separating the pieces, through the use of digital image processing. The vision system uses a camera attached to the robot's end-effector, whose objective is to identify the area, perimeter and position of the centroid of the pieces to be separated. The separation of objects is performed between two classes: with round and non-round shapes. The tests were carried out with objects such as screws and nuts. After obtaining the positions of the objects to be separated by the vision system, some characteristics of these related to the reference system of the robot base are determined. Some difficulties were observed in the recognition of objects related to the colours and lighting conditions of the separation environment. To reduce these problems, black colour objects were used in order to reduce the reflectance. Another problem encountered in the separation process is related to the relative position of objects. In this case, two or more objects can be identified as a single object, depending on their relative positions. The system classifies with a rate of hits greater than 95%.

Design of a Configurable Spike-Encoding Circuit Based on Focal Plane Array

Spiking neural networks inspired by biological models are gaining popularity in artificial intelligence due to their ability to solve diverse problems while reducing energy consumption. As a result of the trade-off between the need to transmit large amounts of data and the power consumption of hardware deployment, artificial vision systems are particularly well-suited to construction using spiking neural networks (SNNs). How to communicate with the neuromorphic network effectively is one of the challenges associated with building systems that utilize SNN systems. It is necessary to convert the data to spike form before they can be processed by an SNN as input, unless neuromorphic or event-triggered sensing systems are employed. We present a configurable circuit based on a focal plane array (FPA) capable of providing spike-encoded readout data at the pixel level. With this type of circuit, the current signal of the photoelectric sensor can be encoded into two spike encodings with different precision, which are sent for processing to SNNs. This provides image information at two different scales for the artificial vision system based on SNNs. With this feature, we can use this circuit and different SNN structures to build an artificial target recognition system that is closer to the biological visual system.

Light-stimulated long-term potentiation behavior enhanced in a HfO2/InGaZnO photonic synapse

Photonic-based neuromorphic systems offer numerous advantages, including low crosstalk, high bandwidth, and low power consumption, making them an ideal choice for ultrafast neuromorphic systems. Although these neuromorphic systems promote the utilization of amorphous oxide semiconductors (AOSs) for promising photonic synapses, precise control of their unique persistent photoconductivity to mimic synaptic plasticity using an external light stimulus has rarely been studied. This study investigates the significant enhancement of the long-term potentiation (LTP) behavior in an InGaZnO (IGZO) photonic synapse through facile HfO2 passivation. In HfO2/IGZO synapses, the HfO2 layer is crucial in enhancing the LTP characteristics. The improvement in the persistent photocurrent behavior of the HfO2/IGZO synapse was attributed to the higher activation energy required for the neutralization of ionized oxygen vacancies, facilitating the LTP behavior. The LTP was dynamically modulated depending on external light pulse conditions. A large number of multilevel states (1024 states) were enabled by consecutive light simulations. Furthermore, HfO2/IGZO synapses exhibit different learning capacities depending on the external stimuli configuration. These findings demonstrate our ability to manipulate synaptic functionality in AOS-based artificial photonic synapses, paving the way for multifunctional photonic synapse circuitry. This advancement holds the potential for realizing an artificial vision system in the future.

Polarization‐Sensitive Optoelectronic Synapse Based on 3D Graphene/MoS2 Heterostructure

AbstractOptoelectronic synapses with information sensing, processing, and memory function are promoting the development of artificial visual perception systems. However, optoelectronic synapses' relatively inferior optoelectronic performance impedes their application in complex neuromorphic computing. Herein, optoelectronic synapses based on 3D graphene/molybdenum disulfide (MoS2) heterostructure field‐effect transistors are developed by using the double stress layer self‐rolled‐up method. The graphene, with excellent electrical properties, enhances the carrier transport capacity of the device. The unique continuous photoconductivity of MoS2 is suitable for simulating various synaptic nerve morphological functions. Meanwhile, the 3D resonant microcavity is added to enhance the optical field and make the device polarization sensitive, which reveals more intangible features of objects. The device demonstrates room‐temperature photodetection at ultraviolet, visible, near‐infrared, and mid‐infrared regions, with photoresponsivity up to 105 A W−1 at 590 nm. Furthermore, multiple synaptic neuromorphic functions, such as inhibitory postsynaptic current, paired‐pulse facilitation, short‐term depression, and long‐term depression, are successfully emulated. Here a new concept is provided for designing high‐performance optoelectronic synapses with polarization sensitivity and excellent potential in artificial intelligence is shown.

Flexible array of high performance and stable formamidinium-based low-n 2D halide perovskite photodetectors for optical imaging

Metal halide perovskites are attracting increasing attention as photodetectors (PDs) materials due to their excellent photoelectric properties. However, integrating perovskite to realize high-resolution perovskite pixel arrays remains critical to enable practical devices. Herein, we demonstrate a high-responsivity and stable 10 × 10 flexible PDs array based on formamidinium-based quasi-2D Ruddlesden-Popper (BA)2FAPb2I7 (n = 2) (BA= butylammonium, C4H9NH3+) perovskite elements for image sensing. The high-performance (BA)2FAPb2I7 arrays are synthesized on a polyethylene terephthalate (PET) flexible substrate, on a large scale, high quality, and controllable manner by introducing Au nanoparticles to enhance film quality upon the annealing process and recrystallization by applying formamidinium chloride (FACl) post-treatment to reduce perovskite film defects, combined with an innovative SiO2-assisted hydrophilic and hydrophobic surface treatment strategy. The prepared flexible PDs arrays show excellent optoelectronic properties with high responsivity (5.9 A W−1), a detectivity of 8.2 × 1012 Jones, large on/off current ratio (up to 3 × 104), ultimate response speed (12.8 μs/10.1 μs), and wide spectral response range. In addition, the device shows outstanding mechanical stability under severe bending (θ = 120°) and excellent folding durability after thousands of bending cycles. It also exhibits excellent environmental stability with only a slight degradation of photoresponse after 100 days of storage under the ambient environment. Finally, the integrated flexible PDs arrays can be used as an effective visible light image sensor with good spatial resolution, paving a multifunctional and competitive way for future wearable sensors and artificial vision systems. One-sentence summaryLow-dimensional halide perovskite layers pixelized on a polyethylene terephthalate (PET) flexible substrate provide high performance photodetectors.

A Novel Artificial Visual System for Motion Direction Detection with Completely Modeled Retinal Direction-Selective Pathway

Some fundamental visual features have been found to be fully extracted before reaching the cerebral cortex. We focus on direction-selective ganglion cells (DSGCs), which exist at the terminal end of the retinal pathway, at the forefront of the visual system. By utilizing a layered pathway composed of various relevant cells in the early stage of the retina, DSGCs can extract multiple motion directions occurring in the visual field. However, despite a considerable amount of comprehensive research (from cells to structures), a definitive conclusion explaining the specific details of the underlying mechanisms has not been reached. In this paper, leveraging some important conclusions from neuroscience research, we propose a complete quantified model for the retinal motion direction selection pathway and elucidate the global motion direction information acquisition mechanism from DSGCs to the cortex using a simple spiking neural mechanism. This mechanism is referred to as the artificial visual system (AVS). We conduct extensive testing, including one million sets of two-dimensional eight-directional binary object motion instances with 10 different object sizes and random object shapes. We also evaluate AVS’s noise resistance and generalization performance by introducing random static and dynamic noises. Furthermore, to thoroughly validate AVS’s efficiency, we compare its performance with two state-of-the-art deep learning algorithms (LeNet-5 and EfficientNetB0) in all tests. The experimental results demonstrate that due to its highly biomimetic design and characteristics, AVS exhibits outstanding performance in motion direction detection. Additionally, AVS possesses biomimetic computing advantages in terms of hardware implementation, learning difficulty, and parameter quantity.

All‐In‐One Hardware Devices with Event‐Based Vision Sensor Arrays for Image Sensing, Computing, and Learning

AbstractMetal oxide semiconductors (MOSs) are considered as potential candidates for the low‐cost, large‐area fabrication of flexible optoelectronic devices. However, the current optoelectronic devices based on MOSs are limited to unidirectional photoresponse, which constrains the performance of MOSs‐based vision sensors for artificial vision systems. Herein, for the first time, a flexible artificial vision system integrated with optical perception, computation, and learning functionalities is demonstrated using SnO optoelectronic synaptic transistor‐based event‐driven vision sensors to enable dynamic image perception, noise reduction, detection, and recognition. Specifically, an ambipolar SnO transistor is demonstrated by introducing HfO2 passivation layer, which facilitates the movement of O atoms around Sn‐vacancy sites to the HfO2 layer to achieve the transformation from p‐type to ambipolar transport behaviors. More importantly, the HfO2‐passivated SnO transistors exhibit gate‐tunable bidirectional photoresponse behavior, which is essential to simulate the neurobiological functionalities of bipolar cells. This way, the multilayer neural network learning circuit built from SnO transistors achieves fast recognition at a 16% Gaussian noise level and high recognition accuracy up to 95.2% for pattern letters. Under the bending states, recognition accuracies are still retained at 91.2%. These properties are well retained even under the influence of 100% offset of the synaptic programming value.

Flexible optoelectronic synaptic transistors for neuromorphic visual systems

Neuromorphic visual systems that integrate the functionalities of sensing, memory, and processing are expected to overcome the shortcomings of conventional artificial visual systems, such as data redundancy, data access delay, and high-energy consumption. Neuromorphic visual systems based on emerging flexible optoelectronic synaptic devices have recently opened up innovative applications, such as robot visual perception, visual prosthetics, and artificial intelligence. Various flexible optoelectronic synaptic devices have been fabricated, which are either two-terminal memristors or three-terminal transistors. In flexible optoelectronic synaptic transistors (FOSTs), the synaptic weight can be modulated by the electricity and light synergistically, which endows the neuromorphic visual systems with versatile functionalities. In this Review, we present an overview of the working mechanisms, device structures, and active materials of FOSTs. Their applications in neuromorphic visual systems for color recognition, image recognition and memory, motion detection, and pain perception are presented. Perspectives on the development of FOSTs are finally outlined.