Visual Perception System Research Articles

CdSe quantum dots photoelectric memristors for simulating biological visual system behavior

The visual system is the most important in the biological neurosensory system, so simulation of the biological visual system is the key for developing artificial nervous system. This work incorporates a CdSe quantum dot layer into silk fibroin based memristor device, endowing the devices with light-responsive capability. Furthermore, interesting phenomena were observed when the device operated as artificial synapses: excessive voltage stimulation led to a reduction in synaptic weight compared to the responses observed under normal electrical stimulation. This behavior mirrors sensations associated with pain, neuroprotection, and potential injuries to the neural system. At the end, we designed a visual perception system, simulating processes of the biological light intensity perception and the visual degradation response under intense light stimulation. Our research demonstrates the feasibility of constructing an artificial visual nervous system using a hardware system based on memristors.

A Near-Infrared Retinomorphic Device with High Dimensionality Reservoir Expression.

Physical reservoir-based reservoir computing (RC) systems for intelligent perception have recently gained attention because they require fewer computing resources. However, the system remains limited in infrared (IR) machine vision, including materials and physical reservoir expression power. Inspired by biological visual perception systems, the study proposes a near-infrared (NIR) retinomorphic device that simultaneously perceives and encodes narrow IR spectral information (at ≈980nm). The proposed device, featuring core-shell upconversion nanoparticle/poly (3-hexylthiophene) (P3HT) nanocomposite channels, enables the absorption and conversion of NIR into high-energy photons to excite more photo carriers in P3HT. The photon-electron-coupled dynamics under the synergy of photovoltaic and photogating effects influence the nonlinearity and high dimensionality of the RC system under narrow-band NIR irradiation. The device also exhibits multilevel data storage capability (≥8 levels), excellent stability (≥2000 s), and durability (≥100 cycles). The system accurately identifies NIR static and dynamic handwritten digit images, achieving recognition accuracies of 91.13% and 90.07%, respectively. Thus, the device tackles intricate computations like solving second-order nonlinear dynamic equations with minimal errors (normalized mean squared errorof 1.06 × 10⁻3 during prediction).

Content adaptive JND profile by leveraging HVS inspired channel modeling and perception oriented energy allocation optimization

The existing just noticeable difference (JND) models consider the effects of various covariates, however, they rarely account for the fusion relationship between the covariates, i.e., they lack a holistic understanding of the mechanisms of visual perception and disregarding the significant impact of energy consumption on visual perception. In fact, visual perception is no exception to the rule that nerve activities and energy supply are inextricably linked. Based on this insight, this paper proposes a novel JND estimation model employing content-adaptive energy allocation. Primarily, the information theory is applied to the visual perception system by conceptualizing human visual system (HVS) as an information communication framework. Then, leveraging the relationship between energy consumption and information perception, this paper quantitatively measures the HVS energy consumption as uniform metric to describe the complicated and heterogeneous HVS perception process, and then construct JND model by fusing low-level and Semantic-level features. Numerous simulation results verify that the proposed JND model is significantly competitive with other frontier models and highly compatible with HVS.

An evolutionarily distinct Hmgn2 variant influences shape recognition in Medaka Fish

Protein sequence diversification significantly impacts physiological traits. In this study, using medaka fish (Oryzias latipes), we identify a novel protein variant affecting shape preference behavior. Re-analysis of sequencing data reveals that LOC101156433 encodes a unique Hmgn2 variant with unusual subnuclear localization, clustered separately from the Hmgn2 clades of other species. Medaka mutants with this variant showed reduce telencephalic regions and altered shape preference, suggesting a link between protein sequence variation and behavioral changes. Additionally, this Hmgn2 variant is common in Acanthopterygii fishes, which are adapted to a variety of environments, indicating its potential evolutionary significance. Our findings highlight the relationship between amino acid sequence variation and the development of new molecular and behavioral adaptations, providing insights into the visual shape perception system in fish.

Visual information perception system of coal mine comprehensive excavation working face for edge computing terminal

AbstractAiming at the problems of low detection accuracy, high computational complexity and long‐time consumption of visual perception model in a complex mining environment, this research designs a visual information perception system of coal mine comprehensive excavation working face for an edge computing terminal. Firstly, the C3‐Fast feature extraction module, spatial pyramid pooling with cross‐stage partial connection (SPPCSPC) pooling module, bi‐directional feature pyramid network and lightweight decoupled detection head are used to optimize the YOLOv5s model, so as to construct the FSBD‐YOLOv5s multi‐object detection model. Secondly, the pruning and distillation algorithm is used to lighten the FSBD‐YOLOv5s model, and the model complexity is greatly reduced while maintaining the model detection accuracy. Further, the lightweight FSBD‐YOLOv5s model is migrated and deployed to the edge computing terminal platform and the TensorRT engine is used to accelerate model inference. Finally, experiments are carried out based on the data set of the coal mine comprehensive excavation working face. The experimental results show that on the edge computing terminal platform, the parameters and computational volume of the lightweight FSBD‐YOLOv5s model are reduced by 50.8% and 34.0%, while its detection accuracy and speed reach 94.0% and 43.7 fps, which can fully satisfy the requirements of the accuracy and real‐time for the coal mine engineering applications.

Light-Adapted Optoelectronic-Memristive Device for the Artificial Visual System.

With the development of artificial intelligence systems, it is necessary to develop optoelectronic devices with photoresponse and storage capacity to simulate human visual perception systems. The key to an artificial visual perception system is to integrate components with both sensing and storage capabilities of illumination information. Although module integration components have made useful progress, they still face challenges such as multispectral response and high energy consumption. Here, we developed a light-adapted optoelectronic-memristive device integrated by an organic photodetector and ferroelectric-based memristor to simulate human visual perception. ITO/P3HT:PC71BM/Au as the light sensor unit shows a high on/off ratio (Iph/Id) reaching ∼5 × 104 at 0 V. The memristor unit, consisting of ITO/CBI@P(VDF-TrFE)/Cu, has a RON/ROFF ratio window of ∼106 under 0.05 V read voltage and ultralow power consumption of ∼1 pW. Moreover, the artificial visual perception unit shows stable light-adapted memory windows under different wavelengths of irradiation light (400, 500, and 600 nm; they meet the spectral range of human visual recognition) and can clearly identify the target image ("T" shape) because of the apparent contrast, which results from the high ROFF/RON ratio values. These results provide a potential design strategy for the development of intelligent artificial vision systems.

Open-Ended Online Learning for Autonomous Visual Perception.

The visual perception systems aim to autonomously collect consecutive visual data and perceive the relevant information online like human beings. In comparison with the classical static visual systems focusing on fixed tasks (e.g., face recognition for visual surveillance), the real-world visual systems (e.g., the robot visual system) often need to handle unpredicted tasks and dynamically changed environments, which need to imitate human-like intelligence with open-ended online learning ability. Therefore, we provide a comprehensive analysis of open-ended online learning problems for autonomous visual perception in this survey. Based on "what to online learn" among visual perception scenarios, we classify the open-ended online learning methods into five categories: instance incremental learning to handle data attributes changing, feature evolution learning for incremental and decremental features with the feature dimension changed dynamically, class incremental learning and task incremental learning aiming at online adding new coming classes/tasks, and parallel and distributed learning for large-scale data to reveal the computational and storage advantages. We discuss the characteristic of each method and introduce several representative works as well. Finally, we introduce some representative visual perception applications to show the enhanced performance when using various open-ended online learning models, followed by a discussion of several future directions.

The Critical Trigger for Cognitive Penetration: Cognitive Processing Priority over Perceptual Processing.

The visual perception system of humans is susceptible to cognitive influence, which implies the existence of cognitive perception. However, the specifical trigger for cognitive penetration is still a matter of controversy. The current study proposed that the cognitive processing priority over perceptual processing might be critical for inducing cognitive penetration. We tested this hypothesis by manipulating the processing priority between cognition and perception across three experiments where participants were asked to complete a size-judging task under different competing conditions between cognition and perception. To sum up, we proved that the cognitive processing priority over perceptual processing is critical for cognitive penetration. This study provided empirical evidence for the critical trigger for cognitive penetration.

Light-Powered Directional Ion Transport via PFN-Br/MoS2 Heterogeneous Membranes: Band Alignment and Activation Energy Barrier Engineering.

Biological photoresponsive ion transport systems consistently attract researchers' attention owing to their remarkable functions of harvesting energy from nature and participating in visual perception systems. Designing and constructing artificial light-driven ion transport devices to mimic biological counterparts remains a challenge owing to fabrication limitations in nanoconfined spaces. Herein, a typical conjugated polyelectrolyte (PFN-Br) was assembled onto a laminated MoS2M using simple solution-processing vacuum filtration, resulting in a heterogeneous three- and two-dimensional nanoporous membrane. The designed band alignment between PFN-Br and MoS2 enables effective directional ion transport under irradiation in an equilibrium solution, even against a 30-fold concentration gradient. The staggered energy structure of PFN-Br and MoS2 enhances charge separation and establishes a photogenerated potential as the driving force for ion transport. Additionally, the activation energy barrier for ion transport across the heterogeneous membrane decreased by 60% after light irradiation, considerably improving ion transport flux. The easy fabrication and high performance of the membrane in light-powered ion transport provide promising approaches for designing nanofluidic devices with possible applications in energy conversion, light-enhanced biosensing, and photoresponsive ionic devices.

A selective harvesting robot for cherry tomatoes: Design, development, field evaluation analysis

AbstractWith the aging population and increasing labor costs, traditional manual harvesting methods have become less economically efficient. Consequently, research into fully automated harvesting using selective harvesting robots for cherry tomatoes has become a hot topic. However, most of the current research is focused on individual harvesting of large tomatoes, and there is less research on the development of complete systems for harvesting cherry tomatoes in clusters. The purpose of this study is to develop a harvesting robot system capable of picking tomato clusters by cutting their fruit‐bearing pedicels and to evaluate the robot prototype in real greenhouse environments. First, to enhance the grasping stability, a novel end‐effector was designed. This end‐effector utilizes a cam mechanism to achieve asynchronous actions of cutting and grasping with only one power source. Subsequently, a visual perception system was developed to locate the cutting points of the pedicels. This system is divided into two parts: rough positioning of the fruits in the far‐range view and accurate positioning of the cutting points of the pedicels in the close‐range view. Furthermore, it possesses the capability to adaptively infer the approaching pose of the end‐effector based on point cloud features extracted from fruit‐bearing pedicels and stems. Finally, a prototype of the tomato‐harvesting robot was assembled for field trials. The test results demonstrate that in tomato clusters with unobstructed pedicels, the localization success rates for the cutting points were 88.5% and 83.7% in the two greenhouses, respectively, while the harvesting success rates reached 57.7% and 55.4%, respectively. The average cycle time to harvest a tomato cluster was 24 s. The experimental results prove the potential for commercial application of the developed tomato‐harvesting robot and through the analysis of failure cases, discuss directions for future work.

Perceptual axioms are irreconcilable with Euclidean geometry.

There are different definitions of axioms, but the one that seems to have general approval is that axioms are statements whose truths are universally accepted but cannot be proven; they are the foundation from which further propositional truths are derived. Previous attempts, led by David Hilbert, to show that all of mathematics can be built into an axiomatic system that is complete and consistent failed when Kurt Gödel proved that there will always be statements which are known to be true but can never be proven within the same axiomatic system. But Gödel and his followers took no account of brain mechanisms that generate and mediate logic. In this largely theoretical paper, but backed by previous experiments and our new ones reported below, we show that in the case of so-called 'optical illusions', there exists a significant and irreconcilable difference between their visual perception and their description according to Euclidean geometry; when participants are asked to adjust, from an initial randomised state, the perceptual geometric axioms to conform to the Euclidean description, the two never match, although the degree of mismatch varies between individuals. These results provide evidence that perceptual axioms, or statements known to be perceptually true, cannot be described mathematically. Thus, the logic of the visual perceptual system is irreconcilable with the cognitive (mathematical) system and cannot be updated even when knowledge of the difference between the two is available. Hence, no one brain reality is more 'objective' than any other.

A novel jujube tree trunk and branch salient object detection method for catch-and-shake robotic visual perception

Visual perception has become a prerequisite for automated jujube harvesting robot operations under complex orchard conditions. Catch-and-Shake harvesting, as the most efficient and common harvesting method, has widely been applied on various manually operated harvesters to complete large-area jujube fruit harvesting. However, the main factors restricting the development of existing harvesters are labor shortage, high labor cost, and low operating efficiency. To address the issues, we designed a catch-and-shake harvesting robot for jujube tree trunks and branches visual perception that can provide a barrier-free catch-and-shake operation area and guide the manipulator to reach the area to complete the harvesting operation. Meanwhile, a visual perception system including tree trunks and branches detection, skeleton extraction, catch-and-shake area confirmation was presented to guide robot intelligent operations. In the visual perception system, a novel salientobjectdetectionmodel called feature intersection and fusion Transformer (FIT-Transformer) network was proposed to split branches and background to provide reference for determining safe catch-and-shake areas. Moreover, we designed a diverse feature aggregation (DFA) and an attention feature fusion module (AFFM) to strengthen feature learning capabilities and obtain robust perception models. Comparative experimental results showed that our proposed FIT-Transformer model outperformed 12 state-of-the-art (SOTA) algorithms including C2FNet, RAS, BASNet, U2Net, SCRNet, PiCANet, EDRNet, EGNet, ICONR, VST, TransSOD and ABiU_Net. Specifically, the segmentation accuracy of jujube tree trunks and branches using our method showed the satisfactory result on five evaluation indexes under natural environment (the EM, SM, WF, FM and MAE reached 0.9713, 0.8991, 0.8854, 0.8905, and 0.0302, respectively). Field experiments also proved that our method could meet the requirements of operational accuracy and real-time operations.

Pose Estimation-Based Visual Perception System for Analyzing Fish Swimming

Pose Estimation-Based Visual Perception System for Analyzing Fish Swimming

Broadband Artificial Tetrachromatic Synaptic Devices Composed of 2D/3D Integrated WSe2‐GaN‐based Dual‐Channel Floating Gate Transistors

AbstractThe development of artificial tetrachromatic vision holds great potential to enhance human color perception and discrimination, thereby enabling more effective navigation in diverse environments. Herein, an artificial tetrachromatic synaptic device is presented built upon 2D‐3D vertically stacked semiconductors composed of tungsten diselenide (WSe2)‐gallium nitride (GaN) configuration, forming a dual‐channel floating gate transistor (FGT). Under the concerted influence of electrical and optical stimulation, the device successfully mimics fundamental tetrachromatic synaptic behaviors, including short‐term potentiation (STP), weak long‐term potentiation (wLTP), long‐term potentiation (LTP), paired‐pulse facilitation (PPF), spike number‐dependent plasticity (SNDP), and spike rate‐dependent plasticity (SRDP). Notably, the plasticity of the device can be further modulated under ultraviolet (UV) stimulation, providing insights into the modulation of synaptic plasticity through the photogenerated carrier dynamics in the GaN channel. These results imply that WSe2‐GaN‐based FGT architecture with dual‐channel characteristics seamlessly integrates optical sensing and synaptic simulation functionalities, representing a promising avenue for the development of next‐generation artificial visual perception systems (AVPS), with a particular advantage for the pursuit of high‐performance artificial tetrachromatic neuromorphic computing applications of the future.

Digital Twin and Deep Reinforcement Learning-Driven Robotic Automation System for Confined Workspaces: A Nozzle Dam Replacement Case Study in Nuclear Power Plants

Robotic automation has emerged as a leading solution for replacing human workers in dirty, dangerous, and demanding industries to ensure the safety of human workers. However, practical implementation of this technology remains limited, requiring substantial effort and costs. This study addresses the challenges specific to nuclear power plants, characterized by hazardous environments and physically demanding tasks such as nozzle dam replacement in confined workspaces. We propose a digital twin and deep-reinforcement-learning-driven robotic automation system with an autonomous mobile manipulator. The study follows a four-step process. First, we establish a simplified testbed for a nozzle dam replacement task and implement a high-fidelity digital twin model of the real-world testbed. Second, we employ a hybrid visual perception system that combines deep object pose estimation and an iterative closest point algorithm to enhance the accuracy of the six-dimensional pose estimation. Third, we use a deep-reinforcement-learning method, particularly the proximal policy optimization algorithm with inverse reachability map, and a centroidal waypoint strategy, to improve the controllability of an autonomous mobile manipulator. Finally, we conduct pre-performed simulations of the nozzle dam replacement in the digital twin and evaluate the system on a robot in the real-world testbed. The nozzle dam replacement with precise object pose estimation, navigation, target object grasping, and collision-free motion generation was successful. The robotic automation system achieved a 92.0%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$92.0\\%$$\\end{document} success rate in the digital twin. Our proposed method can improve the efficiency and reliability of robotic automation systems for extreme workspaces and other perilous environments.

Organic Heterojunction Phototransistors with Bi‐Directional Photoresponse for Vision Biomimetics

AbstractThe scientific community is very interested in artificial visual perception systems because of their neuromorphological capabilities, such as image recognition, learning, and memory. This study reports on an organic heterojunction ambipolar transistor with PTCDI‐C8/C8‐BTBT/PTCDI‐C8 sandwich structure. The proposed transistor efficiently transports both holes and electrons, with a current‐on‐switch ratio of ≈106. Furthermore, hole or electron carrier transmission mechanisms are constructed to achieve the controllable adjustment of bi‐directional photoresponse under identical light illumination conditions by adjusting the gate voltage (positive photoresponse for Vg = 30 V and negative photoresponse for Vg = −30 V). The basic synaptic behaviors are simulated under light illumination, and the power consumption of the device is 18.51 fJ per synaptic event, which is comparable to the energy required for biological synapses. Finally, the simulation of human visual adaptation is completed using the bi‐directional photoresponse effects of the organic heterojunction phototransistor array, which provides a novel idea for neuromorphic visual simulation.

Selecting the Appropriate Speed for Rotational Elements in Human-Machine Interfaces: A Quantitative Study.

The motion of rotation, which served as a dynamic symbol within human-computer interfaces, has garnered extensive attention in interface and graphic design. This study aimed to establish speed benchmarks for interface design by exploring visual system preferences for the perception of both simple and complex rotating icons within the velocity range of 5-1800 degrees per second. The research conducted two experiments with 12 participants to examine the observers' just noticeable difference in speed (JNDS) and perceived speed for rotational icons. Experiment one measured the JNDS over eight-speed levels using a constant stimulus method, achieving a range of 14.9-29%. Building on this, experiment two proposed a sequence of speeds within the given range and used a rating scale method to assess observers ' subjective perception of the speed series' rapidity. The findings indicated that speed increases impacted the ability to differentiate between speeds; key points for categorizing low, medium, and high speeds were identified at 10, 180, and 720 degrees/s, respectively. Shape complexity was found to modulate the visual system's perception of actual speed, such that at rotation speeds above 180 degrees/s, complex icons appeared to rotate faster than simpler ones. Most importantly, the study applied quantitative methods and metrology to interface design, offering a more scientific approach to the design workflow.

Optoelectronic Synapse Based on 2D Electron Gas in Stoichiometry-Controlled Oxide Heterostructures.

Emulating synaptic functionalities in optoelectronic devices is significant in developing artificial visual-perception systems and neuromorphic photonic computing. Persistent photoconductivity (PPC) in metal oxides provides a facile way to realize the optoelectronic synaptic devices, but the PPC performance is often limited due to the oxygen vacancy defects that release excess conduction electrons without external stimuli. Herein, a high-performance optoelectronic synapse based on the stoichiometry-controlled LaAlO3/SrTiO3 (LAO/STO) heterostructure is developed. By increasing La/Al ratio up to 1.057:1, the PPC is effectively enhanced but suppressed the background conductivity at the LAO/STO interface, achieving strong synaptic behaviors. The spectral noise analyses reveal that the synaptic behaviors are attributed to the cation-related point defects and their charge compensation mechanism near the LAO/STO interface. The short-term and long-term plasticity is demonstrated, including the paired-pulse facilitation, in the La-rich LAO/STO device upon exposure to UV light pulses. As proof of concepts, two essential synaptic functionalities, the pulse-number-dependent plasticity and the self-noise cancellation, are emulated using the 5×5 array of La-rich LAO/STO synapses. Beyond the typical oxygen deficiency control, the results show how harnessing the cation stoichiometry can be used to design oxide heterostructures for advanced optoelectronic synapses and neuromorphic applications.

Research on environmental reliability test and assessment of AI devices under vibration stress1

The proliferation of artificial intelligence (AI) devices has generated an increasing demand for reliability in their utilization. Nevertheless, the significant concern persists regarding the absence of suitable assessment and testing techniques to evaluate the performance of these intelligent systems in real-world conditions. In response to these issues, this paper conducts research on the reliability testing and assessment of AI visual perception systems under vibration stress. The paper introduces the working mechanism of the visual perception system and the various testing methods for AI devices. Based on this, a reliability assessment method for intelligent devices is proposed, which uses the Fréchet distance as the measurement function and environmental adaptability as the reliability metric. Additionally, a vibration test platform for the visual perception system is established, which offers a cost-effective and reliable solution to the high cost issue of field testing for AI devices. Finally, the reliability level of the visual perception system under various vibration conditions is tested through vibration testing. The research findings indicate that the reliability of AI models decreases as the degradation caused by vibration increases, following a normal distribution.

Evaluation of visual plasticity in patients with refractive amblyopia treated using a visual perceptual learning system.

Amblyopia is a neurological deficit in binocular vision that affects 3% of the population and is the result of disruptions in early visual development. In this study, we used a visual perceptual learning system for the short-term treatment of children with ametropic amblyopia and evaluated the clinical efficacy of this system in terms of visual plasticity. We conducted a retrospective analysis of the clinical data of 114 children (228 eyes) with refractive amblyopia, who were aged 6.51 ± 1.51 years. Prior to the treatment, we evaluated all children with amblyopia using the visual information processing test. We determined the type of amblyopic defect according to the type of amblyopia, corrected visual acuity, and advanced visual function test results. Based on the type of defect, each child with amblyopia was given short-term visual perception training for 10 days. Finally, we compared the results of visual acuity and visual information processing tests before and after the treatment. The best-corrected visual acuity of patients was better after 10 days of visual training than that before training (P< 0.05). The perceptual eye position after training improved with statistically significant differences in horizontal and vertical perceptual eye position (both P< 0.05) compared to that before training. The number of amblyopic children without suppression in both eyes was 81 cases (71.1%) after training which was higher than that (65 cases, or 57.0%) before training, with a statistically significant difference (P< 0.05). Binocular fine stereopsis and dynamic stereopsis improved after training with a statistically significant difference (both P< 0.05). In this study, it was found that patients with amblyopia showed visual plasticity. Moreover, continuous visual perceptual learning improved the best-corrected visual acuity and recovered stereopsis in children with refractive amblyopia.