Neural Visual System Research Articles

Rigid propagation of visual motion in the insect’s neural system

In the pursuit of developing an efficient artificial visual system for visual motion detection, researchers find inspiration from the visual motion-sensitive neural pathways in the insect’s neural system. Although multiple proposed neural computational models exhibit significant performance aligned with those observed from insects, the mathematical basis for how these models characterize the sensitivity of visual neurons to corresponding motion patterns remains to be elucidated. To fill this research gap, this study originally proposed that the rigid propagation of visual motion is an essential mathematical property of the models for the insect’s visual neural system, meaning that the dynamics of the model output remain consistent with the visual motion dynamics reflected in the input. To verify this property, this study uses the small target motion detector (STMD) neural pathway — one of the visual motion-sensitive pathways in the insect’s neural system — as an exemplar, rigorously demonstrating that the dynamics of translational visual motion are rigidly propagated through the encoding of retinal measurements in STMD computational models. Numerical experiment results further substantiate the proposed property of STMD models. This study offers a novel theoretical framework for exploring the nature of the visual motion perception underlying the insect’s visual neural system and brings an innovative perspective to the broader research field of insect visual motion perception and artificial visual systems.

A Robust Visual System for Looming Cue Detection Against Translating Motion.

Collision detection is critical for autonomous vehicles or robots to serve human society safely. Detecting looming objects robustly and timely plays an important role in collision avoidance systems. The locust lobula giant movement detector (LGMD1) is specifically selective to looming objects which are on a direct collision course. However, the existing LGMD1 models cannot distinguish a looming object from a near and fast translatory moving object, because the latter can evoke a large amount of excitation that can lead to false LGMD1 spikes. This article presents a new visual neural system model (LGMD1) that applies a neural competition mechanism within a framework of separated ON and OFF pathways to shut off the translating response. The competition-based approach responds vigorously to monotonous ON/OFF responses resulting from a looming object. However, it does not respond to paired ON-OFF responses that result from a translating object, thereby enhancing collision selectivity. Moreover, a complementary denoising mechanism ensures reliable collision detection. To verify the effectiveness of the model, we have conducted systematic comparative experiments on synthetic and real datasets. The results show that our method exhibits more accurate discrimination between looming and translational events-the looming motion can be correctly detected. It also demonstrates that the proposed model is more robust than comparative models.

A fractional-order visual neural network for collision sensing in noisy and dynamic scenes

Autonomously sensing collision threats in intelligent mobile machines is a significant challenge. Inspired by the perfect collision sensing system of locusts, various neural networks based on lobular giant motion detectors (LGMD) have been proposed. These networks use integer-order differential operators to simulate the simple and efficient visual neural system of locusts. However, some biological studies have shown that the neurodynamics of neuronal response should be fractional-order, and fractional-order differential operators can accurately describe the biological logic brain neural response process. In view of this, firstly, this paper uses a fractional-order differential operator to simulate the membrane potential response of neurons, and a new fractional-order LGMD collision sensing visual neural network is proposed. This network is designed to accurately detect looming objects and generate reliable response spikes. Then, through experimental analysis, a range [0.36, 0.4] of order in the fractional-order system is given, which is considered to be a reasonable interval for modeling the behavior of biological nervous systems. This maybe contribute to the study of the locust’s fractional-order biological nervous system. Finally, the network was systematically tested in noisy and complex dynamic environments. The experimental results demonstrated that the fractional-order LGMD visual neural network generated responses with high adaptability and reliability, showcasing significantly improved collision detection performance.

A fractional-order visual neural model for small target motion detection

Detecting small moving targets consisting of one or few pixels is technically demanding due to their limited visual features. Motivated by nature, some bio-inspired models have been developed that simulate the behavior of small target motion detectors (STMDs), a class of specialized neurons found in insects’ visual neural systems dominating in detecting small moving targets during activities such as predation or courtship. However, the existing models’ high dependence on the sampling frequency of input videos becomes a bottleneck that seriously hinders their real-time applications in the physical world. This is because massive computational power is required to capture and process high-sampling-frequency videos. While model detection performance in low-sampling-frequency videos is plagued by significant spatial errors and weak responses. To address these issues, we propose an STMD-based visual neural model with a fractional-order difference operator for small target motion detection. The fractional-order operator captures instantaneous luminance change and integrates it with memory information, where the instantaneous information dominates the integrated signal. The STMD network further separates the rising and falling luminance components, which are aligned in the time domain and then multiplied to predict the location of moving small targets. Due to the rapid response of instantaneous information and the supplement of memory information, the proposed model locates the small moving targets accurately and robustly in low-sampling-frequencies. Numerical experiments show that the proposed model significantly improves the detection performance for low-sampling-frequency videos.

A directionally selective collision-sensing visual neural network based on fractional-order differential operator.

In this paper, we propose a directionally selective fractional-order lobular giant motion detector (LGMD) visual neural network. Unlike most collision-sensing network models based on LGMDs, our model can not only sense collision threats but also obtain the motion direction of the collision object. Firstly, this paper simulates the membrane potential response of neurons using the fractional-order differential operator to generate reliable collision response spikes. Then, a new correlation mechanism is proposed to obtain the motion direction of objects. Specifically, this paper performs correlation operation on the signals extracted from two pixels, utilizing the temporal delay of the signals to obtain their position relationship. In this way, the response characteristics of direction-selective neurons can be characterized. Finally, ON/OFF visual channels are introduced to encode increases and decreases in brightness, respectively, thereby modeling the bipolar response of special neurons. Extensive experimental results show that the proposed visual neural system conforms to the response characteristics of biological LGMD and direction-selective neurons, and that the performance of the system is stable and reliable.

CNNs/ViTs-CNNs/ViTs: Mutual distillation for unsupervised domain adaptation

Unsupervised Domain Adaptation (UDA) is a popular machine learning technique to reduce the distribution discrepancy among domains. In previous UDA methods, only convolutional neural networks (CNNs) or vision transformers (ViTs) are used as the backbone. Therefore, these methods are subjected to inherent characteristics of single-CNNs or single-ViTs. In fact, CNNs are more biased towards local textures, while ViTs are inclined to learn the shape information of images. In this paper, we consider that the model combining the above two bias properties is closer to the human visual neural system and achieves more robust performance. A novel mutual distillation method for UDA without any assistant networks is proposed. Mutual distillation between two backbones with complementary properties (i.e. CNNs & ViTs) can promote each other, leading to better domain knowledge transfer. Additionally, traditional domain-mixup approaches can only mix limited cross-domain information through linear interpolation. To encourage more cross-domain information interaction between the two backbones and bridge the domain gap, we propose a patch-mixup method, building a train of mixed intermediate domains composed of augmented patches. Meanwhile, a cross-domain semantic alignment loss is proposed to align the semantic information of these domains. Extensive experiments show that our method achieves state-of-the-art results on several standard UDA benchmarks, such as Office-31, Office-Home, ImageCLEF-DA and VisDA-2017.

Gradient-based fly immune visual recurrent neural network solving large-scale global optimization

The recent achievements claimed that the fly visual neural system could serve a type of artificial computation model for revealing the properties of fly’s learning, memory and decision-making. It, however, still keeps open how to borrow the properties to develop artificial visual neural network optimization models. Hereby, inspired by gradient descent, fly’s visual information-processing and innate immunity mechanisms, this work probes into a fly immune visual recurrent neural network to solve large-scale global optimization (LSGO). As a two-step recurrent network, the neural network updates the output state matrix with the same size as that in the input layer through the gradient descent approach, where the learning rate of state transition is dynamically adjusted by a forward feedback fly immune visual neural network. Also, it is integrated with the conventional convolutional neural network to optimize the ultra-high dimensional weight and threshold parameters in order to acquire a prediction model of visual scene recognition. The theoretical results indicate that the recurrent neural network can converge to an equilibrium point under certain assumptions while the computational complexity is determined by the size of state matrix and the dimension of LSGO. The comparative experiments have confirmed that the neural network is an alternative and potential optimizer for LSGO problems, and in particular the acquired prediction model is a competitive tool for visual scene recognition.

Neuromorphic Motion Detection and Orientation Selectivity by Volatile Resistive Switching Memories

Motion detection is a primary visual function, crucial for the survival of animals in nature. Direction‐selective (DS) neurons can be found in multiple locations in the visual neural system, both in the retina and in the visual cortex. For instance, the DS ganglion cell in the retina provides a real‐time response to moving objects, which is much faster than the image recognition executed in the visual cortex. Such in‐retina biological signal processing capability is enabled by the spatiotemporal correlation within different receptive fields of the DS ganglion cells. Taking inspiration from the biological DS ganglion cells in the retina, the motion detection is demonstrated in an artificial neural network made of volatile resistive switching devices with short‐term memory effects. The motion detection arises from the spatiotemporal correlation between the adjacent excitatory and inhibitory receptive fields with short‐term memory synapses, closely resembling the physiological response of DS ganglion cells in the retina. The work supports real‐time neuromorphic processing of sensor data by exploiting the unique physics of innovative memory devices.

Physiological constraints of visual pathway lead to more efficient coding of information in retina

Nowadays, numerous studies have investigated the modeling of efficient neural encoding processes in the retina of the eye to encode the sensory data. Retina, as the innermost coat of the eye, is the first and the most important area of the visual neural system of mammalians, which is responsible for neural processes. Retina encodes the information of light intensity into a sequence of spikes, and sends them to retinal ganglion cells (RGCs) for further processing. An appropriate retinal encoding model should be adapted to the real retina as much as possible by considering the physiological constraints of the visual pathway to transfer most of the information of the input signal to the brain without too much redundancy of the channel. In this paper, inspired from the existing linear models of retinal encoding process, which have employed input noise and the spatial locality of the RGCs receptive fields (RFs) in the calculation of the encoding matrix, two extra physiological constraints, adapted from the real retina are taken into account so as to achieve a more realistic model for themammalian retina. These new constraints that are the correlation between RGCs and the spatial locality of the photoreceptors’ projective fields (PFs), are modeled in a mathematical form and analyzed in detail. To quantify fidelity of the proposed encoding matrix and prove its superiority over existing models, various parameters of the models are calculated and presented in this paper: mean square error between the original and reconstructed image (MSE), the redundancy of the channel, the amount of information transferred through the channel, and the amount of wasted capacity for carrying input noise, to name a few. The results of these calculations show that the proposed model transfers input information with less redundancy of the channel. In other words, it reduces a portion of channel capacity which is wasted for carrying the input noise in comparison to the existing models. Also, due to considering extra physiological constraints in the proposed model, it is acceptable to have a slightly higher amount of MSE value in order to become similar to the real retina.

Coping With Multiple Visual Motion Cues Under Extremely Constrained Computation Power of Micro Autonomous Robots

The perception of different visual motion cues is crucial for autonomous mobile robots to react to or interact with the dynamic visual world. It is still a great challenge for a micro mobile robot to cope with dynamic environments due to the restricted computational resources and the limited functionalities of its visual systems. In this study, we propose a compound visual neural system to automatically extract and fuse different visual motion cues in real-time using the extremely constrained computation power of micro mobile robots. The proposed visual system contains multiple bio-inspired visual motion perceptive neurons each with a unique role, for example to extract collision visual cues, darker collision cue and directional motion cues. In the embedded system, these multiple visual neurons share a similar presynaptic network to minimise the consumption of computation resources. In the postsynaptic part of the system, visual cues pass results to corresponding action neurons using lateral inhibition mechanism. The translational motion cues, which are identified by comparing pairs of directional cues, are given the highest priority, followed by the darker colliding cues and approaching cues. Systematic experiments with both virtual visual stimuli and real-world scenarios have been carried out to validate the system's functionality and reliability. The proposed methods have demonstrated that (1) with extremely limited computation power, it is still possible for a micro mobile robot to extract multiple visual motion cues robustly in a complex dynamic environment; (2) the cues extracted can be fused with a lateral inhibited postsynaptic network, thus enabling the micro robots to respond effectively with different actions, accordingly to different states, in real-time. The proposed embedded visual system has been modularised and can be easily implemented in other autonomous mobile platforms for real-time applications. The system could also be used by neurophysiologists to test new hypotheses pertaining to biological visual neural systems.

35.3: Invited Paper: Effects of Luminance, Contrast and Saturation of HDR QLED Display on Visual System Based on Eye Movement

This paper investigated the effects of luminance, contrast and saturation of HDR video presented on QLED display on visual system. Results showed that mainly, contrast and saturation influenced visual neural systems controlling eye movements, as well as subjective evaluation of display quality and viewing preference, while luminance generally had no significant influence.

The effects of dexamethasone on 17β-HSD1 levels at the rat optic nerve.

Dexamethasone (DEX) is associated with many inflammation and metabolic diseases. We analyzed the effects of DEX on the expression of estrogen metabolism enzyme 17-HSD1 at the optic nerve. Rats were treated with different concentrations of intraperitoneal DEX. Western Blot analysis showed that 17-HSD protein was expressed in the optic nerve tissue. The enzyme was detected by immunohistochemistry on the terminal foot of Muller cells from the ganglion cell layer of rat retina. ELISA analysis showed that the 17-HSD1 protein expression of DEX-treated group is 2.4 fold comparing to the control group. The results indicated that DMXS sodium phosphate might modulate the expression of 17-HSD1 protein in optic tissue. This study sheds light on understanding of the relationship among DEX, 17-HSD presence and distribution of visual neural systems. At the same time, DEX treatment affects the athletic ability and memory of the animals. Compared with the control group, the experimental group showed slow response to stimulation, inertia, depression, cowardice and lack of appetite. The results of ethology experiments showed that all the parameters decreased by 15–30%.

P‐33: Effects of Luminance, Contrast and Saturation of HDR QLED Display on Visual System Based on Eye Movement

The effects of screen luminance, contrast and saturation of HDR video presented on QLED display on visual system were investigated. Results showed that contrast and saturation, rather than luminance, influenced visual neural systems controlling eye movements, and affected subjective display quality and viewing preference.

Coplanar Multigate MoS2 Electric-Double-Layer Transistors for Neuromorphic Visual Recognition.

Spatial coordinate and visual orientation recognition in cortical cells play important roles in the visual system. Herein, spatiotemporally processed visual neurons are mimicked by a facile coplanar multigate two-dimensional (2D) MoS2 electric-double-layer transistor with proton-conducting poly(vinyl alcohol) electrolytes as laterally coupled gate dielectrics. Fundamental neuromorphic behaviors, e.g., excitatory postsynaptic current and paired-pulse facilitation, were successfully mimicked. For the first time, a proof-of-principle artificial visual neural network system for mimicking spatiotemporal coordinate and orientation recognition was experimentally demonstrated in such devices. The experimental results provide a promising opportunity for adding intelligent spatiotemporally-processed functions in emerging brain-like neuromorphic nanoelectronics.

Temporal Contiguity Training Influences Behavioral and Neural Measures of Viewpoint Tolerance.

Humans can often recognize faces across viewpoints despite the large changes in low-level image properties a shift in viewpoint introduces. We present a behavioral and an fMRI adaptation experiment to investigate whether this viewpoint tolerance is reflected in the neural visual system and whether it can be manipulated through training. Participants saw training sequences of face images creating the appearance of a rotating head. Half of the sequences showed faces undergoing veridical changes in appearance across the rotation (non-morph condition). The other half were non-veridical: during rotation, the face simultaneously morphed into another face. This procedure should successfully associate frontal face views with side views of the same or a different identity, and, according to the temporal contiguity hypothesis, thus enhance viewpoint tolerance in the non-morph condition and/or break tolerance in the morph condition. Performance on the same/different task in the behavioral experiment (N = 20) was affected by training. There was a significant interaction between training (associated/not associated) and identity (same/different), mostly reflecting a higher confusion of different identities when they were associated during training. In the fMRI study (N = 20), fMRI adaptation effects were found for same-viewpoint images of untrained faces, but no adaptation for untrained faces was present across viewpoints. Only trained faces which were not morphed during training elicited a slight adaptation across viewpoints in face-selective regions. However, both in the behavioral and in the neural data the effects were small and weak from a statistical point of view. Overall, we conclude that the findings are not inconsistent with the proposal that temporal contiguity can influence viewpoint tolerance, with more evidence for tolerance when faces are not morphed during training.

Optical and neural anisotropy in peripheral vision.

Optical blur in the peripheral retina is known to be highly anisotropic due to nonrotationally symmetric wavefront aberrations such as astigmatism and coma. At the neural level, the visual system exhibits anisotropies in orientation sensitivity across the visual field. In the fovea, the visual system shows higher sensitivity for cardinal over diagonal orientations, which is referred to as the oblique effect. However, in the peripheral retina, the neural visual system becomes more sensitive to radially-oriented signals, a phenomenon known as the meridional effect. Here, we examined the relative contributions of optics and neural processing to the meridional effect in 10 participants at 0°, 10°, and 20° in the temporal retina. Optical anisotropy was quantified by measuring the eye's habitual wavefront aberrations. Alternatively, neural anisotropy was evaluated by measuring contrast sensitivity (at 2 and 4 cyc/deg) while correcting the eye's aberrations with an adaptive optics vision simulator, thus bypassing any optical factors. As eccentricity increased, optical and neural anisotropy increased in magnitude. The average ratio of horizontal to vertical optical MTF (at 2 and 4 cyc/deg) at 0°, 10°, and 20° was 0.96 ± 0.14, 1.41 ± 0.54 and 2.15 ± 1.38, respectively. Similarly, the average ratio of horizontal to vertical contrast sensitivity with full optical correction at 0°, 10°, and 20° was 0.99 ± 0.15, 1.28 ± 0.28 and 1.75 ± 0.80, respectively. These results indicate that the neural system's orientation sensitivity coincides with habitual blur orientation. These findings support the neural origin of the meridional effect and raise important questions regarding the role of peripheral anisotropic optical quality in developing the meridional effect and emmetropization.

Other ways of seeing: From behavior toneural mechanisms in the online "visual"control of action with sensorysubstitution.

Vision is the dominant sense for perception-for-action in humans and other higher primates. Advances in sight restoration now utilize the other intact senses to provide information that is normally sensed visually through sensory substitution to replace missing visual information. Sensory substitution devices translate visual information from a sensor, such as a camera or ultrasound device, into a format that the auditory or tactile systems can detect and process, so the visually impaired can see through hearing or touch. Online control of action is essential for many daily tasks such as pointing, grasping and navigating, and adapting to a sensory substitution device successfully requires extensive learning. Here we review the research on sensory substitution for vision restoration in the context of providing the means of online control for action in the blind or blindfolded. It appears that the use of sensory substitution devices utilizes the neural visual system; this suggests the hypothesis that sensory substitution draws on the same underlying mechanisms as unimpaired visual control of action. Here we review the current state of the art for sensory substitution approaches to object recognition, localization, and navigation, and the potential these approaches have for revealing a metamodal behavioral and neural basis for the online control of action.

Mouse Ability to Perceive Subjective Contours.

In contrast to the previously held notion that mice have a weak visual system, it is now generally accepted that mice can perceive rather complicated figures in various contexts such as in cognitive experiments and in social settings. Here, we show that mice could even be capable of perceiving a visual illusion--subjective contours. This illusion requires the visual system to compensate for a lack of visual information in compressed 2D images on the retina. In this experiment, we trained mice to respond appropriately to a rectangle-shaped rewarded figure of specific orientation in a two-choice visual discrimination task with a touchscreen monitor. In Transfer Test 1, mice could discriminate illusory rectangle-shaped figures significantly as compared with a figure, which did not induce illusory figures. In Transfer Test 2, the choice rate of targets decreased with imperfect illusory figures, which produced weak perception of rotated or deficient inducers. Moreover, in Transfer Test 3, mice could not discriminate the low-resolution illusory figure, which also induced weak perception. These results demonstrated the possibility that mice might be useful for investigating fundamental properties of the neural visual system.

Immature visual neural system in children reflected by contrast sensitivity with adaptive optics correction.

This study aimed to explore the neural development status of the visual system of children (around 8 years old) using contrast sensitivity. We achieved this by eliminating the influence of higher order aberrations (HOAs) with adaptive optics correction. We measured HOAs, modulation transfer functions (MTFs) and contrast sensitivity functions (CSFs) of six children and five adults with both corrected and uncorrected HOAs. We found that when HOAs were corrected, children and adults both showed improvements in MTF and CSF. However, the CSF of children was still lower than the adult level, indicating the difference in contrast sensitivity between groups cannot be explained by differences in optical factors. Further study showed that the difference between the groups also could not be explained by differences in non-visual factors. With these results we concluded that the neural systems underlying vision in children of around 8 years old are still immature in contrast sensitivity.

The study on neural contrast sensitivity function at temporal frequencies

The neural contrast sensitivity function (NCSF) of human eye at temporal frequencies is acquired with a new method in this paper. Firstly the human eye's contrast sensitivity function (CSF) at temporal frequencies is measured by means of travelling-wave stimuli, and the modulation transfer function (MTF) of the eye's optics is obtained by constructing the eye model of the subject. Then the NCSF at temporal frequencies is calculated with the two functions. It is shown that the overall value of the NCSF decreases as the temporal frequency increases. As the temporal frequency increases from 0 cycles per second (c/s) to 1 c/s, 16 c/s and 30 c/s, the NCSF curve changes from the band-pass shape to the low-pass, and then to almost monotonic variation. The attenuation factor of the NCSF, which represents the sensitivity of visual neural system to the temporal frequency, varies little as spatial frequency increases, while that of the CSF declines dramatically in the region of high spatial frequencies. Because it is the NCSF, rather than the CSF, that reflects the characteristics of visual neural system, the investigation of the NCSF at temporal frequency is more essentially.