Visual Perception Tasks Research Articles

Motion-induced blindness shows spatial anisotropies in conscious perception.

Polar angle asymmetries (PAAs), the differences in perceptual experiences and performance across different regions of the visual field are present in various paradigms and tasks of visual perception. Currently, research in this area is sparse, particularly regarding the influence of PAAs during perceptual illusions, highlighting a gap in visual cognition studies. We aim to fill this gap by measuring PAAs across the visual field during an illusion applied to test conscious vision widely. Motion-induced blindness (MIB) is an illusion when a peripheral target disappears from consciousness as the result of a continuously moving background pattern. During MIB we separately measured the average disappearance time of peripheral targets in eight equidistant visual field positions. Our results indicate a significant variation in MIB disappearance times and frequencies as a function of target location. Specifically, we found shorter and fewer disappearances along the cardinal compared to oblique directions, and along the horizontal compared to the vertical meridian. Our results suggest specific consistencies between visual field asymmetries and conscious visual perception.

Thalamic Roles in Conscious Perception Revealed by Low-Intensity Focused Ultrasound Neuromodulation.

The neural basis of conscious perception remains incompletely understood. While cortical mechanisms of conscious content have been extensively investigated, the role of subcortical structures, including the thalamus, remains less explored. We aim to elucidate the causal contributions of different thalamic regions to conscious perception using transcranial low-intensity focused ultrasound (LIFU) neuromodulation. We hypothesize that modulating different thalamic regions would result in distinct perceptual outcomes. We apply LIFU in human volunteers to investigate region-specific and sonication parameter-dependent effects. We target anterior (transmodal-dominant) and posterior (unimodal-dominant) thalamic regions, further divided into ventral and dorsal regions, while participants perform a near-threshold visual perception task. Task performance is evaluated using Signal Detection Theory metrics. We find that the high duty cycle stimulation of the ventral anterior thalamus enhanced object recognition sensitivity. We also observe a general (i.e., region-independent) effect of LIFU on decision bias (i.e., a tendency toward a particular response) and object categorization accuracy. Specifically, high duty cycle stimulation decreases categorization accuracy, whereas low duty cycle shifts decision bias towards a more conservative stance. In conclusion, our results provide causal insight into the functional organization of the thalamus in shaping human visual experience and highlight the unique role of the transmodal-dominant ventral anterior thalamus.

Shared subcortical arousal systems across sensory modalities during transient modulation of attention.

Subcortical arousal systems are known to play a key role in controlling sustained changes in attention and conscious awareness. Recent studies indicate that these systems have a major influence on short-term dynamic modulation of visual attention, but their role across sensory modalities is not fully understood. In this study, we investigated shared subcortical arousal systems across sensory modalities during transient changes in attention using block and event-related fMRI paradigms. We analyzed massive publicly available fMRI datasets collected while 1,561 participants performed visual, auditory, tactile, and taste perception tasks. Our analyses revealed a shared circuit of subcortical arousal systems exhibiting early transient increases in activity in midbrain reticular formation and central thalamus across perceptual modalities, as well as less consistent increases in pons, hypothalamus, basal forebrain, and basal ganglia. Identifying these networks is critical for understanding mechanisms of normal attention and consciousness and may help facilitate subcortical targeting for therapeutic neuromodulation.

VPA-Net: A visual perception assistance network for 3d lidar semantic segmentation

The semantic segmentation of 3D point clouds holds paramount importance in visual perception tasks of automatic driving, including obstacle avoidance, decision control, path planning, and map construction, etc. Multi-sensor fusion is a rational and pivotal technique for implementing LiDAR semantic segmentation. However, effectively fusing and utilizing multi-source data remains a challenging task. In this work, we introduce a novel visual perception-assisted point cloud segmentation network, termed VPA-Net. This network architecture employs a dual-branch design to effectively combine spatial information from point clouds and visual cues from images, thereby bolstering the performance of 3D LiDAR semantic segmentation. More specifically, the dual-branch network structure processes and fuses multi-modal data from both point clouds and RGB images. Subsequently, the intermediate features from the two branches are merged via the proposed attention-based feature fusion module. Furthermore, to address the challenge of precise boundary prediction in large-scale point cloud scene segmentation, we introduce a refinement module based on 3D sparse convolution to enhance the spatial information of the LiDAR point clouds. The effectiveness of our method is validated on SemanticKITTI and a more challenging 3D semantic segmentation dataset. Experimental results underscore significant improvement on SemanticKITTI, with our approach surpassing the state-of-the-art method, achieving a 7.1% higher mIoU than SalsaNext.

Unsupervised low-light image enhancement by data augmentation and contrastive learning

ABSTRACT Today, with the increasing demand for visual perception and high-level computational vision tasks, the field of low-light enhancement is rapidly developing. However, models trained on existing datasets often fail or suffer significant performance degradation in real-world low-light scenarios. This performance degradation is frequently due to the limitations of current databases, which typically contain small quantities of paired images of a single type. This article proposes an unsupervised model with a unique data augmentation technique that transforms a regular image database into a paired image database. By adjusting image parameters during training to change exposure, a regular image database can be converted into a paired one. The model restores low-exposure images by extracting lighting features through comparative learning. Evaluations of the LOL and DIV2K datasets demonstrate the proposed model's effectiveness, achieving notable results in low-light image enhancement. This method removes dataset restrictions, broadening the model's range of applications.

Auditory cortex encodes lipreading information through spatially distributed activity

Watching a speaker's face improves speech perception accuracy. This benefit is enabled, in part, by implicit lipreading abilities present in the general population. While it is established that lipreading can alter the perception of a heard word, it is unknown how these visual signals are represented in the auditory system or how they interact with auditory speech representations. One influential, but untested, hypothesis is that visual speech modulates the population-coded representations of phonetic and phonemic features in the auditory system. This model is largely supported by data showing that silent lipreading evokes activity in the auditory cortex, but these activations could alternatively reflect general effects of arousal or attention or the encoding of non-linguistic features such as visual timing information. This gap limits our understanding of how vision supports speech perception. To test the hypothesis that the auditory system encodes visual speech information, we acquired functional magnetic resonance imaging (fMRI) data from healthy adults and intracranial recordings from electrodes implanted in patients with epilepsy during auditory and visual speech perception tasks. Across both datasets, linear classifiers successfully decoded the identity of silently lipread words using the spatial pattern of auditory cortex responses. Examining the time course of classification using intracranial recordings, lipread words were classified at earlier time points relative to heard words, suggesting a predictive mechanism for facilitating speech. These results support a model in which the auditory system combines the joint neural distributions evoked by heard and lipread words to generate a more precise estimate of what was said.

Dynamic Feature Focusing Network for small object detection

Deep learning has driven research in object detection and achieved proud results. Despite its significant advancements in object detection, small object detection still struggles with low recognition rates and inaccurate positioning, primarily attributable to their miniature size. The location deviation of small objects induces severe feature misalignment, and the disequilibrium between classification and regression tasks hinders accurate recognition. To address these issues, we propose a Dynamic Feature Focusing Network (DFFN), which contains a duo of crucial modules: Visual Perception Enhancement Module (VPEM) and Task Association Module (TAM). Drawing upon the deformable convolution and attention mechanism, the VPEM concentrates on sparse key features and perceives the misalignment via positional offset. We aggregate multi-level features at identical spatial locations via layer average operation for learning a more discriminative representation. Incorporating class alignment and bounding box alignment parts, the TAM promotes classification ability, refines bounding box regression, and facilitates the joint learning of classification and localization. We conduct diverse experiments, and the proposed method considerably enhances the small object detection performance on four benchmark datasets of MS COCO, VisDrone, VOC, and TinyPerson. Our method has improved by 3.4 and 2.2 in mAP and APs, making solid improvements on COCO. Compared to other classic detection models, DFFN exhibits a high level of competitiveness in precision.

GuidingBand: Evaluating Speed, Precision and Accuracy of a Vibrotactile Hand Guidance System

Abstract Computerized tactile guidance systems can potentially help in human visual perception tasks such as identifying a desired object in a collection of similar objects. We designed a wrist-wearable tactile guidance system called GuidingBand that provides vibrational cues to help the user pick visual targets out of an array. We conducted three studies to systematically investigate the interplay of machine-led tactile guidance and human visual perception. The studies involve presenting visual targets to users on a screen and giving them visual search tasks. In Study 1, we identified the error rate of our guidance system. We presented participants (N=20) with arrays of identical, square targets to pick from, progressively reduced the target sizes and evaluated error rates for each size. Notably, we observed a 4% error rate at a target size of 10 mm. In Study 2, we compared the error rate of the guidance system with and without the help of human visual perception in a visual search task. We constructed a task that involved showing users an array of distractor rectangles (set sizes 2 to 10) varying only in length and asked them to identify the correct target that was previously shown to them. Participants (N=13) had fewer errors when they tried to identify targets with tactile guidance alone, followed by guidance and perception combined and perception alone. In Study 3, we had the same task as Study 2 but the distractor set sizes were larger (varying from 4 to 28), we had more participants (N=30) and measured errors and task time. While perception alone was the fastest, it was also the most error prone. The combined mode was faster and less error prone than guidance only mode, as expected, indicating that human–computer collaboration yielded the best performance.

Respiration recording for fMRI: breathing belt versus spine coil sensor

Abstract Physiological signals such as pulse and respiration strongly contribute to non-neuronal signal change of the blood oxygenation level-dependent (BOLD) contrast in functional magnetic resonance imaging (fMRI). This has been observed not only during task-based but also during resting-state fMRI measurements, where the confounding influence of physiological signals is most pronounced. Over the last decades, a variety of techniques evolved, aiming at detecting and removing physiological artifacts in fMRI time series. These follow either a solely data-driven approach or rely on externally recorded physiological data. To record cardiac and respiratory signals, typically pulse oximetry or electrocardiography (ECG) and a respiration belt are used, respectively. New technologies allow to capture respiratory signal directly with a sensor placed within the spine coil in the patient table, eliminating the need of a respiration belt, which considerably increases participants’ comfort. However, little is known about the effectiveness of these new technologies and how they compare to the standard respiration belt recording. In the current study, we compared the two devices, respiration belt and spine coil sensor, in their suitability for physiological noise removal during a visual perception task and during rest. We did not find any differences in resting-state functional connectivity (RSFC), stimulus-related activity, or residual noise between data corrected with the two recording devices. Our results show that spine coil-derived respiration recordings are as good as belt respiration recordings for physiological noise removal in task-induced activity, with spine coil recordings having an additional advantage in terms of participants’ comfort and artifact susceptibility.

Network mechanisms of ongoing brain activity’s influence on conscious visual perception

Sensory inputs enter a constantly active brain, whose state is always changing from one moment to the next. Currently, little is known about how ongoing, spontaneous brain activity participates in online task processing. We employed 7 Tesla fMRI and a threshold-level visual perception task to probe the effects of prestimulus ongoing brain activity on perceptual decision-making and conscious recognition. Prestimulus activity originating from distributed brain regions, including visual cortices and regions of the default-mode and cingulo-opercular networks, exerted a diverse set of effects on the sensitivity and criterion of conscious recognition, and categorization performance. We further elucidate the mechanisms underlying these behavioral effects, revealing how prestimulus activity modulates multiple aspects of stimulus processing in highly specific and network-dependent manners. These findings reveal heretofore unknown network mechanisms underlying ongoing brain activity’s influence on conscious perception, and may hold implications for understanding the precise roles of spontaneous activity in other brain functions.

Integrating brain function and structure in the study of the human attentional networks: a functionnectome study

Attention is a heterogeneous function theoretically divided into different systems. While functional magnetic resonance imaging (fMRI) has extensively characterized their functioning, the role of white matter in cognitive function has gained recent interest due to diffusion-weighted imaging advancements. However, most evidence relies on correlations between white matter properties and behavioral or cognitive measures. This study used a new method that combines the signal from distant voxels of fMRI images using the probability of structural connection given by high-resolution normative tractography. We analyzed three fMRI datasets with a visual perceptual task and three attentional manipulations: phasic alerting, spatial orienting, and executive attention. The phasic alerting network engaged temporal areas and their communication with frontal and parietal regions, with left hemisphere dominance. The orienting network involved bilateral fronto-parietal and midline regions communicating by association tracts and interhemispheric fibers. The executive attention network engaged a broad set of brain regions and white matter tracts connecting them, with a particular involvement of frontal areas and their connections with the rest of the brain. These results partially confirm and extend previous knowledge on the neural substrates of the attentional system, offering a more comprehensive understanding through the integration of structure and function.

OccTr: A Two-Stage BEV Fusion Network for Temporal Object Detection

Temporal fusion approaches are critical for 3D visual perception tasks in IOV (Internet of Vehicles), but they often rely on intermediate representations without fully utilizing position information from the previous frame’s detection results, which cannot compensate for the lack of depth information in visual data. In this work, we propose a novel framework called OccTr (Occupancy Transformer) that combines two temporal cues, intermediate representation and back-end representation, via occupancy map to enhance temporal fusion in object detection task. OccTr leverages attention mechanisms to perform both intermediate and back-end temporal fusion by incorporating intermediate BEV (bird’s-eye view) features and back-end prediction results of the detector. Our two-stage framework includes occupancy map generation and cross-attention feature fusion. In stage one, the prediction results are converted into occupancy grid map format to generate back-end representation. In stage two, the high-resolution occupancy maps are fused with BEV features using cross-attention layers. This fused temporal cue provides a strong prior for the temporal detection process. Experimental results demonstrate the effectiveness of our method in improving detection performance, achieving an NDS (nuScenes Detection Score) metric score of 37.35% on the nuScenes test set, which is 1.94 points higher than the baseline.

A Faster and Lightweight Lane Detection Method in Complex Scenarios

Lane detection is a crucial visual perception task in the field of autonomous driving, serving as one of the core modules in advanced driver assistance systems (ADASs).To address the insufficient real-time performance of current segmentation-based models and the conflict between the demand for high inference speed and the excessive parameters in resource-constrained edge devices (such as onboard hardware, mobile terminals, etc.) in complex real-world scenarios, this paper proposes an efficient and lightweight auxiliary branch network (CBGA-Auxiliary) to tackle these issues. Firstly, to enhance the model’s capability to extract feature information in complex scenarios, a row anchor-based feature extraction method based on global features was adopted. Secondly, employing ResNet as the backbone network and CBGA (Conv-Bn-GELU-SE Attention) as the fundamental module, we formed the auxiliary segmentation network, significantly enhancing the segmentation training speed of the model. Additionally, we replaced the standard convolutions in the branch network with lightweight GhostConv convolutions. This reduced the parameters and computational complexity while maintaining accuracy. Finally, an additional enhanced structural loss function was introduced to compensate for the structural defect loss issue inherent in the row anchor-based method, further improving the detection accuracy. The model underwent extensive experimentation on the Tusimple dataset and the CULane dataset, which encompass various road scenarios. The experimental results indicate that the model achieved the highest F1 scores of 96.1% and 71.0% on the Tusimple and CULane datasets, respectively. At a resolution of 288 × 800, the ResNet18 and ResNet34 models achieved maximum inference speeds of 410FPS and 280FPS, respectively. Compared to existing SOTA models, it demonstrates a significant advantage in terms of inference speed. The model achieved a good balance between accuracy and inference speed, making it suitable for deployment on edge devices and validates the effectiveness of the model.

EndoSRR: a comprehensive multi-stage approach for endoscopic specular reflection removal.

Specular reflections in endoscopic images not only disturb visual perception but also hamper computer vision algorithm performance. However, the intricate nature and variability of these reflections, coupled with a lack of relevant datasets, pose ongoing challenges for removal. We present EndoSRR, a robust method for eliminating specular reflections in endoscopic images. EndoSRR comprises two stages: reflection detection and reflection region inpainting. In the reflection detection stage, we adapt and fine-tune the segment anything model (SAM) using a weakly labeled dataset, achieving an accurate reflection mask. For reflective region inpainting, we employ LaMa, a fast Fourier convolution-based model trained on a 4.5M-image dataset, enabling effective inpainting of arbitrarily shaped reflection regions. Lastly, we introduce an iterative optimization strategy for dual pre-trained models to refine the results of specular reflection removal, named DPMIO. Utilizing the SCARED-2019 dataset, our approach surpasses state-of-the-art methods in both qualitative and quantitative evaluations. Qualitatively, our method excels in accurately detecting reflective regions, yielding more natural and realistic inpainting results. Quantitatively, our method demonstrates superior performance in both segmentation evaluation metrics (IoU, E-measure, etc.) and image inpainting evaluation metrics (PSNR, SSIM, etc.). The experimental results underscore the significance of proficient endoscopic specular reflection removal for enhancing visual perception and downstream tasks. The methodology and results presented in this study are poised to catalyze advancements in specular reflection removal, thereby augmenting the accuracy and safety of minimally invasive surgery.

Peak Alpha Frequency in Schizophrenia, Bipolar Disorder, and Healthy Volunteers: Associations With Visual Information Processing and Cognition

BackgroundSchizophrenia (SCZ) and bipolar disorder (BD) are associated with information processing abnormalities, including visual perceptual and cognitive impairments, that impact daily functioning. Recent work with healthy samples suggests that peak alpha frequency (PAF) is an electrophysiological index of visual information processing speed that is correlated with cognitive ability. There is evidence that PAF is slowed in SCZ, but it remains unclear whether PAF is reduced in BD or whether slower PAF is associated with impaired visual perception and cognition in these clinical disorders. MethodsWe recorded resting-state brain activity (both eyes open and closed) with electroencephalography in 90 participants with SCZ, 62 participants with BD, and 69 healthy control participants. Most participants also performed a visual perception task (backward masking) and cognitive testing (MATRICS Concensus Cognitive Battery). ResultsWe replicated previous findings of reduced PAF in patients with SCZ compared with healthy control participants. In contrast, PAF in patients with BD did not differ significantly from that in healthy control participants. Furthermore, PAF was significantly correlated with performance on the perceptual and cognitive measures in SCZ but not BD. PAF was also correlated with visual perception in the healthy control group and showed a trend-level correlation with cognition. ConclusionsTogether, these results suggest that PAF deficits characterize SCZ, but not BD, and that individual differences in PAF are related to abnormalities in visual information processing and cognition in SCZ.

Fast, accurate, and interpretable decoding of electrocorticographic signals using dynamic mode decomposition

Dynamic mode (DM) decomposition decomposes spatiotemporal signals into basic oscillatory components (DMs). DMs can improve the accuracy of neural decoding when used with the nonlinear Grassmann kernel, compared to conventional power features. However, such kernel-based machine learning algorithms have three limitations: large computational time preventing real-time application, incompatibility with non-kernel algorithms, and low interpretability. Here, we propose a mapping function corresponding to the Grassmann kernel that explicitly transforms DMs into spatial DM (sDM) features, which can be used in any machine learning algorithm. Using electrocorticographic signals recorded during various movement and visual perception tasks, the sDM features were shown to improve the decoding accuracy and computational time compared to conventional methods. Furthermore, the components of the sDM features informative for decoding showed similar characteristics to the high-γ power of the signals, but with higher trial-to-trial reproducibility. The proposed sDM features enable fast, accurate, and interpretable neural decoding.

SIAM: A parameter-free, Spatial Intersection Attention Module

Attention mechanisms have been shown to play a crucial role in enhancing visual perception tasks. However, in most existing approaches, channel and spatial attention maps are estimated separately without considering the varying importance of each other. This results in a coarse attention weight for objects of interest from a holistic 3-D perspective. To address this issue, we propose a novel Parameter-free Spatial Intersection Attention Module (SIAM), which estimates 3D attention maps with spatial intersection using a parameter-free way. Specifically, SIAM first generates two independent mean queries from two spatial axes and views input as keys. Then, by computing a dot product between these mean queries and keys, SIAM generates two cross-dimension (channel and spatial) attention maps from two spatial directions and combines them into 3-D attention maps. By doing so, the produced attention maps reason important areas with spatial intersection, which can capture location-aware information to facilitate difficult objects’ location in the images. We evaluate our method in image classification, object detection, and object segmentation tasks. Extensive experimental results consistently demonstrate our approach is superior to its counterparts.

Assessing Trial-by-Trial Electrophysiological and Behavioral Markers of Attentional Control and Sensory Precision in Psychotic and Mood Disorders.

The current study investigated the extent to which changes in attentional control contribute to performance on a visual perceptual discrimination task, on a trial-by-trial basis in a transdiagnostic clinical sample. Participants with schizophrenia (SZ; N = 58), bipolar disorder (N = 42), major depression disorder (N = 51), and psychiatrically healthy controls (N = 92) completed a visual perception task in which stimuli appeared briefly. The design allowed us to estimate the lapse rate and the precision of perceptual representations of the stimuli. Electroencephalograms (EEG) were recorded to examine pre-stimulus activity in the alpha band (8-13 Hz), overall and in relation to behavior performance on the task. We found that the attention lapse rate was elevated in the SZ group compared with all other groups. We also observed group differences in pre-stimulus alpha activity, with control participants showing the highest levels of pre-stimulus alpha when averaging across trials. However, trial-by-trial analyses showed within-participant fluctuations in pre-stimulus alpha activity significantly predicted the likelihood of making an error, in all groups. Interestingly, our analysis demonstrated that aperiodic contributions to the EEG signal (which affect power estimates across frequency bands) serve as a significant predictor of behavior as well. These results confirm the elevated attention lapse rate that has been observed in SZ, validate pre-stimulus EEG markers of attentional control and their use as a predictor of behavior on a trial-by-trial basis, and suggest that aperiodic contributions to the EEG signal are an important target for further research in this area, in addition to alpha-band activity.

Multi-task multi-objective evolutionary network for hyperspectral image classification and pansharpening

Multi-task learning has commonly been used and performed well at joint visual perception tasks. Hyperspectral pansharpening (HP) and hyperspectral classification (HC) tasks extract high-frequency information to enhance edges and classify samples, offering potential for performance improvements in multi-task learning. However, differences between tasks can make it challenging to balance their performances. To address this challenge, this paper proposes a multi-task multi-objective evolutionary network (DMOEAD) for joint learning of HC and HP. A multi-task sufficiency-and-diversity sampling method is designed to unify the heterogeneity of sample construction between two types of tasks. Two types of task-specific networks are constructed to decompose high-frequency information. Further, a collaborative learning module is designed to dynamically learn complementary high-frequency information from another task in different layers. To be compatible with the optimization direction of two types of tasks, multi-task optimization is realized using a deep multi-objective evolutionary algorithm (DMEO). In the DMEO, the set of parameters of the DMOEAD is regarded as an individual. A deep mutation operator is designed and used for network optimization, which accelerates large-scale network parameter searching. The DMEO can coordinate the differences between multiple tasks and provide a set of Pareto network parameter solutions. Finally, the experimental results demonstrate that the proposed method can significantly enhance the performance of both pansharpening and classification tasks.

Hypergraph of Functional Connectivity Based on Event-Related Coherence: Magnetoencephalography Data Analysis

We construct hypergraphs to analyze functional brain connectivity, leveraging event-related coherence in magnetoencephalography (MEG) data during the visual perception of a flickering image. Principal network characteristics are computed for the delta, theta, alpha, beta, and gamma frequency ranges. Employing a coherence measure, a statistical estimate of correlation between signal pairs across frequencies, we generate an edge time series, depicting how an edge evolves over time. This forms the basis for constructing an edge-to-edge functional connectivity network. We emphasize hyperedges as connected components in an absolute-valued functional connectivity network. Our coherence-based hypergraph construction specifically addresses functional connectivity among four brain lobes in both hemispheres: frontal, parietal, temporal, and occipital. This approach enables a nuanced exploration of individual differences within diverse frequency bands, providing insights into the dynamic nature of brain connectivity during visual perception tasks. The results furnish compelling evidence supporting the hypothesis of cortico–cortical interactions occurring across varying scales. The derived hypergraph illustrates robust activation patterns in specific brain regions, indicative of their engagement across diverse cognitive contexts and different frequency bands. Our findings suggest potential integration or multifunctionality within the examined lobes, contributing valuable perspectives to our understanding of brain dynamics during visual perception.