Visual Scene Research Articles

Causal inference predicts the transition from integration to segmentation in motion perception.

Motion provides a powerful sensory cue for segmenting a visual scene into objects and inferring the causal relationships between objects. Fundamental mechanisms involved in this process are the integration and segmentation of local motion signals. However, the computations that govern whether local motion signals are perceptually integrated or segmented remain unclear. Hierarchical Bayesian causal inference has recently been proposed as a model for these computations, yet a hallmark prediction of the model - its dependency on sensory uncertainty - has remained untested. We used a recently developed hierarchical stimulus configuration to measure how human subjects integrate or segment local motion signals while manipulating motion coherence to control sensory uncertainty. We found that (a) the perceptual transition from motion integration to segmentation shifts with sensory uncertainty, and (b) that perceptual variability is maximal around this transition point. Both findings were predicted by the model and challenge conventional interpretations of motion repulsion effects.

Robotic Grasping Detection Algorithm Based on 3D Vision Dual-Stream Encoding Strategy

The automatic generation of stable robotic grasping postures is crucial for the application of computer vision algorithms in real-world settings. This task becomes especially challenging in complex environments, where accurately identifying the geometric shapes and spatial relationships between objects is essential. To enhance the capture of object pose information in 3D visual scenes, we propose a planar robotic grasping detection algorithm named SU-Grasp, which simultaneously focuses on local regions and long-distance relationships. Built upon a U-shaped network, SU-Grasp introduces a novel dual-stream encoding strategy using the Swin Transformer combined with spatial semantic enhancement. Compared to existing baseline methods, our algorithm achieves superior performance across public datasets, simulation tests, and real-world scenarios, highlighting its robust understanding of complex spatial environments.

Visual Processing by Hierarchical and Dynamic Multiplexing.

The complexity of natural environments requires highly flexible mechanisms for adaptive processing of single and multiple stimuli. Neuronal oscillations could be an ideal candidate for implementing such flexibility in neural systems. Here, we present a framework for structuring attention-guided processing of complex visual scenes in humans, based on multiplexing and phase coding schemes. Importantly, we suggest that the dynamic fluctuations of excitability vary rapidly in terms of magnitude, frequency and wave-form over time, i.e., they are not necessarily sinusoidal or sustained oscillations. Different elements of single objects would be processed within a single cycle (burst) of alpha activity (7-14 Hz), allowing for the formation of coherent object representations while separating multiple objects across multiple cycles. Each element of an object would be processed separately in time-expressed as different gamma band bursts (>30 Hz)-along the alpha phase. Since the processing capacity per alpha cycle is limited, an inverse relationship between object resolution and size of attentional spotlight ensures independence of the proposed mechanism from absolute object complexity. Frequency and wave-shape of those fluctuations would depend on the nature of the object that is processed and on cognitive demands. Multiple objects would further be organized along the phase of slower fluctuations (e.g., theta), potentially driven by saccades. Complex scene processing, involving covert attention and eye movements, would therefore be associated with multiple frequency changes in the alpha and lower frequency range. This framework embraces the idea of a hierarchical organization of visual processing, independent of environmental temporal dynamics.

EEG frequency tagging reveals the integration of dissimilar observed actions

Extensive research has demonstrated that visual and motor cortices can simultaneously represent multiple observed actions. This ability undoubtedly constitutes a crucial ingredient for the understanding of complex visual scenes involving different agents. However, it is still unclear how these distinct representations are integrated into coherent and meaningful percepts. In line with studies of perceptual binding, we hypothesized that similar movements would be more easily integrated. To test this hypothesis, we developed an EEG frequency tagging experiment in which two hand movements were displayed simultaneously at two different presentation rates. Crucially, the degree of similarity between the two movements varied along two dimensions, namely action identity (i.e., same or different performed movement), and agent identity (i.e., one agent performing a bimanual movement, or two agents moving each one hand). Contrary to our predictions, we found a larger intermodulation oscillatory component, indexing the integrated processing of the two individual movements, when they were less similar. We propose that integration-by-dissimilarity might serve as a top-down process to solve conflict caused by incongruent movements, thus contributing to the global understanding of distinct moving individuals in a complex social scene.

Zero-shot counting with a dual-stream neural network model.

To understand a visual scene, observers need to both recognize objects and encode relational structure. For example, a scene comprising three apples requires the observer to encode concepts of "apple" and "three." In the primate brain, these functions rely on dual (ventral and dorsal) processing streams. Object recognition in primates has been successfully modeled with deep neural networks, but how scene structure (including numerosity) is encoded remains poorly understood. Here, we built a deep learning model, based on the dual-stream architecture of the primate brain, which is able to count items "zero-shot"-even if the objects themselves are unfamiliar. Our dual-stream network forms spatial response fields and lognormal number codes that resemble those observed in the macaque posterior parietal cortex. The dual-stream network also makes successful predictions about human counting behavior. Our results provide evidence for an enactive theory of the role of the posterior parietal cortex in visual scene understanding.

A Possible Neural Basis for Attentional Capture of Faces Revealed by Functional Magnetic Resonance Imaging and Causal Pharmacological Inactivation in Macaques

Abstract In primates, the presence of a face in a visual scene captures attention and rapidly directs the observer's gaze to the face, even when the face is not relevant to the task at hand. Here, we explored a neural circuit that might potentially play a causal role in this powerful behavior. In our previous research, two monkeys received microinfusions of muscimol, a γ-aminobutyric acid type A (GABAA)-receptor agonist, or saline (as a control condition) in separate sessions into individual or pairs of four inferotemporal face patches (middle and anterior lateral and fundal), as identified by an initial localizer experiment. Then, using fMRI, we measured the impact of each inactivation condition on responses in the other face patches relative to the control condition. In this study, we used the same method and measured the impact of each inactivation condition on responses in the FEF and the lateral intraparietal area, two regions associated with attentional processing, while face and nonface object stimuli were viewed. Our results revealed potential relationships between inferotemporal face patches and these two attention-related regions: The inactivation of the middle lateral and anterior fundal face patches had a pronounced impact on FEF, whereas the inactivation of the middle and anterior lateral face patches had a noticeable influence on LIP. Together, these initial exploratory findings document a circuit that potentially underlies the attentional capture of faces. Confirmation of the role of this circuit remains to be accomplished in the context of a paradigm that explicitly tests the attentional capture of faces.

Learning differentiable logic programs for abstract visual reasoning

AbstractVisual reasoning is essential for building intelligent agents that understand the world and perform problem-solving beyond perception. Differentiable forward reasoning has been developed to integrate reasoning with gradient-based machine learning paradigms. However, due to the memory intensity, most existing approaches do not bring the best of the expressivity of first-order logic, excluding a crucial ability to solve abstract visual reasoning, where agents need to perform reasoning by using analogies on abstract concepts in different scenarios. To overcome this problem, we propose NEUro-symbolic Message-pAssiNg reasoNer (NEUMANN), which is a graph-based differentiable forward reasoner, passing messages in a memory-efficient manner and handling structured programs with functors. Moreover, we propose a computationally-efficient structure learning algorithm to perform explanatory program induction on complex visual scenes. To evaluate, in addition to conventional visual reasoning tasks, we propose a new task, visual reasoning behind-the-scenes, where agents need to learn abstract programs and then answer queries by imagining scenes that are not observed. We empirically demonstrate that NEUMANN solves visual reasoning tasks efficiently, outperforming neural, symbolic, and neuro-symbolic baselines.

Language at a glance: How our brains grasp linguistic structure from parallel visual input.

Human brains grasp the gists of visual scenes from a single glance, but to what extent is this possible for language? While we typically think of language in terms of sequential speech, our everyday experience involves numerous rapidly flashing written notifications, which we understand instantly. What do our brains detect in the first few hundred milliseconds after seeing such a stimulus? We flashed short sentences during magnetoencephalography measurement, revealing sentence-sensitive neural activity in left temporal cortex within 130 milliseconds. These signals emerged for subject-verb-object sentences regardless of grammatical or semantic well-formedness, suggesting that at-a-glance language comprehension begins by detecting basic phrase structure, independent of meaning or other grammatical details. Our findings unveil one aspect of how our brains process information rapidly in today's visually saturated world.

Vocabulary Acquisition in Beginner L2 Learners

This project focuses on how different vocabulary teaching methods may develop efficient lexical-semantic networks in L2 learners. Vocabulary that frequently co-occurs in the environment becomes associated in the lexicon (e.g., food + sports) and it strengthens the representation of vocabulary. This experiment tests the effect of co-occurrence presentation and thematic list presentation on vocabulary processing. The goal is to inform vocabulary teaching methods. L2 learners completed a pre-test, four sessions of vocabulary training, and an immediate post-test. During training sessions, participants saw images on a computer and reviewed Spanish vocabulary, under one of two conditions: one word presented at a time in thematic lists (e.g., food), or words displayed in a visual scene or context (e.g., dinner party). During testing, participants completed a picture naming task to assess how different types of vocabulary exposure impacts L2 processing. Other individual differences such as working memory and proficiency were collected as well. The results from the picture naming task show that effectiveness of training in thematic list (TL) verse visual scene (VS) depended on initial proficiency and working memory. Participants with a higher initial proficiency and working memory improved more when trained with the VS condition, while those with lower proficiency and working memory had greater success when trained via the TL condition.

Integrated encoding of relations and objects in visual working memory.

Comprehensive understanding of visual scenes necessitates grasping the relations among visual objects. Given the potentially pivotal role of visual working memory (VWM) in processing visual relations, it is important to investigate the representation of relations in VWM. In our previous study, we proposed the integrated storage hypothesis, postulating that relations and objects are stored together as an integrated structured representation in VWM. The present study aimed to test this hypothesis against the alternative separate encoding hypothesis by probing the irrelevant-distracting effect. Across three experiments, where participants memorized object shapes/colors while disregarding relations, an irrelevant-distracting effect was consistently observed across varying types of changes in relation and set sizes. Critically, recombining the probe with irrelevant relation from another memory item (Experiment 2) or reversing the relational roles of probed objects relative to the memory item (Experiment 3) were perceived as inconsistency with stored representations and impaired change detection. These findings supported the integrated storage hypothesis, indicating that the dynamic relations between the objects are automatically encoded alongside object identities to form an integrated structured representation. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Research on the regulation of emotions in elderly people through bedroom audio-visual environments

The indoor environment of buildings has a long-term impact on the psychological and physiological state of people, and a good indoor environment can help the physical and mental health of the elderly. As the mobility of the elderly weakens, the indoor visual and auditory environments are important channels for them to acquire external perceptions. In order to investigate the effects of visual and auditory environments on the psychological (positive and negative emotions) and physiological (heart rate) states of older adults in their bedrooms, the study designed an audio-visual interaction experiment, selecting three common sounds preferred by older adults and six visual scenes that included two types of visual factors , to measure the emotions and heart rate changes under different sound and visual interaction conditions. The study revealed that natural sounds were more effective in positively modulating mood in warm-toned rooms, while artificial sounds were more appropriate for cool-toned rooms. Regardless of the type of therapeutic sound environment, the addition of greenery significantly contributes to mood regulation in older adults. Combining sound environments with visual factors can be an effective design strategy to promote the physical and mental health of occupants.

A chromatic feature detector in the retina signals visual context changes.

The retina transforms patterns of light into visual feature representations supporting behaviour. These representations are distributed across various types of retinal ganglion cells (RGCs), whose spatial and temporal tuning properties have been studied extensively in many model organisms, including the mouse. However, it has been difficult to link the potentially nonlinear retinal transformations of natural visual inputs to specific ethological purposes. Here, we discover a nonlinear selectivity to chromatic contrast in an RGC type that allows the detection of changes in visual context. We trained a convolutional neural network (CNN) model on large-scale functional recordings of RGC responses to natural mouse movies, and then used this model to search in silico for stimuli that maximally excite distinct types of RGCs. This procedure predicted centre colour opponency in transient suppressed-by-contrast (tSbC) RGCs, a cell type whose function is being debated. We confirmed experimentally that these cells indeed responded very selectively to Green-OFF, UV-ON contrasts. This type of chromatic contrast was characteristic of transitions from ground to sky in the visual scene, as might be elicited by head or eye movements across the horizon. Because tSbC cells performed best among all RGC types at reliably detecting these transitions, we suggest a role for this RGC type in providing contextual information (i.e. sky or ground) necessary for the selection of appropriate behavioural responses to other stimuli, such as looming objects. Our work showcases how a combination of experiments with natural stimuli and computational modelling allows discovering novel types of stimulus selectivity and identifying their potential ethological relevance.

Content-based clustering of hallucinations across sensory modalities in a large online survey

Hallucinations can have rather heterogeneous aetiology and presentation. This inspired the concept of different subtypes based on symptom profiles, especially in the field of auditory hallucinations. As many people experience hallucinations in more than one sensory modality, it seems important to investigate potential hallucination subtypes across different sensory modalities. We assessed the content of hallucinations as part of a large survey among the general Dutch population (n = 10,448) using the Questionnaire for Psychotic Experiences. Based on their descriptions, thematic categories were created in a data-driven cluster analysis. 2594 participants who experienced hallucinations over the past week that contained at least 2 different thematic categories were selected. Clustering of their hallucination content was performed with the HDBSCAN method. We identified 4 clusters, i.e., subtypes, which can be typified as 1. hallucinations of foul odors, 2. complex visual scenes, 3. a vast variety of rather common hallucinations possibly related to heightened alertness, and 4. possibly bereavement hallucinations. The bereavement subtype showed an increase in emotional loneliness and the presence of delusions. Our findings suggest that the content of hallucinations can be informative, especially when investigated across sensory modalities. Such subtypes may help to better understand their underlying mechanisms.

Deep learning-powered visual place recognition for enhanced mobile multimedia communication in autonomous transport systems

The progress of autonomous transport systems (ATS) involves efficient multimedia communication for real-time data tradeoffs and environmental issues. Deep learning (DL) powered visual place recognition (VPR) was developed as an effective tool to improve mobile multimedia communication in ATS. VPR relates to the capability of a method or device to recognize and identify particular places or locations from the visual scene. This procedure involves inspecting visual data, like images or video frames, to control the unique features or features connected with diverse locations. By leveraging camera sensors, VPR allows vehicles to detect their surroundings, enabling context-aware communication and enhancing the entire system's performance. DL-empowered VPR offers a transformative manner to improve mobile multimedia communication in ATS. By identifying and understanding their situation, autonomous vehicles can communicate most effectively and operate reliably and safely, paving the way for a future characterized by seamless and intelligent transportation. This article develops a novel Deep Learning-Powered Visual Place Recognition for Enhanced Multimedia Communication in Autonomous Transport Systems (DLVPR-MCATS) methodology. The main aim of the DLVPR-MCATS methodology is to recognize visual places or not utilize optimal DL approaches. For this purpose, the DLVPR-MCATS approach utilizes a bilateral filtering (BF) based preprocessing model. For the feature fusion model, the DLVPR-MCATS approach follows three models: residual network (ResNet), EfficientNet, and MobileNetv2. Moreover, the hyperparameter tuning method uses the Harris Hawks Optimization (HHO) model. Finally, the bidirectional long short-term memory (BiLSTM) technique is implemented to recognize visual places. A wide range of simulations is executed to validate the solution of the DLVPR-MCATS method. The experimental validation of the DLVPR-MCATS method portrayed a superior performance over other models concerning various aspects.

Application of indoor thermal energy cycle and visual communication simulation scene based on Machine vision and optical image enhancement

Traditional thermal analysis methods have limitations in complex environments, and innovative technologies are urgently needed to improve the accuracy of indoor thermal monitoring and optimization. In this paper, machine vision and optical image enhancement technology are combined to simulate the indoor heat energy cycle process to achieve more efficient heat management and visual communication, so as to improve the energy utilization efficiency and residential comfort of buildings. In this paper, a high-resolution camera is used to capture real-time images of indoor environment, image quality is enhanced by image processing algorithm, and heat distribution characteristics are extracted. The machine learning model is used to analyze the dynamic changes of the thermal energy cycle, and the visual model of the indoor thermal energy cycle is constructed by combining the fluid dynamics theory. The experimental results show that the proposed method can effectively identify and display the key areas of indoor heat circulation and their changing trends. The simulation results show that the optical enhanced images can show different temperature regions more clearly, directly reflect the heat flow situation, and have a high agreement with the actual measurement data. Therefore, the indoor thermal energy cycle simulation method based on machine vision and optical image enhancement not only improves the accuracy and real-time performance of thermal energy analysis, but also provides visual support for indoor environmental management.

Ensemble memory of a scene interacts with current perception regardless of attentional requirements.

How do we maintain a rich and stable perceptual experience across the entire visual scene, even when we are focusing on a subset of visual inputs? The current study explored this question by investigating whether the visual system processes summary statistics of multiple features regardless of task relevance, and how they interact with subsequent perception. To test the processing of multifeature summary statistics under different attentional requirements, we presented multiple Gabor patches with heterogeneous orientations/colors and asked participants to attend to a single feature dimension (Experiments 1 and 3) or a single item (Experiment 2) for the memory task. During the memory maintenance period (before memory response), we asked the participants to perform a discrimination task (Experiments 1 and 2) or a boundary localization task (Experiment 3) to test how the memory of the ensemble representation alters the subsequent perceptual experience. We found evidence for obligatory processing of scene summary statistics presented for the memory task, which interacted with the subsequent perceptual sensitivity. Specifically, not only summary statistics relevant but also those of task-irrelevant feature (Experiments 1 and 3) and outside the focus of attention (Experiment 2) were encoded and bidirectionally interacted with subsequent perception. These results suggest obligatory processing of summary statistics of a scene, which may allow rich and stable visual experience by integrating temporally adjacent visual inputs. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Centrifugal Inertia-Induced Directional Alignment of AgNW Network for Preparing Transparent Electromagnetic Interference Shielding Films with Joule Heating Ability.

Transparent electromagnetic interference (EMI) shielding is highly desired in specific visual scenes, but the challenge remains in balancing their EMI shielding effectiveness (SE) and optical transmittance. Herein, this study proposed a directionally aligned silver nanowire (AgNW) network construction strategy to address the requirement of high EMI SE and satisfactory light transmittance using a rotation spraying technique. The orientation distribution of AgNW is induced by centrifugal inertia force generated by a high-speed rotating roller, which overcomes the issue of high contact resistance in random networks and achieves high conductivity even at low AgNW network density. Thus, the obtained transparent conductive film achieved a high light transmittance of 72.9% combined with a low sheet resistance of 4.5 Ω sq-1 and a desirable EMI SE value of 35.2dB at X band, 38.9dB in the K-band, with the highest SE of 43.4dB at 20.4GHz. Simultaneously, the excellent conductivity endowed the film with outstanding Joule heating performance and defogging/deicing ability, ensuring the visual transparency of windows when shielding electromagnetic waves. Hence, this research presents a highly effective strategy for constructing an aligned AgNW network, offering a promising solution for enhancing the performance of optical-electronic devices.

PV-LaP: Multi-sensor fusion for 3D Scene Understanding in intelligent transportation systems

Intelligent transportation systems are pivotal in modern urban development, aiming to enhance traffic management efficiency, safety, and sustainability. However, existing 3D Visual Scene Understanding methods often face challenges of robustness and high computational complexity in complex traffic environments. This paper proposes a Multi-Sensor Signal Fusion method based on PV-RCNN and LapDepth (PV-LaP) to improve 3D Visual Scene Understanding. By integrating camera and LiDAR data, the PV-LaP method enhances environmental perception accuracy. Evaluated on the KITTI and WHU-TLS datasets, the PV-LaP framework demonstrated superior performance. On the KITTI dataset, our method achieved an Absolute Relative Error (Abs Rel) of 0.079 and a Root Mean Squared Error (RMSE) of 3.014, outperforming state-of-the-art methods. On the WHU-TLS dataset, the method improved 3D reconstruction precision with a PSNR of 19.15 and an LPIPS of 0.299. Despite its high computational demands, PV-LaP offers significant improvements in accuracy and robustness, providing valuable insights for the future development of intelligent transportation systems.

Visual Perceptual Learning of Form-Motion Integration: Exploring the Involved Mechanisms with Transfer Effects and the Equivalent Noise Approach.

Background: Visual perceptual learning plays a crucial role in shaping our understanding of how the human brain integrates visual cues to construct coherent perceptual experiences. The visual system is continually challenged to integrate a multitude of visual cues, including form and motion, to create a unified representation of the surrounding visual scene. This process involves both the processing of local signals and their integration into a coherent global percept. Over the past several decades, researchers have explored the mechanisms underlying this integration, focusing on concepts such as internal noise and sampling efficiency, which pertain to local and global processing, respectively. Objectives and Methods: In this study, we investigated the influence of visual perceptual learning on non-directional motion processing using dynamic Glass patterns (GPs) and modified Random-Dot Kinematograms (mRDKs). We also explored the mechanisms of learning transfer to different stimuli and tasks. Specifically, we aimed to assess whether visual perceptual learning based on illusory directional motion, triggered by form and motion cues (dynamic GPs), transfers to stimuli that elicit comparable illusory motion, such as mRDKs. Additionally, we examined whether training on form and motion coherence thresholds improves internal noise filtering and sampling efficiency. Results: Our results revealed significant learning effects on the trained task, enhancing the perception of dynamic GPs. Furthermore, there was a substantial learning transfer to the non-trained stimulus (mRDKs) and partial transfer to a different task. The data also showed differences in coherence thresholds between dynamic GPs and mRDKs, with GPs showing lower coherence thresholds than mRDKs. Finally, an interaction between visual stimulus type and session for sampling efficiency revealed that the effect of training session on participants' performance varied depending on the type of visual stimulus, with dynamic GPs being influenced differently than mRDKs. Conclusion: These findings highlight the complexity of perceptual learning and suggest that the transfer of learning effects may be influenced by the specific characteristics of both the training stimuli and tasks, providing valuable insights for future research in visual processing.

Visual search and real-image similarity: An empirical assessment through the lens of deep learning.

The ability to predict how efficiently a person finds an object in the environment is a crucial goal of attention research. Central to this issue are the similarity principles initially proposed by Duncan and Humphreys, which outline how the similarity between target and distractor objects (TD) and between distractor objects themselves (DD) affect search efficiency. However, the search principles lack direct quantitative support from an ecological perspective, being a summary approximation of a wide range of lab-based results poorly generalisable to real-world scenarios. This study exploits deep convolutional neural networks to predict human search efficiency from computational estimates of similarity between objects populating, potentially, any visual scene. Our results provide ecological evidence supporting the similarity principles: search performance continuously varies across tasks and conditions and improves with decreasing TD similarity and increasing DD similarity. Furthermore, our results reveal a crucial dissociation: TD and DD similarities mainly operate at two distinct layers of the network: DD similarity at the intermediate layers of coarse object features and TD similarity at the final layers of complex features used for classification. This suggests that these different similarities exert their major effects at two distinct perceptual levels and demonstrates our methodology's potential to offer insights into the depth of visual processing on which the search relies. By combining computational techniques with visual search principles, this approach aligns with modern trends in other research areas and fulfils longstanding demands for more ecologically valid research in the field of visual search.