Visual Scene Research Articles

Verbal Communication in the Upin and Ipin Animated Series (Quantitative Content Analysis of Episode of the Upin and Ipin Series – Gong Xi Fa Cai)

This research will discuss verbal communication in the Upin and Ipin animation series, content analysis of episodes of the Upin and Ipin animation series – Gong Xi Fa cai. This episode was chosen because it has the highest viewership in terms of the regional language. The research uses a quantitative research method that will discuss six aspects of verbal communication. The data collection technique in this study uses initial observations on the series, tracing the episodes with the highest viewing Malay category, encoding each visual scene of the episode and inferring the results of the episode on verbal communication. So it can be concluded that in the episode of the animated series Upin and Ipin – Gong Xi Fa Cai has a score in the vocabulary aspect of 71%, racing 96%. voice intonation 93%, humor 8%, concise and clear 98% and timing 89%. Of the six aspects of verbal communication, the short and clear aspects part that received the highest score of 98% and it can be said that the characters tend to be more consistent with the core message of the conversation discussed when dialogue or communication.

Neuromorphic visual scene understanding with resonator networks

Neuromorphic visual scene understanding with resonator networks

Метод виділення контурів для вирішення завдань моніторингу транспортної інфраструктури

Introduction. Technologies for monitoring transport infrastructure have been rapidly evolving in recent years, absorbing innovations and the latest developments. The main direction of development and use for this technology has been the implementation of continuous monitoring and control of different aspects of transport infrastructure to ensure its safety and allow efficient and timely management. Computer vision has been playing the main role in the evolution of these technologies and has made unmanned aerial vehicles (UAVs) the most cost-efficient remote monitoring tool. The purpose of the paper. Among the main tasks in the field are monitoring traffic and the conditions of road surfaces and markings. Fast and accurate monitoring systems enable quick responses and minimize negative consequences for citizens. Despite the active development of computer vision algorithms, there is no universal algorithm that suits all scenarios. Algorithms depend on the task, conditions, and even UAV trajectory; even a slight change in the visual scene can cause suboptimal results. Lately, significant progress has been made in the development of edge detection algorithms. However, they do not consider the specifics of the task of monitoring road markings. The algorithm should consider the characteristics of the objects of interest – their geometric and color features, and the presence of many other objects in the images. The goal of this paper is to present method crafted specifically for the task of monitoring transport infrastructure. Methods. Computer vision, threshold filtering, Sobel operator, noise removal, probabilistic Hough transform, histograms. Results. The main features of the task of monitoring transport infrastructure using visual data obtained from surveillance cameras or unmanned aerial vehicles have been analyzed. Developed an algorithm for boundary extraction of point clusters using histograms. A method for edge detection in images has been developed, which addresses the shortcomings of known methods and is specifically enhanced for transport infrastructure monitoring. The method leverages the narrow specialization of the task to improve the obtained results. The foundation of the method is based on the features of the HSL color model, filtering in the saturation and lightness channels using gradients obtained from the Sobel operator, segment detection based on the probabilistic Hough transform, and a developed algorithm for boundary extraction of point clusters using histograms. Conclusion. The proposed method can be used in automated and semi-automated decision-making systems, UAV design bureaus, UAV manufacturing enterprises, and information-analytical centers to develop unmanned aviation systems and aerial monitoring technologies to enhance human safety and the economic development of the state. The use of automatic remote monitoring data processing methods allows for faster acquisition of necessary results and improves the efficiency of using geospatial data. Keywords: Computer vision, object detection, edge detection, image filtering, transportation infrastructure, information technology, monitoring.

Reverberation divergence in VR applications

This project aimed to investigate the correlation between virtual reality (VR) imagery and ambisonic sound. With the increasing popularity of VR applications, understanding how sound is perceived in virtual environments is crucial for enhancing the immersiveness of the experience. In the experiment, participants were immersed in a virtual environment that replicated a concert hall. Their task was to assess the correspondence between sound scenes (which differed in reverberation times and their characteristics) and the observed invariant visual scene. The research was conducted using paired tests. Participants were asked to identify the sound scene they considered more closely matched the concert hall seen in the VR goggles for each pair. Each sound scene differed in the employed impulse response. All the impulse responses were recorded in real venues such as concert halls, auditoriums, churches, etc. To provide a realistic auditory experience, the sound scenes were processed using third-order ambisonics and decoded using binaural techniques with HRTFs. The virtual concert hall was generated using the Unreal Engine and was the same for all the tests. One of the major conclusions drawn from the conducted research was confirming the role of spatial sound in creating immersive VR experiences. The study demonstrated that appropriately matching spatial sound to the VR visual scene is essential for achieving complete immersion. Additionally, expectations and preferences regarding reverberation characteristics in different types of spaces were discovered. These findings have significant implications for the design of virtual environments, and understanding these aspects can contribute to improving VR technology and creating more immersive and realistic virtual experiences for users.

Prediction of sensorimotor contingencies generates saccadic omission

With every movement of our eyes, the visual receptors in the retina are swiped across the visual scene. Saccades are the fastest and most frequent movements we perform, yet we remain unaware of the self-produced visual motion. Previous research has tried to identify a dedicated suppression mechanism that either actively or passively cancels vision at the time of saccades.1 Here, we investigated a novel theory, which states that saccadic omission results from habituation to the predicted sensory consequences of our own actions. We experimentally induced novel, i.e., artificial visual consequences of saccade performance by presenting gratings that were drifting faster than the flicker fusion frequency and that became visible only when participants performed saccades. We asked participants to perform more than 100 saccades in each session across these gratings to make the novel contingencies predictable for the sensorimotor system. We found that contrast sensitivity for intra-saccadic motion declined drastically after repeated exposure of such motion. The reduction in sensitivity was even specific to the saccade vector performed in habituation trials. Moreover, when subjects performed the same task in fixation, no reduction in sensitivity was observed. In a motion speed comparison task, we found that the reduction in contrast sensitivity is the consequence of silencing-predicted intra-saccadic visual motion. Our data demonstrate that the sensorimotor system selectively habituates to recurring intra-saccadic visual motion, suggesting an efficient prediction mechanism of visual stability.

Adopting the visual perspective of a group member is influenced by implicit group averaging.

As social entities, individuals' perception and behaviors are susceptible to the influence of their social groups. Previous research has consistently shown that the group context in which individuals are situated significantly influences their perceptual processing. We aim to investigate whether the group context in which another individual is situated alters our understanding of their visual perception, which holds profound implications for interpersonal interactions. To address this inquiry, we conducted three experiments. In Experiment 1, participants were presented with a visual scene depicting multiple avatars seated around a table, all facing an arrow positioned at the center of the table. They were instructed to adopt the visual perspective of a specific avatar within the group to perceive the arrow's orientation, and then reproduce its orientation from their own perspectives. We found that participants exhibited a bias towards the group's average perspective when reproducing the arrow's orientation. Experiment 2 further demonstrated that reinforced group processing could elicit an earlier appearance of this bias. In Experiment 3, we investigated an alternative explanation positing that the aforementioned bias originated from visual ensemble perception rather than group influence by instructing participants to reproduce the target avatar's position relative to the arrow's orientation. If the bias indeed originated from ensemble perception, it should also manifest in this task. However, the absence of any reproduction bias refuted this possibility. Through these experiments, we demonstrate that our understanding of an individual's perceptual experiences is influenced by the social context in which they are situated, which manifests as a convergence phenomenon.

Eye-specific detection and a multi-eye integration model of biological motion perception.

'Biological motion' refers to the distinctive kinematics observed in many living organisms, where visually perceivable points on the animal move at fixed distances from each other. Across the animal kingdom, many species have developed specialized visual circuitry to recognize such biological motion and to discriminate it from other patterns. Recently, this ability has been observed in the distributed visual system of jumping spiders. These eight-eyed animals use six eyes to perceive motion, while the remaining two (the principal anterior medial eyes) are shifted across the visual scene to further inspect detected objects. When presented with a biologically moving stimulus and a random one, jumping spiders turn to face the latter, clearly demonstrating the ability to discriminate between them. However, it remains unclear whether the principal eyes are necessary for this behavior, whether all secondary eyes can perform this discrimination, or whether a single eye-pair is specialized for this task. Here, we systematically tested the ability of jumping spiders to discriminate between biological and random visual stimuli by testing each eye-pair alone. Spiders were able to discriminate stimuli only when the anterior lateral eyes were unblocked, and performed at chance levels in other configurations. Interestingly, spiders showed a preference for biological motion over random stimuli - unlike in past work. We therefore propose a new model describing how specialization of the anterior lateral eyes for detecting biological motion contributes to multi-eye integration in this system. This integration generates more complex behavior through the combination of simple, single-eye responses. We posit that this in-built modularity may be a solution to the limited resources of these invertebrates' brains, constituting a novel approach to visual processing.

UniQRNet: Unifying Referring Expression Grounding and Segmentation with QRNet

Referring expression comprehension aims to align natural language queries with visual scenes, which requires establishing fine-grained correspondence between vision and language. This has important applications in multi-modal reasoning systems. Existing methods typically use text-agnostic visual backbones to extract features independently without considering the specific text input. However, we argue that the extracted visual features can be inconsistent with the referring expression, which hurts multi-modal understanding. To address this, we first propose Query-modulated Refinement Network (QRNet) that leverages language guidance to guide visual feature extraction. However, it only focuses on the grounding task that can only provide coarse-grained annotations in the form of bounding box coordinates. The guidance for the visual backbone is indirect, and the inconsistent issue still exists. To this end, we further propose UniQRNet, a multi-task framework over the QRNet to learn referring expression grounding and segmentation jointly. The framework introduces a multi-task head that leverages fine-grained pixel-level supervision from the segmentation task to directly guide the intermediate layers of QRNet to learn text-consistent visual features. Besides, UniQRNet also includes a loss balance strategy that allows two types of supervision signals to cooperate and optimize the model together. We conduct the most comprehensive comparison experiment covering four major datasets, ten evaluation set and three evaluation metrics used in previous work. UniQRNet outperforms previous state-of-the-art methods by a large margin on both referring comprehensive grounding (1.8%~5.09%) and segmentation tasks (0.57%~5.56%). Ablation and analysis reveal that UniQRNet can improve the consistency of visual features with text input and can bring significant performance improvement.

Visual fidelity in the metaverse matters for memory performance

The metaverse offers unlimited opportunities for interaction, training, business and pleasure across industries. While research is still in its infancy, the concept has been around for more than a decade, experiencing vast technological innovation in recent years. Headsets to experience metaverses via virtual reality (VR) provide increasing visual fidelity, which can alter the metaverse experience for the individual. Visual fidelity refers to the degree to which visual features in the metaverse imitate visual features in the equivalent real-world setting. We use a VR supermarket setting to study consumers' memory processes, dependent on the visual fidelity of the headset as well as task demands. Based on a 2 × 2 between-subjects design, we find that generally a higher visual fidelity improves explicit memory of features in the visual scene. Only for the case of a more abstract task, lower visual fidelity improves implicit memory. Additionally, higher explicit memory is associated with lower spending. We attribute these findings to the level of detail needed to perform the task adequately. We provide recommendations for designing metaverse experiences and research practice. We contribute to the literature on information processing by demonstrating the effect of visual fidelity on memory performance, depending on task demands.

Combined representation of visual features in the scene-selective cortex

Visual features of separable dimensions conjoin to represent an integrated entity. We investigated how visual features bind to form a complex visual scene. Specifically, we focused on features important for visually guided navigation: direction and distance. Previously, separate works have shown that directions and distances of navigable paths are coded in the occipital place area (OPA). Using functional magnetic resonance imaging (fMRI), we tested how separate features are concurrently represented in the OPA. Participants saw eight types of scenes, in which four of them had one path and the other four had two paths. In single-path scenes, path direction was either to the left or to the right. In double-path scenes, both directions were present. A glass wall was placed in some paths to restrict navigational distance. To test how the OPA represents path directions and distances, we took three approaches. First, the independent-features approach examined whether the OPA codes each direction and distance. Second, the integrated-features approach explored how directions and distances are integrated into path units, as compared to pooled features, using double-path scenes. Finally, the integrated-paths approach asked how separate paths are combined into a scene. Using multi-voxel pattern similarity analysis, we found that the OPA’s representations of single-path scenes were similar to other single-path scenes of either the same direction or the same distance. Representations of double-path scenes were similar to the combination of two constituent single-paths, as a combined unit of direction and distance rather than as a pooled representation of all features. These results show that the OPA combines the two features to form path units, which are then used to build multiple-path scenes. Altogether, these results suggest that visually guided navigation may be supported by the OPA that automatically and efficiently combines multiple features relevant for navigation and represent a navigation file.

Comparative temporal dynamics of individuation and perceptual averaging using a biological neural network model

Analyzing visual scenes and computing ensemble statistics, known as perceptual averaging, is crucial for the stable sensory experience of a cognitive agent. Despite the apparent simplicity of applying filters to scenes, the challenge arises from our brain’s seamless transition between summarization and individuation across various reference frames (retinotopic, spatiotopic, and hemispheric). In this study, we explore the capability of a neural network to dynamically switch between individuation and summarization. Our chosen computational model, a fully connected on-center off-surround recurrent neural network previously employed for enumeration/individuation, demonstrates the potential to extract both summary statistics and achieve high individuation accuracy. Notably, our results show that the individuation accuracy can reach close to perfection within a presentation duration of 100 ms, but not so for summarization. We have also shown a spatially varying excitation version of the network that can explain quite a few interesting spatio-temporal patterns of perception. These findings not only highlight the feasibility of such a neural network but also provide insights into the temporal dynamics of ensemble perception.

Visual neurons recognize complex image transformations.

Natural visual scenes are dominated by sequences of transforming images. Spatial visual information is thought to be processed by detection of elemental stimulus features which are recomposed into scenes. How image information is integrated over time is unclear. We explored visual information encoding in the optic tectum. Unbiased stimulus presentation shows that the majority of tectal neurons recognize image sequences. This is achieved by temporally dynamic response properties, which encode complex image transitions over several hundred milliseconds. Calcium imaging reveals that neurons that encode spatiotemporal image sequences fire in spike sequences that predict a logical diagram of spatiotemporal information processing. Furthermore, the temporal scale of visual information is tuned by experience. This study indicates how neurons recognize dynamic visual scenes that transform over time.

Orthogonal neural representations support perceptual judgements of natural stimuli.

In natural behavior, observers must separate relevant information from a barrage of irrelevant information. Many studies have investigated the neural underpinnings of this ability using artificial stimuli presented on simple backgrounds. Natural viewing, however, carries a set of challenges that are inaccessible using artificial stimuli, including neural responses to background objects that are task-irrelevant. An emerging body of evidence suggests that the visual abilities of humans and animals can be modeled through the linear decoding of task-relevant information from visual cortex. This idea suggests the hypothesis that irrelevant features of a natural scene should impair performance on a visual task only if their neural representations intrude on the linear readout of the task relevant feature, as would occur if the representations of task-relevant and irrelevant features are not orthogonal in the underlying neural population. We tested this hypothesis using human psychophysics and monkey neurophysiology, in response to parametrically variable naturalistic stimuli. We demonstrate that 1) the neural representation of one feature (the position of a central object) in visual area V4 is orthogonal to those of several background features, 2) the ability of human observers to precisely judge object position was largely unaffected by task-irrelevant variation in those background features, and 3) many features of the object and the background are orthogonally represented by V4 neural responses. Our observations are consistent with the hypothesis that orthogonal neural representations can support stable perception of objects and features despite the tremendous richness of natural visual scenes.

Multisensory processing impacts memory for objects and their sources.

Multisensory object processing improves recognition memory for individual objects, but its impact on memory for neighboring visual objects and scene context remains largely unknown. It is therefore unclear how multisensory processing impacts episodic memory for information outside of the object itself. We conducted three experiments to test the prediction that the presence of audiovisual objects at encoding would improve memory for nearby visual objects, and improve memory for the environmental context in which they occurred. In Experiments 1a and 1b, participants viewed audiovisual-visual object pairs or visual-visual object pairs with a control sound during encoding and were subsequently tested on their memory for each object individually. In Experiment 2, objects were paired with semantically congruent or meaningless control sounds and appeared within four different scene environments. Memory for the environment was tested. Results from Experiments 1a and 1b showed that encoding a congruent audiovisual object did not significantly benefit memory for neighboring visual objects, but Experiment 2 showed that encoding a congruent audiovisual object did improve memory for the environments in which those objects were encoded. These findings suggest that multisensory processing can influence memory beyond the objects themselves and that it has a unique role in episodic memory formation. This is particularly important for understanding how memories and associations are formed in real-world situations, in which objects and their surroundings are often multimodal.

Realistic About Reference Production: Testing the Effects of Domain Size and Saturation.

Experiments on visually grounded, definite reference production often manipulate simple visual scenes in the form of grids filled with objects, for example, to test how speakers are affected by the number of objects that are visible. Regarding the latter, it was found that speech onset times increase along with domain size, at least when speakers refer to nonsalient target objects that do not pop out of the visual domain. This finding suggests that even in the case of many distractors, speakers perform object-by-object scans of the visual scene. The current study investigates whether this systematic processing strategy can be explained by the simplified nature of the scenes that were used, and if different strategies can be identified for photo-realistic visual scenes. In doing so, we conducted a preregistered experiment that manipulated domain size and saturation; replicated the measures of speech onset times; and recorded eye movements to measure speakers' viewing strategies more directly. Using controlled photo-realistic scenes, we find (1) that speech onset times increase linearly as more distractors are present; (2) that larger domains elicit relatively fewer fixation switches back and forth between the target and its distractors, mainly before speech onset; and (3) that speakers fixate the target relatively less often in larger domains, mainly after speech onset. We conclude that careful object-by-object scans remain the dominant strategy in our photo-realistic scenes, to a limited extent combined with low-level saliency mechanisms. A relevant direction for future research would be to employ less controlled photo-realistic stimuli that do allow for interpretation based on context.

Connectome-driven neural inventory of a complete visual system.

Vision provides animals with detailed information about their surroundings, conveying diverse features such as color, form, and movement across the visual scene. Computing these parallel spatial features requires a large and diverse network of neurons, such that in animals as distant as flies and humans, visual regions comprise half the brain's volume. These visual brain regions often reveal remarkable structure-function relationships, with neurons organized along spatial maps with shapes that directly relate to their roles in visual processing. To unravel the stunning diversity of a complex visual system, a careful mapping of the neural architecture matched to tools for targeted exploration of that circuitry is essential. Here, we report a new connectome of the right optic lobe from a male Drosophila central nervous system FIB-SEM volume and a comprehensive inventory of the fly's visual neurons. We developed a computational framework to quantify the anatomy of visual neurons, establishing a basis for interpreting how their shapes relate to spatial vision. By integrating this analysis with connectivity information, neurotransmitter identity, and expert curation, we classified the ~53,000 neurons into 727 types, about half of which are systematically described and named for the first time. Finally, we share an extensive collection of split-GAL4 lines matched to our neuron type catalog. Together, this comprehensive set of tools and data unlock new possibilities for systematic investigations of vision in Drosophila, a foundation for a deeper understanding of sensory processing.

A robust event-driven approach to always-on object recognition

We propose a neuromimetic architecture capable of always-on pattern recognition, i.e. at any time during processing. To achieve this, we have extended an existing event-based algorithm (Lagorce et al., 2017), which introduced novel spatio-temporal features as a Hierarchy Of Time-Surfaces (HOTS). Built from asynchronous events captured by a neuromorphic camera, these time surfaces allow to encode the local dynamics of a visual scene and to create an efficient event-based pattern recognition architecture. Inspired by neuroscience, we have extended this method to improve its performance. First, we add a homeostatic gain control on the activity of neurons to improve the learning of spatio-temporal patterns (Grimaldi et al., 2021). We also provide a new mathematical formalism that allows an analogy to be drawn between the HOTS algorithm and Spiking Neural Networks (SNN). Following this analogy, we transform the offline pattern categorization method into an online and event-driven layer. This classifier uses the spiking output of the network to define new time surfaces and we then perform the online classification with a neuromimetic implementation of a multinomial logistic regression. These improvements not only consistently increase the performance of the network, but also bring this event-driven pattern recognition algorithm fully online. The results have been validated on different datasets: Poker-DVS (Serrano-Gotarredona and Linares-Barranco, 2015), N-MNIST (Orchard, Jayawant et al., 2015) and DVS Gesture (Amir et al., 2017). This demonstrates the efficiency of this bio-realistic SNN for ultra-fast object recognition through an event-by-event categorization process.

A study on factors influencing the spatial oppressiveness in small courtyard spaces within university residential areas

ABSTRACT A number of obstacles in terms of the growing demand for residential spaces for students in China's education industry. Changing urban development patterns often lead to densely populated, unpleasant university residential neighborhood, posing threats to the well-being of university personnel. This study addresses the limitations of past research, focusing on a real campus's enclosed high-rise student apartment courtyard. Using three key design variables (courtyard area, building height and ground-level interface transparency), 27 sample scenarios were developed for psychological experiments. The findings indicate that while visual scene complexity has minimal impact on oppression to confined environments, building exposure and sky visibility do. People's perception of openness is determined by building exposure, sky visibility, and visual complexity. Higher visual complexity and greater sky visibility enhance the sense of openness. Satisfaction is primarily influenced by sky visibility. The study also conducts a multiple linear regression of spatial and psychological indicators and discusses the tolerance threshold for spatial oppressiveness and the spatial oppressiveness under various spatial types of samples.

Differential cortical and subcortical visual processing with eyes shut.

Closing our eyes largely shuts down our ability to see. That said, our eyelids still pass some light, allowing our visual system to coarsely process information about visual scenes, such as changes in luminance. However, the specific impact of eye closure on processing within the early visual system remains largely unknown. To understand how visual processing is modulated when eyes are shut, we used functional magnetic resonance imaging (fMRI) to measure responses to a flickering visual stimulus at high (100%) and low (10%) temporal contrasts, while participants viewed the stimuli with their eyes open or closed. Interestingly, we discovered that eye closure produced a qualitatively distinct pattern of effects across the visual thalamus and visual cortex. We found that with eyes open, low temporal contrast stimuli produced smaller responses across the lateral geniculate nucleus (LGN), primary (V1) and extrastriate visual cortex (V2). However, with eyes closed, we discovered that the LGN and V1 maintained similar blood oxygenation level-dependent (BOLD) responses as the eyes open condition, despite the suppressed visual input through the eyelid. In contrast, V2 and V3 had strongly attenuated BOLD response when eyes were closed, regardless of temporal contrast. Our findings reveal a qualitatively distinct pattern of visual processing when the eyes are closed-one that is not simply an overall attenuation but rather reflects distinct responses across visual thalamocortical networks, wherein the earliest stages of processing preserve information about stimuli but are then gated off downstream in visual cortex.NEW & NOTEWORTHY When we close our eyes coarse luminance information is still accessible by the visual system. Using functional magnetic resonance imaging, we examined whether eyelid closure plays a unique role in visual processing. We discovered that while the LGN and V1 show equivalent responses when the eyes are open or closed, extrastriate cortex exhibited attenuated responses with eye closure. This suggests that when the eyes are closed, downstream visual processing is blind to this information.

Foreground bias: Semantic consistency effects modulated when searching across depth.

When processing visual scenes, we tend to prioritize information in the foreground, often at the expense of background information. The foreground bias has been supported by data demonstrating that there are more fixations to foreground, and faster and more accurate detection of targets embedded in foreground. However, it is also known that semantic consistency is associated with more efficient search. Here, we examined whether semantic context interacts with foreground prioritization, either amplifying or mitigating the effect of target semantic consistency. For each scene, targets were placed in the foreground or background and were either semantically consistent or inconsistent with the context of immediately surrounding depth region. Results indicated faster response times (RTs) for foreground and semantically consistent targets, replicating established effects. More importantly, we found the magnitude of the semantic consistency effect was significantly smaller in the foreground than background region. To examine the robustness of this effect, in Experiment 2, we strengthened the reliability of semantics by increasing the proportion of targets consistent with the scene region to 80%. We found the overall results pattern to replicate the incongruous effect of semantic consistency across depth observed in Experiment 1. This suggests foreground bias modulates the effects of semantics so that performance is less impacted in near space.