Visual World Research Articles

Teaching deep networks to see shape: Lessons from a simplified visual world.

Deep neural networks have been remarkably successful as models of the primate visual system. One crucial problem is that they fail to account for the strong shape-dependence of primate vision. Whereas humans base their judgements of category membership to a large extent on shape, deep networks rely much more strongly on other features such as color and texture. While this problem has been widely documented, the underlying reasons remain unclear. We design simple, artificial image datasets in which shape, color, and texture features can be used to predict the image class. By training networks from scratch to classify images with single features and feature combinations, we show that some network architectures are unable to learn to use shape features, whereas others are able to use shape in principle but are biased towards the other features. We show that the bias can be explained by the interactions between the weight updates for many images in mini-batch gradient descent. This suggests that different learning algorithms with sparser, more local weight changes are required to make networks more sensitive to shape and improve their capability to describe human vision.

Neural Transformation from Retinotopic to Background-Centric Coordinates in the Macaque Precuneus.

Visual information is initially represented in retinotopic coordinates and later in craniotopic coordinates. Psychophysical evidence suggests that visual information is further represented in more general coordinates related to the external world; however, the neural basis of non-egocentric coordinates remains elusive. This study investigates the automatic transformation from egocentric to non-egocentric coordinates in the macaque precuneus (two males, one female), identified by a functional imaging study as a key area for non-egocentric representation. We found that 6.2% of neurons in the precuneus have receptive fields anchored to the background rather than to the retina or the head, while 16% had traditional retinotopic receptive fields. Notably, these two types were not exclusive: many background-centric neurons initially encode a stimulus's position in retinotopic coordinates (up to ∼90 ms from stimulus onset) but later shift to background coordinates, peaking at ∼150 ms. Regarding retinotopic information, the stimulus dominated the initial period, whereas the background dominated the later period. In the absence of a background, there is a dramatic surge in retinotopic information about the stimulus during the later phase, clearly delineating two distinct periods of retinotopic encoding: one focusing on the figure to be attended and another on the background. These findings suggest that the initial retinotopic information of the stimulus is combined with the background retinotopic information in a subsequent stage, yielding a more stable representation of the stimulus relative to the background through time-division multiplexing.Significance Statement According to psychological literature, the location of visual stimuli is automatically positioned against the background of a scene. This representation relative to the background, not being influenced by eye movements, should be important for stabilizing the visual world. A human functional imaging study suggested that the precuneus in the medial cerebral cortex is a strong candidate. This study recorded neural activity from the precuneus of monkeys and demonstrated the existence of background-centered cells with receptive fields fixed relative to the background.

The perceptual and mnemonic effects of ensemble representation on individual size representation.

Our visual world consists of multiple objects, necessitating the identification of individual objects. Nevertheless, the representation of visual objects often exerts influence on each other. Even when we selectively attend to a subset of visual objects, the representations of surrounding items are encoded and influence the processing of the attended item(s). However, it remains unclear whether the effect of group ensemble representation on individual item representation occurs at the perceptual encoding phase, during the memory maintenance period, or both. Therefore, the current study conducted visual psychophysics experiments to investigate the contributions of perceptual and mnemonic bias on the observed effect of ensemble representation on individual size representation. Across five experiments, we found a consistent pattern of repulsive ensemble bias, such that the size of an individual target circle was consistently reported to be smaller than it actually was when presented alongside other circles with larger mean size, and vice versa. There was a perceptual component to the bias, but mnemonic factors also influenced its magnitude. Specifically, the repulsion bias was strongest with a short retention period (0-50 ms), then reduced within a second to a weaker magnitude that remained stable for a longer retention period (5,000 ms). Such patterns of results persisted when we facilitated the processing of ensemble representation by increasing the set size (Experiment 1B) or post-cueing the target circle so that attention was distributed across all items (Experiment 2B).

The visual experience dataset: Over 200 recorded hours of integrated eye movement, odometry, and egocentric video.

We introduce the Visual Experience Dataset (VEDB), a compilation of more than 240hours of egocentric video combined with gaze- and head-tracking data that offer an unprecedented view of the visual world as experienced by human observers. The dataset consists of 717 sessions, recorded by 56 observers ranging from 7 to 46 years of age. This article outlines the data collection, processing, and labeling protocols undertaken to ensure a representative sample and discusses the potential sources of error or bias within the dataset. The VEDB's potential applications are vast, including improving gaze-tracking methodologies, assessing spatiotemporal image statistics, and refining deep neural networks for scene and activity recognition. The VEDB is accessible through established open science platforms and is intended to be a living dataset with plans for expansion and community contributions. It is released with an emphasis on ethical considerations, such as participant privacy and the mitigation of potential biases. By providing a dataset grounded in real-world experiences and accompanied by extensive metadata and supporting code, the authors invite the research community to use and contribute to the VEDB, facilitating a richer understanding of visual perception and behavior in naturalistic settings.

Insect vision: A steady gaze over different landscapes

To gather visual information effectively, animals must stabilize images when they move. But closely related fruit fly species sometimes occupy strikingly diverse visual habitats. A new study shows that they have adopted gaze strategies tailored to their different visual worlds.

Reduced on-line speech gesture integration during multimodal language processing in adults with moderate-severe traumatic brain injury: Evidence from eye-tracking

BackgroundLanguage is multimodal and situated in rich visual contexts. Language is also incremental, unfolding moment-to-moment in real time, yet few studies have examined how spoken language interacts with gesture and visual context during multimodal language processing. Gesture is a rich communication cue that is integrally related to speech and often depicts concrete referents from the visual world. Using eye-tracking in an adapted visual world paradigm, we examined how participants with and without moderate-severe traumatic brain injury (TBI) use gesture to resolve temporary referential ambiguity. MethodsParticipants viewed a screen with four objects and one video. The speaker in the video produced sentences (e.g., “The girl will eat the very good sandwich”), paired with either a meaningful gesture (e.g., sandwich-holding gesture) or a meaningless grooming movement (e.g., arm scratch) at the verb “will eat.” We measured participants’ gaze to the target object (e.g., sandwich), a semantic competitor (e.g., apple), and two unrelated distractors (e.g., piano, guitar) during the critical window between movement onset in the gesture modality and onset of the spoken referent in speech. ResultsBoth participants with and without TBI were more likely to fixate the target when the speaker produced a gesture compared to a grooming movement; however, relative to non-injured participants, the effect was significantly attenuated in the TBI group. DiscussionWe demonstrated evidence of reduced speech-gesture integration in participants with TBI relative to non-injured peers. This study advances our understanding of the communicative abilities of adults with TBI and could lead to a more mechanistic account of the communication difficulties adults with TBI experience in rich communication contexts that require the processing and integration of multiple co-occurring cues. This work has the potential to increase the ecological validity of language assessment and provide insights into the cognitive and neural mechanisms that support multimodal language processing.

Fair Games—Curating New Music

The authors of “Defragmentation: Curating Contemporary Music,” a special issue of the Darmstädter Beiträge zur Neuen Musik published in 2019, considered all aspects of any given artistic event as fair game for curation. They suggested that not only could the choice of pieces, soloists, conductors, and ensembles be (re)tooled (including even site-specific aspects), but the roles of the musicians and audience and even value regimes could be instrumentalized to accord with the artistic demands of the curator. In her introduction to the special issue, Dorothee Richter argued that curation should be a “practice that is deeply involved in the politics of display, politics of site, politics of transfer and translation, and regimes of visibility.” If we understand politics with Jacques Rancière as “giving form to the social,” then Richter suggests that every thinkable manner in which people relate can, and should, be weighed when programming a concert. Curators need to reflect and act upon the way relationships in our world are displayed, the interplay involved on site, the participation or lack thereof of an audience, the participants’ ability to understand, enjoy, and be entertained, and even the audience’s and presenter’s perceived positions in society and how they play out in concert. Writing as a performer-researcher, a cultural-sociologist, and a musicologist, and drawing on previous research in the visual arts, we discuss the impact of the curator within the European new music world. We question the curator’s central role and probe shifts in power relations between festival directors, receptive venues, and performing ensembles. In our view, the rise of the curator in the new music world leads us in at least two distinct directions. First, as in the visual arts world, the individual curator may assume the role of the new protagonist, displacing the performing artists from their throne. When this occurs, curating takes place in a game of mutual competition, distinction, and exploitation, in which illustration of an abstract idea or concept prevails over musical experience. Alternatively, curating may become a collective affair and responsibility along the lines of commoning principles, in which curating returns to its etymological roots as a form of caring for each other and for both the tradition and the future of music.

Privacy preservation in deep reinforcement learning: A training perspective

Reinforcement learning (RL) is a principled AI framework for autonomous, experience-driven learning. Deep reinforcement learning (DRL) enhances this by incorporating deep learning models, promoting a higher-level understanding of the visual world. However, privacy concerns are emerging in RL applications that involve vast amounts of private information. Recent studies have demonstrated that DRL can leak private information and be vulnerable to attacks aiming to infer the training environment from an agent’s behaviors without direct access to the environment. To address these privacy concerns, we propose a differentially private DRL approach that obfuscates the agent’s observations from each visited state. This defends against privacy leakage attacks and prevents the inference of the agent’s training environment from its optimized policy. We provide a theoretical analysis and design comprehensive experiments to thoroughly reproduce the privacy leakage attack. Both the theoretical analysis and experimental results demonstrate that our method effectively defends against privacy leakage attacks while maintaining the model utility of the RL agent.

Investigation of Secondary School Students' Visual Perception and Attitudes towards Graphic Design

The relationship between visual perceptions and graphic design of secondary school students is of great importance in terms of the development of their artistic skills. This link is directly related to how students interpret the visual world and how they express these interpretations through graphic design. The aim of this study is to examine the relationships between visual perception and attitudes of secondary school students towards graphic design in terms of some variables. The research conducted with the comparative relational screening model was conducted on 168 students studying at a private secondary school in Türkiye. The research data were collected using the ‘Visual Perception Test’ and the ‘Attitude Scale Towards Graphic Design’. The data were analyzed using ‘Descriptive Statistics’, ‘Independent Samples t test’ and ‘Multiple Regression Analysis’ techniques. According to the research findings, the visual perceptions of the participating students were found to be above the average level, while their attitudes towards graphic design were found to be at a high level. Students’ visual perceptions and attitudes towards graphic design differ according to the gender variable. Finally, visual perception significantly predicts students’ attitudes towards graphic design.

Which perceptual categories do observers experience during multistable perception?

Multistable perceptual phenomena provide insights into the mind's dynamic states within a stable external environment and the neural underpinnings of these consciousness changes are often studied with binocular rivalry. Conventional methods to study binocular rivalry suffer from biases and assumptions that limit their ability to describe the continuous nature of this perceptual transitions and to discover what kind of percept was perceived across time. In this study, we propose a novel way to avoid those shortcomings by combining a continuous psychophysical method that estimates introspection during binocular rivalry with machine learning clustering and transition probability analysis. This combination of techniques reveals individual variability and complexity of perceptual experience in 28 normally sighted participants. Also, the analysis of transition probabilities between perceptual categories, i.e., exclusive and different kinds of mixed percepts, suggest that interocular perceptual competition, triggered by low-level stimuli, involves conflict between monocular and binocular neural processing sites rather than mutual inhibition of monocular sites.

Schooling and language usage matter in heritage bilingual processing: Sortal classifiers in Mandarin

Mandarin sortal classifiers simultaneously encode semantic and grammatical form class cues. Building on a second language (L2) study of Grüter et al. we used the same visual world eye-tracking experiment, designed to examine the relative use of the two cues, testing Mandarin heritage speakers (HSs) living in an English-speaking environment. Given the importance of understanding individual differences, the present study also examined if/how individual HSs may systematically differ as a function of experience with Mandarin. As a group, HSs – like the first language (L1) group in Grüter et al.’s study – showed a clear reliance on the grammatical form class cue. Nevertheless, individual HSs with Mandarin schooling and with more Mandarin social exposure/use showed more reliance on semantics, the L2 pattern observed in Grüter et al. We discuss why this latter pattern might have obtained (formal and informal literacy), while highlighting the value of individual difference approaches to understanding HS processing.

Online Metaphor Comprehension in Adults with Autism Spectrum Disorders: An Eye Tracking Study

AbstractThe aim of this study was to investigate novel metaphor comprehension in adults with autism spectrum disorder (ASD). Previous literature is conflicting about whether individuals with ASD have impairment in this particular type of figurative language. Participants in the study completed a visual world paradigm eye-tracking task, which involved selecting an interpretation of an auditorily presented sentence (i.e. a picture-sentence matching task), where images corresponded to literal and metaphorical interpretations. Thus, the study also investigated online processing, via reaction times and eye movements. Forty adults participated in the study (18 with ASD and 22 typically-developing controls). Each participant completed the AQ questionnaire and had their vocabulary assessed. Results showed that participants with ASD comprehended metaphorical utterances with the same accuracy as controls. However, they had significantly slower reaction times, and specifically, were approximately 800 ms slower. Analysis of eye movements revealed that participants with ASD showed significantly longer fixation times on both the target and distractor image, the latter of which suggests difficulty overcoming the literal interpretation. Consistent with some prior studies, we showed that adults with ASD are not impaired in novel metaphor comprehension, but they were clearly less efficient. Verbal abilities did not significantly relate to performance. Finally, our online processing measure (eye tracking) provided us with insights into the nature of the ASD inefficiency (i.e. a literality bias).

Study on the Strategy of Creating Teaching Situations in High School Art Appreciation Classes

Art appreciation focuses on guiding students to use their perception, experience and knowledge to carry out learning activities such as appreciation, identification and commentary around the visual world of natural beauty and works of art, so as to obtain aesthetic enjoyment, form aesthetic interest, and cultivate aesthetic ability and artistic accomplishment. In art appreciation teaching, situation creation is an important teaching strategy that can help students better understand and apply learning knowledge. This paper first introduces the definition and characteristics of situation creation, and then discusses the importance of situation creation in middle school English classroom teaching, as well as how to create effective teaching situations, so as to build a better classroom for students that is more emotional, more interesting, more abundant, and can enhance their aesthetic ability, so that students can fall in love with art appreciation classes.

Visual place recognition from end-to-end semantic scene text features.

We live in a visual world where text cues are abundant in urban environments. The premise for our work is for robots to capitalize on these text features for visual place recognition. A new technique is introduced that uses an end-to-end scene text detection and recognition technique to improve robot localization and mapping through Visual Place Recognition (VPR). This technique addresses several challenges such as arbitrary shaped text, illumination variation, and occlusion. The proposed model captures text strings and associated bounding boxes specifically designed for VPR tasks. The primary contribution of this work is the utilization of an end-to-end scene text spotting framework that can effectively capture irregular and occluded text in diverse environments. We conduct experimental evaluations on the Self-Collected TextPlace (SCTP) benchmark dataset, and our approach outperforms state-of-the-art methods in terms of precision and recall, which validates the effectiveness and potential of our proposed approach for VPR.

Presynaptic Proteins and Their Roles in Visual Processing by the Retina.

The sense of vision begins in the retina, where light is detected and processed through a complex series of synaptic connections into meaningful information relayed to the brain via retinal ganglion cells. Light responses begin as tonic and graded signals in photoreceptors, later emerging from the retina as a series of spikes from ganglion cells. Processing by the retina extracts critical features of the visual world, including spatial frequency, temporal frequency, motion direction, color, contrast, and luminance. To achieve this, the retina has evolved specialized and unique synapse types. These include the ribbon synapses of photoreceptors and bipolar cells, the dendritic synapses of amacrine and horizontal cells, and unconventional synaptic feedback from horizontal cells to photoreceptors. We review these unique synapses in the retina with a focus on the presynaptic molecules and physiological properties that shape their capabilities.

Is there a cost when predictions are not met? A VWP study investigating L1 and L2 speakers.

Research has found that both first language (L1) and second language (L2) speakers make predictions about upcoming linguistic information, with predictive behaviour being impacted by individual differences and methodological factors. However, it is not clear whether a cost is incurred when a prediction is made, but not met. L2 speakers have less experience with their L2 and parsing can be cognitively demanding, which together may lead L2 speakers to incur prediction costs differently relative to L1 speakers. In this study using the visual world paradigm, we test whether L1 and L2 speakers predict in the same way, within the same time frame, and incur the same costs if predictions are not met. We also explore the role of proficiency and speech rate. We found that both groups predict in a similar way and within a similar time frame. In addition, neither group incurred a prediction cost when the target was the most likely alternative, though L2 speakers take longer to shift their attention to the target object when predictions are not met. We argue that this reflects a slowing of lexical access rather than a specific cost of prediction. We only found prediction differences when speech rate was included in the analysis, highlighting the importance of attending to speech rate in studies using the visual world paradigm. Overall, this study supports research showing that both L1 and L2 speakers may make multiple partial predictions about upcoming information rather than predicting one specific lexical candidate while inhibiting less likely lexical candidates.

Redistributing the sensible: exploring aesthetic practices in a photography course

In Western industrialised contexts, ecological problems such as climate change are sensed and come largely to be known through media representations. In this article, we consider aesthetic practices as a concept for understanding the ways in which the visual sensible world relating to climate change is continuously distributed. Furthermore, we study photography as a pedagogical means to redistribute the sensible. Our data are gathered from a photography course in which participants explored four alternative aesthetic practices: revealing connectedness, recognising agency, compromising the attractions of consumerism, and illuminating alternatives. Through their photographic explorations, participants were encouraged to intervene in two problematic consensual frameworks: human-centredness and consumption-centredness. We found that exploring aesthetic practices through photography helped redistribute the sensible in two significant ways: participants recognised visible traces of the consensual frameworks and shifted the emphasis of perception beyond their personal viewpoints.

Underlying dimensions of real-time word recognition in cochlear implant users

Word recognition is a gateway to language, linking sound to meaning. Prior work has characterized its cognitive mechanisms as a form of competition between similar-sounding words. However, it has not identified dimensions along which this competition varies across people. We sought to identify these dimensions in a population of cochlear implant users with heterogenous backgrounds and audiological profiles, and in a lifespan sample of people without hearing loss. Our study characterizes the process of lexical competition using the Visual World Paradigm. A principal component analysis reveals that people’s ability to resolve lexical competition varies along three dimensions that mirror prior small-scale studies. These dimensions capture the degree to which lexical access is delayed (“Wait-and-See”), the degree to which competition fully resolves (“Sustained-Activation”), and the overall rate of activation. Each dimension is predicted by a different auditory skills and demographic factors (onset of deafness, age, cochlear implant experience). Moreover, each dimension predicts outcomes (speech perception in quiet and noise, subjective listening success) over and above auditory fidelity. Higher degrees of Wait-and-See and Sustained-Activation predict poorer outcomes. These results suggest the mechanisms of word recognition vary along a few underlying dimensions which help explain variable performance among listeners encountering auditory challenge.

Neuronal and Behavioral Responses to Naturalistic Texture Images in Macaque Monkeys.

The visual world is richly adorned with texture, which can serve to delineate important elements of natural scenes. In anesthetized macaque monkeys, selectivity for the statistical features of natural texture is weak in V1, but substantial in V2, suggesting that neuronal activity in V2 might directly support texture perception. To test this, we investigated the relation between single cell activity in macaque V1 and V2 and simultaneously measured behavioral judgments of texture. We generated stimuli along a continuum between naturalistic texture and phase-randomized noise and trained two macaque monkeys to judge whether a sample texture more closely resembled one or the other extreme. Analysis of responses revealed that individual V1 and V2 neurons carried much less information about texture naturalness than behavioral reports. However, the sensitivity of V2 neurons, especially those preferring naturalistic textures, was significantly closer to that of behavior compared with V1. The firing of both V1 and V2 neurons predicted perceptual choices in response to repeated presentations of the same ambiguous stimulus in one monkey, despite low individual neural sensitivity. However, neither population predicted choice in the second monkey. We conclude that neural responses supporting texture perception likely continue to develop downstream of V2. Further, combined with neural data recorded while the same two monkeys performed an orientation discrimination task, our results demonstrate that choice-correlated neural activity in early sensory cortex is unstable across observers and tasks, untethered from neuronal sensitivity, and therefore unlikely to directly reflect the formation of perceptual decisions.

Exploration of the Historical and Social Significance of One of the First Cinematographic Devices Based on Gender Roles in the Andalusian Environment

In 1914, El Noticiero Sevillano and other Spanish newspapers published a piece about the Cinémhymen, a cinematographic device designed to capture and sell images of prospective wives. This article explores why this advertisement was not considered derogatory and examines the construction of a patriarchy during a time when the term “feminist” was already appearing in the Spanish press. In our methodology, we analyzed the name of the device and the business, both based on the word Hymen, used a bibliographic review of Spanish feminism of those years, and researched the film technology of the time. The Manzano’s pyramid of oppression served us in establishing the control operations underlying the advertisement. Our study reveals the patriarchal principles of Cinémhymen, which stigmatized women once they conformed to the expected role. The objectifying gaze present in Cinémhymen provides insight into the progression of patriarchy in a visual world that subjugates women. The camera could see through the female masquerade (as Joan Rivière explained) and explore the “true” body underneath, the very core of the female (or what is considered to be). In some ways, Cinémhymen serves as a precursor to the current subjugation seen in online pornography and represents a distorted evolution of the panopticon principle as applied to women.