Visual Scene Research Articles

Gordon Crook: Banners and Wall Hangings

Gordon Crook was a British textile artist who came to live in Wellington in 1972, aged 51. Through contacts at the Dowse Art Museum and the Ministry of Foreign Affairs and Trade, Crook was commissioned by the architect Miles Warren to make banners for the New Zealand Chancery in Washington D.C. (1979–80), followed by banners and wall hangings for the Michael Fowler Centre in Wellington (1981–83). Crook’s work was suigeneric, idiosyncratic in its imagery and development, a world feeding on itself. Outside any national or current artworld style, Crook extended and enriched New Zealand’s public visual art scene.[i] [i] See my three articles on the work of Gordon Crook: “Gordon Crook: tapestries,” Tuhinga 31 (2020): 70–90; “Gordon Crook: The Pastel Triptychs,” Tuhinga 32 (2021): 120–34; and “Gordon Crook and the Wolf-Man,” Tuhinga 33 (2022): 1–29.

Deep scene understanding with extended text description for human object interaction detection

Human–object interaction (HOI) detection plays a pivotal role in scene understanding, enabling the identification, localization, and behavioral intention prediction of humans and objects within a visual scene. Conventional approaches, such as graph-based networks, have demonstrated effectiveness in capturing spatio-temporal interaction cues. However, relying solely on visual information within these networks limits their ability to comprehensively grasp the intricate aspects of human interactions. To address this shortcoming, we propose a novel deep scene understanding graph network that harnesses the power of extended text descriptions to represent and interpret interactions between humans and objects effectively. Text serves as a rich source of information, directly conveying the nature of interactions within a visual scene. Our model seamlessly integrates text descriptions with visual features extracted from the entire video sequence, enabling it to capture the context and nuances of human–object interactions. This approach significantly enhances the model’s ability to accurately predict ambiguous human actions and anticipate future interactions. Extensive experiments on two benchmark datasets, CAD-120, and Something-Else, demonstrate that our proposed method achieves remarkable improvements in the F1-Score for both HOI detection and anticipation tasks. This work paves the way for more accurate and comprehensive HOI understanding in various real-world scenarios, highlighting the potential of text-augmented graph networks for effective interaction modeling.

Portable technology for postural control measurement: Comparing head position with center of pressure data.

Standing is a basic human function that healthy adults take for granted, yet it is a complex perceptual-motor process that requires sensation of position and motion from the sensory systems. We assessed agreement between center of pressure data from a laboratory force-platform and head position data from an HTC Vive head-mounted display (HMD) for the evaluation of standing postural control. We investigated the impact of different statistical choices when assessing the relationship between two measurements. Specifically: 1) How does correlation and agreement statistics relate before and after logarithmic transformation? 2) Is there systemic or proportional bias between the force-platform and HMD measurements? We tested 37 adults (26 controls, 11 with unilateral vestibular hypofunction) standing on foam, observing a static or dynamic visual scene projected from the HMD. We quantified anterior-posterior and medio-lateral sway via Directional Path, Root Mean Square Velocity, Variance, and Power Spectral Density (PSD) from a force-platform and the HMD. Intra-class correlations (ICCs) were moderate-to-good for the non-transformed data and good-to-excellent after logarithmic transformation for all outcomes except for PSD above 1 Hz. Correlations were higher than ICCs. Bland-Altman plots indicated proportional bias but not after logarithmic transformation. Both devices correlated linearly, and measure people's postural responses but cannot be used interchangeably, mostly because they appear to diverge with larger sway as evident on Bland-Altman plots of non-transformed data. Agreement between devices was excellent for low frequency movement but poor for high frequency small corrective movements.

A Lightweight Visual Understanding System for Enhanced Assistance to the Visually Impaired Using an Embedded Platform

Visually impaired individuals often face significant challenges in navigating their environments due to limited access to visual information. To address this issue, a portable, cost-effective assistive tool is proposed to operate on a low-power embedded system such as the Jetson Nano. The novelty of this research lies in developing an efficient, lightweight video captioning model within constrained resources to ensure its compatibility with embedded platforms. This research aims to enhance the autonomy and accessibility of visually impaired people by providing audio descriptions of their surroundings through the processing of live-streaming videos. The proposed system utilizes two distinct lightweight deep learning modules: an object detection module based on the state-of-the-art YOLOv7 model, and a video captioning module that utilizes both the Video Swin Transformer and 2D-CNN for feature extraction, along with the Transformer network for caption generation. The goal of the object detection module is for providing real-time multiple object identification in the surrounding environment of the blind while the video captioning module is to provide detailed descriptions of the entire visual scenes and activities including objects, actions, and relationships between them. The user interacts via a headphone with the proposed system using a specific audio command to trigger the corresponding module even object detection or video captioning and receiving an audio description output for the visual contents. The system demonstrates satisfactory results, achieving inference speeds between 0.11 to 1.1 seconds for object detection and 0.91 to 1.85 seconds for video captioning, evaluated through both quantitative metrics and subjective assessments.

Looming Detection in Complex Dynamic Visual Scenes by Interneuronal Coordination of Motion and Feature Pathways

Looming Detection in Complex Dynamic Visual Scenes by Interneuronal Coordination of Motion and Feature Pathways

Bridging Implicit and Explicit Geometric Transformation for Single-Image View Synthesis.

Creating novel views from a single image has achieved tremendous strides with advanced autoregressive models, as unseen regions have to be inferred from the visible scene contents. Although recent methods generate high-quality novel views, synthesizing with only one explicit or implicit 3D geometry has a trade-off between two objectives that we call the "seesaw" problem: 1) preserving reprojected contents and 2) completing realistic out-of-view regions. Also, autoregressive models require a considerable computational cost. In this paper, we propose a single-image view synthesis framework for mitigating the seesaw problem while utilizing an efficient non-autoregressive model. Motivated by the characteristics that explicit methods well preserve reprojected pixels and implicit methods complete realistic out-of-view regions, we introduce a loss function to complement two renderers. Our loss function promotes that explicit features improve the reprojected area of implicit features and implicit features improve the out-of-view area of explicit features. With the proposed architecture and loss function, we can alleviate the seesaw problem, outperforming autoregressive-based state-of-the-art methods and generating an image ≈ 100 times faster. We validate the efficiency and effectiveness of our method with experiments on RealEstate10 K and ACID datasets.

The Types of Discrimination and Princess Diana’s Ways to Pursue Her Freedom in Spencer (2021)

This article elaborates the film "Spencer," characterized as "a fable from true tragedy" based on Princess Diana's story. The focus of the research is on how Princess Diana pursued her freedom. The methodology involves a direct viewing and interpretation of "Spencer," analyzing visual scenes and audio dialogues. Following Roland Barthes' semiotic approach, the researcher identifies denotative signs, transitions them into connotative markers, and ultimately explores the mythic stage. Specific scenes, such as Princess Diana encountering a sign urging silence and her secretive actions in the palace, are dissected to reveal layers of meaning. Through semiotic analysis, the article illustrates Princess Diana's attempts to break free from royal constraints. Each scene reflects the conflict between individual desires for freedom and the norms imposed by the royal institution. The study offers a nuanced understanding of Princess Diana's struggle within the royal family's rules and pressures. The analysis provides profound insights into how "Spencer" portrays Princess Diana's quest for personal freedom amidst the rigid regulations of the royal household, employing semiotics as a valuable approach to unravel the meanings embedded in visual and auditory elements of the film.

Movie trailer genre classification using multimodal pretrained features

We introduce a novel method for movie genre classification, capitalizing on a diverse set of readily accessible pretrained models. These models extract high-level features related to visual scenery, objects, characters, text, speech, music, and audio effects. To intelligently fuse these pretrained features, we train small classifier models with low time and memory requirements. Employing the transformer model, our approach utilizes all video and audio frames of movie trailers without performing any temporal pooling, efficiently exploiting the correspondence between all elements, as opposed to the fixed and low number of frames typically used by traditional methods. Our approach fuses features originating from different tasks and modalities, with different dimensionalities, different temporal lengths, and complex dependencies as opposed to current approaches. Our method outperforms state-of-the-art movie genre classification models in terms of precision, recall, and mean average precision (mAP). To foster future research, we make the pretrained features for the entire MovieNet dataset, along with our genre classification code and the trained models, publicly available.

Fixational Eye Movements Enhance the Precision of Visual Information Transmitted by the Primate Retina.

Fixational eye movements alter the number and timing of spikes transmitted from the retina to the brain, but whether these changes enhance or degrade the retinal signal is unclear. To quantify this, we developed a Bayesian method for reconstructing natural images from the recorded spikes of hundreds of retinal ganglion cells (RGCs) in the macaque retina (male), combining a likelihood model for RGC light responses with the natural image prior implicitly embedded in an artificial neural network optimized for denoising. The method matched or surpassed the performance of previous reconstruction algorithms, and provides an interpretable framework for characterizing the retinal signal. Reconstructions were improved with artificial stimulus jitter that emulated fixational eye movements, even when the eye movement trajectory was assumed to be unknown and had to be inferred from retinal spikes. Reconstructions were degraded by small artificial perturbations of spike times, revealing more precise temporal encoding than suggested by previous studies. Finally, reconstructions were substantially degraded when derived from a model that ignored cell-to-cell interactions, indicating the importance of stimulus-evoked correlations. Thus, fixational eye movements enhance the precision of the retinal representation.

Natural Language Processing Applied to Spontaneous Recall of Famous Faces Reveals Memory Dysfunction in Temporal Lobe Epilepsy Patients.

Epilepsy patients rank memory problems as their most significant cognitive comorbidity. Current clinical assessments are laborious to administer and score and may not always detect subtle memory decline. The Famous Faces Task (FF) has robustly demonstrated that left temporal lobe epilepsy (LTLE) patients remember fewer names and biographical details compared to right TLE (RTLE) patients and healthy controls (HCs). We adapted the FF task to capture subjects' entire spontaneous spoken recall, then scored responses using manual and natural language processing (NLP) methods. We expected to replicate previous group level differences using spontaneous speech and semi-automated analysis. Seventy-three (N=73) adults (28 LTLE, 18 RTLE, and 27 HCs) were included in a case-control prospective study design. Twenty FF in politics, sports, and entertainment (active 2008-2017) were shown to subjects, who were asked if they could recognize and spontaneously recall as much biographical detail as possible. We created human-generated and automatically-generated keyword dictionaries for each celebrity, based on a randomly selected training set of half of the HC transcripts. To control for speech output, we measured the speech duration, total word count and content word count for the FF task and a Cookie Theft Control Task (CTT), in which subjects were merely asked to describe a visual scene. Subjects' responses to FF and CTT tasks were recorded, transcribed, and analyzed in a blinded manner with a combination of manual and automated NLP approaches. Famous face recognition accuracy was similar between groups. LTLE patients recalled fewer biographical details compared to HCs and RTLEs using both the gold-standard human-generated dictionary (24%±12% vs. 31%±12% and 30%±12%, p=0.007) and the automated dictionary (24%±12% vs. 31%±12% and 32%±13%, p=0.007). There were no group level differences in speech duration, total word count, or content word count for either the FF and CTT to explain difference in recall performance. There was a positive, statistically significant relationship between MOCA score and FF recall performance as scored by the human-generated (ρ= .327, p= .029) and automatically-generated dictionaries (ρ= .422, p= .004) for TLE subjects, but not HCs, an effect that was driven by LTLE subjects. LTLE patients remember fewer details of famous people than HCs or RTLE patients, as discovered by NLP analysis of spontaneous recall. Decreased biographical memory was not due to decreased speech output and correlated with lower MOCA scores. NLP analysis of spontaneous recall can detect memory dysfunction in clinical populations in a semi-automated, objective, and sensitive manner.

Search for neurophysiological mechanisms of configurational learning

Configural learning is a form of associative learning in which the conditioned stimulus is a holistic set of stimulus elements rather than individual stimuli or their isolated properties. Successfully solving the task of such associative learning requires a holistic analysis of the entire configuration as a whole. The ability to analyze not only individual physical aspects of a stimulus or single objects in a visual scene, but also their holistic combinations, offers significant evolutionary advantages, as configurations often have substantially greater predictive power compared to individual stimulus elements or features. Moreover, the ability to holistically analyze combinations of stimulus field elements or features can be considered an initial, primitive manifestation of consciousness. In the present review, we consider the history of the development of the concept of configural learning, the main methodological avenues of investigation, and currently available neurophysiological data on the putative neural basis of this phenomenon. We find it most interesting to study the processes of configural learning in humans using modern neuroimaging methods, as they provide a glimpse into the holistic brain functioning. Finally, we consider the future tasks aimed to provide a more complete understanding of the neurophysiology of the configural learning phenomenon.

Neuro-symbolic Predicate Invention: Learning relational concepts from visual scenes

Neuro-symbolic Predicate Invention: Learning relational concepts from visual scenes

Multi-modal deep learning framework for damage detection in social media posts.

In crisis management, quickly identifying and helping affected individuals is key, especially when there is limited information about the survivors' conditions. Traditional emergency systems often face issues with reachability and handling large volumes of requests. Social media has become crucial in disaster response, providing important information and aiding in rescues when standard communication systems fail. Due to the large amount of data generated on social media during emergencies, there is a need for automated systems to process this information effectively and help improve emergency responses, potentially saving lives. Therefore, accurately understanding visual scenes and their meanings is important for identifying damage and obtaining useful information. Our research introduces a framework for detecting damage in social media posts, combining the Bidirectional Encoder Representations from Transformers (BERT) architecture with advanced convolutional processing. This framework includes a BERT-based network for analyzing text and multiple convolutional neural network blocks for processing images. The results show that this combination is very effective, outperforming existing methods in accuracy, recall, and F1 score. In the future, this method could be enhanced by including more types of information, such as human voices or background sounds, to improve its prediction efficiency.

Cortical dynamics of perception as trains of coherent gamma oscillations, with the pulvinar as central coordinator

Synchronization of spikes carried by the visual streams is strategic for the proper binding of cortical assemblies, hence for the perception of visual objects as coherent units. Perception of a complex visual scene involves multiple trains of gamma oscillations, coexisting at each stage in visual and associative cortex. Here, we analyze how this synchrony is managed, so that the perception of each visual object can emerge despite this complex interweaving of cortical activations. After a brief review of structural and temporal facts, we analyze the interactions which make the oscillations coherent for the visual elements related to the same object. We continue with the propagation of these gamma oscillations within the sensory chain. The dominant role of the pulvinar and associated reticular thalamic nucleus as cortical coordinator is the common thread running through this step-by-step description. Synchronization mechanisms are analyzed in the context of visual perception, although the present considerations are not limited to this sense. A simple experiment is described, with the aim of assessing the validity of the elements developed here. A first set of results is provided, together with a proposed method to go further in this investigation.

Learning to use landmarks for navigation amplifies their representation in retrosplenial cortex.

Visual landmarks provide powerful reference signals for efficient navigation by altering the activity of spatially tuned neurons, such as place cells, head direction cells, and grid cells. To understand the neural mechanism by which landmarks exert such strong influence, it is necessary to identify how these visual features gain spatial meaning. In this study, we characterized visual landmark representations in mouse retrosplenial cortex (RSC) using chronic two-photon imaging of the same neuronal ensembles over the course of spatial learning. We found a pronounced increase in landmark-referenced activity in RSC neurons that, once established, remained stable across days. Changing behavioral context by uncoupling treadmill motion from visual feedback systematically altered neuronal responses associated with the coherence between visual scene flow speed and self-motion. To explore potential underlying mechanisms, we modeled how burst firing, mediated by supralinear somatodendritic interactions, could efficiently mediate context- and coherence-dependent integration of landmark information. Our results show that visual encoding shifts to landmark-referenced and context-dependent codes as these cues take on spatial meaning during learning.

Enhancing Sound Source Localization via False Negative Elimination.

Sound source localization aims to localize objects emitting the sound in visual scenes. Recent works obtaining impressive results typically rely on contrastive learning. However, the common practice of randomly sampling negatives in prior arts can lead to the false negative issue, where the sounds semantically similar to visual instance are sampled as negatives and incorrectly pushed away from the visual anchor/query. As a result, this misalignment of audio and visual features could yield inferior performance. To address this issue, we propose a novel audio-visual learning framework which is instantiated with two individual learning schemes: self-supervised predictive learning (SSPL) and semantic-aware contrastive learning (SACL). SSPL explores image-audio positive pairs alone to discover semantically coherent similarities between audio and visual features, while a predictive coding module for feature alignment is introduced to facilitate the positive-only learning. In this regard SSPL acts as a negative-free method to eliminate false negatives. By contrast, SACL is designed to compact visual features and remove false negatives, providing reliable visual anchor and audio negatives for contrast. Different from SSPL, SACL releases the potential of audio-visual contrastive learning, offering an effective alternative to achieve the same goal. Comprehensive experiments demonstrate the superiority of our approach over the state-of-the-arts. Furthermore, we highlight the versatility of the learned representation by extending the approach to audio-visual event classification and object detection tasks. Code and models are available at: https://github.com/zjsong/SACL.

Brain-like border ownership signals support prediction of natural videos.

To make sense of visual scenes, the brain must segment foreground from background. This is thought to be facilitated by neurons in the primate visual system that encode border ownership (BOS), i.e. whether a local border is part of an object on one or the other side of the border. It is unclear how these signals emerge in neural networks without a teaching signal of what is foreground and background. In this study, we investigated whether BOS signals exist in PredNet, a self-supervised artificial neural network trained to predict the next image frame of natural video sequences. We found that a significant number of units in PredNet are selective for BOS. Moreover these units share several other properties with the BOS neurons in the brain, including robustness to scene variations that constitute common object transformations in natural videos, and hysteresis of BOS signals. Finally, we performed ablation experiments and found that BOS units contribute more to prediction than non-BOS units for videos with moving objects. Our findings indicate that BOS units are especially useful to predict future input in natural videos, even when networks are not required to segment foreground from background. This suggests that BOS neurons in the brain might be the result of evolutionary or developmental pressure to predict future input in natural, complex dynamic visual environments.

The Visual Systems of Zebrafish.

The zebrafish visual system has become a paradigmatic preparation for behavioral and systems neuroscience. Around 40 types of retinal ganglion cells (RGCs) serve as matched filters for stimulus features, including light, optic flow, prey, and objects on a collision course. RGCs distribute their signals via axon collaterals to 12 retinorecipient areas in forebrain and midbrain. The major visuomotor hub, the optic tectum, harbors nine RGC input layers that combine information on multiple features. The retinotopic map in the tectum is locally adapted to visual scene statistics and visual subfield-specific behavioral demands. Tectal projections to premotor centers are topographically organized according to behavioral commands. The known connectivity in more than 20 processing streams allows us to dissect the cellular basis of elementary perceptual and cognitive functions. Visually evoked responses, such as prey capture or loom avoidance, are controlled by dedicated multistation pathways that-at least in the larva-resemble labeled lines. This architecture serves the neuronal code's purpose of driving adaptive behavior.

The Visual Scene Network and Hippocampus Represent Spatial Boundary Structures for Temporal Episodic Memory Organization.

Crossing a spatial boundary such as a doorway plays an important role in the temporal organization of episodic memory. However, despite the wealth of evidence showing that visual boundary structures in scenes affect our representation of space, no studies have tested the possibility that they also function as event boundaries even without active navigation. In this study, we used a nonnavigational scene-based memory task that required participants to remember a sequence of objects moving to various baskets in a scene. In the boundary condition, there was a freestanding boundary in the middle of the room, low enough to see the rest of the room beyond it. We found that the additional boundary within the scene was sufficient to trigger a larger response in the cortical visual scene network, the hippocampus, and their coordinated activity. These results suggest that active navigation across a spatial boundary such as a doorway into another room is not necessary to form an event boundary and that a visual representation of boundaries is sufficient to influence the organization of a hippocampal episodic memory.

Decoding dynamic visual scenes across the brain hierarchy.

Understanding the computational mechanisms that underlie the encoding and decoding of environmental stimuli is a crucial investigation in neuroscience. Central to this pursuit is the exploration of how the brain represents visual information across its hierarchical architecture. A prominent challenge resides in discerning the neural underpinnings of the processing of dynamic natural visual scenes. Although considerable research efforts have been made to characterize individual components of the visual pathway, a systematic understanding of the distinctive neural coding associated with visual stimuli, as they traverse this hierarchical landscape, remains elusive. In this study, we leverage the comprehensive Allen Visual Coding-Neuropixels dataset and utilize the capabilities of deep learning neural network models to study neural coding in response to dynamic natural visual scenes across an expansive array of brain regions. Our study reveals that our decoding model adeptly deciphers visual scenes from neural spiking patterns exhibited within each distinct brain area. A compelling observation arises from the comparative analysis of decoding performances, which manifests as a notable encoding proficiency within the visual cortex and subcortical nuclei, in contrast to a relatively reduced encoding activity within hippocampal neurons. Strikingly, our results unveil a robust correlation between our decoding metrics and well-established anatomical and functional hierarchy indexes. These findings corroborate existing knowledge in visual coding related to artificial visual stimuli and illuminate the functional role of these deeper brain regions using dynamic stimuli. Consequently, our results suggest a novel perspective on the utility of decoding neural network models as a metric for quantifying the encoding quality of dynamic natural visual scenes represented by neural responses, thereby advancing our comprehension of visual coding within the complex hierarchy of the brain.