Visual Scene Research Articles

MRAM: Multi-scale Regional Attribute-weighting via Meta-learning for Personalized Image Aesthetics Assessment

Image aesthetics assessment (IAA) is inherently subjective, with generic image aesthetics assessment (GIAA) often overlooking individual user preferences. Personalized aesthetic assessment (PIAA) addresses this by accounting for user-specific inconsistencies. However, the limited annotated data from individual users and the scarcity of user-jointly annotated image data pose challenges. To cope with these problems, we propose a multi-scale regional attribute-weighting via meta-learning for PIAA, namely MRAM, which aims to simulate how the brain computes aesthetic values from visual stimuli. The human brain is to predict the subjective aesthetic value of the individual to the image by combining the features of the visual scene segmentation through the weighting method. MRAM introduces a multi-scale attention module to capture visual scene regions segmented at different scales, adjusting feature selection based on user attention. Additionally, a regional attribute-weighting module (RAM) is proposed to weigh these features using aesthetic attribute representations. In the meta-training phase, MRAM learns aesthetic prior knowledge from a diverse set of PIAA tasks. Transitioning to the personalization phase, MRAM is fine-tuned with only a small number of specific user samples, enabling it to rapidly adapt to the unique aesthetic preferences of individuals. Experimental results demonstrate the superior performance of the proposed MRAM compared to state-of-the-art PIAA methods.

Attention redirection transformer with semantic oriented learning for unbiased scene graph generation

Scene Graph Generation (SGG) plays an important role in scene understanding because all of the objects and relations in an image can be abstracted into a concise topological graph. Due to the complexity of visual scenes, including mutual occlusion between objects and semantic ambiguity, SGG is still a challenging task. Most of the existing models only focused on the context of a single object while contexts provided by paired objects are ignored. In this paper, we propose an Attention Redirection Transformer (ART) to extract pair-level contexts specifically, which is divided into an attention distraction stage and an attention integration stage. In this way, the attention of the model is forced to be redirected, which explores the implicit information in the background. In addition, to incorporate the semantic information of predicates, a Semantic Oriented Learning Module (SOL) is designed, which may assist in getting better textual semantics and also prompts cross-modal information fusion. At last, a self-diversity driven Dual Translation Embedding Module (DTM) is designed, which refines representations of subject and object and makes them distinct. Experimental results on the Visual Genome dataset demonstrate the effectiveness of our proposed method. Moreover, our method outperforms state-of-the-art methods on the mR@K metric. The source codes are released on Github: https://github.com/Nora-Zhang98/ART-SOL.

Length of Stay and Destination Image: Insights from Social Media Content of Day Trippers and Tourists

ABSTRACT Social media platforms have a significant role in determining destination image. This study examines the impact of length of stay on destination image as reflected on visitors’ posts on social media through a comparative analysis of day trippers and tourists. Two text mining methods (Term Frequency-Inverse Document Frequency and Latent Dirichlet Allocation) and an image classification model (ResNet) were applied to Weibo data to identify day trippers’ and tourists’ online destination image. Differences were found between day trippers and tourists when examining keywords, topics, and visual scenes. The keyword-based image was predominantly utilitarian for day trippers and more recreational-oriented for tourists. Moreover, tourists shared a greater variety of topic-based image attributes (e.g. accommodation, catering, activity) than day trippers, and form an abundant holistic image. Both groups were highly consistent in the visual category of common images and top-10 visual scenes of unique features, though significant differences were recorded when considering gender.

The role of binocular disparity and attention in the neural representation of multiple moving stimuli in the visual cortex.

Segmenting visual scenes into distinct objects and surfaces is a fundamental visual process, with stereoscopic depth and motion serving as crucial cues. However, how the visual system uses these cues to segment multiple objects is not fully understood. We investigated how neurons in the middle-temporal (MT) cortex of macaque monkeys represent overlapping surfaces at different depths, moving in different directions. Neuronal activity was recorded from three male monkeys during discrimination tasks under varying attention conditions. We found that neuronal responses to overlapping surfaces showed a robust bias toward the binocular disparity of one surface over the other. The disparity bias of a neuron was positively correlated with the neuron's disparity preference for a single surface. In two animals, neurons preferring near disparities of single surfaces (near neurons) showed a near bias for overlapping stimuli, while neurons preferring far disparities (far neurons) showed a far bias. In the third animal, both near and far neurons displayed a near bias, though the near neurons showed a stronger near bias. All three animals exhibited an initial near bias across neurons relative to the average of the responses to the individual surfaces. Although attention modulated neuronal responses, the disparity bias was not caused by attention. We also found that the effect of attention was consistent with object-based, rather than feature-based attention. We proposed a model in which the pool size of the neuron population that weighs the responses to individual stimulus components can be variable. This model is a novel extension of the standard normalization model and provides a unified explanation for the disparity bias across animals. Our results reveal how MT neurons encode multiple stimuli moving at different depths and present new evidence of response modulation by object-based attention. The disparity bias allows subgroups of neurons to preferentially represent individual surfaces of multiple stimuli at different depths, thereby facilitating segmentation.

Zero-shot visual grounding via coarse-to-fine representation learning

Visual grounding (VG) locates target objects in visual scenes by understanding given natural language queries. Current methods for VG mainly focus on grounding referring expressions or noun phrases covered in the labeled training samples. Despite their grounding prowess, these approaches struggle in grounding novel query–image pairs excluded from the training data. This shortage is usually caused by the deficiency of discriminative representation learning both in images and queries. To address these issues, we propose a one-stage coarse-to-fine framework for zero-shot VG to ground novel query-image samples. Specifically, in the coarse stage, we mine the global context information in the visual features and query embeddings by employing a multi-head self-attention block, strengthening the intra-modality relations in the visual and textual features. In the fine stage, we first learn the query-aware visual representations based on the acquired global context information via a multi-modal relation-enhanced transformer block, which explores the informative information by modeling the cross-modal interaction among visual and textual domains. We further excavate target-oriented discriminative representations from the acquired query-aware visual representations by a noun phrase-guided multi-modal interaction network, which augments the interaction between target-related phrases and the obtained query-aware visual representations to enhance the distinction of target regions, enhancing the subsequent referred target grounding and generalizing. In order to validate the proposed approach, we implement extensive experiments and ablation studies on public benchmark datasets, including RefCOCO, RefCOCO+, RefCOCOg, Flickr30K Entity, Flickr-Split-0 and Flickr-Split-1. Experimental results demonstrate that our approach substantially improves the grounding accuracy and achieves new state-of-the-art performance under single-stage training and testing.

Using imagination and the contents of memory to create new scene and object representations: A functional MRI study

Humans can use the contents of memory to construct scenarios and events that they have not encountered before, a process colloquially known as imagination. Much of our current understanding of the neural mechanisms mediating imagination is limited by paradigms that rely on participants' subjective reports of imagined content. Here, we used a novel behavioral paradigm that was designed to systematically evaluate the contents of an individual's imagination. Participants first learned the layout of four distinct rooms containing five wall segments with differing geometrical characteristics, each associated with a unique object. During functional MRI, participants were then shown two different wall segments or objects on each trial and asked to first, retrieve the associated objects or walls, respectively (retrieval phase) and then second, imagine the two objects side-by-side or combine the two wall segments (imagination phase). Importantly, the contents of each participant's imagination were interrogated by having them make a same/different judgment about the properties of the imagined objects or scenes. Using univariate and multivariate analyses, we observed widespread activity across occipito-temporal cortex for the retrieval of objects and for the imaginative creation of scenes. Interestingly, a classifier, whether trained on the imagination or retrieval data, was able to successfully differentiate the neural patterns associated with the imagination of scenes from that of objects. Our results reveal neural differences in the cued retrieval of object and scene memoranda, demonstrate that different representations underlie the creation and/or imagination of scene and object content, and highlight a novel behavioral paradigm that can be used to systematically evaluate the contents of an individual's imagination.

Gordon Crook: Banners and Wall Hangings

Gordon Crook was a British textile artist who came to live in Wellington in 1972, aged 51. Through contacts at the Dowse Art Museum and the Ministry of Foreign Affairs and Trade, Crook was commissioned by the architect Miles Warren to make banners for the New Zealand Chancery in Washington D.C. (1979–80), followed by banners and wall hangings for the Michael Fowler Centre in Wellington (1981–83). Crook’s work was suigeneric, idiosyncratic in its imagery and development, a world feeding on itself. Outside any national or current artworld style, Crook extended and enriched New Zealand’s public visual art scene.[i] [i] See my three articles on the work of Gordon Crook: “Gordon Crook: tapestries,” Tuhinga 31 (2020): 70–90; “Gordon Crook: The Pastel Triptychs,” Tuhinga 32 (2021): 120–34; and “Gordon Crook and the Wolf-Man,” Tuhinga 33 (2022): 1–29.

Deep scene understanding with extended text description for human object interaction detection

Human–object interaction (HOI) detection plays a pivotal role in scene understanding, enabling the identification, localization, and behavioral intention prediction of humans and objects within a visual scene. Conventional approaches, such as graph-based networks, have demonstrated effectiveness in capturing spatio-temporal interaction cues. However, relying solely on visual information within these networks limits their ability to comprehensively grasp the intricate aspects of human interactions. To address this shortcoming, we propose a novel deep scene understanding graph network that harnesses the power of extended text descriptions to represent and interpret interactions between humans and objects effectively. Text serves as a rich source of information, directly conveying the nature of interactions within a visual scene. Our model seamlessly integrates text descriptions with visual features extracted from the entire video sequence, enabling it to capture the context and nuances of human–object interactions. This approach significantly enhances the model’s ability to accurately predict ambiguous human actions and anticipate future interactions. Extensive experiments on two benchmark datasets, CAD-120, and Something-Else, demonstrate that our proposed method achieves remarkable improvements in the F1-Score for both HOI detection and anticipation tasks. This work paves the way for more accurate and comprehensive HOI understanding in various real-world scenarios, highlighting the potential of text-augmented graph networks for effective interaction modeling.

Portable technology for postural control measurement: Comparing head position with center of pressure data.

Standing is a basic human function that healthy adults take for granted, yet it is a complex perceptual-motor process that requires sensation of position and motion from the sensory systems. We assessed agreement between center of pressure data from a laboratory force-platform and head position data from an HTC Vive head-mounted display (HMD) for the evaluation of standing postural control. We investigated the impact of different statistical choices when assessing the relationship between two measurements. Specifically: 1) How does correlation and agreement statistics relate before and after logarithmic transformation? 2) Is there systemic or proportional bias between the force-platform and HMD measurements? We tested 37 adults (26 controls, 11 with unilateral vestibular hypofunction) standing on foam, observing a static or dynamic visual scene projected from the HMD. We quantified anterior-posterior and medio-lateral sway via Directional Path, Root Mean Square Velocity, Variance, and Power Spectral Density (PSD) from a force-platform and the HMD. Intra-class correlations (ICCs) were moderate-to-good for the non-transformed data and good-to-excellent after logarithmic transformation for all outcomes except for PSD above 1 Hz. Correlations were higher than ICCs. Bland-Altman plots indicated proportional bias but not after logarithmic transformation. Both devices correlated linearly, and measure people's postural responses but cannot be used interchangeably, mostly because they appear to diverge with larger sway as evident on Bland-Altman plots of non-transformed data. Agreement between devices was excellent for low frequency movement but poor for high frequency small corrective movements.

A Lightweight Visual Understanding System for Enhanced Assistance to the Visually Impaired Using an Embedded Platform

Visually impaired individuals often face significant challenges in navigating their environments due to limited access to visual information. To address this issue, a portable, cost-effective assistive tool is proposed to operate on a low-power embedded system such as the Jetson Nano. The novelty of this research lies in developing an efficient, lightweight video captioning model within constrained resources to ensure its compatibility with embedded platforms. This research aims to enhance the autonomy and accessibility of visually impaired people by providing audio descriptions of their surroundings through the processing of live-streaming videos. The proposed system utilizes two distinct lightweight deep learning modules: an object detection module based on the state-of-the-art YOLOv7 model, and a video captioning module that utilizes both the Video Swin Transformer and 2D-CNN for feature extraction, along with the Transformer network for caption generation. The goal of the object detection module is for providing real-time multiple object identification in the surrounding environment of the blind while the video captioning module is to provide detailed descriptions of the entire visual scenes and activities including objects, actions, and relationships between them. The user interacts via a headphone with the proposed system using a specific audio command to trigger the corresponding module even object detection or video captioning and receiving an audio description output for the visual contents. The system demonstrates satisfactory results, achieving inference speeds between 0.11 to 1.1 seconds for object detection and 0.91 to 1.85 seconds for video captioning, evaluated through both quantitative metrics and subjective assessments.

Looming Detection in Complex Dynamic Visual Scenes by Interneuronal Coordination of Motion and Feature Pathways

Looming Detection in Complex Dynamic Visual Scenes by Interneuronal Coordination of Motion and Feature Pathways

Bridging Implicit and Explicit Geometric Transformation for Single-Image View Synthesis.

Creating novel views from a single image has achieved tremendous strides with advanced autoregressive models, as unseen regions have to be inferred from the visible scene contents. Although recent methods generate high-quality novel views, synthesizing with only one explicit or implicit 3D geometry has a trade-off between two objectives that we call the "seesaw" problem: 1) preserving reprojected contents and 2) completing realistic out-of-view regions. Also, autoregressive models require a considerable computational cost. In this paper, we propose a single-image view synthesis framework for mitigating the seesaw problem while utilizing an efficient non-autoregressive model. Motivated by the characteristics that explicit methods well preserve reprojected pixels and implicit methods complete realistic out-of-view regions, we introduce a loss function to complement two renderers. Our loss function promotes that explicit features improve the reprojected area of implicit features and implicit features improve the out-of-view area of explicit features. With the proposed architecture and loss function, we can alleviate the seesaw problem, outperforming autoregressive-based state-of-the-art methods and generating an image ≈ 100 times faster. We validate the efficiency and effectiveness of our method with experiments on RealEstate10 K and ACID datasets.

The Types of Discrimination and Princess Diana’s Ways to Pursue Her Freedom in Spencer (2021)

This article elaborates the film "Spencer," characterized as "a fable from true tragedy" based on Princess Diana's story. The focus of the research is on how Princess Diana pursued her freedom. The methodology involves a direct viewing and interpretation of "Spencer," analyzing visual scenes and audio dialogues. Following Roland Barthes' semiotic approach, the researcher identifies denotative signs, transitions them into connotative markers, and ultimately explores the mythic stage. Specific scenes, such as Princess Diana encountering a sign urging silence and her secretive actions in the palace, are dissected to reveal layers of meaning. Through semiotic analysis, the article illustrates Princess Diana's attempts to break free from royal constraints. Each scene reflects the conflict between individual desires for freedom and the norms imposed by the royal institution. The study offers a nuanced understanding of Princess Diana's struggle within the royal family's rules and pressures. The analysis provides profound insights into how "Spencer" portrays Princess Diana's quest for personal freedom amidst the rigid regulations of the royal household, employing semiotics as a valuable approach to unravel the meanings embedded in visual and auditory elements of the film.

Movie trailer genre classification using multimodal pretrained features

We introduce a novel method for movie genre classification, capitalizing on a diverse set of readily accessible pretrained models. These models extract high-level features related to visual scenery, objects, characters, text, speech, music, and audio effects. To intelligently fuse these pretrained features, we train small classifier models with low time and memory requirements. Employing the transformer model, our approach utilizes all video and audio frames of movie trailers without performing any temporal pooling, efficiently exploiting the correspondence between all elements, as opposed to the fixed and low number of frames typically used by traditional methods. Our approach fuses features originating from different tasks and modalities, with different dimensionalities, different temporal lengths, and complex dependencies as opposed to current approaches. Our method outperforms state-of-the-art movie genre classification models in terms of precision, recall, and mean average precision (mAP). To foster future research, we make the pretrained features for the entire MovieNet dataset, along with our genre classification code and the trained models, publicly available.

Fixational Eye Movements Enhance the Precision of Visual Information Transmitted by the Primate Retina.

Fixational eye movements alter the number and timing of spikes transmitted from the retina to the brain, but whether these changes enhance or degrade the retinal signal is unclear. To quantify this, we developed a Bayesian method for reconstructing natural images from the recorded spikes of hundreds of retinal ganglion cells (RGCs) in the macaque retina (male), combining a likelihood model for RGC light responses with the natural image prior implicitly embedded in an artificial neural network optimized for denoising. The method matched or surpassed the performance of previous reconstruction algorithms, and provides an interpretable framework for characterizing the retinal signal. Reconstructions were improved with artificial stimulus jitter that emulated fixational eye movements, even when the eye movement trajectory was assumed to be unknown and had to be inferred from retinal spikes. Reconstructions were degraded by small artificial perturbations of spike times, revealing more precise temporal encoding than suggested by previous studies. Finally, reconstructions were substantially degraded when derived from a model that ignored cell-to-cell interactions, indicating the importance of stimulus-evoked correlations. Thus, fixational eye movements enhance the precision of the retinal representation.

Natural Language Processing Applied to Spontaneous Recall of Famous Faces Reveals Memory Dysfunction in Temporal Lobe Epilepsy Patients.

Epilepsy patients rank memory problems as their most significant cognitive comorbidity. Current clinical assessments are laborious to administer and score and may not always detect subtle memory decline. The Famous Faces Task (FF) has robustly demonstrated that left temporal lobe epilepsy (LTLE) patients remember fewer names and biographical details compared to right TLE (RTLE) patients and healthy controls (HCs). We adapted the FF task to capture subjects' entire spontaneous spoken recall, then scored responses using manual and natural language processing (NLP) methods. We expected to replicate previous group level differences using spontaneous speech and semi-automated analysis. Seventy-three (N=73) adults (28 LTLE, 18 RTLE, and 27 HCs) were included in a case-control prospective study design. Twenty FF in politics, sports, and entertainment (active 2008-2017) were shown to subjects, who were asked if they could recognize and spontaneously recall as much biographical detail as possible. We created human-generated and automatically-generated keyword dictionaries for each celebrity, based on a randomly selected training set of half of the HC transcripts. To control for speech output, we measured the speech duration, total word count and content word count for the FF task and a Cookie Theft Control Task (CTT), in which subjects were merely asked to describe a visual scene. Subjects' responses to FF and CTT tasks were recorded, transcribed, and analyzed in a blinded manner with a combination of manual and automated NLP approaches. Famous face recognition accuracy was similar between groups. LTLE patients recalled fewer biographical details compared to HCs and RTLEs using both the gold-standard human-generated dictionary (24%±12% vs. 31%±12% and 30%±12%, p=0.007) and the automated dictionary (24%±12% vs. 31%±12% and 32%±13%, p=0.007). There were no group level differences in speech duration, total word count, or content word count for either the FF and CTT to explain difference in recall performance. There was a positive, statistically significant relationship between MOCA score and FF recall performance as scored by the human-generated (ρ= .327, p= .029) and automatically-generated dictionaries (ρ= .422, p= .004) for TLE subjects, but not HCs, an effect that was driven by LTLE subjects. LTLE patients remember fewer details of famous people than HCs or RTLE patients, as discovered by NLP analysis of spontaneous recall. Decreased biographical memory was not due to decreased speech output and correlated with lower MOCA scores. NLP analysis of spontaneous recall can detect memory dysfunction in clinical populations in a semi-automated, objective, and sensitive manner.

Search for neurophysiological mechanisms of configurational learning

Configural learning is a form of associative learning in which the conditioned stimulus is a holistic set of stimulus elements rather than individual stimuli or their isolated properties. Successfully solving the task of such associative learning requires a holistic analysis of the entire configuration as a whole. The ability to analyze not only individual physical aspects of a stimulus or single objects in a visual scene, but also their holistic combinations, offers significant evolutionary advantages, as configurations often have substantially greater predictive power compared to individual stimulus elements or features. Moreover, the ability to holistically analyze combinations of stimulus field elements or features can be considered an initial, primitive manifestation of consciousness. In the present review, we consider the history of the development of the concept of configural learning, the main methodological avenues of investigation, and currently available neurophysiological data on the putative neural basis of this phenomenon. We find it most interesting to study the processes of configural learning in humans using modern neuroimaging methods, as they provide a glimpse into the holistic brain functioning. Finally, we consider the future tasks aimed to provide a more complete understanding of the neurophysiology of the configural learning phenomenon.

Neuro-symbolic Predicate Invention: Learning relational concepts from visual scenes

Neuro-symbolic Predicate Invention: Learning relational concepts from visual scenes

Multi-modal deep learning framework for damage detection in social media posts.

In crisis management, quickly identifying and helping affected individuals is key, especially when there is limited information about the survivors' conditions. Traditional emergency systems often face issues with reachability and handling large volumes of requests. Social media has become crucial in disaster response, providing important information and aiding in rescues when standard communication systems fail. Due to the large amount of data generated on social media during emergencies, there is a need for automated systems to process this information effectively and help improve emergency responses, potentially saving lives. Therefore, accurately understanding visual scenes and their meanings is important for identifying damage and obtaining useful information. Our research introduces a framework for detecting damage in social media posts, combining the Bidirectional Encoder Representations from Transformers (BERT) architecture with advanced convolutional processing. This framework includes a BERT-based network for analyzing text and multiple convolutional neural network blocks for processing images. The results show that this combination is very effective, outperforming existing methods in accuracy, recall, and F1 score. In the future, this method could be enhanced by including more types of information, such as human voices or background sounds, to improve its prediction efficiency.

Cortical dynamics of perception as trains of coherent gamma oscillations, with the pulvinar as central coordinator

Synchronization of spikes carried by the visual streams is strategic for the proper binding of cortical assemblies, hence for the perception of visual objects as coherent units. Perception of a complex visual scene involves multiple trains of gamma oscillations, coexisting at each stage in visual and associative cortex. Here, we analyze how this synchrony is managed, so that the perception of each visual object can emerge despite this complex interweaving of cortical activations. After a brief review of structural and temporal facts, we analyze the interactions which make the oscillations coherent for the visual elements related to the same object. We continue with the propagation of these gamma oscillations within the sensory chain. The dominant role of the pulvinar and associated reticular thalamic nucleus as cortical coordinator is the common thread running through this step-by-step description. Synchronization mechanisms are analyzed in the context of visual perception, although the present considerations are not limited to this sense. A simple experiment is described, with the aim of assessing the validity of the elements developed here. A first set of results is provided, together with a proposed method to go further in this investigation.