Visual Context Research Articles

Enabling Harmonious Human-Machine Interaction with Visual-Context Augmented Dialogue System: A Review

The intelligent dialogue system, aiming at communicating with humans harmoniously with natural language, is brilliant for promoting the advancement of human-machine interaction in the era of artificial intelligence. With the gradually complex human-computer interaction requirements, it is difficult for traditional text-based dialogue system to meet the demands for more vivid and convenient interaction. Consequently, Visual-Context Augmented Dialogue System (VAD), which has the potential to communicate with humans by perceiving and understanding multimodal information (i.e., visual context in images or videos, textual dialogue history), has become a predominant research paradigm. Benefiting from the consistency and complementarity between visual and textual context, VAD possesses the potential to generate engaging and context-aware responses. To depict the development of VAD, we first characterize the concept model of VAD and then present its generic system architecture to illustrate the system workflow, followed by a summary of multimodal fusion techniques. Subsequently, several research challenges and representative works are investigated, followed by the summary of authoritative benchmarks and real-world application of VAD. We conclude this paper by putting forward some open issues and promising research trends for VAD, e.g., the cognitive mechanisms of human-machine dialogue under cross-modal dialogue context, mobile and lightweight deployment of VAD.

The Lucerne Lion Monument of Bertel Thorvaldsen and ‘A Song of Liberty’ Poem of William Blake

The Purpose of the Study: Lucerne City, a canton of Switzerland, shelters the famous Lion Monument (the Lucerne Lion) which represents Swiss Guards’ who lost their lives while defending the Monarchy during The French Revolution. On the other hand, William Blake, who lived during the French Revolution period and wrote poems on the revolution, wrote the lines ‘Empire is no more! And now the lion and wolf shall cease’ in his poem ‘A Song of Liberty’. The main idea of this article is to expose the intuitive connection between the Lucerne Lion and the lion metaphor in William Blake’s verse. Method: The reference works of this article have consisted of art history, history, and literature. Historical information and visual materials, which are related to the topic of the article, were obtained from local researchers of Lucerne and official web pages of various facilities and museums in Switzerland, France, Italy, and Norway. Results: It is deduced that the Metaphorical expression of the lion theme in both the sculpture and the poem in the article by illuminating visual and written sources. Within this direction of these facts, an intertextuality is evaluated between the Lion Monument and the A Song of Liberty poem. Conclusion: The historical steps during the consisting of this sculpture express the collective formation stages of a monument from the perspective of art history. Individuals and mass facts have carried today the Lucerne Lion, an artifact of this collective memory. The Song of Liberty poem written by Willam Blake against the monarchy in the French Revolution matches the metaphoric content of Lucerne Lion. In the direction of these detections, these two works were reviewed in the historical, visual, and text contexts, thus the intuitional connection was exposed between them.

The 1265 Dual Stelae

Abstract This article explores an understudied aspect of the influential Quanzhen Daoist movement in thirteenth-century North China under Mongol rule: installing genealogical stelae as a critical mechanism in lineage making. It does so through a case study of two parallel genealogical stelae produced by a local Daoist community in 1265 for its male and female monastics. The article makes two arguments, one historical and the other methodological. With respect to the historical argument, it demonstrates how genealogical stelae not only reified Quanzhen lineage-making but also projected hereditary political power and reinforced a troubling discourse on women's subservience within religious institutions. Regarding the methodological argument, it highlights the importance of situating epigraphic narratives in the material, visual, and spatial contexts in which they were produced and represented. Detailed attention to all the dimensions of stelae allows us to discover surprisingly rich information stored in and communicated by them.

Visual priming and parsing preferences: A self-paced reading study of PP-attachment ambiguity in Dutch verb-final structures

Understanding how language users navigate through syntactic ambiguities is crucial for unravelling the intricacies of sentence processing. This study explored ambiguity resolution within verb-final double prepositional phrase structures in Dutch. While previous research has largely focused on SVO structures such as those found in English, there is a scarcity of evidence regarding initial syntactic decisions in verb-final structures. The study pursued two aims: firstly, to investigate whether visual cues can direct the parser towards an intended interpretation; and secondly, to examine the potential of visual cues in priming verb subcategorization information within ambiguous sentences in the absence of linguistic input. A reading task was administered to address these objectives by assessing per-word reading times. Furthermore, ambiguity within the structures was resolved by presenting an animation clip that displayed the intended interpretation of each sentence before reading it. The results indicated that the animations provided before the reading task modulated the processing of NP- and VP-attachment interpretations of the ambiguous structures under study. However, these contextual cues did not prime subcategorized information of verbs earlier than their appearance within the sentence, as evidenced by unchanged reading profiles for the regions preceding the verb. This study demonstrates that visual contexts modulate the word-by-word processing of Dutch verb-final structures, and that verb argument structure appears to influence this process. The results are examined through various sentence processing theories, lending support to constraint-based models.

Entanglements for Visualization: Changing Research Outcomes through Feminist Theory.

A growing body of work draws on feminist thinking to challenge assumptions about how people engage with and use visualizations. This work draws on feminist values, driving design and research guidelines that account for the influences of power and neglect. This prior work is largely prescriptive, however, forgoing articulation of how feminist theories of knowledge - or feminist epistemology - can alter research design and outcomes. At the core of our work is an engagement with feminist epistemology, drawing attention to how a new framework for how we know what we know enabled us to overcome intellectual tensions in our research. Specifically, we focus on the theoretical concept of entanglement, central to recent feminist scholarship, and contribute: a history of entanglement in the broader scope of feminist theory; an articulation of the main points of entanglement theory for a visualization context; and a case study of research outcomes as evidence of the potential of feminist epistemology to impact visualization research. This work answers a call in the community to embrace a broader set of theoretical and epistemic foundations and provides a starting point for bringing feminist theories into visualization research.

Learning with semantic ambiguity for unbiased scene graph generation.

Scene graph generation (SGG) aims to identify and extract objects from images and elucidate their interrelations. This task faces two primary challenges. Firstly, the long-tail distribution of relation categories causes SGG models to favor high-frequency relations, such as "on" and "in". Secondly, some subject-object pairs may have multiple reasonable relations, which often possess a certain degree of semantic similarity. However, the use of one-hot ground-truth relation labels does not effectively represent the semantic similarities and distinctions among relations. In response to these challenges, we propose a model-agnostic method named Mixup and Balanced Relation Learning (MBRL). This method assigns soft labels to samples exhibiting semantic ambiguities and optimizes model training by adjusting the loss weights for fine-grained and low-frequency relation samples. Its model-agnostic design facilitates seamless integration with diverse SGG models, enhancing their performance across various relation categories. Our approach is evaluated on widely-used datasets, including Visual Genome and Generalized Question Answering, both with over 100,000 images, providing rich visual contexts for scene graph model evaluation. Experimental results show that our method outperforms state-of-the-art approaches on multiple scene graph generation tasks, demonstrating significant improvements in both relation prediction accuracy and the handling of imbalanced data distributions.

A Study of the Visual Meaning in the Comic Hai Miiko by Ono Eriko (Based on Ferdinand de Saussure's Semiotics)

This study examines the Hai Miiko comic, popular among Indonesian teenagers, which tells the story of Miiko, a cheerful fifth-grader experiencing puberty. Using Ferdinand de Saussure's semiotic theory, the research analyzes the comic's visual elements, language, and context to uncover deeper meanings and symbolism. Key concepts such as synchronic and diachronic analysis, langue and parole, signifier and signified, syntagmatic and paradigmatic structures, and denotation and connotation are applied to dissect the semiotic aspects of the comic. The findings reveal that Hai Miiko not only depicts the daily lives of teenagers but also serves as a visual representation of the journey to adulthood. Through symbolic imagery and a distinct artistic style, Ono Eriko effectively portrays emotional growth and self-discovery, inviting readers to reflect on childhood nostalgia and the challenges of growing up. This study emphasizes that comics, beyond being entertainment, can convey profound meanings and cultural insights through their visuals and narratives.

Subtitle Translation from a Multimodal Perspective: Text and Visual Integration in Animated Film

The global appeal of animated films has increased significantly, leading to an urgent demand for high-quality and easily comprehensible subtitle translations. This research investigates the complexities of subtitle translation in the animated film Frozen (2013) by exploring the interaction between visual elements and dialogue. Employing multimodal discourse analysis, the study evaluates how subtitles can enhance or diminish the narrative experience by aligning with or diverging from the film’s visual context. The findings indicate that while translations generally capture the intended meaning and tone, there are critical moments—especially in humorous scenes—where subtitles fail to fully resonate with the accompanying visuals. This misalignment underscores the importance of a refined approach to translation that considers the interplay of various expressive modes. The research ultimately aims to contribute to best practices in audiovisual translation, emphasizing the need for coherence between subtitles and visual storytelling to enrich audience engagement.

Your smile works: understanding smiling face emojis in social media interactions

Previous studies on smiling face emojis mainly focus on how they are employed by the addresser for rapport management, with limited attention given to exploring the addressee’s interpretation of smiling face emojis. This study aims to explore whether smiling face emojis are interpreted as (im)polite in a virtual communication context and what factors might influence the interpretation. When carrying out the research, the author designed stimuli with 10 commonly used smiling face emojis in WeChat and then recruited 192 participants to take part in an experimental study. The results indicate that smiling face emojis enhance the politeness degree of an utterance while mitigating potential impolite intentions. They also indicate that the interpretation of politeness in utterances with smiling face emojis is influenced by age and gender. In addition, the interpretation of smiling face emojis is tightly related to the culture of politeness (礼) and internet culture. This article helps us to understand the use of smiling face emojis and the management of interpersonal relationships in a visual communication context.

When abstract becomes concrete, naturalistic encoding of concepts in the brain.

Language is acquired and processed in complex and dynamic naturalistic contexts, involving the simultaneous processing of connected speech, faces, bodies, objects, etc. How words and their associated concepts are encoded in the brain during real-world processing is still unknown. Here, the representational structure of concrete and abstract concepts was investigated during movie watching to address the extent to which brain responses dynamically change depending on visual context. First, across contexts, concrete and abstract concepts are shown to encode different experience-based information in separable sets of brain regions. However, these differences are reduced when multimodal context is considered. Specifically, the response profile of abstract words becomes more concrete-like when these are processed in visual scenes highly related to their meaning. Conversely, when the visual context is unrelated to a given concrete word, the activation pattern resembles more that of abstract conceptual processing. These results suggest that while concepts generally encode habitual experiences, the underlying neurobiological organisation is not fixed but depends dynamically on available contextual information.

Seeing the unseen: adapting Landscape and Visual Impact Assessment (LVIA) to examine Neolithic stone circles in the English Lake District

This article presents an in-depth exploration of the application of 3D techniques frequently used in Landscape and Visual Impact Assessment, with a focus on the adaptation of Accurate Visual Representation for use in landscape archaeology. Specifically, this paper explores the visual environment surrounding mid-to-late Neolithic enclosures in the English county of Cumbria.This study explores how 3D modelling, photography, and GIS analyses can enhance our understanding of archaeological landscapes. It focuses on the relationship between stone circles and their visual context, using the concept of ‘visual magnitude’ as a phenomenological lens. The results indicate that the mid-to-late Neolithic Cumbrian stone circles (circa. 3200 – 2500 BC) exhibit similarities to contemporary ‘henge enclosures’ across England.

RESMatch: Referring expression segmentation in a semi-supervised manner

Referring Expression segmentation (RES), a task that involves localizing specific instance-level objects on the basis of free-form linguistic descriptions, has emerged as a crucial frontier in human–AI interactions. It demands an intricate understanding of both the visual and textual contexts, and often requires extensive training data. This paper introduces RESMatch, the first semi-supervised learning (SSL) approach for RES aimed at reducing reliance on exhaustive data annotation. Moreover, our proposed RESMatch can leverage the abundance of image–text paired data available in the current era of training large models, resulting in an improved RES task performance without the need for costly semantic annotation, as evidenced by our experimental results. Moreover, although existing SSL techniques are effective in image segmentation, we find that they fall short in RES. Facing challenges such as the comprehension of free-form linguistic descriptions and the variability in object attributes, RESMatch introduces a trifecta of adaptations: revised strong perturbation, text augmentation, and adjustments for pseudolabel quality and strong–weak supervision. RESMatch has demonstrated its effectiveness by achieving state-of-the-art (SOTA) results in various experimental settings. This pioneering work lays the groundwork for future research in semi-supervised learning for referring expression segmentation.

Exacerbation of paranoia-like thoughts following exposure to common misophonia trigger sounds

Paranoia-like thoughts refer to heightened suspicions and unfounded beliefs about being watched or persecuted by others. Recent research has found a significant correlation between misophonia symptoms, a form of decreased sound tolerance, and paranoia-like thoughts, both of which are linked to heightened negative emotions in clinical and non-clinical populations. Notably, it has been observed that misophonia may also be associated with the tendency to attribute hostile intent to those producing triggering sounds, a feature consistent with paranoid ideation (i.e., perceptions of intentional harm). However, existing research is based on correlational data, limiting causal inference. Therefore, an online study involving a non-clinical sample (N = 487) employed an experimental approach to examine the relationship between misophonia symptoms, negative emotional response, and paranoia-like thoughts. Participants were randomly assigned to one of four task conditions, each related to exposure to different stimulus types: orofacial human-produced sounds, non-human sounds, sounds without visual context, or visuals devoid of sound. The results of mixed model ANOVA and mediation analysis revealed that exposure to common misophonia trigger sounds with a human-related visual context raised, but not significantly, the levels of paranoia-like thoughts. However, it did lead to a significant increase in negative emotions, which, in turn, proved to be a significant mediator of an increase in paranoia-like thoughts. Conversely, exposure to non-human sounds or to only audio/visual stimuli either decreased both negative emotions and paranoia-like thoughts or showed no significant change. This emphasized the role of context and the involvement of negative emotional response to human-made sounds in amplifying paranoia-like thoughts. Importantly, this effect was observed in individuals who do not meet the provisional diagnostic criteria for misophonia, suggesting that symptoms of misophonia may extend beyond clinical diagnoses, with milder manifestations potentially being present within the general population.

Ants integrate proprioception as well as visual context and efference copies to make robust predictions

Forward models are mechanisms enabling an agent to predict the sensory outcomes of its actions. They can be implemented through efference copies: copies of motor signals inhibiting the expected sensory stimulation, literally canceling the perceptual outcome of the predicted action. In insects, efference copies are known to modulate optic flow detection for flight control in flies. Here we investigate whether forward models account for the detection of optic flow in walking ants, and how the latter is integrated for locomotion control. We mounted Cataglyphis velox ants in a virtual reality setup and manipulated the relationship between the ants’ movements and the optic flow perceived. Our results show that ants compute predictions of the optic flow expected according to their own movements. However, the prediction is not solely based on efference copies, but involves proprioceptive feedbacks and is fine-tuned by the panorama’s visual structure. Mismatches between prediction and perception are computed for each eye, and error signals are integrated to adjust locomotion through the modulation of internal oscillators. Our work reveals that insects’ forward models are non-trivial and compute predictions based on multimodal information.

ArtEyer: Enriching GPT-based agents with contextual data visualizations for fine art authentication

Fine art authentication plays a significant role in protecting cultural heritage and ensuring the integrity of artworks. Traditional authentication methods require professionals to collect many reference materials and conduct detailed analyses. To ease the difficulty, we collaborate with domain experts to develop a GPT-based agent, namely ArtEyer, that offers accurate attributions, determines the origin and authorship, and executes visual analytics. Despite the convenience of the conversational user interface, novice users may still face challenges due to the hallucination issue and the steep learning curve associated with prompting. To face these obstacles, we propose a novel solution that places interactive data visualizations into the conversations. We create contextual visualizations from an external domain-dependent database to ensure data trustworthiness and allow users to provide precise instructions to the agent by interacting directly with these visualizations, thus overcoming the vagueness inherent in natural language-based prompting. We evaluate ArtEyer through an in-lab user study and demonstrate its usage with a real-world case.

Context-dependent orientation discontinuity encoding by gamma rhythms in mouse primary visual cortex.

Through the modulation of its surround, an identical visual stimulus can be perceived as more or less salient, allowing it to either stand out or seamlessly integrate with the rest of the visual scene. Gamma rhythms are associated with processing stimulus features across extensive areas of the visual field. Consistent with this concept, the magnitude of visually induced gamma rhythm depends on how well stimulus features aligned both within and outside the classical receptive field (CRF) at the recording site. However, there still exists some uncertainty regarding the encoding of context-modulated orientation discontinuity by gamma rhythms. To address this concern, we conducted extracellular recordings in layers II/III and IV of area V1 using lightly anaesthetized mice to investigate the gamma tuning for stimuli with orientation discontinuity. Our study revealed that gamma rhythms exhibit a preference for stimuli with orientation discontinuity similar to the spiking responses observed in V1, which contradicts the findings of previous studies. Furthermore, the gamma tuning of discontinuous orientations exhibits a moderate correlation with spike tuning and a positive correlation with the strength of surround suppression. Therefore, our study suggests a close association between gamma tuning and nearby spiking tuning; additionally, it highlights the connection between the encoding of visual features by gamma rhythms and functional architecture, as well as neural signal integration. KEY POINTS: Visual context modulates the gamma rhythms in the primary visual cortex. Discontinuous orientation elicits significantly enhanced gamma rhythms compared to the iso-orientation stimulus. The gamma tuning of discontinuous orientations exhibits a moderate correlation with spike tuning. Gamma tuning of orientation discontinuity exhibits a positive correlation with the strength of surround suppression.

The phantom effect in information visualization

Recent research focuses on understanding what triggers cognitive biases and how to alleviate them in the context of visualization use. Given its role in decision-making in other research fields, the Phantom Effect may hold exciting prospects among known biases. The Phantom Effect belongs to the category of decoy effects, where the decoy is an optimal yet unavailable alternative. We conducted a hybrid design experiment (N=76) where participants performed decision tasks based on information represented in different visualization idioms and phantom alternative’s unavailability presentation delays. We measured participants’ perceptual speed and visual working memory to study their impact on the expression of the Phantom Effect. Results show that visualization usually triggers the Phantom Effect, but two-sided bar charts mitigate this bias more effectively. We also found that waiting until the participant decides before presenting the decoy as unavailable helps alleviate the Phantom Effect. Although we did not find measurable effects, results also suggest that visual working memory and visualization literacy play a role in bias susceptibility. Our findings extend prior research in visualization-based decoy effects. They are the first steps to understanding the role of individual differences in the susceptibility to cognitive bias in visualization contexts.

Visual search for hazardous items: using virtual reality (VR) in laypersons to test wearable displays for firefighters

In virtual reality (VR), we assessed how untrained participants searched for fire sources with the digital twin of a novel augmented reality (AR) device: a firefighter’s helmet equipped with a heat sensor and an integrated display indicating the heat distribution in its field of view. This was compared to the digital twin of a current state-of-the-art device, a handheld thermal imaging camera. The study had three aims: (i) compare the novel device to the current standard, (ii) demonstrate the usefulness of VR for developing AR devices, (iii) investigate visual search in a complex, realistic task free of visual context. Users detected fire sources faster with the thermal camera than with the helmet display. Responses in target-present trials were faster than in target-absent trials for both devices. Fire localization after detection was numerically faster and more accurate, in particular in the horizontal plane, for the helmet display than for the thermal camera. Search was strongly biased to start on the left-hand side of each room, reminiscent of pseudoneglect in scene viewing. Our study exemplifies how VR can be used to study vision in realistic settings, to foster the development of AR devices, and to obtain results relevant to basic science and applications alike.

Revealing Religious Tolerance and Moderation in Upin & Ipin Series

Children’s animated films play an important role in shaping children’s perceptions of social and cultural values, including tolerance and religious moderation. This study aimed to understand how messages about religious tolerance and moderation are communicated through symbols, scenes, and character interactions in the Upin & Ipin series. This study used a qualitative approach with a visual discourse analysis method to examine the visual representation in the series. The data analysis technique involved observing scenes that show interreligious interactions, religious celebrations, and respectful behavior between characters with different cultural and religious backgrounds. Each scene was analyzed based on the visual and narrative context to identify messages of religious moderation. Data were analyzed using descriptive coding and thematic analysis to find patterns of visual depictions of religious moderation. The results showed that Upin & Ipin series consistently displays the values ​​of religious moderation through harmonious characters in participating in religious celebrations. This visual message emphasizes the importance of mutual respect in a multicultural society. This study implies that children’s media, especially cartoon film, can be used to teach the values ​​of religious moderation and tolerance from an early age.

Reduced on-line speech gesture integration during multimodal language processing in adults with moderate-severe traumatic brain injury: Evidence from eye-tracking

BackgroundLanguage is multimodal and situated in rich visual contexts. Language is also incremental, unfolding moment-to-moment in real time, yet few studies have examined how spoken language interacts with gesture and visual context during multimodal language processing. Gesture is a rich communication cue that is integrally related to speech and often depicts concrete referents from the visual world. Using eye-tracking in an adapted visual world paradigm, we examined how participants with and without moderate-severe traumatic brain injury (TBI) use gesture to resolve temporary referential ambiguity. MethodsParticipants viewed a screen with four objects and one video. The speaker in the video produced sentences (e.g., “The girl will eat the very good sandwich”), paired with either a meaningful gesture (e.g., sandwich-holding gesture) or a meaningless grooming movement (e.g., arm scratch) at the verb “will eat.” We measured participants’ gaze to the target object (e.g., sandwich), a semantic competitor (e.g., apple), and two unrelated distractors (e.g., piano, guitar) during the critical window between movement onset in the gesture modality and onset of the spoken referent in speech. ResultsBoth participants with and without TBI were more likely to fixate the target when the speaker produced a gesture compared to a grooming movement; however, relative to non-injured participants, the effect was significantly attenuated in the TBI group. DiscussionWe demonstrated evidence of reduced speech-gesture integration in participants with TBI relative to non-injured peers. This study advances our understanding of the communicative abilities of adults with TBI and could lead to a more mechanistic account of the communication difficulties adults with TBI experience in rich communication contexts that require the processing and integration of multiple co-occurring cues. This work has the potential to increase the ecological validity of language assessment and provide insights into the cognitive and neural mechanisms that support multimodal language processing.