Visual Modality Research Articles

Software Review

D-ID Creative Reality Studio (D-ID) is a platform for creating Artificial Intelligence (AI) presenter (digital human) videos, translating videos, and designing conversational agents. D-ID seamlessly integrates deep-learning face animation technology, large language models (LLMs), natural language processing (NLP), and speech synthesis and recognition (SSR), offering new possibilities for immersive language teaching and learning. As highlighted in the slogan, “Amaze your audience with your art” (D-ID, 2024), both language learners and teachers can utilise D-ID to create personalised multimodal digital learning resources. Aligning with the interaction hypothesis (Long, 1996), a conversational agent can serve as a language partner, offering accurate linguistic input and facilitating conversational practice anytime and anywhere. Additionally, the incorporation of a digital human is consistent with the Cognitive Theory of Multimedia Learning (Mayer & Moreno, 2003), which asserts that dual-channel input—integrating visual and auditory modalities—enhances memory retention and overall learning effectiveness. This software review centres on the web-based version of D-ID and provides a detailed analysis of the main features that can contribute to second language acquisition.

Stereo Event-Based Visual–Inertial Odometry

Event-based cameras are a new type of vision sensor in which pixels operate independently and respond asynchronously to changes in brightness with microsecond resolution, instead of providing standard intensity frames. Compared with traditional cameras, event-based cameras have low latency, no motion blur, and high dynamic range (HDR), which provide possibilities for robots to deal with some challenging scenes. We propose a visual–inertial odometry for stereo event-based cameras based on Error-State Kalman Filter (ESKF). The vision module updates the pose by relying on the edge alignment of a semi-dense 3D map to a 2D image, while the IMU module updates the pose using median integration. We evaluate our method on public datasets with general 6-DoF motion (three-axis translation and three-axis rotation) and compare the results against the ground truth. We compared our results with those from other methods, demonstrating the effectiveness of our approach.

Working Memory Deficits in School-Age Children With Cochlear Implants Are Primarily Explained by Deficits in the Processing of Auditory and Lexical Information.

Prelingual deaf children with cochlear implants show lower digit span test scores compared to normal-hearing peers, suggesting a working memory impairment. To pinpoint more precisely the subprocesses responsible for this impairment, we designed a sequence reproduction task with varying length (two to six stimuli), modality (auditory or visual), and compressibility (sequences with more or less regular patterns). Results on 22 school-age children with cochlear implants and 21 normal-hearing children revealed a deficit of children with cochlear implants only in the auditory modality. We observed no deficit in the visual modality and no deficit in the ability to detect and use regular patterns to improve memorization. These results suggest that the working memory deficit of children with cochlear implants is explained by an impairment in the processing, encoding, and/or storage of the auditory and lexical information, as opposed to a global storage deficit or an inability to use compressibility strategies to improve memorization. https://doi.org/10.23641/asha.28216088.

A cross sectional investigation of the development of rhythmic preferences with motor and perceptual tests

The preferred period hypothesis posits a slowing down of motor and perceptual rhythmic preferences with age, both reflecting an increase in the common internal oscillation period. This study further investigates the preferred period hypothesis by improving the measurement of perceptual rhythmic preferences through two tasks, tempo adjustment and tempo judgment, conducted in auditory and visual modalities. The study was conducted with three groups of children (5–6, 8–9, and 11–12 years old), and a group of young adults (21 to 30 years old) during the same time of the day. Correlational analyses show that the relationship between motor and perceptual rhythmic preferences strengthens between childhood and adulthood. Multilevel analyses reveal a general slowing down of motor and perceptual rhythmic preferences, contingent on the task and modality, with distinct trajectories for males and females. Our study indicate that the range of preferred rhythms narrows with age, becoming less variable in young adulthood, at the expense of a single and consistent, default rhythm. In brief, the preferred period appears only in young adults, with no support for the idea of a slowing down of an internal oscillation that would be reflected by both motor and perceptual rhythmic preferences.

Surgical video workflow analysis via visual-language learning

Surgical video workflow analysis has made intensive development in computer-assisted surgery by combining deep learning models, aiming to enhance surgical scene analysis and decision-making. However, previous research has primarily focused on coarse-grained analysis of surgical videos, e.g., phase recognition, instrument recognition, and triplet recognition that only considers relationships within surgical triplets. In order to provide a more comprehensive fine-grained analysis of surgical videos, this work focuses on accurately identifying triplets <instrument, verb, target> from surgical videos. Specifically, we propose a vision-language deep learning framework that incorporates intra- and inter- triplet modeling, termed I2TM, to explore the relationships among triplets and leverage the model understanding of the entire surgical process, thereby enhancing the accuracy and robustness of recognition. Besides, we also develop a new surgical triplet semantic enhancer (TSE) to establish semantic relationships, both intra- and inter-triplets, across visual and textual modalities. Extensive experimental results on surgical video benchmark datasets demonstrate that our approach can capture finer semantics, achieve effective surgical video understanding and analysis, with potential for widespread medical applications.

MEM-8X8: A Multimodal Framework for Emotion Classification and Mismatch Analysis Using the ICE Dataset

Aims: The study aims to evaluate visual and textual emotions, understand mismatches and overall explainability using the proposed multimodal emotion mapping-8X8 (MEM-8X8) framework on the developed IndiComics Emotion (ICE) dataset. Background: Understanding and aligning emotions in the visual and linguistic modes of comics is difficult due to their stylized and abstract nature. However, existing unimodal methods fail to obtain the interplay between text and visuals, highlighting the requirement for multimodal techniques for advanced emotion classification and mismatch assessment. Objective: This study proposes a robust MEM-8X8 framework backed by VisualBERT to classify and analyze the ICE dataset into visual and nuanced textual emotions, analyze mismatches, and provide comprehensive explainability with advanced interpretation tools. Methods: The proposed MEM-8X8 framework combines the Vision Transformers (ViT) and the bidirectional encoder representations from transformers (BERT) embeddings to evaluate the multimodal emotion classification and mapping using VisualBERT. Chi-square tests are conducted to analyze if the given emotions are in agreement, mismatched, and statistically significant. Along with this, the proposed model has outperformed its performance when compared with existing models. Results: The VisualBERT model outperformed the unimodal ViT and BERT with an average accuracy of 93.5% across all 8 emotions and a gain of 6.5% over single-modality approaches. The best agreement score of 93.6% has been recorded for the “Neutral” emotion. In comparison, the “Angry-Disgust” emotion pair has the highest mismatch frequency of 18.4%, highlighting challenges in their alignment across visual and textual modalities in comics. Conclusion: The proposed VisualBERT-based MEM-8X8 framework performs better in classifying 8 emotions, addressing mismatch alignment across visual and textual modalities. This study lays the foundation for future comics and multimodal emotion recognition analysis advancements.

Posterior parietal cortex regulates neural timescales and stimulus-driven attentional modulation in the prefrontal cortex.

Intrinsic neural timescales characterize the dynamics of endogenous fluctuations in neural activity. We measured the intrinsic timescales of prefrontal neurons and examined their changes during posterior parietal cortex (PPC) inactivation. Frontal eye field (FEF) neurons within the prefrontal cortex (PFC) exhibited a bimodal distribution of timescales: short-timescale neurons showed stronger transient visual responses, while long-timescale neurons exhibited stronger sustained modulation during stimulus-driven attention. PPC inactivation increased intrinsic timescales in both neuron types, with a 15-fold greater increase in short-timescale neurons. Additionally, it reduced visual and attentional modulation, particularly impairing attentional modulation in long-timescale neurons. Our results provide the first causal evidence that the intrinsic timescales of FEF neurons depend on PPC and suggest the presence of at least two network motifs with distinct timescales that contribute to neuronal dynamics and functional computations within FEF. The heterogeneity in these timescales may facilitate cognitive flexibility, allowing PFC to adapt to varying task demands.

Multimodal language processing in school-aged Mandarin-speaking children: the role of beat gesture in enhancing memory for discourse information

Abstract Communication is typically multimodal, meaning that communication involves the utilization of various cues from different modalities. While spoken words in the auditory modality primarily convey semantic information, gestures from the visual modality complement and enhance the communication process. However, the role of visual cues, specifically beat gestures, a type of non-verbal co-speech gesture used to emphasize certain information, in language processing remains largely underexplored. The present study, using a memory task where 90 Mandarin-speaking schoolchildren aged 6–9 viewed stories individually and were later asked about what happened in the stories, investigates the role of beat gestures in enhancing the memory for discourse information in Mandarin. The results show that words with beat gestures were generally recalled better for children of all grades, indicating that by age 6, Mandarin children have already acquired the ability to utilize beat gestures to encode a discourse. This study contributes significantly to our limited understanding of the significance of visual cues in multimodal language processing.

Large language model-augmented learning for auto-delineation of treatment targets in head-and-neck cancer radiotherapy.

Radiation therapy (RT) is highly effective, but its success depends on accurate, manual target delineation, which is time-consuming, labor-intensive, and prone to variability. Despite AI advancements in auto-contouring normal tissues, accurate RT target volume delineation remains challenging. This study presents Radformer, a novel visual language model that integrates text-rich clinical data with medical imaging for accurate automated RT target volume delineation. We developed Radformer, an innovative network that utilizes a hierarchical vision transformer as its backbone and integrates large language models (LLMs) to extract and embed clinical data in text-rich form. The model features a novel visual language attention module (VLAM) to combine visual and linguistic features, enabling language-aware visual encoding (LAVE). The Radformer was evaluated on a dataset of 2985 patients with head-and-neck cancer who underwent RT. Quantitative evaluations were performed utilizing metrics such as the Dice similarity coefficient (DSC), intersection over union (IOU), and 95th percentile Hausdorff distance (HD95). The Radformer demonstrated superior performance in segmenting RT target volumes compared to state-of-the-art models. On the head-and-neck cancer dataset, Radformer achieved a mean DSC of 0.76 ± 0.09 versus 0.66 ± 0.09, a mean IOU of 0.69 ± 0.08 versus 0.59 ± 0.07, and a mean HD95 of 7.82 ± 6.87 mm versus 14.28 ± 6.85 mm for gross tumor volume delineation, compared to the baseline 3D-UNETR. The Radformer model offers a clinically optimal means of RT target auto-delineation by integrating both imaging and clinical data through a visual language model. This approach improves the accuracy of RT target volume delineation, facilitating broader AI-assisted automation in RT treatment planning.

Effect of Temporal Modulation of Tactile Stimulus on Time Perception

Abstract Time perception varies across different sensory modalities, with previous research indicating that visual flicker can dilate perceived duration while auditory flutter can sometimes compress it. This study aimed to explore duration perception in the relatively understudied tactile modality and compare it with the visual modality. In Experiment 1A, we investigated the effect of temporal modulation on perceived duration using 1-s tactile flutter and visual flicker stimuli, either stationary or modulated at 6.5 Hz, 12.5 Hz, or 24.5 Hz. Participants reported which of two sequentially presented stimuli appeared longer in duration. Results demonstrated that both tactile flutter and visual flicker induced significant time distortions, with higher frequencies leading to greater duration dilation. Notably, the magnitudes of time distortion were significantly correlated between the two modalities. Experiment 1B assessed the roles of sensitivity and subjective saliency in these time distortions. Contrary to previous findings, saliency did not correlate with the observed time distortions in either modality, and sensitivity did not correlate with time distortions in the visual modality. These results suggest that, while tactile duration perception exhibits similarities to visual duration perception, the lack of correlation with saliency challenges the idea that time distortion is driven by subjective saliency.

Measuring working memory span with WAIS-IV: Digit sequence is the superior span test

The Digit Span test has been part of the Wechsler tests from the first version. In the WAIS-IV the Digit Span Sequencing subtest (DSS) was introduced and in the forthcoming WAIS-5 working memory span will also be measured in the visual modality. The present study analyzes WAIS-IV Digit Span, Letter- Number Span (LNS) and WMS-III Spatial Span (SS) performance in a mixed clinical sample, expecting to find that Digit Span Forwards (DSF) lacks sensitivity to the Working Memory impairment evident in D-KEFS Trail Making Test-4 (TMT-4) scores ≤1 SD below normative means in the sample. The results showed DSF score above normative means, Digit Span Backwards (DSB) around mean, while SS and LNS was slightly impaired and DSS impaired at the same level as TMT- 4. A double dissociation was observed in DSF and SS performance between subjects with Language- and Non-verbal learning disorders. Most subjects scored in the average range on the LNS-span and high kurtosis reduced sensitivity. Taking LNS and TMT-4 as criterion measures of WM, regression analyses showed no unique contribution of DSF to the variance in these tests. The study supports prior critiques regarding the composite Digit Span measure and demonstrates that the DSS are more effective than the current version of LNS in identifying reduced Working Memory capacity.

A Multimodal Social Semiotic Analysis of Typography and Layout in ‘’Alexander Master’s Novel Stuart: A Life Backwards’’

The output of multimodal texts has increased over the past 20 years due to advances in technology and culture, which has accelerated the demand for tackling multimodal additional semiotic modes for meaning making, such as typography, layout ,visual images, colour, etc. The theory of multimodality, as supposed by Kress and Van Leeuwen (2001–2006), Baldry and Thibaut (2006), is applied to a potential framework for the analysis of multimodal novels that use not only wording structure but also many semiotic modes in the formation of meaning. The social approach to multimodality, in brief, is based on Halliday's linguistic description and adhered to the multimodal assumption that "mutual propositions are employed and worked together in various modes" (Kress and Leeuwen,2002:3). Thus, Kress and Van Leeuwen (1996:39), in their book ‘’Reading Images’’ investigate the level for which the underlying theories of Halliday's framework to language are feasible and applicative to visual communicative management, and they create a visual system that mostly makes use of Halliday's principles and concepts. The study is concerned with the investigation of a multimodal analysis of the multiple elements of various modes of typographic devices, and layout design used in the selected novel. This selected text ‘’Stuart: A Life Backwards’’ is a multimodal novel since the writer Masters does not rely on linguistic verbal tools, he extends them to incorporate a variety of visual semiotic modes and graphic communicative elements that will intensify his understanding of storytelling and narrative development. The study has shown that an extension to the mode of wording formation is performed in the multimodal analysis by incorporating more multimodal tackling features of typographic principles and layout framing devices in creating an appropriate comprehensive meaning construction in Masters’ selected narration.

Flexibility in choosing decision policies in gathering discrete evidence over time.

The brain can remarkably adapt its decision-making process to suit the dynamic environment and diverse aims and demands. The brain's flexibility can be classified into three categories: flexibility in choosing solutions, decision policies, and actions. We employ two experiments to explore flexibility in decision policy: a visual object categorization task and an auditory object categorization task. Both tasks required participants to accumulate discrete evidence over time, with the only difference being the sensory state of the stimuli. We aim to investigate how the brain demonstrates flexibility in selecting decision policies in different sensory contexts when the solution and action remain the same. Our results indicate that the decision policy of the brain in integrating information is independent of inter-pulse interval across these two tasks. However, the decision policy based on how the brain ranks the first and second pulse of evidence changes flexibly. We show that the sequence of pulses does not affect the choice accuracy in the auditory mode. However, in the visual mode, the first pulse had the larger leverage on decisions. Our research underscores the importance of incorporating diverse contexts to improve our understanding of the brain's flexibility in real-world decision-making.

Advertising fragrance through visual and audible information: a multimodal metaphor analysis of perfume commercials

Abstract According to conceptual metaphor, this study finds and categorizes three metaphors in perfume commercials: FRAGRANCE IS ATTRACTION, FRAGRANCE IS EMOTION, and FRAGRANCE IS OBJECT. Drawing on the analytical tool of multimodal metaphor analysis, the study further analyzes how perfume commercials complete the metaphorical operation mechanism of mapping from visual and auditory modes to olfactory. It is found that, different from the traditional definition of a concrete source domain and an abstract target domain in conceptual metaphors, the joint participation of multiple modes in commercials may cause a reversal of the degree of abstraction between the source and the target. Due to the limitation of transmitting olfactory information in fragrance commercials where the audience can only receive audible and visual information, “scent” becomes a more abstract concept, while the abstract concepts of “attraction” and “emotion” become a more concrete source. Additionally, the three metaphors’ advertising effects in fragrance commercials are evaluated. Theoretical implications for future studies in multimodal metaphor analysis are also discussed.

Video Description Generation Method Based on Contrastive Language–Image Pre-Training Combined Retrieval-Augmented and Multi-Scale Semantic Guidance

In view of the insufficiency of the text encoder using CLIP and the insufficiency of the interaction between the two towers using CLIP, a CLIP-based video description model RAMSG is proposed, which combines retrieval augmentation with multi-scale semantic guidance. Firstly, RAMSG uses the visual and text encoder module of CLIP to achieve cross-modal retrieval and extract relevant text as a supervisory signal. Then, the intrinsic order of semantic words is obtained by a semantic detector and ranker. Finally, the global and local semantic guidance of the multi-scale semantic guidance module is used to improve the video description generation effect of the decoder module. Experimental results on the video description datasets MSR-VTT and VATEX show that the RAMSG model has a significant improvement in several performance indicators compared with other work, and the additional text semantics obtained through videotext matching tasks greatly improve the performance of the model.

Virtual simulators in the automated control system subsystem for training operators of metal-cutting machines

The article describes the proposed method of automated construction of virtual simulators for the subsystem of personnel training of the automated control systems of production. Its distinctive feature is the work with the customer based on the provision of functional videos from production, as well as the formalization of the design and development process in the form of structures for the process of creating software and visual modules taking into account the specifics of the development of virtual reality simulators. An ontological model has been developed for the construction of virtual simulators for training machine tool operators in the automated control systems of production, which uses the identified basic concepts and relationships between them in the subject area of work of machine tool operators. Its implementation is based on the Protege software environment. The following diagrams are proposed: functional and contextual IDEF0 of the functioning of virtual simulators for training in working on universal metal-cutting machines: milling and turning. The architecture of a virtual simulator for use in the subsystem of training personnel of automated control systems of production, as well as a structural diagram of the formation of practice-oriented skills in working on metal-cutting machines using virtual simulators are described.

Efficient Multi-Task Training with Adaptive Feature Alignment for Universal Image Segmentation.

Universal image segmentation aims to handle all segmentation tasks within a single model architecture and ideally requires only one training phase. To achieve task-conditioned joint training, a task token needs to be used in the multi-task training to condition the model for specific tasks. Existing approaches generate the task token from a text input (e.g., "the task is panoptic"). However, such text-based inputs merely serve as labels and fail to capture the inherent differences between tasks, potentially misleading the model. In addition, the discrepancy between visual and textual modalities limits the performance gains in existing text-involved segmentation models. Nevertheless, prevailing modality-alignment methods rely on large-scale uni-modal encoders for both modalities and an extremely large amount of paired data for training, and therefore it is hard to apply these existing models to lightweight segmentation models and resource-constrained devices. In this paper, we propose Adaptive Feature Alignment (AFA) integrated with a learnable task token to address these issues. The learnable task token automatically captures inter-task differences from both image features and text queries during training, providing a more effective and efficient solution than a predefined text-based token. To efficiently align the two modalities without introducing extra complexity, we reconsider the differences between a text token and an image token and replace image features with class-specific means in our proposed AFA. We evaluate our model performance on the ADE20K and Cityscapes datasets. Experimental results demonstrate that our model surpasses baseline models in both efficiency and effectiveness, achieving state-of-the-art performance among segmentation models with a comparable amount of parameters.

MKER: multi-modal knowledge extraction and reasoning for future event prediction

Humans can predict what will happen shortly, which is essential for survival, but machines cannot. To equip machines with the ability, we introduce the innovative multi-modal knowledge extraction and reasoning (MKER) framework. This framework combines external commonsense knowledge, internal visual relation knowledge, and basic information to make inference. This framework is built on an encoder-decoder structure with three essential components: a visual language reasoning module, an adaptive cross-modality feature fusion module, and a future event description generation module. The visual language reasoning module extracts the object relationships among the most informative objects and the dynamic evolution of the relationship, which comes from the sequence scene graphs and commonsense graphs. The long short-term memory model is employed to explore changes in the object relationships at different times to form a dynamic object relationship. Furthermore, the adaptive cross-modality feature fusion module aligns video and language information by using object relationship knowledge as guidance to learn vision-language representation. Finally, the future event description generation module decodes the fused information and generates the language description of the next event. Experimental results demonstrate that MKER outperforms existing methods. Ablation studies further illustrate the effectiveness of the designed module. This work advances the field by providing a way to predict future events, enhance machine understanding, and interact with dynamic environments.

Touching the Ground: Evaluating the Effectiveness of Data Physicalizations for Spatial Data Analysis Tasks.

Inspired by recent advances in digital fabrication, artists and scientists have demonstrated that physical data encodings (i.e., data physicalizations) can increase engagement with data, foster collaboration, and in some cases, improve data legibility and analysis relative to digital alternatives. However, prior empirical studies have only investigated abstract data encoded in physical form (e.g., laser cut bar charts) and not continuously sampled spatial data fields relevant to climate and medical science (e.g., heights, temperatures, densities, and velocities sampled on a spatial grid). This paper presents the design and results of the first study to characterize human performance in 3D spatial data analysis tasks across analogous physical and digital visualizations. Participants analyzed continuous spatial elevation data with three visualization modalities: (1) 2D digital visualization; (2) perspective-tracked, stereoscopic "fishtank" virtual reality; and (3) 3D printed data physicalization. Their tasks included tracing paths downhill, looking up spatial locations and comparing their relative heights, and identifying and reporting the minimum and maximum heights within certain spatial regions. As hypothesized, in most cases, participants performed the tasks just as well or better in the physical modality (based on time and error metrics). Additional results include an analysis of open-ended feedback from participants and discussion of implications for further research on the value of data physicalization. All data and supplemental materials are available at https://osf.io/7xdq4/.

Neural correlates of phonetic categorization under auditory (phoneme) and visual (grapheme) modalities

This study assessed the neural mechanisms and relative saliency of categorization for speech sounds and comparable graphemes (i.e., visual letters) of the same phonetic label. Given that linguistic experience shapes categorical processing, and letter-speech sound matching plays a crucial role during early reading acquisition, we hypothesized sound phoneme and visual grapheme tokens representing the same linguistic identity might recruit common neural substrates, despite originating from different sensory modalities. Behavioral and neuroelectric brain responses (ERPs) were acquired as participants categorized stimuli from sound (phoneme) and homologous letter (grapheme) continua each spanning a /da/−/ga/ gradient. Behaviorally, listeners were faster and showed stronger categorization of phoneme compared to graphemes. At the neural level, multidimensional scaling of the EEG revealed responses self-organized in a categorial fashion such that tokens clustered within their respective modality beginning ∼150–250 ms after stimulus onset. Source-resolved ERPs further revealed modality-specific and overlapping brain regions supporting phonetic categorization. Left inferior frontal gyrus and auditory cortex showed stronger responses for sound category members compared to phonetically ambiguous tokens, whereas early visual cortices paralleled this categorical organization for graphemes. Auditory and visual categorization also recruited common visual association areas in extrastriate cortex but in opposite hemispheres (auditory = left; visual = right). Our findings reveal both auditory and visual sensory cortex supports categorical organization for phonetic labels within their respective modalities. However, a partial overlap in phoneme and grapheme processing among occipital brain areas implies the presence of an isomorphic, domain-general mapping for phonetic categories in dorsal visual system