Multimodal Annotations Research Articles

PilotAR: Streamlining Pilot Studies with OHMDs from Concept to Insight

Pilot studies in HCI research serve as a cost-effective approach to validate potential ideas and identify impactful findings before extensive studies. Yet, the additional requirements of AR/MR, such as multi-view observations and increased multitasking, make it challenging to conduct pilot studies effectively, hindering innovations in this field. Based on interviews with 12 AR/MR researchers, we identified the key challenges associated with conducting AR/MR pilot studies with Optical See-Through Head-Mounted Displays (OST-HMDs, OHMDs), including the inability to observe and record in-context user interactions, increased task load, and difficulties with in-context data analysis and discussion. To tackle these challenges, we introduce PilotAR, a desktop-based tool designed iteratively to enhance OHMD-based AR/MR pilot studies. PilotAR facilitates data collection via live first-person and third-person views, multi-modal annotations, and flexible wizarding interfaces. It also accommodates multi-experimenter settings, streamlines the study process with configurable workflows and shortcuts, records annotated data, and eases results sharing. Formative testing, conducted using three case studies, has highlighted the significant benefits of PilotAR, as well as its potential for further development and refinement.

Editable Co-Speech Gesture Synthesis Enhanced with Individual Representative Gestures

Co-speech gesture synthesis is a challenging task due to the complexity and uncertainty between gestures and speech. Gestures that accompany speech (i.e., Co-Speech Gesture) are an essential part of natural and efficient embodied human communication, as they work in tandem with speech to convey information more effectively. Although data-driven approaches have improved gesture synthesis, existing deep learning-based methods use deterministic modeling which could lead to averaging out predicted gestures. Additionally, these methods lack control over gesture generation such as user editing of generated results. In this paper, we propose an editable gesture synthesis method based on a learned pose script, which disentangles gestures into individual representative and rhythmic gestures to produce high-quality, diverse and realistic poses. Specifically, we first detect the time of occurrence of gestures in video sequences and transform them into pose scripts. Regression models are then built to predict the pose scripts. Next, learned pose scripts are used for gesture synthesis, while rhythmic gestures are modeled using a variational auto-encoder and a one-dimensional convolutional network. Moreover, we introduce a large-scale Chinese co-speech gesture synthesis dataset with multimodal annotations for training and evaluation, which will be publicly available to facilitate future research. The proposed method allows for the re-editing of generated results by changing the pose scripts for applications such as interactive digital humans. The experimental results show that this method generates more quality, more diverse, and realistic gestures than other existing methods.

OcularSeg: Accurate and Efficient Multi-Modal Ocular Segmentation in Non-Constrained Scenarios

Multi-modal ocular biometrics has recently garnered significant attention due to its potential in enhancing the security and reliability of biometric identification systems in non-constrained scenarios. However, accurately and efficiently segmenting multi-modal ocular traits (periocular, sclera, iris, and pupil) remains challenging due to noise interference or environmental changes, such as specular reflection, gaze deviation, blur, occlusions from eyelid/eyelash/glasses, and illumination/spectrum/sensor variations. To address these challenges, we propose OcularSeg, a densely connected encoder–decoder model incorporating eye shape prior. The model utilizes Efficientnetv2 as a lightweight backbone in the encoder for extracting multi-level visual features while minimizing network parameters. Moreover, we introduce the Expectation–Maximization attention (EMA) unit to progressively refine the model’s attention and roughly aggregate features from each ocular modality. In the decoder, we design a bottom-up dense subtraction module (DSM) to amplify information disparity between encoder layers, facilitating the acquisition of high-level semantic detailed features at varying scales, thereby enhancing the precision of detailed ocular region prediction. Additionally, boundary- and semantic-guided eye shape priors are integrated as auxiliary supervision during training to optimize the position, shape, and internal topological structure of segmentation results. Due to the scarcity of datasets with multi-modal ocular segmentation annotations, we manually annotated three challenging eye datasets captured in near-infrared and visible light scenarios. Experimental results on newly annotated and existing datasets demonstrate that our model achieves state-of-the-art performance in intra- and cross-dataset scenarios while maintaining efficient execution.

MACSA: A multimodal aspect-category sentiment analysis dataset with multimodal fine-grained aligned annotations

MACSA: A multimodal aspect-category sentiment analysis dataset with multimodal fine-grained aligned annotations

Uni2Mul: A Conformer-Based Multimodal Emotion Classification Model by Considering Unimodal Expression Differences with Multi-Task Learning

Multimodal emotion classification (MEC) has been extensively studied in human–computer interaction, healthcare, and other domains. Previous MEC research has utilized identical multimodal annotations (IMAs) to train unimodal models, hindering the learning of effective unimodal representations due to differences between unimodal expressions and multimodal perceptions. Additionally, most MEC fusion techniques fail to consider the unimodal–multimodal inconsistencies. This study addresses two important issues in MEC: learning satisfactory unimodal representations of emotion and accounting for unimodal–multimodal inconsistencies during the fusion process. To tackle these challenges, the authors propose the Two-Stage Conformer-based MEC model (Uni2Mul) with two key innovations: (1) in stage one, unimodal models are trained using independent unimodal annotations (IUAs) to optimize unimodal emotion representations; (2) in stage two, a Conformer-based architecture is employed to fuse the unimodal representations learned in stage one and predict IMAs, accounting for unimodal–multimodal differences. The proposed model is evaluated on the CH-SIMS dataset. The experimental results demonstrate that Uni2Mul outperforms baseline models. This study makes two key contributions: (1) the use of IUAs improves unimodal learning; (2) the two-stage approach addresses unimodal–multimodal inconsistencies during Conformer-based fusion. Uni2Mul advances MEC by enhancing unimodal representation learning and Conformer-based fusion.

A Multimodal Corpus-Based Study of american career weblogs

The article describes the application of multimodal corpus-based methodologies to the study of the popular American career blogs. The research is carried out within the framework of the theory of multimodal discourse analysis. Quantitative and qualitative methods have been applied in order to identify and explain different aspects of verbal and graphic modes convergence. Two corpora have been compiled by the authors for the purposes of the research. Both corpora contain multimodal annotations to browse and search for relevant textual and visual materials. The corpus software used in this research is UAM CorpusTool and UAM ImageTool, they include frequency and keywords search, statistical processing of visual images annotations and their combinations with keywords. A collection of texts along with the images (photos, graphics and illustrations) has been analyzed to investigate how verbal and non-verbal modes work together to convey the holistic meaning of the message. The results suggest that verbal and graphic modes are non-competing but mutually supportive and complementary. The analysis has proved that the specified modes have different meaning potential: words can add focus, identification, precision, and perspective to a pictorial representation while images add emotional, expressive and connotative value to the text.

Medical lesion segmentation by combining multimodal images with modality weighted UNet.

Automatic segmentation of medical lesions is a prerequisite for efficient clinic analysis. Segmentation algorithms for multimodal medical images have received much attention in recent years. Different strategies for multimodal combination (or fusion), such as probability theory, fuzzy models, belief functions, and deep neural networks, have also been developed. In this paper, we propose the modality weighted UNet (MW-UNet) and attention-based fusion method to combine multimodal images for medical lesion segmentation. MW-UNet is a multimodal fusion method which is based on UNet, but we use a shallower layer and fewer feature map channels to reduce the amount of network parameters, and our method uses the new multimodal fusion method called fusion attention. It uses weighted sum rule and fusion attention to combine feature maps in intermediate layers. During training, all the weight parameters are updated through backpropagation like other parameters in the network. We also incorporate residual blocks into MW-UNet to further improve segmentation performance. The comparison between the automatic multimodal lesion segmentations and the manual contours was quantified by (1) five metrics including Dice, 95% Hausdorff Distance (HD95), volumetric overlap error (VOE), relative volume difference (RVD), and mean-Intersection-over-Union (mIoU); (2) Number of parameters and flops to calculate the complexity of thenetwork. The proposed method is verified on ZJCHD, which is the data set of contrast-enhanced computed tomography (CECT) for Liver Lesion Segmentation taken from Cancer Hospital of the University of Chinese Academy of Sciences (Zhejiang Cancer Hospital), Hangzhou, China. For accuracy evaluation, we use 120 patients with liver lesions from ZJCHD, of which 100 are used for fourfold cross-validation (CV) and 20 are used for hold-out (HO) test. The mean Dice was and for HO and CV tests, respectively. The corresponding HD95, VOE, RVD, and mIoU of the two tests are 1.95 ± 1.83 and 2.67 ± 3.35 mm, 13.11 ± 15.83 and , 12.20 ± 18.20 and , and 83.79 ± 15.83 and . The parameters and flops of our method is 4.04 M and 18.36 G,respectively. The results show that our method performs well on multimodal liver lesion segmentation. It can be easily extended to other multimodal data sets and other networks for multimodal fusion. Our method is the potential to provide doctors with multimodal annotations and assist them with clinicaldiagnosis.

Audible Sniffs: Smelling-in-Interaction

ABSTRACTThis article contributes to the study of nonlexical sound resources in social interaction by describing sniffs as sounds made by the body that have a physiological origin but also assume an interactional relevance: sniffing sounds made by participants when smelling. On the basis of a video-recorded tasting session in which participants engage in describing aromas, the article details the systematic organization of sniffing in a diversity of sequential environments—in which sniff-prefaced turns offer aroma descriptions in response to olfactory inquiries, confirm previous descriptions, and give alternative descriptions. Analyzing audible sounds made while visibly smelling in face-to-face interaction, the article’s aims are twofold: to contribute to the multimodal study of sounds in interaction and also to the study of sensoriality as an intersubjective practice through the systematic investigation of smelling-in-interaction. Data are in French with English translation and multimodal annotations.

On the benefits of multimodal annotations for vocabulary uptake from reading

ABSTRACTSeveral research articles published in the realm of Computer Assisted Language Learning (CALL) have reported evidence of the benefits of multimodal annotations, i.e. the provision of pictorial as well as verbal clarifications, for vocabulary uptake from reading. Almost invariably, these publications account for the observed benefits with reference to Paivio's Dual Coding Theory, suggesting it is the visual illustration of word meaning that enhances the quality of processing and hence makes new words more memorable. In this discussion article, we explore the possibility that it is not necessarily the multimodality per se that accounts for the reported benefits. Instead, we argue that the provision of multimodal annotations is one of several possible means of inviting more and/or longer attention to the annotations – with amounts of attention given to words being a significant predictor of their retention in memory. After reviewing the available research on the subject and questioning whether invoking Paivio's Dual Coding Theory is an optimal account for reported findings, we report an eye-tracking study the results of which are consistent with the alternative thesis that the advantage of multimodal glosses for word learning lies with the greater quantity of attention these glosses attract in comparison with single-mode glosses. We conclude with a call for further research on combinations and sequences of annotation types, regardless of multimodality, as ways of promoting vocabulary uptake from reading.

Multimodal Saliency and Fusion for Movie Summarization Based on Aural, Visual, and Textual Attention

Multimodal streams of sensory information are naturally parsed and integrated by humans using signal-level feature extraction and higher level cognitive processes. Detection of attention-invoking audiovisual segments is formulated in this work on the basis of saliency models for the audio, visual, and textual information conveyed in a video stream. Aural or auditory saliency is assessed by cues that quantify multifrequency waveform modulations, extracted through nonlinear operators and energy tracking. Visual saliency is measured through a spatiotemporal attention model driven by intensity, color, and orientation. Textual or linguistic saliency is extracted from part-of-speech tagging on the subtitles information available with most movie distributions. The individual saliency streams, obtained from modality-depended cues, are integrated in a multimodal saliency curve, modeling the time-varying perceptual importance of the composite video stream and signifying prevailing sensory events. The multimodal saliency representation forms the basis of a generic, bottom-up video summarization algorithm. Different fusion schemes are evaluated on a movie database of multimodal saliency annotations with comparative results provided across modalities. The produced summaries, based on low-level features and content-independent fusion and selection, are of subjectively high aesthetic and informative quality.

Interactive state-transition diagrams for visualization of multimodal annotation

Multimodal Analysis is a branch of Social Semiotics that studies how different resources, such as language, gesture, music, imagery, cinematography, etc. are used in modern media, and by humans in general, to construct and communicate meaning. Though theoretical aspects of multimodal analysis are w ell developed in the literature, and there is progress in the development of annotation software, the interpretation of annotated data still rarely uses algorithmic and data visualization approaches, limiting practical research in multimodal analysis. In the current work we propose how time stamped, tier-based multimodal annotations can be seen as finite state automatons and visualized as state-transition diagrams. We propose an algorithm converting time stamped annotation into a state-transition diagram and discuss how the resulting state-transition diagram can be visualized using freely available graph visualization tools. Then, we discuss common drawbacks of the existing tools, and propose an interactive visualization software tool allowing the analyst to synchronize the annotation state-transition diagram with the original media file. Finally, a case study on how the proposed visualization tools can assist media researchers in detecting patterns in multimodal analyses of news videos is presented.

Learning human multimodal dialogue strategies

AbstractWe investigate the use of different machine learning methods in combination with feature selection techniques to explore human multimodal dialogue strategies and the use of those strategies for automated dialogue systems. We learn policies from data collected in a Wizard-of-Oz study where different human ‘wizards’ decide whether to ask a clarification request in a multimodal manner or else to use speech alone. We first describe the data collection, the coding scheme and annotated corpus, and the validation of the multimodal annotations. We then show that there is a uniform multimodal dialogue strategy across wizards, which is based on multiple features in the dialogue context. These are generic features, available at runtime, which can be implemented in dialogue systems. Our prediction models (for human wizard behaviour) achieve a weighted f-score of 88.6 per cent (which is a 25.6 per cent improvement over the majority baseline). We interpret and discuss the learned strategy. We conclude that human wizard behaviour is not optimal for automatic dialogue systems, and argue for the use of automatic optimization methods, such as Reinforcement Learning. Throughout the investigation we also discuss the issues arising from using small initial Wizard-of-Oz data sets, and we show that feature engineering is an essential step when learning dialogue strategies from such limited data.

Du corpus vidéo à l'agent expressif. Utilisation des différents niveaux de représentation multimodale et émotionnelle

Relations between emotions and multimodal behaviors have mostly been studied in the case of acted basic emotions. In this paper, we describe two experiments studying these relations with a copy-synthesis approach. We start from video clips of TV interviews including real-life behaviors. A protocol and a coding scheme have been defined for annotating these clips at several levels (context, emotion, multimodality). The first experiment enabled to manually identify the levels of representation required for replaying the annotated behaviors by an expressive agent. The second experiment involved automatic extraction of information from the multimodal annotations. Such an approach enables to study the complex relations between emotion and multimodal behaviors.