Multimodal Corpus Research Articles

Multimodal Sarcasm Detection via Hybrid Classifier with Optimistic Logic

This work aims to provide a novel multimodal sarcasm detection model that includes four stages: pre-processing, feature extraction, feature level fusion, and classification. The pre-processing uses multimodal data that includes text, video, and audio. Here, text is pre-processed using tokenization and stemming, video is pre-processed during the face detection phase, and audio is pre-processed using the filtering technique. During the feature extraction stage, such text features as TF-IDF, improved bag of visual words, n-gram, and emojis as well on the video features using improved SLBT, and constraint local model (CLM) are extraction. Similarly the audio features like MFCC, chroma, spectral features, and jitter are extracted. Then, the extracted features are transferred to the feature level fusion stage, wherein an improved multilevel canonical correlation analysis (CCA) fusion technique is performed. The classification is performer using a hybrid classifier (HC), e.g. bidirectional gated recurrent unit (Bi-GRU) and LSTM. The outcomes of Bi-GRU and LSTM are averaged to obtain an effective output. To make the detection results more accurate, the weight of LSTM will be optimally tuned by the proposed opposition learning-based aquila optimization (OLAO) model. The MUStARD dataset is a multimodal video corpus used for automated sarcasm Discovery studies. Finally, the effectiveness of the proposed approach is proved based on various metrics.

한영 영화 번역에 나타나는 멀티모달 결속성 - 오프스크린 및 온뮤트 더빙 분석을 중심으로 -

This study aims to analyze various aspects of the multimodal cohesion formulated between visual elements on screen and offscreen/on-mute dubbing dialogues in a personally built multimodal corpus, named Multimodal Corpus of English-dubbed Korean Films, consisting of 12 films released in a DVD format in North America from 2002 to 2017. Based on the model of visual-verbal cohesion and the concept of information linking, this study identifies that the offscreen and on-mute dubbing dialogues show multiple forms of multimodal cohesion. Visual elements and dubbing dialogues form the relations of elaboration: dubbing dialogue making the image more specific (specification), and dubbing dialogues paraphrasing the image (explanation). The relation can be extended to make dubbing dialogues narratively cohesive or give a spoiler (extension). This study shows that visual elements and dialogues create visually cohesive relations, enhancing target audience’s understanding of dubbed films.

Deep Learning-Based Construction and Processing of Multimodal Corpus for IoT Devices in Mobile Edge Computing.

Dialogue sentiment analysis is a hot topic in the field of artificial intelligence in recent years, in which the construction of multimodal corpus is the key part of dialogue sentiment analysis. With the rapid development of the Internet of Things (IoT), it provides a new means to collect the multiparty dialogues to construct a multimodal corpus. The rapid development of Mobile Edge Computing (MEC) provides a new platform for the construction of multimodal corpus. In this paper, we construct a multimodal corpus on MEC servers to make full use of the storage space distributed at the edge of the network according to the procedure of constructing a multimodal corpus that we propose. At the same time, we build a deep learning model (sentiment analysis model) and use the constructed corpus to train the deep learning model for sentiment on MEC servers to make full use of the computing power distributed at the edge of the network. We carry out experiments based on real-world dataset collected by IoT devices, and the results validate the effectiveness of our sentiment analysis model.

When hands stop moving, interaction keeps going

Abstract This study explores moments in signed and spoken conversation when manual production is on hold and its resulting interactive ramifications. Typically, the temporal structure of gesture and sign can be decomposed into a stream of distinct manual phases. There are moments, however, when this activity is stopped. This may happen for various reasons, e.g., when seeking attention, holding the floor or during overlaps. Holds have mostly been examined in sign languages regarding prosody, syntax, and corresponding to vowel lengthening in spoken languages. In gesture studies, they have been overlooked for not deemed relevant in the gesture-speech interface. By combining contrastive and multimodal analyses, this paper examines the relevance of holds as potential meaning-making practices deployed by LSFB signers and its comparison to Belgian French speakers. In 3 hours of video-recorded material drawn from 3 multimodal corpora, the following question is addressed: what are the roles of holds in the management of interaction within and across languages/modalities? While most of linguistic work considers manual movements to express referential content, the observations here push to reconsider the common boundary set between what constitutes gestural/linguistic phenomena in one language and what does not.

A method of multimodal machine sign language translation for natural human-computer interaction

This paper aims to investigate the possibility of robustness enhancement as applied to an automatic system for isolated signs and sign languages recognition, through the use of the most informative spatiotemporal visual features. The authors present a method for the automatic recognition of gestural information, based on an integrated neural network model, which analyses spatiotemporal visual features: 2D and 3D distances between the palm and the face; the area of the hand and the face intersection; hand configuration; the gender and the age of signers. A 3DResNet-18-based neural network model for hand configuration data extraction was elaborated. Deepface software platform neural network models were embedded in the method in order to extract gender and age-related data. The proposed method was tested on the data from the multimodal corpus of sign language elements TheRuSLan, with the accuracy of 91.14 %. The results of this investigation not only improve the accuracy and robustness of machine sign language translation, but also enhance the naturalness of human-machine interaction in general. Besides that, the results have application in various fields of social services, medicine, education and robotics, as well as different public service centers.

Understanding Laughter in Dialog

This work explores laughter within a corpus of three-party, task-based dialogs with native and non-native speakers of English, each consisting of two players and a facilitator, in relation to whether the laughter is perceived as serving discourse functions or rather as genuinely mirthful according to a small number of annotators’ (2) inspection of a substantical multimodal dialog corpus (18 interactions of approximately 10 min each). We test the hypothesis that those different types of laughter have occurrence patterns that relate in different ways to the topical structure of the conversations, with discourse laughter showing a stronger tendency to occur at topic termination points. All laughter events (569) are assigned to one of three values, discourse, mirthful or ambiguous, and are studied with respect to their distribution across the dialog topic sections. The analysis explores interactions among laughter type and section type, also with respect to other variables such as the facilitators’ feedback and the speakers’ conversational role and gender. Discourse laughter is more frequent at topic termination points than at topic beginnings, also in comparison to mirthful laughter. Discourse laughter is also highly associated with facilitators’ feedback type, especially at topic ends. Finally, there are few distinctive effects of gender, and an interaction among speaker role and laughter type. The results strengthen the hypothesis of the discourse function of laughter, indicating a systematicity in discourse laughter, in that it is more predictable and highly associated with the dialog topic termination points, and, on the contrary, a less systematic distribution of mirthful laughter, which shows no particular pattern in relation to topic boundaries.

Action request episodes in trauma team interactions in Japan and the UK - A multimodal analysis of joint actions in medical simulation

Grounding is a fundamental human practice for cooperation and collaboration in a joint activity, when more than two people interact. Emergency care is one such interactive situation, and whether a trauma team can efficiently establish and increment their common ground at an appropriate timing during the complex and fluid activity of emergency medical treatment is key to maximise collective competence to best perform as a trauma team. This article investigates recurrent patterns in the grounding process between the trauma team leader and the members, comparing the practices between Japan and the UK, using an eye-tracking device. The embodied practice of grounding was multimodally described, applying both quantitative multimodal corpus analytic and qualitative interactional linguistic approaches. The analysis has shown that five grounding episodes reoccurred, most of which were more ego-centric and one of them ba-centric interactions, drawing on intersubjectivity and the theory of ba in Western and Eastern philosophy respectively.

Construction of Shared Semantic Spaces through Gestures in Interpreter-Mediated Psychotherapy Sessions

Gestures are an inherent part of all face-to-face interactions. Nevertheless, their role in interpreter-mediated events has so far been marginalized in the field of interpreting studies. Based on a multimodal corpus of interactions in child psychiatry with migrant patients, this research focuses on the use of iconic gestures, their role in bridging the linguistic gap and in performing cultural mediation. The results of the study suggest that gestures play a part in the construction of shared semantic spaces and act as a marker of interpreting fidelity, hence, becoming corner stones in the trust-building process between the migrant patients and the public service interpreters.

Construction of a Multimodal Corpus of College Students’ Spoken English Based on Semantic Concepts

With the rapid development of multimedia technology, multimodal corpus came into being, which also opened up a new idea for oral English grammar teaching. In view of the problems existing in the current traditional English grammar textbooks, this paper applies the concepts of “corpus driven” and “multimodality” to grammar textbooks and tries to describe language use in a more authentic, reliable, and up-to-date form. In view of the problems existing in the current traditional English grammar teaching materials, this paper applies the semantic concept to the grammar teaching materials and tries to describe the language use in a more authentic, reliable, and up-to-date form. Based on the semantic concept, this paper studies the construction of college students’ oral English multimodal corpus. After the experiment, the results of the language fluency, accuracy, and complexity of the subjects in the experimental group and the control group are tested. The differences between the two groups were analyzed by one-way ANOVA as follows: in terms of language fluency, F 1 , 62 = 0.835 , P > 0.04 , which showed that there was no significant difference between the two groups after the experiment. In terms of accuracy, F 1 , 62 = 10.693 , P < 0.04 . There was significant difference between the two groups after the experiment. In this study, by analyzing and counting the distribution of conjunctions in students’ oral discourse, the types of conjunctions, their frequency of use, and the types of errors in use, we hope to understand the level and limitations of students’ use of conjunctions in oral discourse, so that students can better apply semantic conceptual cohesion and speak a coherent discourse, First of all, we should help students establish the awareness of textual cohesion and coherence.

Multimodal Behavioral Cues Analysis of the Sense of Presence and Social Presence During a Social Interaction With a Virtual Patient

User's experience evaluation is a key challenge when studying human-agent interaction. Besides user's satisfaction, this question is addressed in virtual reality through the sense ofpresenceandsocial presence, generally assessed thanks to subjective post-experience questionnaires. We propose in this article a novel approach making it possible to evaluate automatically these notions by correlating objective multimodal cues produced by users to their subjective sense of presence and social presence. This study is based on a multimodal human-agent interaction corpus collected in a task-oriented context: a virtual environment aiming at training doctors to break bad news to a patient played by a virtual agent. Based on a corpus study, we applied machine learning approaches to build a model predicting the user's sense of presence and social presence thanks to specific multimodal behavioral cues. We explore different classification algorithms and machine learning techniques (oversampling and clustering) to cope with the dimensionality of the dataset and to optimize the prediction performance. We obtain models to automatically and accurately predict the level of presence and social presence. The results highlight the relevance of a multimodal model, based both on verbal and non-verbal cues as objective measures of (social) presence. The main contribution of the article is two-fold: 1/ proposing the first presence and social prediction presence models offering a way to automatically provide a user's experience evaluation and 2/ showing the importance of multimodal information for describing these notions.

A cross-linguistic comparison of the propositional content of laughter in American English and Central Thai

Abstract Research suggests that laughter can serve several communicative functions beyond indicating mirth, and as such, may hold propositional meaning. The present study analyzes cross-linguistic differences in the propositional content of laughter in American English and Central Thai television shows. A framework for classifying laughter by propositional content was first developed by drawing on existing literature and bottom-up analysis of the laughter found in American English and Thai shows. The framework includes categories of positive valency, negative valency, and humor, along with subcategories of disbelief, support, expressive, and pride. A multi-modal corpus of laughter was then created by compiling all laughter instances in the first 100 min of three American English television shows and three Thai television shows. The meanings of all 848 laughter instances in the corpus were categorized by propositional content of laughter. Results show that humor laughter and negative-support laughter are more frequent in American English, and positive-support laughter and negative-pride laughter are more frequent in Central Thai. These findings provide further evidence that laughter contains propositional content because they indicate that laughter use is subject to cross-linguistic variation that aligns with existing linguistic patterns and cultural values.

Dual Coding or Cognitive Load? Exploring the Effect of Multimodal Input on English as a Foreign Language Learners' Vocabulary Learning.

In the era of eLearning 4.0, many researchers have suggested that multimodal input helps to enhance second language (L2) vocabulary learning. However, previous studies on the effects of multimodal teaching have failed to yield definitive conclusions. Furthermore, only few studies on the multimodal input of vocabulary learning have aimed at junior high school students and have focused on explicit vocabulary instruction in class. To explore the effects of multimodal input on English as a foreign language (EFL) learners’ vocabulary learning and summarize effective methods, this study adopts a mixed-method approach. Based on dual coding theory and cognitive load theory, the teaching materials in this study were designed using the resources provided by the multimodal corpus iWeb and other websites. A total of 60 junior high school students who learned EFL and had a similar English proficiency level were divided into an experimental group (EG) and a control group (CG). Target words were selected through questionnaire I. During the experiment, the CG learned from monomodal materials while the EG received multimodal input, and an immediate post-test was delivered to the two groups. Questionnaire II was distributed in the EG, and five students of the EG were randomly selected for an interview. One week later, a delayed post-test was conducted on the EG and CG. The results showed that the EG performed better in the post-test but did worse than the CG in the delayed post-test. The results of the questionnaire and the interview suggest that students held both positive and negative attitudes toward the multimodal input approach in vocabulary learning. The study concludes with some implications for choosing a multimodal input approach in vocabulary learning, along with a number of suggestions on how to optimize its positive influence and minimize its negative effects.

A Study of Word Complexity Under Conditions of Non-experimental, Natural Overt Speech Production Using ECoG.

The linguistic complexity of words has largely been studied on the behavioral level and in experimental settings. Only little is known about the neural processes underlying it in uninstructed, spontaneous conversations. We built up a multimodal neurolinguistic corpus composed of synchronized audio, video, and electrocorticographic (ECoG) recordings from the fronto-temporo-parietal cortex to address this phenomenon based on uninstructed, spontaneous speech production. We performed extensive linguistic annotations of the language material and calculated word complexity using several numeric parameters. We orthogonalized the parameters with the help of a linear regression model. Then, we correlated the spectral components of neural activity with the individual linguistic parameters and with the residuals of the linear regression model, and compared the results. The proportional relation between the number of consonants and vowels, which was the most informative parameter with regard to the neural representation of word complexity, showed effects in two areas: the frontal one was at the junction of the premotor cortex, the prefrontal cortex, and Brodmann area 44. The postcentral one lay directly above the lateral sulcus and comprised the ventral central sulcus, the parietal operculum and the adjacent inferior parietal cortex. Beyond the physiological findings summarized here, our methods may be useful for those interested in ways of studying neural effects related to natural language production and in surmounting the intrinsic problem of collinearity between multiple features of spontaneously spoken material.

Basic Research on Construction of Multimodal Parallel Corpus of Tourism Translation in New Media Era

Parallel corpus are bilingual or multilingual corpus composed of the original text and its parallel counterparts in the target language. Bilingual parallel corpushas the closest relationship with translation, providing rich translated texts and convenient translation means. The new media era has put forward new requirements for tourism translation. As the number of foreign tourists coming to Liaoning increases year by year, the problems in tourism translation in Liaoning become more and more obvious. The basic research on Liaoning tourism translation multimodal parallel corpus in the new media era, including theoretical basis, technical support, development trend and standard formulation, will provide basic support for the construction of multimodal corpus and realize the integration of various media forms, such as words, images, voice and video, which can fully mobilize users' multiple sensory systems and provide comprehensive tourism translation services.

Design and release of a multimodal dialogue corpus

Design and release of a multimodal dialogue corpus

A database of North American double modals and self-repairs from YouTube

AbstractSequences of two modal verbs in spoken English can represent use of a nonstandard syntactic feature (double modal) or a corrected utterance in which a speaker begins with one modal auxiliary, but switches to another (self-repair). This article presents the Double Modals and Self-Repairs (DMSR) database, a table of naturalistic double modals and self-repairs in videos from local government entities in North America, created from theCorpus of North American Spoken English(CoNASE). The paper describes the procedures used for the database’s creation, discusses potential uses, and presents an exploratory analysis in which a logistic regression classifier is trained withCoNASEdata to distinguish authentic double modals from self-repair sequences on the basis of local discourse context. The analysis demonstrates how large corpora of speech can be used to investigate the links between syntactic and pragmatic phenomena and shows specifically that double modals are an interactive device, while two-modal sequences as self-repairs may be the result of high cognitive load. The paper concludes with a discussion of multimodal corpus creation from YouTube for the study of lexical, syntactic, and interactional phenomena in speech as well as for the analysis of complex, multilevel computer-mediated communication (CMC) phenomena.

Cross-modal Translation: A Study of English Subtitle Translation from the Perspective of Multimodal Interaction

Under the guidance of the Systemic-functional synthetic framework for multimodal discourse analysis, this paper collects multimodal corpus from the classical American TV show <i>Friends</i> and undertakes a thorough qualitative analysis of how cross-modal relations influence the Chinese translation of English subtitles. As is shown, in order to give Chinese viewers a better understanding of English subtitles in multimodal contexts, different translational approaches should be used for the translation of subtitles that are in different cross-modal relations, which are categorized into two general types, that is, complementary relation on the one hand and non-complementary relation on the other hand. The qualitative analysis in this paper finds that though strategies of translation that are appropriate to make use of in the process of English translation into Chinese include amplification, paraphrasing, annotation, direct translation, free translation, and linear translation, the choice of translation approach is directly influenced by cross-modal relations. For example, for subtitles that belong to multimodal discourse of reinforcing complementary relation, it is advisable for translators to make use of translational strategies of amplification, paraphrasing and annotation to highlight the foregrounded meaning expressed by the dominant modality on the one hand, and on the other hand to explicitly and verbally express the background information provided by the backgrounded modality if discourses of two modalities in the same scene form a relation of highlighting. As cross-modal communication becomes continuously widespread and the popularity of English shows keep arising in China, this paper endeavors to provide some useful implications for the translation of English subtitles, which as verbal and auditory discourse is part of a larger multimodal discourse in films.

Signaling irony, displaying politeness, replacing words

Abstract This paper aims to identify and define the different linguistic functions that emoji can carry in French and English computer-mediated discourse (CMD). Emoji are graphical icons, representing attitudes or concepts, that can be used on any smartphone or computer with a variety of communicative roles inside a conversation. Using a multimodal corpus containing 1200 emoji and the messages in which they were used, we tested an initial typology of emoji functions inspired by previous frameworks. The quantitative analysis of the data, along with the detailed examination of specific examples, led to the construction of a new, more refined typology consisting of three primary functions (expressive, interpretative and referential) and five secondary functions (relational, politeness, emphatic, structural and aesthetic).

From X to Y: Anatomy of a Constructional Pattern

Compositionality is undoubtedly one of the hardest problems in linguistics. In decoding theories, the speaker occupies a leading role, having to carefully choose the form that better encodes the meaning to be communicated. In contrast, in inferential theories, the burden is shifted from speaker to hearer: linguistic information typically underspecifies meaning and the hearer must make a number of inferences to bridge the gap between what is said and what is meant. In this article, I argue that constructional meaning can aid the process of sentence meaning formation by providing a scaffold that can help the hearer with the construal operations. Constructions, by providing an additional layer of meaning, constrain the range of possible meanings activated by words thereby reducing the combinatorial explosion when several words are joined together. This process is examined here by analysing the meanings associated with the grammatical construction [from X to Y], which is connected to a polysemy network of related senses, using examples extracted from a multimodal corpus. A preliminary analysis of the gesturing behaviour associated with the different senses proposed is also included, which can be seen to contribute to the characterisation of the different senses of the polysemy network.

Engaging students in multimodal literacy practices in a university ESP context: towards understanding identity and ideology in government debates

Abstract This paper focuses on issues of multimodal literacy practices in ESP higher education settings. In particular, the research explores how students become engaged in various literacy activities aimed at enhancing their critical-thinking skills and interpretation of images. For this purpose, two datasets consisting of video clips were extracted from a larger multimodal corpus and developed for teaching applications: one involved a UK live parliament debate and the other a US House of Representatives debate. The main objective is to identify the key verbal strategies reflecting persuasive, argumentative rhetoric and the non-verbal features accompanying these verbal utterances such as prosodic stress, body/head movements, gaze, gesture. Thus, the focus of the analysis is on how different semiotic modes of communication construct meaning, especially in terms of how they reinforce the construction of identity and ideological stance. The results were systematically categorized and applied on a practical level to a teaching unit on ‘identity and ideology’.