Multimodal Content Research Articles

LiVLR: A Lightweight Visual-Linguistic Reasoning Framework for Video Question Answering

Video Question Answering (VideoQA), aiming to correctly answer a given question based on understanding multimodal video content, is challenging due to the richness of the video content. From the perspective of video understanding, a complete VideoQA framework needs to understand the video content at different semantic levels and flexibly integrate diverse video content to distill question-related content. To this end, we propose a Lightweight Visual-Linguistic Reasoning framework named <inline-formula><tex-math notation="LaTeX">$\text{LiVLR}$</tex-math></inline-formula>. Specifically, <inline-formula><tex-math notation="LaTeX">$\text{LiVLR}$</tex-math></inline-formula> first utilizes graph-based visual and linguistic encoders to obtain multi-grained visual and linguistic representations, respectively. Subsequently, the obtained representations are integrated with the devised Diversity-aware Visual-Linguistic Reasoning module (<inline-formula><tex-math notation="LaTeX">$\text{DaVL}$</tex-math></inline-formula>). <inline-formula><tex-math notation="LaTeX">$\text{DaVL}$</tex-math></inline-formula> distinguishes different types of representations with the learnable index embedding in graph embedding. Therefore, <inline-formula><tex-math notation="LaTeX">$\text{DaVL}$</tex-math></inline-formula> can flexibly adjust the importance of different representations when generating the question-related joint representation. The proposed <inline-formula><tex-math notation="LaTeX">$\text{LiVLR}$</tex-math></inline-formula> is lightweight and shows its performance advantage on three VideoQA benchmarks, MRSVTT-QA, KnowIT VQA, and TVQA. Extensive ablation studies demonstrate the effectiveness of the key components of <inline-formula><tex-math notation="LaTeX">$\text{LiVLR}$</tex-math></inline-formula>.

Multimodal Topic Modeling by Exploring Characteristics of Short Text Social Media

Millions of people post images and texts to express their feelings and point of views on social media everyday, especially on the short text social media such as Twitter or Weibo. As the images can provide important supplementary information for the text, many multimodal topic models have been developed to mine the topics from the multimodal social media content. We summarize three fundamental characteristics of the short text multimodal social media. The first is that the text of a short social media document generally belong to only one topic. The second is that the attached images can be relevant to multiple topics due to the rich information expressed in the images. The last is that although in most cases, text and images in social media posts are relevant, it should be noted that in a small number of cases, text and pictures are not relevant. However, most of the current multimodal topic models fail to model the these characteristics, and thus may produce low-quality topics. Based on these characteristics, we propose an unsupervised multimodal topic model SMMTM to model the short text multimodal social media documents. In the SMMTM model, only one topic is sampled for the the text while an image can belong to different topics. The correlation of the topics between the text and the images in a document are also formulated in an appropriate way. The experiments on three short text social media datasets with four evaluation metrics show the advantages of our model over the existing models.

End-to-End Multimodal COVID-19 Content Quantitative Safety Detection Algorithm

This paper takes the multimodal COVID-19 related dataset as the research object and carries out research on the content safety monitoring algorithm in the field of COVID-19. It is proposed to carry out research work in the following two aspects. The first is end-to-end topic and sentiment modeling. Turning topic modeling and sentiment analysis into multi-label classification problems essentially unifies topic modeling and sentiment analysis, with the model directly outputting the predicted sample's topic and sentiment. The second is to quantify sentiment analysis results. While modeling sentiment polarity, it also models sentiment intensity. By analyzing public opinion through sentiment polarity and judging people's attention to a topic and the degree of speech safety based on sentiment intensity values, we propose an end-to-end multimodal COVID-19 content quantitative safety detection algorithm (EMQD). The EMQD algorithm is tested on the KJZY2020 dataset and compared with other algorithms.

Inter-Intra Modal Representation Augmentation With DCT-Transformer Adversarial Network for Image-Text Matching

Image-text matching has become a challenging task in the multimedia analysis field. Many advanced methods have been used to explore local and global cross-modal correspondence in matching. However, most methods ignore the importance of eliminating potential irrelevant features in the original features of each modality and cross-modal common feature. Moreover, the features extracted from regions in images and words in sentences contain cluttered background noise and different occlusion noise, which negatively affects alignment. Different from these methods, we propose a novel DCT-Transformer Adversarial Network (DTAN) for image-text matching in this paper. This work can obtain an effective metric based on two aspects: i) DCT-Transformer uses DCT (Discrete Cosine Transform) method based on a transformer mechanism to extract multi-domain common representations and eliminate irrelevant features from different modalities (inter-modal). Among them, DCT divides multi-modal content into chunks of different frequencies and quantifies them. ii) The adversarial network introduces an adversary idea by combining the original features of various single modalities and the multi-domain common representation, alleviating the background noise within each modality (intra-modal). The proposed adversarial feature augmentation method can easily obtain the common representation that is only useful for alignment. Extensive experiments are completed on the benchmark datasets Flickr30K and MS-COCO, demonstrating the superiority of the DTAN model over the state-of-the-art methods.

Examining the Multimodal Design of Explainer Videos: A Multimodal Content Analysis of Khan Academy Online Resources

Examining the Multimodal Design of Explainer Videos: A Multimodal Content Analysis of Khan Academy Online Resources

Translation of multilingual films in Iran in Persian dubbing

ABSTRACT Filmmakers increasingly resort to using multiple languages in their work to realistically reflect today's globalised world. However, this multiplicity poses specific challenges in the process of translation for dubbing. This study explores the rendition of Western multilingual films into Persian dubbed versions for the Iranian audience. Films as audiovisual texts have multimodal content, so both verbal and non-verbal elements were analysed. We compared the original versions of a selection of ten multilingual films with the Persian dubbed versions based on two models of analyses [i.e. Sanz Ortega, E. (2011). Subtitling and the relevance of Non-verbal information in polyglot films. New Voices in Translation Studies, 7, 19–34; Zabalbeascoa, P., & Voellmer, E. (2014). Accounting for multilingual films in translation studies. Intratextual translation in dubbing. In D. Abend-David (Ed.), Media and translation. An interdisciplinary approach (pp. 25–51). Bloomsbury Academic]. The results reveal that the most frequently used solution for translating multilingual films at the verbal level is neutralisation, which eliminates linguistic variation, and consequently, the multiplicity of languages in multilingual films is not maintained. Incidentally, a vast range of non-verbal signs is altered to conform to the socio-cultural norms of the target culture.

Multimodal false information detection method based on Text-CNN and SE module.

False information detection can detect false information in social media and reduce its negative impact on society. With the development of multimedia, the multimodal content contained in false information is increasing, so it is important to use multimodal features to detect false information. This paper mainly uses information from two modalities, text and image. The features extracted by the backbone network are not further processed in the previous work, and the problems of noise and information loss in the process of fusing multimodal features are ignored. This paper proposes a false information detection method based on Text-CNN and SE modules. We use Text-CNN to process the text and image features extracted by BERT and Swin-transformer to enhance the quality of the features. In addition, we use the modified SE module to fuse text and image features and reduce the noise in the fusion process. Meanwhile, we draw on the idea of residual networks to reduce information loss in the fusion process by concatenating the original features with the fused features. Our model improves accuracy by 6.5% and 2.0% on the Weibo dataset and Twitter dataset compared to the attention based multimodal factorized bilinear pooling. The comparative experimental results show that the proposed model can improve the accuracy of false information detection. The results of ablation experiments further demonstrate the effectiveness of each module in our model.

Huế dental students' use and perception of an online dental learning platform: A pilot study.

Online educational materials are growing in use, and dental students worldwide can benefit from higher quality and more accessible online supplemental resources. This study was created to evaluate the learning resources non-English speaking dental students desire and to pilot My Dental Key (MDK), an English, evidence-based, online dental educational platform. Third to sixth year dental students at the Huế University of Medicine and Pharmacy were asked to pilot MDK over a 5-week period and were invited to answer three surveys throughout the study. A preliminary survey was given to gauge the participants' (n = 209) preferences regarding the use of English-based dental educational resources. Participants (n = 58) completed a presurvey prior to accessing MDK. After the 5-week period, participants (n = 38) were given a postsurvey to evaluate the platform's effectiveness as a supplemental educational resource. Overall, we found that: (1) students desire credible online supplemental resources in addition to current resources provided by their school, (2) the multimodal content that MDK provides is a strength that bridges language barriers (3) participants perceived that the content on MDK would help them in class and when treating patients. Improving the quality of online supplemental dental resources will have the capability to progress the current educational landscape, and further resources should be created to best serve the global dental community.

CrowdGraph: A Crowdsourcing Multi-modal Knowledge Graph Approach to Explainable Fauxtography Detection

Human-centric fauxtography is a category of multi-modal posts that spread misleading information on online information distribution and sharing platforms such as online social media. The reason of a human-centric post being fauxtography is closely related to its multi-modal content that consists of diversified human and non-human subjects with complex and implicit relationships. In this paper, we focus on an explainable fauxtography detection problem where the goal is to accurately identify and explain why a human-centric social media post is fauxtography (or not). Our problem is motivated by the limitations of current fauxtography detection solutions that focus primarily on the detection task but ignore the important aspect of explaining their results (e.g., why a certain component of the post delivers the misinformation). Two important challenges exist in solving our problem: 1) it is difficult to capture the implicit relations and attributions of different subjects in a fauxtography post given the fact that many of such knowledge is shared between different crowd workers; 2) it is not a trivial task to create a multi-modal knowledge graph from crowd workers to identify and explain human-centric fauxtography posts with multi-modal contents. To address the above challenges, we develop CrowdGraph, a crowdsourcing based multi-modal knowledge graph approach to address the explainable fauxtography detection problem. We evaluate the performance of CrowdGraph by creating a real-world dataset that consists of human-centric fauxtography posts from Twitter and Reddit. The results show that CrowdGraph not only detects the fauxtography posts more accurately than the state-of-the-arts but also provides well-justified explanations to the detection results with convincing evidence.

Identity performance and self-branding in social commerce: A multimodal content analysis of Chinese wanghong women’s video-sharing practice on TikTok

Social media have facilitated the development of a new wanghong profession and the burgeoning social commerce in China, where young women capitalize on their femininity to promote beauty products. Against this background, this study aims to investigate how Chinese wanghong women craft their identities in their self-branding videos on the leading social commerce platform of TikTok. A framework is developed to model their projected identities as evaluative attributes and to elucidate how the identities are constructed using verbal and visual resources. A multimodal content analysis of a corpus of 1258 videos posted by 6 top-ranked wanghong women on TikTok shows that their identities are defined in terms of three marketable attributes: (1) the celebrity self, in which they highlight their glamorous appearance, aspirational lifestyle, and social responsibility, (2) the entrepreneur self, which includes their accentuation of their professionalism and self-empowerment, and (3) the ordinary woman self, in which they emphasize their intimate relationship with their consumer-audience on the one hand, and construct a cheerful and amiable self-image on the other. The multi-faceted identities shed new light on the evolving Chinese femininity shaped by the entangled forces of the development of the neoliberal social commerce and wanghong economy in China on the one hand, and the unique socio-political context on the other hand.

Multimodal Sentiment Sensing and Emotion Recognition Based on Cognitive Computing Using Hidden Markov Model with Extreme Learning Machine

In today's competitive business environment, exponential increase of multimodal content results in a massive amount of shapeless data. Big data that is unstructured has no specific format or organisation and can take any form, including text, audio, photos, and video. Many assumptions and algorithms are generally required to recognize different emotions as per literature survey, and the main focus for emotion recognition is based on single modality, such as voice, facial expression and bio signals. This paper proposed the novel technique in multimodal sentiment sensing with emotion recognition using artificial intelligence technique. Here the audio and visual data has been collected based on social media review and classified using hidden Markov model based extreme learning machine (HMM_ExLM). The features are trained using this method. Simultaneously, these speech emotional traits are suitably maximised. The strategy of splitting areas is employed in the research for expression photographs and various weights are provided to each area to extract information. Speech as well as facial expression data are then merged using decision level fusion and speech properties of each expression in region of face are utilized to categorize. Findings of experiments show that combining features of speech and expression boosts effect greatly when compared to using either speech or expression alone. In terms of accuracy, recall, precision, and optimization level, a parametric comparison was made.

A novel label-based multimodal topic model for social media analysis

Extracting useful knowledge from multimodal data is the core of many multimedia applications, such as recommendation systems, and cross-modal retrieval. In this paper, we propose a label-based multimodal topic (LB-MMT) model to jointly model text and image data tagged with multiple labels. Specifically, we use the labels as supervised information to generate the text and image data. In the LB-MMT model, we assume that the textual words and visual words related to each text and image are drawn from a mixture of latent topics, where each topic is represented as a group of textual words and visual words. Moreover, we introduce multiple topics for each label, to build the top-down relationship from label to text and image. To investigate the effectiveness of the proposed approach, we conduct extensive experiments on a real-world multimodal dataset with labels. The results show the proposed approach obtains superior performances on topic coherence and label prediction compared with previous competitors. In addition, we show that our model yields interesting insights about multimodal topics. The proposed model provides important practical implications, e.g., designing more attractive multimodal contents for marketers.

Fusing visual and textual content for knowledge graph embedding via dual-track model

Large-scale knowledge graphs are usually incomplete. Knowledge graph embedding has achieved encouraging performance in alleviating the incompleteness of knowledge graphs. There are approaches to leverage the multi-modal content, such as text description and images, to improve the performance of knowledge graph embedding. However,due to the heterogeneity across different modalities, current methods are not effective to fuse the multi-modal content and network structure information to learn the embedding. In this work, a dual-track model DuMF for knowledge graph embedding enhancement is proposed. The model includes two tracks to fuse multi-modal content and network structure respectively. In each track, the expressiveness of joint features is improved by the bilinear method, and meanwhile the task-specific important features are learned by deliberate attention. Finally, the fused features are generated by the gating network. To extensively evaluate the model, two challenging datasets are enriched with additional multi-modal data. Experimental results show that DuMF is superior to baselines in link prediction. The flexibility of the model is promising for the further improvement of model performance.

The use of the physical laboratory of computer networks as a learning tool

ABSTRACT This study contributes to learning improvement in practical classes in Computer Network technology courses, using the Physical Technological Laboratory (PTL) as a tool. Multimodal narration content analysis was used, which aggregates and organises the data collected in the PTL environment. Based on the results, we infer that both the student and the teacher use the physical laboratory as a tool since the detected physical interactions prove its use and reuse. Evidence of causality between teacher epistemic movements and learning in terms of physical interactions, epistemic practices, and student autonomy was also noted. Contributions were: (1) In the context of work in networks PTL, the variety and quality of epistemic practices of students are enhanced if there is autonomous work concomitant with the physical interaction of students with the respective artifacts. (2) Teacher action can better promote epistemic practices, stretching beyond direct action if there is an ‘orchestration’ of teacher mediation patterns.

Multimodal Event Processing: A Neural-Symbolic Paradigm for the Internet of Multimedia Things

With the Internet of Multimedia Things (IoMT) becoming a reality, new approaches are needed to process real-time multimodal event streams. Existing approaches to event processing have limited consideration for the challenges of multimodal events, including the need for complex content extraction, and increased computational and memory costs. This article explores event processing as a basis for processing real-time IoMT data. This article introduces the multimodal event processing (MEP) paradigm, which provides a formal basis for native approaches to neural multimodal content analysis (i.e., computer vision, linguistics, and audio) with symbolic event processing rules to support real-time queries over multimodal data streams using the multimodal event processing language to express single, primitive multimodal, and complex multimodal event patterns. The content of multimodal streams is represented using multimodal event knowledge graphs to capture the semantic, spatial, and temporal content of the multimodal streams. The approach is implemented and evaluated within a MEP engine using single and multimodal queries achieving near real-time performance with a throughput of ~30 frames processed per second (fps) and subsecond latency of 0.075–0.30 s for video streams of 30 fps input rate. Support for high input stream rates (45 fps) is achieved through content-aware load-shedding techniques with a ~127X latency improvement resulting in only a minor decrease in accuracy.

#Mixedcouples on TikTok: Performative Hybridization and Identity in the Face of Discrimination

Spanish-Moroccan mixed couples exemplify a case of cultural hybridization that reflects the colonial past between these two countries. In this context, social media is a space of opportunity and risk to face discrimination and othering. In this article, we draw upon postcolonial theory and Internet studies to consider TikTok affordances as tools that are relevant to understand how cultural, national, and religious identities are shaped and presented in digital spaces. In this sense, this study constitutes the first exploratory analysis of Moroccan-Spanish mixed couples’ (self)representations and identity construction as reflected by their TikTok content. The memetic and intersectional aspects of the mixed-couple identity emerge as central in the convergent nature of the platform and their lived experiences. Through an initial analysis of 8,653 TikTok videos, we identified 6 creator accounts with more than 10K followers to conduct a deeper multimodal content and discourse analysis of 146 videos. This enabled us to explore how these creators portray their hybrid identity through short, vertical videos harnessing TikTok’s vernaculars and affordances. The results enable us to propose the concept of performative hybridization, which involves discursive markers within these couples’ TikTok content that reflect a fluid integration of two cultures, while the pervading visual and narrative components reflect the dominance of one set of identity traits over the other. In this case, Moroccan culture is more prevalent.

MAGIC: Multimodal relAtional Graph adversarIal inferenCe for Diverse and Unpaired Text-Based Image Captioning

Text-based image captioning (TextCap) requires simultaneous comprehension of visual content and reading the text of images to generate a natural language description. Although a task can teach machines to understand the complex human environment further given that text is omnipresent in our daily surroundings, it poses additional challenges in normal captioning. A text-based image intuitively contains abundant and complex multimodal relational content, that is, image details can be described diversely from multiview rather than a single caption. Certainly, we can introduce additional paired training data to show the diversity of images' descriptions, this process is labor-intensive and time-consuming for TextCap pair annotations with extra texts. Based on the insight mentioned above, we investigate how to generate diverse captions that focus on different image parts using an unpaired training paradigm. We propose the Multimodal relAtional Graph adversarIal InferenCe (MAGIC) framework for diverse and unpaired TextCap. This framework can adaptively construct multiple multimodal relational graphs of images and model complex relationships among graphs to represent descriptive diversity. Moreover, a cascaded generative adversarial network is developed from modeled graphs to infer the unpaired caption generation in image–sentence feature alignment and linguistic coherence levels. We validate the effectiveness of MAGIC in generating diverse captions from different relational information items of an image. Experimental results show that MAGIC can generate very promising outcomes without using any image–caption training pairs.

Multimodal speech-gesture training in patients with schizophrenia spectrum disorder: Effects on quality of life and neural processing

Dysfunctional social communication is one of the most stable characteristics in patients with schizophrenia spectrum disorder (SSD) that severely affects quality of life. Interpreting abstract speech and integrating nonverbal information is particularly affected.Considering the difficulty to treat communication dysfunctions with usual intervention, we investigated the possibility to apply a multimodal speech-gesture (MSG) training.In the MSG training, we offered 8 sessions (60 min each) including perceptive and expressive tasks as well as meta-learning elements and transfer exercises to 29 patients with SSD. In a within-group crossover design, patients were randomized to a TAU-first (treatment as usual first, then MSG training) group (N = 20) or a MSG-first (MSG training first, then TAU only) group (N = 9), and were compared to healthy controls (N = 17). Outcomes were quality of life and related changes in the neural processing of abstract speech-gesture information, which were measured pre-post training through standardized psychological questionnaires and functional Magnetic Resonance Imaging, respectively.Pre-training, patients showed reduced quality of life as compared to controls but improved significantly during the training. Strikingly, this improvement was correlated with neural activation changes in the middle temporal gyrus for the processing of abstract multimodal content. Improvement during training, self-report measures and ratings of relatives confirmed the MSG-related changes.Together, we provide first promising results of a novel multimodal speech-gesture training for patients with schizophrenia. We could link training induced changes in speech-gesture processing to changes in quality of life, demonstrating the relevance of intact communication skills and gesture processing for well-being.

Intelligent Video Tag Recommendation System Improving Video Popularity

A successful marketing activity requires attracting high social popularity to their contents, since higher popularity usually indicates stronger influence, more fame and higher revenue. In this, we focus on the question of how to improve popularity of videos sharing on websites like YouTube in mobile computing environment. Obviously, composing high quality titles and tags is beneficial for viewers to discover videos of their interests and increase their tendency to watch more videos. However, it is not an easy task for uploaders,which is especially true since the screen is tight for most mobile devices. To this end, this proposes a novel hybrid method based on multi-modal content analysis that recommends keywords for video uploaders to compose titles and tags of their videos and then to gain higher popularity. The method generates candidate keywords by integrating techniques of textual semantic analysis of original tags and recognition of video content. On one hand, taking the original keywords of a video as input, the method obtains most relevant words from WordNet and related video titles gathered from the three top video sharing sites (YouTube, Yahoo Video, Bing Video). On the other hand, through recognizing video content with deep learning technology,the method extracts the entity name of video content as candidate keywords. Finally, a TFSIM algorithm is proposed to rank the candidate keywords and the most relevant keywords are recommended to uploaders for optimizing the titles and tags of their videos

Gesture, Music and Computer: The Centro di Sonologia Computazionale at Padova University, a 50-Year History.

With the advent of digital technologies, the computer has become a generalized tool for music production. Music can be seen as a creative form of human–human communication via a computer, and therefore, research on human–computer and computer–human interfaces is very important. This paper, for the Sensors Special Issue on 800 Years of Research at Padova University, presents a review of the research in the field of music technologies at Padova University by the Centro di Sonologia Computazionale (CSC), focusing on scientific, technological and musical aspects of interaction between musician and computer and between computer and audience. We discuss input devices for detecting information from gestures or audio signals and rendering systems for audience and user engagement. Moreover, we discuss a multilevel conceptual framework, which allows multimodal expressive content processing and coordination, which is important in art and music. Several paradigmatic musical works that stated new lines of both musical and scientific research are then presented in detail. The preservation of this heritage presents problems very different from those posed by traditional artworks. CSC is actively engaged in proposing new paradigms for the preservation of digital art.