Multimodal Posts Research Articles

End-to-end multi-perspective multimodal posts relevance score reasoning prediction

Driven by the surge in students sharing insights on online platforms, evaluating discussion forum posts has emerged as a significant research focus. This assessment is crucial for both students and educators, as it enables teachers to efficiently gauge student comprehension and helps students tailor their learning approaches to suit their individual needs, thereby fostering personalized growth. However, existing research on post-quality assessment faces challenges: 1) They overlook multimodal information interaction between post and topic, leading to inaccurate evaluations. 2) They are usually classification tasks that fail to provide feedback on the minor and relative differences between posts on the same topic. 3) They are usually evaluated using a single perspective, which is insufficient for capturing the complexity of the relationship between post and topic. Based on the above challenges, we propose a new assessment task, Multimodal Topic-Post Relevance Score Prediction (MTRSP), which analyzes whether a student's post comprehensively answers the question of the discussion topic by combining text and images to predict topic-post relevance scores, i.e., the degree of correctness of the answer. We develop an end-to-end Multi-perspective Topic-Post Relevance Score Reasoning (MTRSR) Method to solve the MTRSP Task, which leverages images and text from both the post and topic to infer topic-post relevance scores based on semantic similarity and logical coherence. Specifically, the topic-post content relevance reasoning module uses multimodal fusion to learn the semantic similarity of posts and topics. The logical coherence inference module examines the logical connections between posts and topics. Finally, we use three newly collected multimodal topic-post datasets and the public dataset Lazada-Home as an evaluation benchmark for the MTRSP task. Experimental results show that our MTRSR method can bring up to 9.02% in the NDCG@3 (Normalized Discounted Cumulative Gain) metric compared to the best-performing text-only model. The source code and dataset will be made public.

Multimodal online dissident culture in Instagram: A critique of the Turkish economy

Ever-present mass surveillance has blocked the flourishing of a traditional dissident culture in Turkey. Focusing on popular ‘just for fun’ Instagram accounts during the lira's freefall that began in the autumn of 2021, this study seeks to identify the creative strategies for digital social resistance embedded in multimodal content sharing of posts, which are composed of visuals, text, and sound. For this, we employed a multimodal-type analysis of Instagram posts regarding Turkey's economic crisis, followed by an interpretative content analysis aiming to (1) identify, categorize, and compile a typology of the main countersurveillance strategies inherent in multimodal posts, such as memes, edited videos, and animations, Photoshop-crafted still images, and (2) explore the contextual traits of the connected dissident culture. We discuss how these multimodal-type posts support connected dissident group formation while maintaining confidentiality while criticizing governmental conduct of economic policy making in Turkey.

Fueling divisions: a multimodal analysis of Canadian petro-nationalism in the social media discourse of “oil sands strong”

ABSTRACT This research conducts a multimodal critical discourse analysis of social media posts by Oil Sands Strong (OSS), examining their role in advocating for resource extraction through the use of “petro-nationalism” discourse. The analysis uncovers three key aspects: First, the posts employ multimodal semiotic resources to create dichotomies and blend nationalist and xenophobic discourses with energy discourse. Second, they integrate diverse multimodal symbolic elements from various discursive frameworks to forge a collective identity for the local Canadian community. Lastly, the posts stigmatize foreign oil producers, new energy industries, and environmentalists, employing multimodal symbols to obfuscate the argument's focal point and generate a hybrid discourse. Through visual symbols, assertive text, and argumentation of topoi, the posts present an exaggerated and distorted image of petro-nationalism. The study concludes that these multimodal posts aim to reshape Canadian energy politics, promote biased sentiments, exploit cultural prejudices, and formulate petro-nationalism narratives for resource extraction advocacy.

Planting in the pandemic: surveillance on social media

ABSTRACT This article looks at interest-driven and informal social media practices that have flourished in the pandemic period and its ensuing renaissance of domesticity. It investigates how tending plants and discussing them on social media serve as a particular site for connecting around loving and taking care of plants. Its focus is on the discursive means with which posters – guided by social media algorithms – rhetorically co-construct a morally acceptable version of a pandemic lifestyle around houseplants. More specifically, drawing on multimodal discourse studies, critical sociolinguistics and work on digital surveillance, it investigates how members of a Finland-based social media site observe and monitor themselves and others via their linguistically heterogeneous and multimodal posts. The paper demonstrates how constructions of tending plants highlight a normative subject who besides cultivating plants also cultivates themselves and others in the allegedly safe microcosm of the home, surrounded by the risk-ridden, tension-full, dangerous pandemic world. In the same way, as in many other types of informal and interest-driven social media activities, surveillance forms a crucial part of the routine digital activities and interactions about and around plants. Three manifestations of surveillance are discussed in detail: site-specific panoptic surveillance, peer surveillance and self-surveillance.

What and Why? Towards Duo Explainable Fauxtography Detection Under Constrained Supervision

Fauxtography is a category of multi-modal posts that spread misleading information on various big data online social platforms that generate billions of posts on a daily basis (e.g., Facebook, Twitter, Reddit). A fauxtography post usually consists of an image, a text description and comments from its readers. In this paper, we focus on explaining fauxtography posts by identifying <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">what</i> specific component and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">why</i> that component of a post leads to the fauxtography (i.e., duo explanations). This problem is motivated by the limitations of current fauxtography detection solutions that only focus on the detection but ignore the important <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">explanation</i> aspect of their results. Two critical challenges exist in solving our problem: i) it is difficult to accurately identify the “guilty” component of a fauxtography post given the fact that different components of the post and their associations could all lead to the fauxtography; ii) it is expensive and time-consuming to obtain a good training set with fine-grained labels of fauxtography posts in terms of explainability, making it challenging to develop fully supervised explainable solutions. To address the above challenges, we develop a <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</b> uo <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Ex</b> plainable <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</b> auxtography Detection Framework under a <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</b> onstrained Supervision (DExFC) to generate duo explanations from both content and comment parts of fauxtography posts. We evaluate the DExFC by creating real-world datasets from different online social media platforms (Twitter and Reddit). The results show that DExFC not only detects the fauxtography posts more accurately than the state-of-the-art solutions but also provides well-justified explanations to its results without the full supervision.

Enhanced Multimodal Fake News Detection with Optimal Feature Fusion and Modified Bi-LSTM Architecture

Numerous enhancements have been made to the mobile internet, which leads to an increase the people’s attention to posting more multi-modal posts among the social media platforms. Hence, this paper aims to design a multimodal fake news detection model with enhanced deep learning architecture. Initially, the multi-modal information including images and text are gathered from real-time social media. Then, pre-processing of both images and text is carried out. Further, the text features are extracted using Word2vector and glove embedding techniques. Then, the optimal features from both image features and text features are attained using the Adaptive Water Strider Algorithm (A-WSA). The achieved optimal features are forwarded to the new feature fusion concept based on the weight factor that is optimized using the same A-WSA. Finally, the fused features are forwarded to the fake news classification stage with the help of “Optimized-Bidirectional Long Short-Term Memory (O-BiLSTM),” where the hyperparameter of BiLSTM is optimized through similar A-WSA. Throughout the result analysis, the accuracy rate of the designed A-WSA-BiLSTM method is attained at 96.51%. The experimental result demonstrates that the proposed model is effective in using multi-modal data in automated fake news classification.

CrowdGraph: A Crowdsourcing Multi-modal Knowledge Graph Approach to Explainable Fauxtography Detection

Human-centric fauxtography is a category of multi-modal posts that spread misleading information on online information distribution and sharing platforms such as online social media. The reason of a human-centric post being fauxtography is closely related to its multi-modal content that consists of diversified human and non-human subjects with complex and implicit relationships. In this paper, we focus on an explainable fauxtography detection problem where the goal is to accurately identify and explain why a human-centric social media post is fauxtography (or not). Our problem is motivated by the limitations of current fauxtography detection solutions that focus primarily on the detection task but ignore the important aspect of explaining their results (e.g., why a certain component of the post delivers the misinformation). Two important challenges exist in solving our problem: 1) it is difficult to capture the implicit relations and attributions of different subjects in a fauxtography post given the fact that many of such knowledge is shared between different crowd workers; 2) it is not a trivial task to create a multi-modal knowledge graph from crowd workers to identify and explain human-centric fauxtography posts with multi-modal contents. To address the above challenges, we develop CrowdGraph, a crowdsourcing based multi-modal knowledge graph approach to address the explainable fauxtography detection problem. We evaluate the performance of CrowdGraph by creating a real-world dataset that consists of human-centric fauxtography posts from Twitter and Reddit. The results show that CrowdGraph not only detects the fauxtography posts more accurately than the state-of-the-arts but also provides well-justified explanations to the detection results with convincing evidence.

Global Local Fusion Neural Network for Multimodal Sentiment Analysis

With the popularity of social networking services, people are increasingly inclined to share their opinions and feelings on social networks, leading to the rapid increase in multimodal posts on various platforms. Therefore, multimodal sentiment analysis has become a crucial research field for exploring users’ emotions. The complex and complementary interactions between images and text greatly heighten the difficulty of sentiment analysis. Previous works conducted rough fusion operations and ignored the study for fine fusion features for the sentiment task, which did not obtain sufficient interactive information for sentiment analysis. This paper proposes a global local fusion neural network (GLFN), which comprehensively considers global and local fusion features, aggregating these features to analyze user sentiment. The model first extracts overall fusion features by attention modules as modality-based global features. Then, coarse-to-fine fusion learning is applied to build local fusion features effectively. Specifically, the cross-modal module is used for rough fusion, and fine-grained fusion is applied to capture the interaction information between objects and words. Finally, we integrate all features to achieve a more reliable prediction. Extensive experimental results, comparisons, and visualization of public datasets demonstrate the effectiveness of the proposed model for multimodal sentiment classification.

More than words can say: a multimodal approach to understanding meaning and sentiment in social media

ABSTRACT The authors develop a multimodal social listening analysis (MSLA) approach as a framework for managers to understand how meaning is constructed in social media posts using both text and other media. The research adds to AI and text analysis approaches by considering the whole meaning of a post rather than an analysis of subsets of information in text and other media. The use of MSLA is validated across the social media platforms of Facebook, Twitter and Instagram. The findings show that MSLA helps (i) reveal structures in what appear to be unstructured multimodal posts; (ii) identify all the sentiment items in a post; (iii) identify implicit meanings, such as irony, humour and sarcasm; and (iv) further identify emotions and judgements in multimodal communication. Importantly, this paper explains how decisions and opinions are made online and how marketing strategies can be tailored towards meanings derived from multimodal communication in social media.

Philosophical and Discursive Approaches to the Categorisation of Modal Meanings in Multi-modal Texts

The question of overcoming barriers between the study of language and philosophical, immanent perception of the image and other aspects that formulate a coherent text is still open. A multimodal approach to understanding the news coverage strategy in the Telegram can reveal the discursive function and the multimodal perspective. One uses a combination of linguistic and philosophical approaches to analyze multimodal texts. This combination proves that the homogeneous use of the model of proving empirical reality employing philosophical and linguistic segments reveals the modulations. The article aims to explore the theoretical foundations of philosophical and discursive approaches to categorizing modal meanings in multimodal texts. The study results of the ideational architectonics of the 65 analysed multimodal posts on the Ukrainian Telegram channel “SUSPILNE NEWS” as of April 18, 2022 have pointed to a tendency of disconnection of categorical modalities. 19 multimodal texts out of 65 revealed a complete symbiosis of philosophical and linguistic modus operandi (the presence of all linguistic and extralinguistic resources, including video sequences), while 46 posts acted as monomodal tests. Regarding the number of “likes” and views, the posts with all extralinguistic and semiotically heterogeneous resources were more popular and, therefore, more exciting and easier to perceive.

Profiling high school students’ multimodal posting in a digital literacy SPOC and examining teachers’ and students’ perceptions

ABSTRACT Given the changing nature of literacy, it is critical to investigate how learners communicate using the multimodal affordances of digital platforms. Guided by metafunctions of Systemic Functional Linguistics, this study developed a multimodal analytical framework to quantify multimodal composing with indicators of functional meaning to evaluate students’ multimodal literacy practices from an overall perspective. Based on a high school digital literacy SPOC (Small Private Online Course), this study adopted a mixed-method approach to profile students’ multimodal posting and examine teachers’ and students’ perceptions. The empirical findings included that: (1) Five profiles were identified based on students’ ideational, interpersonal, and textual meaning differences: academic posters, sensational posters, visual posters, all-round posters, and perfunctory posters. Academic posters viewed multimodal posting as a presentation of research findings; sensational posters viewed multimodal posting as a connection with classmates; visual posters valued the formal beauty of multimodal posts; all-round posters remixed the ideational, interpersonal, and textual functions of multimodal posting; perfunctory posters viewed multimodal posting as a have-to-do task. (2) Teachers underestimated multimodal posting’s interpersonal meaning that students valued. The SFL-based multimodal analytical framework and empirical findings provide implications for explicitly evaluating and guiding students’ multimodal composition in digital environments.

Nice Perfume. How Long Did You Marinate in It? Multimodal Sarcasm Explanation

Sarcasm is a pervading linguistic phenomenon and highly challenging to explain due to its subjectivity, lack of context and deeply-felt opinion. In the multimodal setup, sarcasm is conveyed through the incongruity between the text and visual entities. Although recent approaches deal with sarcasm as a classification problem, it is unclear why an online post is identified as sarcastic. Without proper explanation, end users may not be able to perceive the underlying sense of irony. In this paper, we propose a novel problem -- Multimodal Sarcasm Explanation (MuSE) -- given a multimodal sarcastic post containing an image and a caption, we aim to generate a natural language explanation to reveal the intended sarcasm. To this end, we develop MORE, a new dataset with explanation of 3510 sarcastic multimodal posts. Each explanation is a natural language (English) sentence describing the hidden irony. We benchmark MORE by employing a multimodal Transformer-based architecture. It incorporates a cross-modal attention in the Transformer's encoder which attends to the distinguishing features between the two modalities. Subsequently, a BART-based auto-regressive decoder is used as the generator. Empirical results demonstrate convincing results over various baselines (adopted for MuSE) across five evaluation metrics. We also conduct human evaluation on predictions and obtain Fleiss' Kappa score of 0.4 as a fair agreement among 25 evaluators.

Visual small stories of #JESUISAYLAN: sharing cosmopolitan emotions on insagram

This article examines the circulation of Instagram posts using the hashtag#JeSuisAylan on the 3rd September 2015 as a reaction to the breaking news of threeyearold Alan Kurdi who drowned in the Mediterranean. Posts are analysed as visualsmall stories (Georgakopoulou 2016) and coded for types of story frames, which helpto clarify users’ participation patterns (De Fina 2016). Findings indicate thatparticipation to this shared story involves acts of sharing transportable stances andcosmopolitan emotions, articulated mainly in the subjunctive mode.. The articlecontributes to the analysis of multimodal posts as visual small stories and the study ofsharing emotions online.

Hierarchical self-adaptation network for multimodal named entity recognition in social media

Multimodal Named Entity Recognition task aims to identify named entities in user-generated posts containing both images and texts. Previous multimodal named entity recognition methods greatly benefit from visual features when the text and the image are well aligned, but this is not always the case in social media. On condition that the image is missing or mismatched with the text, these models usually fail to provide excellent performance. Besides, previous models use only single attention to capture the semantic interaction between different modalities, which largely ignore the existence of multiple entity objects in images and texts of the posts. To alleviate these issues, we present a novel model named Hierarchical Self-adaptation Network (HSN) to address these issues. The HSN contains 1) a Cross-modal Interaction Module to promote semantic interaction for the multiple entity objects in different modalities, which is proved to suppress wrong or incomplete attention in multimodal interactivity; 2) a Self-adaptive Multimodal Integration module to handle the problems that the images are missing or mismatched with the texts. Additionally, to evaluate the adaptability of HSN in real-life social media, we construct a Real-world NER dataset consisting of plain text posts and multimodal posts from Twitter. Extensive experiments demonstrate that our model achieves state-of-the-art results on the Real-world multimodal NER dataset and the Twitter multimodal NER dataset.

Fake News Detection via Multi-Modal Topic Memory Network

With the development of the Mobile Internet, more and more people create and release multi-modal posts on social media platforms. Fake news detection has become an increasingly challenging task. Although many current works focus on constructing models extracting abstract features from the content of each post, they neglect the intrinsic semantic architecture such as latent topics, etc. These models only learn patterns in content coupled with certain specific latent topics on the training set to distinguish real and fake posts, which will suffer generalization and discriminating ability decline, especially when posts are associated with rare or new topics. Moreover, most existing works using deep schemes to extract and integrate textual and visual representation in post have not effectively modeled and sufficiently utilized the complementary and noisy multi-modal information containing semantic concepts and entities to complement and enhance each modal. In this paper, to deal with the above problems, we propose a novel end-to-end Multi-modal Topic Memory Network (MTMN), which obtains and combines post representations shared across latent topics together with global features of latent topics while modeling intra-modality and inter-modality information in a unified framework. (1) To tackle real scenarios where newly arriving posts with different topic distribution from the training data, our method incorporates a topic memory module to explicitly characterize final representation as post feature shared across topics and global features of latent topics. These two kinds of features are jointly learned and then combined to generate robust representation. (2) To effectively integrate multi-modality information in posts, we propose a novel blended attention module for multi-modal fusion, which can simultaneously exploit the intra-modality relation within each modal and the inter-modality relation between text words and image regions to complement and enhance each other for high-quality representation. Extensive experiments on two public real-world datasets demonstrate the superior performance of MTMN compared with other state-of-the-art algorithms.

Multi-Level Multi-Modal Cross-Attention Network for Fake News Detection

With the development of the Mobile Internet, more and more users publish multi-modal posts on social media platforms. Fake news detection has become an increasingly challenging task. Although there are many works using deep schemes to extract and combine textual and visual representation in the post, most existing methods do not sufficiently utilize the complementary multi-modal information containing semantic concepts and entities to complement and enhance each modality. Moreover, these methods do not model and incorporate the rich multi-level semantics of text information to improve fake news detection tasks. In this paper, we propose a novel end-to-end Multi-level Multi-modal Cross-attention Network (MMCN) which exploits the multi-level semantics of textual content and jointly integrates the relationships of duplicate and different modalities (textual and visual modality) of social multimedia posts in a unified framework. Pre-trained BERT and ResNet models are employed to generate high-quality representations for text words and image regions respectively. A multi-modal cross-attention network is then designed to fuse the feature embeddings of the text words and image regions by simultaneously considering data relationships in duplicate and different modalities. Specially, due to different layers of the transformer architecture have different feature representations, we employ a multi-level encoding network to capture the rich multi-level semantics to enhance the presentations of posts. Extensive experiments on the two public datasets (WEIBO and PHEME) demonstrate that compared with the state-of-the-art models, the proposed MMCN has an advantageous performance.

The shared story of #JeSuisAylan on Twitter

Abstract Despite an increased interest in the discourse representations of refugees in the media, little attention has been paid so far to the circulation and uptake of such portrayals in social media. This article addresses this gap by examining networked users’ reactions to the iconic image of Alan Kurdi, which quickly turned into a shared story. By analyzing story frames, i.e. orientations to the Storyrealm, the Taleworld, or the Outside world (De Fina 2016) in multimodal posts (dated September 3rd 2015), which feature the hashtag #JeSuisAylan, we show how hashtags, comments, and images combine into visual small stories (Georgakopoulou 2016) that prompt acts of affective and narrative stancetaking. Our analysis calls attention to stancetaking as embedded in storytelling activities and calls for extending the critical examination of discourse representations to the study of their uptake in practices of story participation online.