Multimodal Content Research Articles

Design of a Trusted Content Authorization Security Framework for Social Media

Artificial intelligence-generated content (AIGC) technology has had disruptive results in AI, representing a new trend in research and application and promoting a new era of AI. The potential benefits of this technology are both profound and diverse. However, the benefits of generative tools are accompanied by a series of significant challenges, the most critical of which is that it may cause AI information pollution on social media and mislead the public. Traditional network security models have shown their limitations in dealing with today’s complex network threats, so ensuring that generated content published on social media accurately reflects the true intentions of content creators has become particularly important. This paper proposes a security framework called “secToken”. The framework adopts multi-level security and privacy protection measures. It combines deep learning and network security technology to ensure users’ data integrity and confidentiality while ensuring credibility of the published content. In addition, the framework introduces the concept of zero trust security, integrates OAuth2.0 ideas, and provides advanced identity authentication, fine-grained access control, continuous identity verification, and other functions, to comprehensively guarantee the published content’s reliability on social media. This paper considers the main issues of generative content management in social media and offers some feasible solutions. Applying the security framework proposed in this paper, the credibility of generated content published on social media can be effectively ensured and can help detect and audit published content on social media. At the operational level, when extracting key information summaries from user-generated multimodal artificial intelligence-generated content and binding them to user identity information as a new token to identify user uniqueness, it can effectively associate user identity information with the current network status and the generated content to be published on the platform. This method significantly enhances system security and effectively prevents information pollution caused by generative artificial intelligence on social media platforms. This innovative method provides a powerful solution for addressing social and ethical challenges and network security issues.

Supporting newcomer children's cultural transitions with contemporary Canadian picture books

AbstractThis article examines major themes in 24 contemporary Canadian picture books to identify their potential to support children who are newcomers to Canada. Transition to a new country can be challenging for children and picture books are an effective tool for supporting their cultural and often linguistic transition. Through multimodal content analysis, the authors identify major themes that emerge from the visual and textual data in the books. Analysis identified prevailing themes related to family, food, multiculturalism, and geographic information. Each theme offers valuable insights into the diverse perspectives represented in the texts, providing a foundation for meaningful pedagogical applications for immigrant and newcomer children.

Cross-modal Multiple Granularity Interactive Fusion Network for Long Document Classification

Long Document Classification (LDC) has attracted great attention in Natural Language Processing and achieved considerable progress owing to the large-scale pre-trained language models. In spite of this, as a different problem from the traditional text classification, LDC is far from being settled. Long documents, such as news and articles, generally have more than thousands of words with complex structures. Moreover, compared with flat text, long documents usually contain multi-modal content of images, which provide rich information but not yet being utilized for classification. In this article, we propose a novel cross-modal method for long document classification, in which multiple granularity feature shifting networks are proposed to integrate the multi-scale text and visual features of long documents adaptively. Additionally, a multi-modal collaborative pooling block is proposed to eliminate redundant fine-grained text features and simultaneously reduce the computational complexity. To verify the effectiveness of the proposed model, we conduct experiments on the Food101 dataset and two constructed multi-modal long document datasets. The experimental results show that the proposed cross-modal method outperforms the single-modal text methods and defeats the state-of-the-art related multi-modal baselines.

THE PRE-PRESIDENTIAL ELECTION 2024: UNCOVERING THE CONSISTENCY OF RELIGIOUS POLITICIZATION

Social media is an effective communication tool for facilitating political campaign activities. However, social media promotes the enormous politicization of religion, such as the controversy surrounding the mixing of politics and religion in Indonesia's 2017 and 2019 general elections. This study aims at responding to the causes of social media supporting the consistency of the politicization of religion in the Pre-Presidential General Election 2024 by employing the politicization of religion in the past as a reflection of the current phenomenon. The research was conducted using a qualitative strategy and multimodal discourse analysis as an instrument for data analysis. Data was collected from the Twitter account @aniesbaswedan from October to December 2022 and previous studies. The research results indicate that the nuances can demonstrate indicators of the consistency of the politicization of religion through multimodal content, such as text, images, video, audio, and animation. The multimodal aspect can frame politics with an intriguing blend of religious attributes to obtain responses from netizens. Media convergence also stimulates interactive communication between netizens with others. In conclusion, politicians implement religious symbols to increase netizens' response in its campaign.

Research on the perception of the terrain image of the tourism destination based on multimodal user-generated content data.

Destination image is a powerful means by which destinations compete in the tourism industry, and the accurate identification of a destination image better serves destination marketing and management. This study uses multimodal data, such as text, images, and videos uploaded by tourists, to construct a comprehensive and systematic destination image process. The "cognitive-emotional-overall image" model, latent Dirichlet allocation (LDA) model, and deep residual neural networks are implemented to build a framework to examine the perception of a destination image, travelogues, and short videos from the sources called Ctrip, Qunar, and TikTok. The results show that tourists' overall perception of Sanya is based mainly on the cognitive image of natural scenery, human resources, and food. In addition, there are differences between textual and visual cognitive images among the perceptual images when multimodal data is under consideration. Furthermore, tourists have an overall positive affective image of Sanya as a destination.

Conception and prospective multicentric validation of a Robotic Surgery Training Curriculum (RoSTraC) for surgical residents: from simulation via laboratory training to integration into the operation room

There is a lack of training curricula and educational concepts for robotic-assisted surgery (RAS). It remains unclear how surgical residents can be trained in this new technology and how robotics can be integrated into surgical residency training. The conception of a training curriculum for RAS addressing surgical residents resulted in a three-step training curriculum including multimodal learning contents: basics and simulation training of RAS (step 1), laboratory training on the institutional robotic system (step 2) and structured on-patient training in the operating room (step 3). For all three steps, learning content and video tutorials are provided via cloud-based access to allow self-contained training of the trainees. A prospective multicentric validation study was conducted including seven surgical residents. Transferability of acquired skills to a RAS procedure were analyzed using the GEARS score. All participants successfully completed RoSTraC within 1 year. Transferability of acquired RAS skills could be demonstrated using a RAS gastroenterostomy on a synthetic biological organ model. GEARS scores concerning this procedure improved significantly after completion of RoSTraC (17.1 (±5.8) vs. 23.1 (±4.9), p < 0.001). In step 3 of RoSTraC, all participants performed a median of 12 (range 5–21) RAS procedures on the console in the operation room. RoSTraC provides a highly standardized and comprehensive training curriculum for RAS for surgical residents. We could demonstrate that participating surgical residents acquired fundamental and advanced RAS skills. Finally, we could confirm that all surgical residents were successfully and safely embedded into the local RAS team.

Framing bilingual parenting as market advantage: the neoliberal discourses of ‘grassroots’ media

ABSTRACT Little research has examined media dimensions of family language policy. This project addresses this gap by analysing marketing efforts targeted at media consumers seeking ‘grassroots’ advice on bilingual parenting. Here, we refer to ‘grassroots media marketing’ as purposeful attempts to target, inform or persuade a niche demographic with ‘grassroots’-style media methods and tools. We did Google Searches to identify 40 high-profile corporate media, non-profit, and independent bilingual parenting blogs written for Spanish- and for English-language readers. We found English- and Spanish-language keyword searches generated distinct blog-style texts (i.e. advice sites). While English-language searches were more likely to generate returns from small-owned blogs, Spanish-language searches tended to yield corporate news and non-profit sites. Frame analysis of multimodal media blog content revealed gaps and contradictions regarding how bilingual parenting in the U.S is portrayed in Spanish and in English, as well as a neoliberal framing across texts. We describe three neoliberal ideologies: (1) ‘bilingual parenting as individual choice’, (2) ‘bilingual parenting as competitive edge’, and (3) ‘bilingual parents as consumers’. Analysis reveals that documented racist, anti-immigrant discourses widely circulating in U.S. society are absent from these neoliberal framings, resulting in public contradictions of what it means to be a ‘good’ bilingual parent.

MultArtRec: A Multimodal Neural Topic Modeling for Integrating Image and Text Features in Artwork Recommendation

Recommender systems help users obtain the content they need from massive amounts of information. Artwork recommender systems is a topic that has attracted attention. However, existing art recommender systems rarely consider user preferences and multimodal information at the same time, while utilizing all the information has the potential to help make better personalized recommendations. To better apply recommender systems to the artwork-recommendation scenario, we propose a new neural topic modeling (NTM)-based multimodal artwork recommender system (MultArtRec), that can take all the information into account at the same time and extract effective features representing user preferences from multimodal content. Also, to improve MultArtRec’s performance on monomodal feature extraction, we add a novel topic loss term to the conventional NTM loss. The first two experiments in this study compare the performances of different models with different monomodal inputs. The results show that MultArtRec can improve the performance with image modality inputs by up to 174.8% compared to the second-best model and improve the performance with text modality inputs by up to 10.7% compared to the second-best model. The third experiment is conducted to compare the performance of MultArtRec with monomodal inputs and multimodal inputs. The results show that the performance of MultArtRec with multimodal inputs can be improved by up to 15.9% compared to monomodal inputs. The last experiment preliminarily tests the versatility of MultArtRec on a fashion recommendation scenario that considers clothing image content and user preferences. The results show that MultArtRec outperforms the other methods across all the metrics.

Sustainability data distortions: the use of visual impression management techniques in corporate sustainability reporting

PurposeThe purpose of this study is to conduct a comprehensive investigation into the utilization of visual impression management techniques within sustainability reporting. Specifically, the study aims to determine whether Italian companies employ impression management tactics in the presentation of graphs within their sustainability reports and, thus, problematize visual data communication in corporate social responsibility (CSR).Design/methodology/approachThe research adopts a multimodal content analysis of the 58 sustainability reports from Italian listed companies that are GRI-compliant. The analysis focused on three types of graphs: pie charts, line graphs and bar graphs. In total, 860 graphs have been examined.FindingsThe study found evidence of graphical distortion techniques being employed by companies in their sustainability reports to create a favorable impression. Specifically, graph distortions are found in column graphs and not in line or pie charts. In particular, selectivity, presentation enhancement and measurement distortion techniques seem to be extensively used when adopting column graphs in sustainability communication. Moreover, social sustainability–related topics tend to be more represented of other area of CSR reporting. This suggests that companies, whether consciously or unconsciously, engage in impression management techniques when using graphs in their sustainability reports.Social implicationsThe study findings suggest that more consciousness is needed for companies when engaging in the construction and selection of graphs in their sustainability reports and that decision-makers should develop a clear guide for ethical visual communication.Originality/valueThe paper systematically analyzes visual impression management techniques in communicating sustainability data and, in particular, advances literature on graphical distortion. The value lies in empirical evidence of distortion adoption in GRI-compliant reports as well as problematizing visual data communication as a fundamental challenge for sustainability communication management.

Implementing Universal Design through augmented-reality game-based learning

Technology integration in higher education (HE) provides educators with the opportunity to design stimulating learning environments, especially in design and engineering education (DEE) where it plays a unique role in nurturing creativity, problem-solving and innovation. This study investigates the integration of an augmented reality (AR) educational game in DEE, focusing on the teaching of Design for Manufacturing and Assembly (DfMA) principles. The primary aim of this research is to enhance traditional DfMA teaching and learning (T&L) practices by applying innovative T&L strategies. The resulting AR DfMA game, developed using digital game-based learning, microlearning and collaborative learning techniques, aligns with the Universal Design for Learning framework to create an inclusive learning environment that encourages participation from all students and supports several learning styles.The study tests the individual features designed for personal study opportunities to discover which elements are optimal in that environment, and then on the proposed in-class group components of the AR game, involving 68 participants from the appropriate program, though it is not currently implemented as part of a module. We utilize a mixed methods approach to examine the students’ experience and identify key design features that contribute to an inclusive educational experience. The findings highlight positive student experiences, and preferences for hands-on engagement, multimodal content, and collaborative activities. The AR DfMA game also has the potential to enhance intrinsic motivation and create an active and inclusive learning environment. Challenges and areas for improvement are also discussed.

Fake News Detection Using Stance Extracted Multimodal Fusion-Based Hybrid Neural Network

Public and governmental concerns over online rumors’ widespread diffusion and deceptive impact on social media have increased. For users to obtain accurate information and preserve social peace, finding and controlling social media rumors is challenging. Automatically, identifying fake news (FN) is a critical yet challenging topic that is still little understood because the consequences are so high. The text, visual features, the acceptance of the user’s reply, stance, and social context are a few aspects of FN that are universally acknowledged. Current research has concentrated on modifying results to one specific trait, which has been partially the reason for their success. This article proposes Fakefind, a convolutional neural network (CNN) <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$+$</tex-math> </inline-formula> recurrent neural networks (RNNs) hybrid model that integrates multimodal features for efficient rumor detection (RD). Additionally, the stance is extracted from indirectly implied postreply pairs using a CNN-based knowledge extractor (KE), and the stance representations are integrated for FN detection (FND). Extensive research findings are based on three multimedia rumor datasets from Weibo, Fakeddit, and PHEME. The outcomes show how well the recommended Fakefind identifies rumors with multimodal content.

The Construction of a Multimodal Teaching Content Analysis System for English in Colleges and Universities

Abstract Based on multimodal theory, this paper constructs a multimodal content analysis system for college English with the digital model of teaching content, teaching interactive technology, and virtual reality technology as the main functional modules. The digital transformation of teaching content is done through the use of the particle swarm optimization algorithm and Wiki collaboration, and the learning objects in the learning content library are processed and transformed. The main technical aspects of shot detection, key frame determination, feature extraction, and feature matching are improved to establish the interactive function of English teaching videos. The quadratic error measurement algorithm enhances the edge folding algorithm and streamlines the 3D model to realize the virtual teaching simulation function. Students majoring in business English at a university in Kunming, Yunnan Province, China, are used as research objects to carry out the teaching application practice of this paper’s system. The total average English unit scores of the experimental class increased from 68.2 to 72.1 in the pre-test, and the recognition, comprehension, and utilization dimensions of the learning effect were higher than those of the control class by 0.46, 0.66, and 0.51, respectively, which showed a significant difference (P&lt;0.05). The experimental group of subject students also had a better experience using the system in this paper.

Analysis of the value of news and cultural communication using multimodal learning

Abstract News serves as a critical medium for mass communication, characterized by its distinct purpose, comprehensiveness, and authenticity. It is essential to explore the developmental trends within the news sector from the perspective of communicative value and to provide appropriate guidance. This study employs multimodal learning techniques to develop a new content detection model that leverages multimodal fusion for effective content identification. Building on this framework, a sentiment analysis model utilizing multimodal data is constructed to assess the emotional impact of news communication. Additionally, the cognitive mediation model is used to establish an analysis model for determining the value of news communication. Empirical analyses indicate that the multimodal news content detection model presented in this study achieves high-performance metrics, with each index exceeding 0.9. Similarly, the sentiment direction model demonstrates robust accuracy, precision, recall, and F1 values, all-surpassing 0.83. The models exhibit Pearson correlation coefficients (Corr) of 0.758 and 0.785 and mean absolute errors (MAE) of 0.624 and 0.948, respectively, outperforming comparable models. Further analysis into the value of news and cultural communication reveals significant impacts of sound attention (0.328), sound-associated thinking (0.210), and image attention (0.284) on communicative value. Conversely, leisure Internet time shows a negative correlation with knowledge acquisition (-0.090). The findings underscore a positive correlation between the value of news and cultural communication and sound content, thereby affirming that the methodologies employed in this study provide a viable approach to analyzing the communicative value of news. This study not only advances the theoretical understanding of news communication but also enhances practical approaches to media analysis.

MSCMGTB: A Novel Approach for Multimodal Social Media Content Moderation Using Hybrid Graph Theory and Bio-Inspired Optimization

MSCMGTB: A Novel Approach for Multimodal Social Media Content Moderation Using Hybrid Graph Theory and Bio-Inspired Optimization

Positive Unlabeled Fake News Detection via Multi-Modal Masked Transformer Network

Fake news detection has gotten continuous attention during these years with more and more people have been posting and reading news online. To enable fake news detection, existing researchers usually assume labeled posts are provided for two classes (true or false) so that the model can learn a discriminative classifier from the labeled data. However, this supposition may not hold true in reality, as most users may only label a small number of posts in a single category that they are interested in. Furthermore, most existing methods fail to mask the noise or irrelevant context (i.e., regions or words) between different modalities to assist in strengthening the correlations between relevant contexts. To tackle these issues, we present a curriculum-based multi-modal masked transformer network (CMMTN) for positive unlabeled multi-modal fake news detection by jointly modeling the inter-modality and intra-modality relationships of multi-modal information and masking the irrelevant context between modalities. In particular, we adopt BERT and ResNet to obtain better representations for texts and images, separately. Then, the extracted features of images and texts are fed into a multi-modal masked transformer network to fuse the multi-modal content and mask the irrelevant context between modalities by calculating the similarity between inter-modal contexts. Finally, we design a curriculum-based PU learning method to handle the positive and unlabeled data. Massive experiments on three public real datasets prove the effectiveness of the CMMTN.

Multimodal High-Order Relationship Inference Network for Fashion Compatibility Modeling in Internet of Multimedia Things

Recent progress in artificial intelligence (AI) have broadened various intelligent application scenarios on the Internet of Multimedia Things (IoMT). Due to the urgent demands for intelligence in the online fashion industry, using AI techniques to explore user’s clothing collocations from fashion data generated by the IoMT system is a challenging task. Under the background, fashion compatibility modeling (FCM), which aims to estimate the matching degree of a given outfit, has attracted great attention in the multimedia analysis field. However, most of the studies often fail to fully leverage multimodal content or ignore the sparse associations between fashion items. In this paper, we propose a novel multimodal high-order relationship inference network (MHRIN) for fashion compatibility modeling task. In MHRIN, we focus on enriching multimodal representations of fashion items by means of incorporating the category correlations and injecting high-order item-item connectivity. Concretely, considering that fashion collocations depend on the semantic relevance patterns between categories, we design a category correlations learning module to adaptively learn category representations. On this basis, multiple modality representations are aggregated by a hierarchical multimodal fusion module to generate visual-semantic embeddings. To address the item-item matching interactions issue, we further refine the final representations by a high-order message propagation module to absorb rich connection information. Experiments on the publicly available dataset demonstrate the superiority of our MHRIN over state-of-the-art methods.

Cyber vaccinator for image tamper resilient and recovery using invertible neural network

People frequently interact with their families, friends, and colleagues through Online Social Networks (OSNs). People post and share their photos in online communities and content-sharing sites. The problem addressed in this paper is the susceptibility of digital images to tampering, which compromises security and privacy. Traditional image forgery detection methods face challenges in reproducing original content after manipulation. This paper introduces an advanced Image Immunization System leveraging Invertible Neural Networks. The system, which comprises the cyber vaccinator, vaccine validator, forward pass for tamper detection, and backward pass for image self-recovery, aims to proactively immunize images against various attacks. The run-length encoding in the backward pass transforms hidden perturbations into information, facilitating the recovery of the authentic image. The middleware's expansion to multimodal content analysis, including videos and audio, provides a more comprehensive defense against digital manipulation within OSNs. These advancements reflect a commitment to robust security and holistic content integrity. The Cyber Vaccinator, using Invertible Neural Networks (INNs) for image tamper resilience and recovery, demonstrates significant effectiveness in detecting tampering and restoring images, providing a robust solution for maintaining image integrity. The Cyber Vaccinator uses an Invertible Neural Network (INN) to safeguard image integrity. It detects tampering by analyzing invariant features and responds with precise recovery methods. By continuously monitoring images, it ensures real-time tamper detection and efficient restoration, maintaining image authenticity through advanced neural network resilience and recovery techniques.

The pragmatics of sharing memes on Twitter

This paper explores the interactional pragmatic characteristics of multimodal memetic content facilitated by Twitter affordances, notably the functions of retweeting and quote tweeting. These Twitter affordances enable reposting a previous multimodal tweet (with or without a preceding commentary), which is conducive to nested memetic posts involving entextualised multimodal quotations and/or to memes interwoven into conversational exchanges. These retweeting practices also promote different forms of recontextualisation and user engagement with the original multimodal post, ranging from content co-construction (dubbed “pragmatic entrenchment”) to repurposing (referred to as “repragmatisation”). The theoretical proposals concerning Twitter memes are illustrated with examples taken from a manually compiled dataset of 893 (quote) tweeted memes collected between 25th and 30th November 2022 and originating from two viral memes that satirically announce the formation of NAFO's New Fella Land and Královec on Russian lands in response to Russia's illegitimate annexation of Ukrainian territories based on sham referenda held in late September 2022. These two memetic constructs are shown to have inspired pragmatic entrenchment (involving joint fantasising among other things), as well as repragmatisation through quote tweets which query or question the parent memes or their spin-offs and apply them for new communicative purposes.

Multimodal online dissident culture in Instagram: A critique of the Turkish economy

Ever-present mass surveillance has blocked the flourishing of a traditional dissident culture in Turkey. Focusing on popular ‘just for fun’ Instagram accounts during the lira's freefall that began in the autumn of 2021, this study seeks to identify the creative strategies for digital social resistance embedded in multimodal content sharing of posts, which are composed of visuals, text, and sound. For this, we employed a multimodal-type analysis of Instagram posts regarding Turkey's economic crisis, followed by an interpretative content analysis aiming to (1) identify, categorize, and compile a typology of the main countersurveillance strategies inherent in multimodal posts, such as memes, edited videos, and animations, Photoshop-crafted still images, and (2) explore the contextual traits of the connected dissident culture. We discuss how these multimodal-type posts support connected dissident group formation while maintaining confidentiality while criticizing governmental conduct of economic policy making in Turkey.

Feminism as a polarizing axis of the political conversation on Twitter: the case of #IreneMonteroDimision

The fragmentation of the Spanish party system has led to a growing political polarization, particularly evident on the Internet. The discursive strategies of political actors online, combined with the particularities of communication on social media platforms such as the disinhibition effect, echo chambers, and filter bubbles, are likely to promote a state of tension among digital citizens. This tension is particularly pronounced in gender issues, which have become key positions of political parties, with convergence of sexism in the community and a highly reactive misogynistic online culture that turns female politicians into easy targets. Starting from the hashtag #IreneMonteroDimision, which conveys a paradigmatic criticism movement against the Spanish Minister of Equality, an analysis of the social conversation on Twitter is developed to determine its intentionality, tone, and orientation, as well as the themes and users that generated the most tension. In this way, multimodal content and discourse analysis is applied to the 418 tweets with the greatest interaction and explicitly directed at the minister. The hypothesis is that the criticism of Irene Montero goes beyond the management of her Ministry to place her at the center of a power struggle in which political ideology, hate speech, and antifeminism converge. The study has shown that attacks on Montero extend to the Executive and the feminist movement; they are promoted by a mass of politicized and polarized profiles, with constant activism and a tendency towards contagious replication of messages. These practices, as well as the participation of artificial profiles, allow us to appreciate signs of astroturfing; behind this seemingly natural critical reaction, there are orchestrated movements of antifeminist and far-right profiles (gender trolling), but also of trans-exclusionary feminists opposed to legislation promoted by Montero.