Multimodal Representation Research Articles

Drug Sensitivity Prediction Based on Multi-stage Multi-modal Drug Representation Learning.

Accurate prediction of anticancer drug responses is essential for developing personalized treatment plans in order to improve cancer patient survival rates and reduce healthcare costs. To this end, we propose a drug sensitivity prediction model based on multi-stage multi-modal drug representations (ModDRDSP) to reflect the properties of drugs more comprehensively, and to better model the complex interactions between cells and drugs. Specifically, we adopt the SMILES representation learning method based on the deep hierarchical bi-directional GRU network (DSBiGRU) and the molecular graph representation learning method based on the deep message-crossing network (DMCN) for the multi-modal information of drugs. Additionally, we integrate the multi-omics information of cell lines based on a convolutional neural network (CNN). Finally, we use an ensemble deep forest algorithm for the prediction of drug sensitivity. After validation, the ModDRDSP shows impressive performance which outperforms the four current industry-leading models. More importantly, ablation experiments demonstrate the validity of each module of the proposed model, and case studies show the good results of ModDRDSP for predicting drug sensitivity, further establishing the superiority of ModDRDSP in terms of performance.

Multimodal Representations of the American Dream Theme: A Comparative Analysis of The Great Gatsby in Film and Novel

This research analyzes the film adaptation of The Great Gatsby and its written text. It aims to examine the film adaptation through the perspective of multimodal analysis to determine how it expands or contracts the viewersꞌ comprehension of the complex themes and characters from the source material. Another goal is to determine the effect of the use of multimodal elements in the film on viewersꞌ engagement compared to the capacity of text to achieve the same function. According to this research hypothesis, the film adaptation of The Great Gatsby relies heavily on multimodal modes of communication to convey themes, character development, and emotions, influencing its viewers more than the textual descriptions do. Bateman and Schmidt's (2011) model, which includes paradigmatic and syntagmatic analysis, underpins the evaluation of the film. A thematic sample was selected for the multimodality analysis to compare the novel by F. Scott Fitzgerald to the film version directed by Baz Luhrmann in 2013, featuring Leonardo DiCaprio and Carey Mulligan. This research finds that the film adaptation presents a distinct interpretation of the American Dream compared to that of the novel. Fitzgerald's work conveys ideas through introspective narration and lush, descriptive language, allowing for a more nuanced and personal understanding of the American Dream.

Multi-modal representation learning in retinal imaging using self-supervised learning for enhanced clinical predictions

Self-supervised learning has become the cornerstone of building generalizable and transferable artificial intelligence systems in medical imaging. In particular, contrastive representation learning techniques trained on large multi-modal datasets have demonstrated impressive capabilities of producing highly transferable representations for different downstream tasks. In ophthalmology, large multi-modal datasets are abundantly available and conveniently accessible as modern retinal imaging scanners acquire both 2D fundus images and 3D optical coherence tomography (OCT) scans to assess the eye. In this context, we introduce a novel multi-modal contrastive learning-based pipeline to facilitate learning joint representations for the two retinal imaging modalities. After self-supervised pre-training on 153,306 scan pairs, we show that such a pre-training framework can provide both a retrieval system and encoders that produce comprehensive OCT and fundus image representations that generalize well for various downstream tasks on three independent external datasets, explicitly focusing on clinically pertinent prediction tasks. In addition, we show that interchanging OCT with lower-cost fundus imaging can preserve the predictive power of the trained models.

CE-DCVSI: Multimodal relational extraction based on collaborative enhancement of dual-channel visual semantic information

Visual information implied by the images in multimodal relation extraction (MRE) usually contains details that are difficult to describe in text sentences. Integrating textual and visual information is the mainstream method to enhance the understanding and extraction of relations between entities. However, existing MRE methods neglect the semantic gap caused by data heterogeneity. Besides, some approaches map the relations between target objects in image scene graphs to text, but massive invalid visual relations introduce noise. To alleviate the above problems, we propose a novel multimodal relation extraction method based on cooperative enhancement of dual-channel visual semantic information (CE-DCVSI). Specifically, to mitigate the semantic gap between modalities, we realize fine-grained semantic alignment between entities and target objects through multimodal heterogeneous graphs, aligning feature representations of different modalities into the same semantic space using the heterogeneous graph Transformer, thus promoting the consistency and accuracy of feature representations. To eliminate the effect of useless visual relations, we perform multi-scale feature fusion between different levels of visual information and textual representations to increase the complementarity between features, improving the comprehensiveness and robustness of the multimodal representation. Finally, we utilize the information bottleneck principle to filter out invalid information from the multimodal representation to mitigate the negative impact of irrelevant noise. The experiments demonstrate that the method achieves 86.08% of the F1 score on the publicly available MRE dataset, which outperforms other baseline methods.

Hypercomplex Graph Neural Network: Towards Deep Intersection of Multi-modal Brain Networks.

The multi-modal neuroimage study has provided insights into understanding the heteromodal relationships between brain network organization and behavioral phenotypes. Integrating data from various modalities facilitates the characterization of the interplay among anatomical, functional, and physiological brain alterations or developments. Graph Neural Networks (GNNs) have recently become popular in analyzing and fusing multi-modal, graph-structured brain networks. However, effectively learning complementary representations from other modalities remains a significant challenge due to the sophisticated and heterogeneous inter-modal dependencies. Furthermore, most existing studies often focus on specific modalities (e.g., only fMRI and DTI), which limits their scalability to other types of brain networks. To overcome these limitations, we propose a HyperComplex Graph Neural Network (HC-GNN) that models multi-modal networks as hypercomplex tensor graphs. In our approach, HC-GNN is conceptualized as a dynamic spatial graph, where the attentively learned inter-modal associations are represented as the adjacency matrix. HC-GNN leverages hypercomplex operations for inter-modal intersections through cross-embedding and cross-aggregation, enriching the deep coupling of multi-modal representations. We conduct a statistical analysis on the saliency maps to associate disease biomarkers. Extensive experiments on three datasets demonstrate the superior classification performance of our method and its strong scalability to various types of modalities. Our work presents a powerful paradigm for the study of multi-modal brain networks.

A Multimodal Metaphorical Representation of Selected Political Cartoons in Nigeria

Complementing studies on the representation of political cartoons which apply metaphorical representation in the portrayal of societal ills, this study explores a multimodal metaphorical representation of selected political cartoons in Nigeria to investigate how cartoonists employed metaphors to represent the current political situation in the country. In the face of the ongoing election petitions and handover of office to the President, among other activities within the political domain in Nigeria, a number of political cartoonists have flooded the online platform with cartoons depicting visual metaphorical representations of happenings within the nation. Such metaphorical representations are semiotically showcased via such cartoons as Go to Court, the Scape Goat, Buhari’s Pet, Corruption, and Original Wahala Received by Me in the portrayal of reality that Nigeria faces today. Given the foregoing, this study combines linguistic and visual modes to form a multimodal representation of the metaphors used in the selected political cartoons. The study adopts a qualitative analysis of 10 political cartoons in Nigeria. It uses Kress and van Leeuwen’s (2006, 2020) approach to multimodality and Forceville’s (1996, 2016) view of visual realisations of conceptual metaphor in constructing meaning to investigate how visual images are constructed to cue conceptual metaphors. The results of the study suggest that Nigerian political cartoons rely mainly on visual metaphors as a means of communicating their messages. In addition, the metaphors used often rely on comic, exaggerated and simplistic depictions to convey their meaning effectively and to stir a strong emotional reaction from the readers. Furthermore, the findings of the study emphasise the importance of metaphorical representation in political cartoons and its implications for public discourse. The results reveal ways to improve the power of the metaphorical messages and the impact of the cartoons on public opinion. The study contributes to the multimodal metaphor research field and reveals the Nigerian public’s underlying beliefs and ideologies.

PII-GCNet: Lightweight Multi-Modal CNN Network for Efficient Crowd Counting and Localization in UAV RGB-T Images

With the increasing need for real-time crowd evaluation in military surveillance, public safety, and event crowd management, crowd counting using unmanned aerial vehicle (UAV) captured images has emerged as an essential research topic. While conventional RGB-based methods have achieved significant success, their performance is severely hampered in low-light environments due to poor visibility. Integrating thermal infrared (TIR) images can address this issue, but existing RGB-T crowd counting networks, which employ multi-stream architectures, tend to introduce computational redundancy and excessive parameters, rendering them impractical for UAV applications constrained by limited onboard resources. To overcome these challenges, this research introduces an innovative, compact RGB-T framework designed to minimize redundant feature processing and improve multi-modal representation. The proposed approach introduces a Partial Information Interaction Convolution (PIIConv) module to selectively minimize redundant feature computations and a Global Collaborative Fusion (GCFusion) module to improve multi-modal feature representation through spatial attention mechanisms. Empirical findings indicate that the introduced network attains competitive results on the DroneRGBT dataset while significantly reducing floating-point operations (FLOPs) and improving inference speed across various computing platforms. This study’s significance is in providing a computationally efficient framework for RGB-T crowd counting that balances accuracy and resource efficiency, making it ideal for real-time UAV deployment.

Beyond Text: Multimodal Credibility Assessment Approaches for Online User-Generated Content

User-generated content (UGC) is increasingly becoming prevalent on various digital platforms. The content generated on social media, review forums, and question–answer platforms impacts a larger audience and influences their political, social, and other cognitive abilities. Traditional credibility assessment mechanisms involve assessing the credibility of the source and the text. However, with the increase in how user content can be generated and shared (audio, video, and images), multimodal representation of UGC has become increasingly popular. This article reviews the credibility assessment of UGC in various domains, particularly identifying fake news, suspicious profiles, and fake reviews and testimonials, focusing on both textual content and the source of the content creator. Next, the concept of multimodal credibility assessment is presented, which also includes audio, video, and images in addition to text. After that, the article presents a systematic review and comprehensive analysis of work done in the credibility assessment of UGC considering multimodal features. Additionally, the article provides extensive details on the publicly available multimodal datasets for the credibility assessment of UGC. In the end, the research gaps, challenges, and future directions in assessing the credibility of multimodal UGC are presented.

Multimodal meaning making in news communication about immigration: using the NewsScape corpus to explore co-verbal images in TV news

The communication of news relies on semiotic resources besides language, including various audiovisual modes of representation. Owing to the difficulties associated with obtaining televisual data, the vast majority of research addressing multimodality in the news has been targeted at print news media, where various strategies in visual representation and patterns of interaction between verbal and visual modes have been discerned. Where televisual data has been interrogated, this has been based on a very limited number of data points. In this study, I exploit the NewsScape library – a massive multimodal corpus of news communication – to investigate multimodal representations of immigration in television news. Accessible via CQPWeb, the corpus is searched for target utterances refugees/(im)migrants have VERBed and refugees/(im)migrants are VERBing. The co-verbal images accompanying 474 utterances describing motion events are analyzed quantitatively and qualitatively. Among the results discussed are that refugees/migrants are depicted in large rather than small groups, that they are depicted in transit somewhere along the migratory journey rather than in countries of origin or destination countries, that they are depicted on land more than at sea, that they are depicted in security contexts, and that they are erased represented instead through abstract forms such as maps. Differences in the visual representation of people designated as ‘refugee’ versus ‘migrant’ are also observed and discussed.

Robust Multimodal Representation under Uncertain Missing Modalities

Multimodal representation learning has gained significant attention across various fields, yet it faces challenges when dealing with missing modalities in real-world applications. Existing solutions are confined to specific scenarios, such as single-modality missing or missing modalities in test cases, thereby restricting their applicability. To address a more general scenario of uncertain missing modalities in both training and testing phases, we propose Robust Multimodal Representation under Uncertain Missing Modalities (RMRU). This framework projects each modality's representation into a shared subspace, enabling the reconstruction of any missing modalities within a unified model. We propose an interaction refinement module that utilizes cross-modal attention to enhance these reconstructions, particularly beneficial in scenarios with limited complete modality data. Furthermore, we introduce an iterative training strategy that alternately trains different modules to effectively utilize both complete and incomplete modality data. Experimental results on four benchmark datasets demonstrate the superiority of RMRU over existing baselines, particularly in scenarios with a high rate of missing modalities. Remarkably, our proposed RMRU can be broadly applied to diverse scenarios, regardless of modality types and quantities.

A Riemannian multimodal representation to classify parkinsonism-related patterns from noninvasive observations of gait and eye movements

A Riemannian multimodal representation to classify parkinsonism-related patterns from noninvasive observations of gait and eye movements

Students' Meaning‐Making of Nature of Science: Interaction Between Visual, Verbal, and Written Modes of Representation

ABSTRACTStudents' understanding of nature of science (NOS) has been largely examined primarily in written or verbal modes. The visual, verbal, and written modes are essential for students' meaning‐making of NOS. However, research has sidelined the interaction among these three modes in understanding students' collaborative discourse of NOS. Informed by theories of multimodality and social semiotics, this paper investigates the interactions between the visual, verbal, and written modes as groups of students engaged in explicit‐reflective multimodal representation during NOS instruction. Utilizing a collective case study approach, we planned NOS instruction with teachers, and videotaped how each focal group of students in two grade seven classes in Hong Kong constructed multimodal representations of NOS. Multimodal discourse analysis revealed that the three modes fulfill various purposes during students' co‐construction of multimodal representations of NOS. The interaction between the three modes facilitates meaning‐making of NOS in four ways: (a) students' re‐semiotization of discursive scientific practices into their multimodal ensembles; (b) bridging students' writing of scientific reports to scientists' social certification and dissemination; (c) connecting students' decontextualized meaning‐making to contextualized meaning‐making of methods and methodological rules; and (d) facilitating students' embodied semiosis in social organizations and interactions of science. Focusing on four episodes of co‐constructing multimodal representations of NOS, we illustrate how students' meaning‐making of NOS is multimodal in nature and how various modes have their own affordances. We discuss future research directions on how multimodality can facilitate students' meaning‐making of NOS.

Multimodal Representation Learning via Graph Isomorphism Network for Toxicity Multitask Learning.

Toxicity is paramount for comprehending compound properties, particularly in the early stages of drug design. Due to the diversity and complexity of toxic effects, it became a challenge to compute compound toxicity tasks. To address this issue, we propose a multimodal representation learning model, termed multimodal graph isomorphism network (MMGIN), to address this challenge for compound toxicity multitask learning. Based on fingerprints and molecular graphs of compounds, our MMGIN model incorporates a multimodal representation learning model to acquire a comprehensive compound representation. This model adopts a two-channel structure to independently learn fingerprint representation and molecular graph representation. Subsequently, two feedforward neural networks utilize the learned multimodal compound representation to perform multitask learning, encompassing compound toxicity classification and multiple compound category classification simultaneously. To test the effectiveness of our model, we constructed a novel data set, termed the compound toxicity multitask learning (CTMTL) data set, derived from the TOXRIC data set. We compare our MMGIN model with other representative machine learning and deep learning models on the CTMTL and Tox21 data sets. The experimental results demonstrate significant advancements achieved by our MMGIN model. Furthermore, the ablation study underscores the effectiveness of the introduced fingerprints, molecular graphs, the multimodal representation learning model, and the multitask learning model, showcasing the model's superior predictive capability and robustness.

Reconstructing representations using diffusion models for multimodal sentiment analysis through reading comprehension

The primary challenge in multimodal sentiment analysis (MSA), which utilizes textual, audio, and visual information to analyze speakers' emotions, lies in constructing representation vectors that incorporate both unimodal semantic and multimodal interaction information. While existing research has extensively focused on multimodal fusion strategies, there remains insufficient exploration of the intrinsic potential within concurrently enhancing unimodal and multimodal representations. To address this gap, we introduce two additional steps to traditional three-step MSA: text modality enhancement through the machine reading comprehension (MRC) framework and multimodal representation reconstruction via proposing the diverse diffusion denoising autoencoder (D3AE) module. The MRC queries are integrated to locate sentiment-related prior knowledge, thereby deepening textual semantic understanding from a pretrained language model. Meanwhile, D3AE employs a single-step denoising strategy along with diffusion models across multiple time intervals, enabling efficient reconstruction and enhancement of multimodal representations. Extensive experiments conducted on two benchmark datasets, CMU-MOSI and CMU-MOSEI, validate that our model, MRC-D3AE, achieves state-of-the-art performance. The superiority of our model over existing baselines is primarily attributed to integrating MRC for enhancing text modality and D3AE for reconstructing multimodal representations.

The Analysis of Smarter Future

With the continuous development of technology, smart home control systems, as one of the applications of artificial intelligence in daily life, are receiving increasing attention. However, smart home control systems face challenges in practical applications, such as complex scenarios and multimodal information. In this context, this paper introduces Visual Question Answering (VQA) technology to enhance the intelligence and user experience of smart home control systems. Despite the potential advantages of VQA technology in smart home control systems, current models still have certain shortcomings in handling local image information and integrating visual-language multimodal features. To address this, this paper proposes an innovative Transformer-based Multimodal Fusion Network (TMFNet) model. TMFNet aims to overcome the limitations of existing models in dealing with complex smart home scenarios by introducing a global-local feature attention mechanism, deep encoding-decoding modules, and multimodal representation modules.

Augmenting Multimodal Content Representation with Transformers for Misinformation Detection

Information sharing on social media has become a common practice for people around the world. Since it is difficult to check user-generated content on social media, huge amounts of rumors and misinformation are being spread with authentic information. On the one hand, most of the social platforms identify rumors through manual fact-checking, which is very inefficient. On the other hand, with an emerging form of misinformation that contains inconsistent image–text pairs, it would be beneficial if we could compare the meaning of multimodal content within the same post for detecting image–text inconsistency. In this paper, we propose a novel approach to misinformation detection by multimodal feature fusion with transformers and credibility assessment with self-attention-based Bi-RNN networks. Firstly, captions are derived from images using an image captioning module to obtain their semantic descriptions. These are compared with surrounding text by fine-tuning transformers for consistency check in semantics. Then, to further aggregate sentiment features into text representation, we fine-tune a separate transformer for text sentiment classification, where the output is concatenated to augment text embeddings. Finally, Multi-Cell Bi-GRUs with self-attention are used to train the credibility assessment model for misinformation detection. From the experimental results on tweets, the best performance with an accuracy of 0.904 and an F1-score of 0.921 can be obtained when applying feature fusion of augmented embeddings with sentiment classification results. This shows the potential of the innovative way of applying transformers in our proposed approach to misinformation detection. Further investigation is needed to validate the performance on various types of multimodal discrepancies.

Multi-Modal Pose Representations for 6-DOF Object Tracking

Pose estimation methods for robotics should return a distribution of poses rather than just a single pose estimate. Motivated by this, in this work we investigate multi-modal pose representations for reliable 6-DoF object tracking. A neural network architecture for simultaneous object segmentation and estimation of fiducial points of the object on RGB images is proposed. Given a priori probability distribution of object poses a particle filter is employed to estimate the posterior probability distribution of object poses. An advanced observation model relying on matching the projected 3D model with the segmented object and a distance transform-based object representation is used to weight samples representing the probability distribution. Afterwards, the object pose determined by the PnP algorithm is included in the probability distribution via replacing a particle with the smallest weight. Next, a k-means++ algorithm is executed to determine modes in a multi-modal probability distribution. A multi-swarm particle swarm optimization is then executed to determine the finest modes in the probability distribution. A subset of particles for final pose optimization is found in a multi-criteria analysis using the TOPSIS algorithm. They are verified using conflicting criteria that are determined on the basis of object keypoints, segmented object, and the distance transform. On the challenging YCB-Video dataset it outperforms recent algorithms for both object pose estimation and object pose tracking.

Correlation-aware Cross-modal Attention Network for Fashion Compatibility Modeling in UGC Systems

Empowered by the continuous integration of social multimedia and artificial intelligence, the application scenarios of information retrieval (IR) progressively tend to be diversified and personalized. Currently, User-Generated Content (UGC) systems have great potential to handle the interactions between large-scale users and massive media contents. As an emerging multimedia IR, Fashion Compatibility Modeling (FCM) aims to predict the matching degree of each given outfit and provide complementary item recommendation for user queries. Although existing studies attempt to explore the FCM task from a multimodal perspective with promising progress, they still fail to fully leverage the interactions between multimodal information or ignore the item-item contextual connectivities of intra-outfit. In this paper, a novel fashion compatibility modeling scheme is proposed based on Correlation-aware Cross-modal Attention Network. To better tackle these issues, our work mainly focuses on enhancing comprehensive multimodal representations of fashion items by integrating the cross-modal collaborative contents and uncovering the contextual correlations. Since the multimodal information of fashion items can deliver various semantic clues from multiple aspects, a modality-driven collaborative learning module is presented to explicitly model the interactions of modal consistency and complementarity via a co-attention mechanism. Considering the rich connections among numerous items in each outfit as contextual cues, a correlation-aware information aggregation module is further designed to adaptively capture significant intra-correlations of item-item for characterizing the content-aware outfit representations. Experiments conducted on two real-world fashion datasets demonstrate the superiority of our approach over state-of-the-art methods.

Leveraging Contrastive Language–Image Pre-Training and Bidirectional Cross-attention for Multimodal Keyword Spotting

In resource-limited keyword spotting scenarios, the scarcity of annotated corpora hinders deep learning’s ability to develop robust models for representing acoustic features. Recent studies focus on contrastive learning, using paired examples for self-supervision, indicating a growing research interest in keyword spotting. Typically, models are trained on segmented spoken words. However, the absence of distinct word boundaries in lengthy audio poses significant challenges. This study presents a novel approach that integrates Contrastive Language–Image Pre-Training with Cross-Attention for Self-Supervised Alignment in the field of Multimodal Keyword Spotting. The proposed method introduces a cross-modal process for matching word pairs, enhancing collaborative effectiveness between audio and text embeddings, achieving word-level congruence across different modalities. Employing a self-supervised learning approach, we use a restricted number of annotated audio–text pairs to discern semantic congruities and divergences, resulting in improved multimodal feature representation. Using this strategy, we input into subsequent keyword detection tasks, adopting a Bidirectional Cross-Attention block for detailed coordination across modalities, bridging the semantic gap between diverse audio and text representations, consequently improving keyword localization precision. Comprehensive experiments conducted on the Aishell-2 and Librispeech datasets demonstrate that the proposed method markedly surpasses existing techniques, especially regarding the comprehensive performance metric F1 for keyword spotting. It attains state-of-the-art outcomes for both pristine and noisy speech, showing respective enhancements of 5.2% and 5.4% on Aishell-2, as well as 5.4% and 5.6% on Librispeech.

VIEPred: Predicting Viral Immune Evasion with Multimodal Representation Learning

Abstract Predicting coronaviral immune evasion is crucial for identifying and responding to new COVID-19 variants in advance, thereby optimizing vaccine development and public health strategies to prevent further outbreaks. Hence, we present VIEPred, a model designed to predict mutations in every amino acid of the RBD. However, protein representation learning is a challenge task due to the hierarchical structure of proteins. Our method uses a sequence encoder based on dilated convolutions and attention mechanisms to generate sequence embeddings and a structure encoder employing geometric vector perceptrons (GVP) and graph neural networks (GNN) to produce protein structure representations. These features are combined using a cross-attention fusion module and subsequently processed through a fully connected layer to output the probability of viral immune evasion. Extensive ablation studies led to an optimal model with ACC of 88.4% and AUC of 86.8%. Compared to existing prediction methods, VIEPred demonstrates superior predictive performance across most evaluation metrics for both SARS-CoV-2 and influenza viruses.