Multimodal Transformer Research Articles

Video–text retrieval via multi-modal masked transformer and adaptive attribute-aware graph convolutional network

Video–text retrieval via multi-modal masked transformer and adaptive attribute-aware graph convolutional network

Integrating GIN-based multimodal feature transformation and multi-feature combination voting for irony-aware cyberbullying detection

With the increasing diversity of expressions, irony-aware cyberbullying has emerged as a significant issue in online social networks. However, detecting irony-aware cyberbullying is challenging, as it requires a comprehensive understanding of context and external factors beyond literal meanings. To take full advantage of multiple features of multimodal data to detect challenging irony-aware cyberbullying, we propose an integration framework (GINBV_MFCV). The multimodal feature construction method with Graph Isomorphism Network (GIN) feature transformation (GINBV) leverages the message passing and aggregation operations of GIN to extract the potential representations of text-image features, which enriches the structural information of multimodal data. In addition, the multi-feature combination voting strategy (MFCV) soft-votes the prediction results of constructed multimodal features and multiple combinations of GIN, Bidirectional Encoder Representations from Transformers (BERT), and Vision Transformers (ViT) embedded features to reduce the data structure information bias, which has a positive effect on irony-aware cyberbullying detection. Experimental results on a real-world dataset from Weibo demonstrate that GINBV_MFCV achieves an F1-score of 83.29% and an AUC of 91.21% in irony-aware cyberbullying detection, improving 8.65% and 15.73% over the baseline algorithm, respectively. These promising results confirm the potential of GINBV_MFCV for detecting irony-aware cyberbullying.

Accurate estimation of biological age and its application in disease prediction using a multimodal image Transformer system.

Aging in an individual refers to the temporal change, mostly decline, in the body's ability to meet physiological demands. Biological age (BA) is a biomarker of chronological aging and can be used to stratify populations to predict certain age-related chronic diseases. BA can be predicted from biomedical features such as brain MRI, retinal, or facial images, but the inherent heterogeneity in the aging process limits the usefulness of BA predicted from individual body systems. In this paper, we developed a multimodal Transformer-based architecture with cross-attention which was able to combine facial, tongue, and retinal images to estimate BA. We trained our model using facial, tongue, and retinal images from 11,223 healthy subjects and demonstrated that using a fusion of the three image modalities achieved the most accurate BA predictions. We validated our approach on a test population of 2,840 individuals with six chronic diseases and obtained significant difference between chronological age and BA (AgeDiff) than that of healthy subjects. We showed that AgeDiff has the potential to be utilized as a standalone biomarker or conjunctively alongside other known factors for risk stratification and progression prediction of chronic diseases. Our results therefore highlight the feasibility of using multimodal images to estimate and interrogate the aging process.

Multimodal Transformer of Incomplete MRI Data for Brain Tumor Segmentation.

Accurate segmentation of brain tumors plays an important role for clinical diagnosis and treatment. Multimodal magnetic resonance imaging (MRI) can provide rich and complementary information for accurate brain tumor segmentation. However, some modalities may be absent in clinical practice. It is still challenging to integrate the incomplete multimodal MRI data for accurate segmentation of brain tumors. In this paper, we propose a brain tumor segmentation method based on multimodal transformer network with incomplete multimodal MRI data. The network is based on U-Net architecture consisting of modality specific encoders, multimodal transformer and multimodal shared-weight decoder. First, a convolutional encoder is built to extract the specific features of each modality. Then, a multimodal transformer is proposed to model the correlations of multimodal features and learn the features of missing modalities. Finally, a multimodal shared-weight decoder is proposed to progressively aggregate the multimodal and multi-level features with spatial and channel self-attention modules for brain tumor segmentation. A missing-full complementary learning strategy is used to explore the latent correlation between the missing and full modalities for feature compensation. For evaluation, our method is tested on the multimodal MRI data from BraTS 2018, BraTS 2019 and BraTS 2020 datasets. The extensive results demonstrate that our method outperforms the state-of-the-art methods for brain tumor segmentation on most subsets of missing modalities.

Positive Unlabeled Fake News Detection via Multi-Modal Masked Transformer Network

Fake news detection has gotten continuous attention during these years with more and more people have been posting and reading news online. To enable fake news detection, existing researchers usually assume labeled posts are provided for two classes (true or false) so that the model can learn a discriminative classifier from the labeled data. However, this supposition may not hold true in reality, as most users may only label a small number of posts in a single category that they are interested in. Furthermore, most existing methods fail to mask the noise or irrelevant context (i.e., regions or words) between different modalities to assist in strengthening the correlations between relevant contexts. To tackle these issues, we present a curriculum-based multi-modal masked transformer network (CMMTN) for positive unlabeled multi-modal fake news detection by jointly modeling the inter-modality and intra-modality relationships of multi-modal information and masking the irrelevant context between modalities. In particular, we adopt BERT and ResNet to obtain better representations for texts and images, separately. Then, the extracted features of images and texts are fed into a multi-modal masked transformer network to fuse the multi-modal content and mask the irrelevant context between modalities by calculating the similarity between inter-modal contexts. Finally, we design a curriculum-based PU learning method to handle the positive and unlabeled data. Massive experiments on three public real datasets prove the effectiveness of the CMMTN.

Multi-Modal Structure-Embedding Graph Transformer for Visual Commonsense Reasoning

Visual commonsense reasoning (VCR) is a challenging reasoning task that aims to not only answer the question based on a given image but also provide a rationale justifying for the choice. Graph-based networks are appropriate to represent and extract the correlation between image and language for reasoning, where how to construct and learn graphs based on such multi-modal Euclidean data is a fundamental problem. Most existing graph-based methods view visual regions and linguistic words as identical graph nodes, ignoring inherent characteristics of multi-modal data. In addition, these approaches typically only have one graph-learning layer, and the performance declines as the model goes deeper. To address these issues, a novel method named Multi-modal Structure-embedding Graph Transformer (MSGT) is proposed. Specifically, an answer-vision graph and an answer-question graph are constructed to represent and model intra-modal and inter-modal correlations in VCR simultaneously, where additional multi-modal structure representations are initialized and embedded according to visual region distances and linguistic word orders for more reasonable graph representation. Then, a structure-injecting graph transformer is designed to inject embedded structure priors into the semantic correlation matrix for the evolution of node features and structure representations, which can stack more layers to make model deeper and extract more powerful features with instructive priors. To adaptively fuse graph features, a scored pooling mechanism is further developed to select valuable clues for reasoning from learnt node features. Experiments demonstrate the superiority of the proposed MSGT framework compared with state-of-the-art methods on the VCR benchmark dataset. The source code of this work can be found in <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://mic.tongji.edu.cn</uri> .

Strategies for Multimodal Image Data Transformation to a Common Format for Cloud Integration and Visualization

Strategies for Multimodal Image Data Transformation to a Common Format for Cloud Integration and Visualization

CAM-Vtrans: real-time sports training utilizing multi-modal robot data.

Assistive robots and human-robot interaction have become integral parts of sports training. However, existing methods often fail to provide real-time and accurate feedback, and they often lack integration of comprehensive multi-modal data. To address these issues, we propose a groundbreaking and innovative approach: CAM-Vtrans-Cross-Attention Multi-modal Visual Transformer. By leveraging the strengths of state-of-the-art techniques such as Visual Transformers (ViT) and models like CLIP, along with cross-attention mechanisms, CAM-Vtrans harnesses the power of visual and textual information to provide athletes with highly accurate and timely feedback. Through the utilization of multi-modal robot data, CAM-Vtrans offers valuable assistance, enabling athletes to optimize their performance while minimizing potential injury risks. This novel approach represents a significant advancement in the field, offering an innovative solution to overcome the limitations of existing methods and enhance the precision and efficiency of sports training programs.

Local Climate Zone Classification via Semi-Supervised Multimodal Multiscale Transformer

Local Climate Zone Classification via Semi-Supervised Multimodal Multiscale Transformer

Integrating Neural-Symbolic Reasoning With Variational Causal Inference Network for Explanatory Visual Question Answering.

Recently, a novel multimodal reasoning task named Explanatory Visual Question Answering (EVQA) has been introduced, which combines answering visual questions with multimodal explanation generation to expound upon the underlying reasoning processes. In contrast to conventional Visual Question Answering (VQA) that merely concentrates on providing answers, EVQA aims to improve the explainability and verifiability of reasoning by providing user-friendly explanations. Despite the improved explainability of inferred results, the existing EVQA models still adopt black-box neural networks to infer results, lacking the explainability of the reasoning process. Moreover, existing EVQA models commonly predict answers and explanations in isolation, overlooking the inherent causal correlation between them. To handle these challenges, we propose a Program-guided Variational Causal Inference Network (Pro-VCIN) that integrates neural-symbolic reasoning with variational causal inference and constructs causal correlations between the predicted answers and explanations. First, we utilize pretrained models to extract visual features and convert questions into the corresponding programs. Secondly, we propose a multimodal program Transformer to translate programs and the related visual features into coherent and rational explanations of the reasoning processes Finally, we propose a variational causal inference to construct the target structural causal model and predict answers based on the causal correlation to explanations. Comprehensive experiments conducted on EVQA benchmark datasets reveal the superiority of Pro-VCIN in terms of both performance and explainability over state-of-the-art EVQA methods.

MBGT: Encoding Brain Signals With Multimodal Brain Graph Transformer

mBGT: Encoding Brain Signals With Multimodal Brain Graph Transformer

Learning to Answer Visual Questions from Web Videos.

Recent methods for visual question answering rely on large-scale annotated datasets. Manual annotation of questions and answers for videos, however, is tedious, expensive and prevents scalability. In this work, we propose to avoid manual annotation and generate a large-scale training dataset for video question answering making use of automatic cross-modal supervision. We leverage a question generation transformer trained on text data and use it to generate question-answer pairs from transcribed video narrations. Given narrated videos, we then automatically generate the HowToVQA69M dataset with 69M video-question-answer triplets. To handle the open vocabulary of diverse answers in this dataset, we propose a training procedure based on a contrastive loss between a video-question multi-modal transformer and an answer transformer. We introduce the zero-shot VideoQA task and the VideoQA feature probe evaluation setting and show excellent results. Furthermore, our method achieves competitive results on MSRVTT-QA, ActivityNet-QA, MSVD-QA and How2QA datasets. We also show that our approach generalizes to another source of web video and text data. We generate the WebVidVQA3M dataset from videos with alt-text annotations, and show its benefits for training VideoQA models. Finally, for a detailed evaluation we introduce iVQA, a new VideoQA dataset with reduced language bias and high-quality manual annotations.

RemixFormer++: A Multi-modal Transformer Model for Precision Skin Tumor Differential Diagnosis with Memory-efficient Attention.

Diagnosing malignant skin tumors accurately at an early stage can be challenging due to ambiguous and even confusing visual characteristics displayed by various categories of skin tumors. To improve diagnosis precision, all available clinical data from multiple sources, particularly clinical images, dermoscopy images, and medical history, could be considered. Aligning with clinical practice, we propose a novel Transformer model, named Remix-Former++ that consists of a clinical image branch, a dermoscopy image branch, and a metadata branch. Given the unique characteristics inherent in clinical and dermoscopy images, specialized attention strategies are adopted for each type. Clinical images are processed through a top-down architecture, capturing both localized lesion details and global contextual information. Conversely, dermoscopy images undergo a bottom-up processing with two-level hierarchical encoders, designed to pinpoint fine-grained structural and textural features. A dedicated metadata branch seamlessly integrates non-visual information by encoding relevant patient data. Fusing features from three branches substantially boosts disease classification accuracy. RemixFormer++ demonstrates exceptional performance on four single-modality datasets (PAD-UFES-20, ISIC 2017/2018/2019). Compared with the previous best method using a public multi-modal Derm7pt dataset, we achieved an absolute 5.3% increase in averaged F1 and 1.2% in accuracy for the classification of five skin tumors. Furthermore, using a large-scale in-house dataset of 10,351 patients with the twelve most common skin tumors, our method obtained an overall classification accuracy of 92.6%. These promising results, on par or better with the performance of 191 dermatologists through a comprehensive reader study, evidently imply the potential clinical usability of our method.

An Improved ConvNeXt with Multimodal Transformer for Physiological Signal Classification

An Improved ConvNeXt with Multimodal Transformer for Physiological Signal Classification

MMSFormer: Multimodal Transformer for Material and Semantic Segmentation

MMSFormer: Multimodal Transformer for Material and Semantic Segmentation

Exploring Multi-modal Spatial-Temporal Contexts for High-performance RGB-T Tracking.

In RGB-T tracking, there exist rich spatial relationships between the target and backgrounds within multi-modal data as well as sound consistencies of spatial relationships among successive frames, which are crucial for boosting the tracking performance. However, most existing RGB-T trackers overlook such multi-modal spatial relationships and temporal consistencies within RGB-T videos, hindering them from robust tracking and practical applications in complex scenarios. In this paper, we propose a novel Multi-modal Spatial-Temporal Context (MMSTC) network for RGB-T tracking, which employs a Transformer architecture for the construction of reliable multi-modal spatial context information and the effective propagation of temporal context information. Specifically, a Multi-modal Transformer Encoder (MMTE) is designed to achieve the encoding of reliable multi-modal spatial contexts as well as the fusion of multi-modal features. Furthermore, a Quality-aware Transformer Decoder (QATD) is proposed to effectively propagate the tracking cues from historical frames to the current frame, which facilitates the object searching process. Moreover, the proposed MMSTC network can be easily extended to various tracking frameworks. New state-of-the-art results on five prevalent RGB-T tracking benchmarks demonstrate the superiorities of our proposed trackers over existing ones.

Efficient Multimodal Transformer With Dual-Level Feature Restoration for Robust Multimodal Sentiment Analysis

With the proliferation of user-generated online videos, Multimodal Sentiment Analysis (MSA) has attracted increasing attention recently. Despite significant progress, there are still two major challenges on the way towards robust MSA: 1) inefficiency when modeling cross-modal interactions in unaligned multimodal data; and 2) vulnerability to random modality feature missing which typically occurs in realistic settings. In this paper, we propose a generic and unified framework to address them, named Efficient Multimodal Transformer with Dual-Level Feature Restoration (EMT-DLFR). Concretely, EMT employs utterance-level representations from each modality as the global multimodal context to interact with local unimodal features and mutually promote each other. It not only avoids the quadratic scaling cost of previous local-local cross-modal interaction methods but also leads to better performance. To improve model robustness in the incomplete modality setting, on the one hand, DLFR performs low-level feature reconstruction to implicitly encourage the model to learn semantic information from incomplete data. On the other hand, it innovatively regards complete and incomplete data as two different views of one sample and utilizes siamese representation learning to explicitly attract their high-level representations. Comprehensive experiments on three popular datasets demonstrate that our method achieves superior performance in both complete and incomplete modality settings.

Multimodal transformer with adaptive modality weighting for multimodal sentiment analysis

Multimodal Sentiment Analysis (MSA) constitutes a pivotal technology in the realm of multimedia research. The efficacy of MSA models largely hinges on the quality of multimodal fusion. Notably, when conveying information pertinent to specific tasks or applications, not all modalities hold equal importance. Previous research, however, has either disregarded the importance of modalities altogether or solely focused on the importance of linguistic and non-linguistic modalities while neglecting the importance between non-linguistic modalities. To facilitate effective multimodal information fusion based on the relative importance of modalities, a novel multimodal fusion mode named Multimodal Transformer with Adaptive Modality Weighting (MTAMW) is proposed in this paper. Specifically, we introduce a multimodal adaptive weight matrix that allocates appropriate weights to each modality based on its contribution to sentiment analysis. Furthermore, a multimodal attention mechanism is introduced, utilizing multiple Softmax functions to compute attention weights, thereby efficiently fusion multimodal information via a single-stream Transformer. By meticulously considering the relative importance of each modality during the fusion process, more effective multimodal information fusion is achievable. Extensive experiments on benchmark datasets show that it is superior to or comparable to state-of-the-art methods on MSA tasks. The codes for our experiments are available at https://github.com/Vamos66/MTAMW.

Network security situation assessment and prediction method based on multimodal transformation in edge computing

Network security play a critical role in ensuring the robustness and integrity of network systems. We propose a novel method based on multimodal transformation in edge computing for accurate and reliable data communication. The method leverages Graph Convolutional Networks (GCNs) to capture and analyze the complex relationships and dependencies among network entities, enabling enhanced prediction capabilities. By integrating multimodal transformation, diverse data sources are fused in the model. The assessments and comprehensive comparisons with established algorithms unequivocally establish the supremacy of our proposed approach, particularly with respect to metrics such as accuracy, precision, recall, F1-score, and the area under the receiver operating characteristic curve (AUC-ROC). Our method exhibits promising performance, enabling proactive threat detection and mitigation in network security by more then 90% accuracy. In the future work, we have included the different multimodal transformation techniques, addressing interoperability challenges, scaling the method for large-scale networks, and adapting the model to dynamic network environments.

Dual-adaptive interactive transformer with textual and visual context for image captioning

The multimodal Transformer, which integrates visual and textual contextual information, has recently shown success in image captioning tasks. However, there is still natural complementarity and redundancy between text and vision, and effectively integrating the information from both modalities is crucial for comprehending the content of an image. In this paper, we propose the Dual-Adaptive Interactive Transformer (DAIT), which incorporates similar textual and visual contextual information into the encoding and decoding stages. Specifically, during encoding, we propose the Adaptive Interactive Encoder (AIE), which expands the feature vectors for both modalities through the introduction of new operations. Furthermore, we also introduce normalization gate factors to mitigate noise caused by the interaction between the two modalities. During decoding, we propose the Adaptive Interactive Decoder (AID), which adaptively adjusts the multimodal features at each moment through similarity-weighted textual and visual branches. To validate our model, we conducted extensive experiments on the MS COCO image captioning dataset and achieved outstanding performance compared to many state-of-the-art methods.