Multimodal Transformer Research Articles

Diffusion-driven Incomplete Multimodal Learning for Air Quality Prediction

Predicting air quality using multimodal data is crucial to comprehensively capture the diverse factors influencing atmospheric conditions. Therefore, this study introduces a multimodal learning framework that integrates outdoor images with traditional ground-based observations to improve the accuracy and reliability of air quality predictions. However, aligning and fusing these heterogeneous data sources pose a formidable challenge, further exacerbated by pervasive data incompleteness issues in practice. In this paper, we propose a novel incomplete multimodal learning approach (iMMAir) to recovery missing data for robust air quality prediction. Specifically, we first design a shallow feature extractor to capture modal-specific features within the latent representation space. Then we develop a conditional diffusion-driven recovery module to mitigate the distribution gap between the recovered and true data. This module further incorporates two conditional constraints of temporal correlation and semantic consistency for effective modal completion. Finally, we reconstruct incomplete modalities and fuse available data using a multimodal transformer network to predict the air quality. To alleviate the modality imbalance problem, we employ an adaptive gradient modulation strategy to adjust the optimization of each modality. Experimental results demonstrate that iMMAir significantly reduces prediction errors, outperforming baseline models by an average of 5.6% and 2.5% in air quality regression and classification tasks. Our source code and data are available at https://github.com/pestasu/IMMAir.

Multi-Modal Graph Aggregation Transformer for image captioning

The current image captioning directly encodes the detected target area and recognizes the objects in the image to correctly describe the image. However, it is unreliable to make full use of regional features because they cannot convey contextual information, such as the relationship between objects and the lack of object predicate level semantics. An effective model should contain multiple modes and explore their interactions to help understand the image. Therefore, we introduce the Multi-Modal Graph Aggregation Transformer (MMGAT), which uses the information of various image modes to fill this gap. It first represents an image as a graph consisting of three sub-graphs, depicting context grid, region, and semantic text modalities respectively. Then, we introduce three aggregators that guide message passing from one graph to another to exploit context in different modalities, so as to refine the features of nodes. The updated nodes have better features for image captioning. We show significant performance scores of 144.6% CIDEr on MS-COCO and 80.3% CIDEr on Flickr30k compared to state of the arts, and conduct a rigorous analysis to demonstrate the importance of each part of our design.

Semantic-aware frame-event fusion based pattern recognition via large vision–language models

Pattern recognition through the fusion of RGB frames and Event streams has emerged as a novel research area in recent years. Current methods typically employ backbone networks to individually extract the features of RGB frames and event streams, and subsequently fuse these features for pattern recognition. However, we posit that these methods may suffer from two key issues: (1). They attempt to directly learn a mapping from the input vision modality to the semantic labels. This approach often leads to sub-optimal results due to the disparity between the input and semantic labels; (2). They utilize small-scale backbone networks for the extraction of RGB and Event input features, thus these models fail to harness the recent performance advancements of large-scale visual-language models. In this study, we introduce a novel pattern recognition framework that consolidates the semantic labels, RGB frames, and event streams, leveraging pre-trained large-scale vision–language models. Specifically, given the input RGB frames, event streams, and all the predefined semantic labels, we employ a pre-trained large-scale vision model (CLIP vision encoder) to extract the RGB and event features. To handle the semantic labels, we initially convert them into language descriptions through prompt engineering and polish using ChatGPT, and then obtain the semantic features using the pre-trained large-scale language model (CLIP text encoder). Subsequently, we integrate the RGB/Event features and semantic features using multimodal Transformer networks. The resulting frame and event tokens are further amplified using self-attention layers. Concurrently, we propose to enhance the interactions between text tokens and RGB/Event tokens via cross-attention. Finally, we consolidate all three modalities using self-attention and feed-forward layers for recognition. Comprehensive experiments on the HARDVS and PokerEvent datasets fully substantiate the efficacy of our proposed SAFE model. The source code has been released at https://github.com/Event-AHU/SAFE_LargeVLM.

Social Event Classification Based on Multimodal Masked Transformer Network

The key to multimodal social event classification is to fully and accurately utilize the features of both image and text modalities. However, most existing methods have the following limitations: (1) they simply concatenate the image features and text features of the event, and (2) there is irrelevant contextual information between different modalities, which leads to mutual interference. Therefore, it is not enough to only consider the relationship between the modalities of multimodal data, but also the irrelevant contextual information (i.e., regions or words) between the modalities. To overcome these limitations, a novel social event classification method based on multimodal masked transformer network (MMTN) is proposed. A better representation of text and image is learned through an image-text encoding network. Then, the obtained image and text representations are input into the multimodal masked transformer network to fuse the multimodal information, and the relationship between the modalities of multimodal information is modeled by calculating the similarity between the multimodal information, masking the irrelevant context between the modalities. Extensive experiments on two benchmark datasets show that the proposed multimodal masked transformer network model achieves state-of-the-art performance.

Design of multimodal transformable wheels for amphibious robotic vehicles

Amphibious vehicles are versatile for locomotion on both water and land; to reduce the complexities of structure and control, an amphibious vehicle is expected to use a single propulsion system without additional screw propellers or water jets; as vehicles are typically driven by wheels that are efficient for smooth land locomotion, how to make the wheels also as water propellors for multimodal transformations is meaningful, while such wheels are less explored. From this motivation, in this short article, we report the design of a multimodal transformable wheel with three driving modes (wheel mode, forward-propellor mode, and backward-propellor mode) and the integrated mode-locks for each driving mode. To achieve the multimodal transformations of the wheel, we design the integrated mode-transform mechanism and mode-lock mechanism, the two mechanical implementations are driven by only two actuators and are compact with low inertia and low control complexity, with the kinematic properties provided during the wheel transformations.

Paying attention to uncertainty: A stochastic multimodal transformers for post-traumatic stress disorder detection using video

Background and Objectives:Post-traumatic stress disorder is a debilitating psychological condition that can manifest following exposure to traumatic events. It affects individuals from diverse backgrounds and is associated with various symptoms, including intrusive thoughts, nightmares, hyperarousal, and avoidance behaviors. Methods:To address this challenge this study proposes a decision support system powered by a novel multimodal deep learning approach, based on a stochastic Transformer and video data. This Transformer has the ability to take advantage of its stochastic activation function and layers that allow it to learn sparse representations of the inputs. The method leverages a combination of low-level features extracted using three modalities, including Mel-frequency cepstral coefficients extracted from audio recordings, Facial Action Units captured from facial expressions, and textual data obtained from the audio transcription. By considering these modalities, our proposed model captures a comprehensive range of information related to post-traumatic stress disorder symptoms, including vocal cues, facial expressions, and linguistic content. Results:The deep learning model was trained and evaluated on the eDAIC dataset, which consists of clinical interviews with individuals with and without post-traumatic disorder. The model achieved state-of-the-art results, demonstrating its effectiveness in accurately detecting PTSD, showing an impressive Root Mean Square Error of 1.98, and a Concordance Correlation Coefficient of 0.722, signifying the model’s superior performance compared to existing approaches. Conclusion:This work introduces a new method for post-traumatic stress disorder detection from videos by utilizing a multimodal stochastic Transformer model. The model makes use of a variety of modalities, such as text, audio, and visual data, to gather comprehensive and varied information in order to make the detection.

HMTV: hierarchical multimodal transformer for video highlight query on baseball

HMTV: hierarchical multimodal transformer for video highlight query on baseball

Zero-shot visual grounding via coarse-to-fine representation learning

Visual grounding (VG) locates target objects in visual scenes by understanding given natural language queries. Current methods for VG mainly focus on grounding referring expressions or noun phrases covered in the labeled training samples. Despite their grounding prowess, these approaches struggle in grounding novel query–image pairs excluded from the training data. This shortage is usually caused by the deficiency of discriminative representation learning both in images and queries. To address these issues, we propose a one-stage coarse-to-fine framework for zero-shot VG to ground novel query-image samples. Specifically, in the coarse stage, we mine the global context information in the visual features and query embeddings by employing a multi-head self-attention block, strengthening the intra-modality relations in the visual and textual features. In the fine stage, we first learn the query-aware visual representations based on the acquired global context information via a multi-modal relation-enhanced transformer block, which explores the informative information by modeling the cross-modal interaction among visual and textual domains. We further excavate target-oriented discriminative representations from the acquired query-aware visual representations by a noun phrase-guided multi-modal interaction network, which augments the interaction between target-related phrases and the obtained query-aware visual representations to enhance the distinction of target regions, enhancing the subsequent referred target grounding and generalizing. In order to validate the proposed approach, we implement extensive experiments and ablation studies on public benchmark datasets, including RefCOCO, RefCOCO+, RefCOCOg, Flickr30K Entity, Flickr-Split-0 and Flickr-Split-1. Experimental results demonstrate that our approach substantially improves the grounding accuracy and achieves new state-of-the-art performance under single-stage training and testing.

Multi-modal transformer with language modality distillation for early pedestrian action anticipation

Language-vision integration has become an increasingly popular research direction within the computer vision field. In recent years, there has been a growing recognition of the importance of incorporating linguistic information into visual tasks, particularly in domains such as action anticipation. This integration allows anticipation models to leverage textual descriptions to gain deeper contextual understanding, leading to more accurate predictions. In this work, we focus on pedestrian action anticipation, where the objective is the early prediction of pedestrians’ future actions in urban environments. Our method relies on a multi-modal transformer model that encodes past observations and produces predictions at different anticipation times, employing a learned mask technique to filter out redundancy in the observed frames. Instead of relying solely on visual cues extracted from images or videos, we explore the impact of integrating textual information in enriching the input modalities of our pedestrian action anticipation model. We investigate various techniques for generating descriptive captions corresponding to input images, aiming to enhance the anticipation performance. Evaluation results on available public benchmarks demonstrate the effectiveness of our method in improving the prediction performance at different anticipation times compared to previous works. Additionally, incorporating the language modality in our anticipation model proved significant improvement, reaching a 29.5% increase in the F1 score at 1-second anticipation and a 16.66% increase at 4-second anticipation. These results underscore the potential of language-vision integration in advancing pedestrian action anticipation in complex urban environments.

MMCA-NET: A Multimodal Cross Attention Transformer Network for Nasopharyngeal Carcinoma Tumor Segmentation Based on a Total-Body PET/CT System.

Nasopharyngeal carcinoma (NPC) is a malignant tumor primarily treated by radiotherapy. Accurate delineation of the target tumor is essential for improving the effectiveness of radiotherapy. However, the segmentation performance of current models is unsatisfactory due to poor boundaries, large-scale tumor volume variation, and the labor-intensive nature of manual delineation for radiotherapy. In this paper, MMCA-Net, a novel segmentation network for NPC using PET/CT images that incorporates an innovative multimodal cross attention transformer (MCA-Transformer) and a modified U-Net architecture, is introduced to enhance modal fusion by leveraging cross-attention mechanisms between CT and PET data. Our method, tested against ten algorithms via fivefold cross-validation on samples from Sun Yat-sen University Cancer Center and the public HECKTOR dataset, consistently topped all four evaluation metrics with average Dice similarity coefficients of 0.815 and 0.7944, respectively. Furthermore, ablation experiments were conducted to demonstrate the superiority of our method over multiple baseline and variant techniques. The proposed method has promising potential for application in other tasks.

A new multi-modal time series transformation method and multi-scale convolutional attention network for railway wagon bearing fault diagnosis

To align with the evolving trends in intelligent railway wagon operation and maintenance and to enhance the precision of railway wagon bearing fault diagnosis, this paper introduces a novel method for bearing fault diagnosis. The method comprises two key innovations. Firstly, a multi-modal time series transformation method is proposed. This method extracts time series data from the original time domain signals via self-adaptive ensemble empirical mode decomposition with adaptive noise, transforms them into 2D matrices, and captures inter- and intra-period information relationships through convolution. Secondly, a multi-scale convolutional attention network is introduced, enriching fault information by utilizing parallel multi-scale convolution for down-sampling. To prevent feature loss, sliding convolution is adopted instead of pooling. Additionally, the model incorporates the convolutional block attention module to focus on critical information. Experimental validation conducted in a laboratory using a self-developed railway wagon bearing dynamic performance tester demonstrates high diagnostic accuracy and strong overall performance. The method’s generalizability is further confirmed through validation using publicly available datasets. This method could find practical use in railway maintenance, improving the accuracy of bearing fault diagnosis, and making operations more efficient.

Multimodal Artificial Intelligence in Medicine.

Traditional medical Artificial Intelligence models, approved for clinical use, restrict themselves to single-modal data e.g. images only, limiting their applicability in the complex, multimodal environment of medical diagnosis and treatment. Multimodal Transformer Models in healthcare can effectively process and interpret diverse data forms such as text, images, and structured data. They have demonstrated impressive performance on standard benchmarks like USLME question banks and continue to improve with scale. However, the adoption of these advanced AI models is not without challenges. While multimodal deep learning models like Transformers offer promising advancements in healthcare, their integration requires careful consideration of the accompanying ethical and environmental challenges.

Multi-modal transformer architecture for medical image analysis and automated report generation

Medical practitioners examine medical images, such as X-rays, write reports based on the findings, and provide conclusive statements. Manual interpretation of the results and report generation by examiners are time-consuming processes that lead to potential delays in diagnosis. We propose an automated report generation model for medical images leveraging an encoder–decoder architecture. Our model utilizes transformer architectures, including Vision Transformer (ViT) and its variants like Data Efficient Image Transformer (DEiT) and BERT pre-training image transformer (BEiT), as an encoder. These transformers are adapted for processing to extract and gain visual information from medical images. Reports are transformed into text embeddings, and the Generative Pre-trained Transformer (GPT2) model is used as a decoder to generate medical reports. Our model utilizes a cross-attention mechanism between the vision transformer and GPT2, which enables it to create detailed and coherent medical reports based on the visual information extracted by the encoder. In our model, we have extended the report generation with general knowledge, which is independent of the inputs and provides a comprehensive report in a broad sense. We conduct our experiments on the Indiana University X-ray dataset to demonstrate the effectiveness of our models. Generated medical reports from the model are evaluated using word overlap metrics such as Bleu scores, Rouge-L, retrieval augmentation answer correctness, and similarity metrics such as skip thought cs, greedy matching, vector extrema, and RAG answer similarity. Results show that our model is performing better than the recurrent models in terms of report generation, answer similarity, and word overlap metrics. By automating the report generation process and incorporating advanced transformer architectures and general knowledge, our approach has the potential to significantly improve the efficiency and accuracy of medical image analysis and report generation.

Automatic movie genre classification & emotion recognition via a BiProjection Multimodal Transformer

Analyzing, manipulating, and comprehending data from multiple sources (e.g., websites, software applications, files, or databases) and of diverse modalities (e.g., video, images, audio and text) has become increasingly important in many domains. Despite recent advances in multimodal classification (MC), there are still several challenges to be addressed, such as: the combination of modalities of very diverse nature, the optimal feature engineering for each modality, as well as the semantic alignment between text and images. Accordingly, the main motivation of our research relies in devising a neural architecture that effectively processes and combines text, image, video and audio modalities, so it can offer a noteworthy performance in different MC tasks. In this regard, the Multimodal Transformer (MulT) model is a cutting-edge approach often employed in multimodal supervised tasks, which, although effective, has the problem of having a fixed architecture that limits its performance in specific tasks as well as its contextual understanding, meaning it may struggle to capture fine-grained temporal patterns in audio or effectively model spatial relationships in images. To address these issues, our research modifies and extends the MulT model in several aspects. Firstly, we focus on leveraging the Gated Multimodal Unit (GMU) module within the architecture to efficiently and dynamically weigh modalities at the instance level and to visualize the use of modalities. Secondly, to overcome the problem of vanishing and exploding gradients we focus on strategically placing residual connections in the architecture. The proposed architecture is evaluated in two different and complex classification tasks, on the one hand, the movie genre categorization (MGC) and, on the other hand, the multimodal emotion recognition (MER). The results obtained are encouraging as they indicate that the proposed architecture is competitive against state-of-the-art (SOTA) models in MGC, outperforming them by up to 2% on the Moviescope dataset, and by 1% on the MM-IMDB datasets. Furthermore, in the MER task the unaligned version of the datasets was employed, which is considerably more difficult; we improve accuracy SOTA results by up to 1% on the IEMOCAP dataset, and attained a competitive outcome on the CMU-MOSEI11Dai et al., (2021) collection, outperforming SOTA results in several emotions.

Adding compact parameter blocks to multimodal transformers to detect harmful memes

Harmful content serves as a potent tool for spreading destructive ideologies and creating divisions among people, often exploiting factors like race, religion, and socioeconomic status. Social media platforms are especially conducive to such content, mainly through memes. Hence, the development of tools that can detect and eliminate harmful memes before they reach a wide audience and cause harm is a very important issue. However, this task represents a critical challenge, as memes can take various forms, typically combining images with text. Recent research has focused on building multimodal machine learning models, especially transformers-based models, to uncover harmful content within memes. Although these models have shown effectiveness, they often require substantial computational resources for training and incorporate additional information besides the textual and visual components of the memes, making them complex and challenging to reproduce. This study introduces a novel approach to enhance the performance of established multimodal transformer models without needing extra information. It works by adding a specific module called Compact Parameter Blocks (CPB) into the encoder segments of these models. This module enhances the quality of input data, leading to improvements in the model’s attention mechanism and in its overall regularization. This results in better performance across all relevant metrics. In this paper, we use this approach with two variations of multimodal models — VisualBERT (Visual Bidirectional Encoder Representations from Transformers) and VilBERT (Vision and Language Bidirectional Encoder Representations from Transformers) — both built on canonical transformers to identify harmful content in memes across four different datasets. The experimental results demonstrated that this method outperforms several robust and complex approaches.

Interactive Multi-scale Fusion: Advancing Brain Tumor Detection Through Trans-IMSM Model.

Multi-modal medical image (MI) fusion assists in generating collaboration images collecting complement features through the distinct images of several conditions. The images help physicians to diagnose disease accurately. Hence, this research proposes a novel multi-modal MI fusion modal named guided filter-based interactive multi-scale and multi-modal transformer (Trans-IMSM) fusion approach to develop high-quality computed tomography-magnetic resonance imaging (CT-MRI) fused images for brain tumor detection. This research utilizes the CT and MRI brain scan dataset to gather the input CT and MRI images. At first, the data preprocessing is carried out to preprocess these input images to improve the image quality and generalization ability for further analysis. Then, these preprocessed CT and MRI are decomposed into detail and base components utilizing the guided filter-based MI decomposition approach. This approach involves two phases: such as acquiring the image guidance and decomposing the images utilizing the guided filter. A canny operator is employed to acquire the image guidance comprising robust edge for CT and MRI images, and the guided filter is applied to decompose the guidance and preprocessed images. Then, by applying the Trans-IMSM model, fuse the detail components, while a weighting approach is used for the base components. The fused detail and base components are subsequently processed through a gated fusion and reconstruction network, and the final fused images for brain tumor detection are generated. Extensive tests are carried out to compute the Trans-IMSM method's efficacy. The evaluation results demonstrated the robustness and effectiveness, achieving an accuracy of 98.64% and an SSIM of 0.94.

Detecting and Grounding Multi-Modal Media Manipulation and Beyond.

Misinformation has become a pressing issue. Fake media, in both visual and textual forms, is widespread on the web. While various DeepFake detection and text fake news detection methods have been proposed, they are only designed for single-modality forgery based on binary classification, let alone analyzing and reasoning subtle forgery traces across different modalities. In this paper, we highlight a new research problem for multi-modal fake media, namely Detecting and Grounding Multi-Modal Media Manipulation (DGM 4). DGM 4 aims to not only detect the authenticity of multi-modal media, but also ground the manipulated content (i.e., image bounding boxes and text tokens), which requires deeper reasoning of multi-modal media manipulation. To support a large-scale investigation, we construct the first DGM 4 dataset, where image-text pairs are manipulated by various approaches, with rich annotation of diverse manipulations. Moreover, we propose a novel HierArchical Multi-modal Manipulation rEasoning tRansformer (HAMMER) to fully capture the fine-grained interaction between different modalities. HAMMER performs: 1) manipulation-aware contrastive learning between two uni-modal encoders as shallow manipulation reasoning and 2) modality-aware cross-attention by multi-modal aggregator as deep manipulation reasoning. Dedicated manipulation detection and grounding heads are integrated from shallow to deep levels based on the interacted multi-modal information. To exploit more fine-grained contrastive learning for cross-modal semantic alignment, we further integrate Manipulation-Aware Contrastive Loss with Local View and construct a more advanced model HAMMER++. Finally, we build an extensive benchmark and set up rigorous evaluation metrics for this new research problem. Comprehensive experiments demonstrate the superiority of HAMMER and HAMMER++; several valuable observations are also revealed to facilitate future research in multi-modal media manipulation.

Free Vibration analysis of C/SiC blisk based on modified global mode method

The bladed disk is the core component which is under load in aero-engine, rocket engine and gas turbine. In recent years, the bladed disk has developed towards the direction of integrated bladed disk (blisk) and being applied with ceramic matrix composites. However, there is no accurate semi-analytical dynamic model to describe the dynamic characteristic of ceramic matrix composite blisk. In this paper, a new semi-analytical method, modified global mode method (MGMM) is proposed to model the 2D C/SiC laminated blisk. In proposed method, Chebyshev polynomials series are used to expand the displacements of the blades and disk, constraints between blades and disk is strictly satisfied by multi-modal transformation and integrated into governing equation, and high-order shear deformation theory is combined to establish the dynamic model of the blisk of 2D C/SiC laminated composite material. The proposed method avoids the matrix singular problem appearing in traditional global mode method when modeling of combined structure and makes the system perform dynamic analysis and mode prediction without mode extraction and reconstruction. Then, amplitude frequency response and modal experiment are carried out to verify the correctness and convergence of the proposed method. Finally, under the framework of proposed method, the effects of material parameters and geometric parameters on the modal characteristics of the blisk are analyzed. The results show that compared with the geometric parameters, the material parameters have less influence on the modal characteristics of blisk, additionally, a series of modal steering is observed. The work in this paper can provide theoretical guidance for the dynamic design of composite blisk.

Cognitive digital twin-enabled multi-robot collaborative manufacturing: Framework and approaches

Multi-robot collaborative manufacturing (MRCMfg) is vital to ensure effective and efficient operation in the manufacturing industry. In recent years, enabled by the new generation of information technology, represented by the digital twin (DT), the digitalization of MRCMfg has progressed steadily. However, more complex manufacturing tasks and more customized manufacturing demands pose challenges to MRCMfg, which limit the applicability of MRCMfg in practice. Recent successes in intelligent manufacturing show that empowering DT with the intelligence and cognition capabilities to adapt to the above challenges is a promising approach. Inspired by that, a cognitive digital twin (CDT) framework with multiple artificial intelligence algorithms for MRCMfg is proposed. The framework consists of three critical spaces: (i) Physical-virtual space: multi-agent system-based DT; (ii) Cognition space: multi-behavior tree (BT) cognition models; (iii) Data space: data-driven cognition evolution. In the data space, multimodal perception is a important way to improve the cognition capabilities of MRCMfg. To alleviate the multimodal data quality requirements as well as to improve the robustness of the perception model, an incomplete multimodal Transformer with deep autoencoder (IMTDAE) approach is proposed. The effectiveness and feasibility of this framework and IMTDAE are verified by case studies. The results provide compelling evidence that this framework has the potential to significantly enhance the intelligence and cognition capabilities of MRCMfg.

Can physician judgment enhance model trustworthiness? A case study on predicting pathological lymph nodes in rectal cancer

Explainability is key to enhancing the trustworthiness of artificial intelligence in medicine. However, there exists a significant gap between physicians’ expectations for model explainability and the actual behavior of these models. This gap arises from the absence of a consensus on a physician-centered evaluation framework, which is needed to quantitatively assess the practical benefits that effective explainability should offer practitioners. Here, we hypothesize that superior attention maps, as a mechanism of model explanation, should align with the information that physicians focus on, potentially reducing prediction uncertainty and increasing model reliability. We employed a multimodal transformer to predict lymph node metastasis of rectal cancer using clinical data and magnetic resonance imaging. We explored how well attention maps, visualized through a state-of-the-art technique, can achieve agreement with physician understanding. Subsequently, we compared two distinct approaches for estimating uncertainty: a standalone estimation using only the variance of prediction probability, and a human-in-the-loop estimation that considers both the variance of prediction probability and the quantified agreement. Our findings revealed no significant advantage of the human-in-the-loop approach over the standalone one. In conclusion, this case study did not confirm the anticipated benefit of the explanation in enhancing model reliability. Superficial explanations could do more harm than good by misleading physicians into relying on uncertain predictions, suggesting that the current state of attention mechanisms should not be overestimated in the context of model explainability.