Cross-modal Attention Research Articles

Multimodal registration across 3D point clouds and CT-volumes

Multimodal registration is a challenging problem in visual computing, commonly faced during medical image-guided interventions, data fusion and 3D object retrieval. The main challenge of multimodal registration is finding accurate correspondence between modalities, since different modalities do not exhibit the same characteristics. This paper explores how the coherence of different modalities can be utilized for the challenging task of 3D multimodal registration. A novel deep learning multimodal registration framework is proposed by introducing a siamese deep learning architecture, especially designed for aligning and fusing modalities of different structural and physical principles. The cross-modal attention blocks lead the network to establish correspondences between features of different modalities. The proposed framework focuses on the alignment of 3D point clouds and the micro-CT 3D volumes of the same object. A multimodal dataset consisting of real micro-CT scans and their synthetically generated 3D models (point clouds) is presented and utilized for evaluating our methodology.

Efficient dual attention SlowFast networks for video action recognition

Video data mainly differ in temporal dimension compared with static image data. Various video action recognition networks choose two-stream models to learn spatial and temporal information separately and fuse them to further improve performance. We proposed a cross-modality dual attention fusion module named CMDA to explicitly exchange spatial–temporal information between two pathways in two-stream SlowFast networks. Besides, considering the computational complexity of these heavy models and the low accuracy of existing lightweight models, we proposed several two-stream efficient SlowFast networks based on well-designed efficient 2D networks, such as GhostNet, ShuffleNetV2 and so on. Experiments demonstrate that our proposed fusion model CMDA improves the performance of SlowFast, and our efficient two-stream models achieve a consistent increase in accuracy with a little overhead in FLOPs. Our code and pre-trained models will be made available at https://github.com/weidafeng/Efficient-SlowFast.

SSLNet: A network for cross-modal sound source localization in visual scenes

Sound source localization in visual scenes is to associate sounds and their visual producers. Although great progress has been made in this field, the mixed sounds from multiple objects make it intractable to perform efficient localization in the unconstrained scenarios. In this paper, we propose a novel cross-modal sound source localization networks (SSLNet), which is accessible to dispose audio and visual information. First, to encode spatial and temporal information in the spectrogram, we combine convolutional neural networks (CNN) and bidirectional long short-term memory networks (BiLSTM). For different time-step inputs of BiLSTMs, we propose a novel operation, called as grouped global average pooling (GGAP), to divide the 3-D spectrogram feature block into several 1-D vectors. Then, a cross-modal channel attention mechanism is introduced to alleviate the discrepancy of different modalities. Finally, for achieving the pixel-level localization, we propose a fusion approach using cosine and L2 distances to measure the similarity between audio and visual vectors. We implement extensive experiments in benchmark and FAIR-Play datasets. The qualitative results demonstrate the effectiveness of sound source localization. On benchmark dataset, the quantitative results show that SSLNet can improve the localization accuracy by 1.2% in consensus intersection over union (cIoU) and 16.4% in area under the curve (AUC). On FAIR-Play dataset, SSLNet also achieves superior performance with Pearson’s correlation coefficient (CC) of 0.779 and similarity metric (SIM) of 0.650.

A Method Based on Dual Cross-Modal Attention and Parameter Sharing for Polyphonic Sound Event Localization and Detection

Sound event localization and detection (SELD) is a joint task that unifies sound event detection (SED) and direction-of-arrival estimation (DOAE). The task has become such a popular topic that it was introduced into the challenge on Detection and Classification of Acoustic Scenes and Events (DCASE) Task3 in 2019. In this paper, we propose a method based on dual cross-modal attention (DCMA) and parameter sharing to simultaneously detect and localize sound events. In particular, the DCMA-based decoder commonly used for multiple predictions efficiently learns the associations between SED and DOAE features by exchanging SED and DOAE information in the process of attention, in addition to the encoder with parameter sharing. Furthermore, acoustic features that have not been usually used in the SELD task are additionally adopted to improve the performance, and data augmentation techniques of the mixup to simulate polyphonic events and channel rotation for spatial augmentation are conducted for this task. Experimental results demonstrate that our efficient model using one common decoder block based on the DCMA to predict multiple events in the track-wise output format is effective for the SELD task with up to three overlapping events.

Multimodality Alzheimer's Disease Analysis in Deep Riemannian Manifold

Structural Magnetic Resonance Image (sMRI) and functional MRI (fMRI) are two of the most important modalities to unveil brain disorders for Alzheimer's disease (AD) analysis. Comprehensively utilizing both modalities is the way to ensure an accurate AD diagnosis. Currently, the most common computational approach to aid the AD diagnosis is to formulate the sMRI and fMRI into graphs and then extract discriminative features through Graph Neural Networks (GNNs). However, most GNNs rely heavily on the aggregation operation on each node, which exploits the local topological information from the neighborhood nodes but does not fully respect the characteristics of the global graph topology. Also, only a few works addressed the structural and functional coupling problem on the graphs. In this paper, a novel Riemannian manifold-based model, called Cross-Modal Riemannian Network (CMRN), is proposed to solve the above issues, which respects the global topologies and invariant characteristics of the sMRI and fMRI graphs by fully operating on the Riemannian Manifold. Furthermore, a novel cross-modal attention mechanism is proposed to enable the interactions between two modalities on the Riemannian manifold, which helps the model comprehensively utilize both modalities to identify the most discriminative information for AD diagnosis. Extensive experimental results on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset demonstrate the effectiveness of the proposed method.

AI Ekphrasis: Multi-Modal Learning with Foundation Models for Fine-Grained Poetry Retrieval

Artificial intelligence research in natural language processing in the context of poetry struggles with the recognition of holistic content such as poetic symbolism, metaphor, and other fine-grained attributes. Given these challenges, multi-modal image–poetry reasoning and retrieval remain largely unexplored. Our recent accessibility study indicates that poetry is an effective medium to convey visual artwork attributes for improved artwork appreciation of people with visual impairments. We, therefore, introduce a deep learning approach for the automatic retrieval of poetry suitable to the input images. The recent state-of-the-art CLIP provides a way for multi-modal visual and text features matched using cosine similarity. However, it lacks shared cross-modality attention features to model fine-grained relationships. The proposed approach in this work takes advantage of strong pre-training of the CLIP model and overcomes its limitations by introducing shared attention parameters to better model the fine-grained relationship between both modalities. We test and compare our proposed approach using the expertly annotated MiltiM-Poem dataset, which is considered the largest public image–poetry pair dataset for English poetry. The proposed approach aims to solve the problems of image-based attribute recognition and automatic retrieval for fine-grained poetic verses. The test results reflect that the shared attention parameters alleviate fine-grained attribute recognition, and the proposed approach is a significant step towards automatic multi-modal retrieval for improved artwork appreciation of people with visual impairments.

Triple Pseudo-Siamese network with hybrid attention mechanism for welding defect detection

Most of the existing methods neglect their complementary relation and only use welding pool images to detect welding defects. Therefore, a new triple pseudo-siamese network to improve identification performance using their complementarity is proposed in this paper. The network consists of three branches that extract the effective features of welding pool image, sound, current and voltage, respectively. Moreover, we employ a convolutional block attention module (CBAM) and propose cross-modal attention (CMA) to focus on the vitalregions of the welding pool image. To train our model, the experimental setup was first designed. Then, it was deployed in an actual body production workshop of high-speed train to collect welding pool images, as well as sound and current and voltage data. Hence, a new dataset for detecting welding defects is presented. Experimental results on the dataset indicate that the proposed model can effectively detect different defects and actions. Furthermore, when the output of the shallow layer, the layer near the input layer, is fed into the CBAM and the output of the deep layer, the layer near the output layer, is fed into the CMA. In addition, they are highly complementary to each other.

FCMNet: Frequency-aware cross-modality attention networks for RGB-D salient object detection

RGB-D saliency detection aims to comprehensively use RGB images and depth maps to detect object saliency. This field still faces two challenges: 1) how to extract representative multimodal features and 2) how to effectively fuse them. Most of the previous methods in this field equally treat RGB and depth information as two modalities, while not considering the difference in the frequency domain of the two modalities, and may lose some complementary information. In this paper, we introduce the frequency channel attention mechanism into the fusion process. First, we design a frequency-aware cross-modality attention (FACMA) module to interweave adequate channel features and select representative features. In the FACMA module, we also propose a spatial frequency channel attention (SFCA) module to introduce more complementary information in different channels. Second, we develop a weighted cross-modality fusion (WCMF) module to adaptively fuse multimodality features by learning the content-dependent weight maps. Comprehensive experiments on several benchmark datasets demonstrate that the proposed framework outperforms seventeen state-of-the-art methods.

FLeak-Seg: Automated Fundus Fluorescein Leakage Segmentation via Cross-Modal Attention Learning

Despite the recent success of deep learning-based models for medical image segmentation and the importance of automated fluorescein leakage segmentation for the diagnosis of advanced diabetic retinopathy, segmentation of fluorescein leakage has been neglected because 1) there are no publicly available databases with sufficient annotations to train segmentation models and 2) supervised models struggle to accurately distinguish between different types of fluorescein leakage and localize leakages at different imaging angles. To tackle these challenges, this work presents FLeak-Seg, a cross-modal dual attention learning method to jointly capture visual and language information, for end-to-end fluorescein leakage segmentation in fundus fluorescein angiography. Specifically, both image and text data are used as input, where visual and linguistic features are captured by a cross-modal attention learning module to compensate for the lack of annotations. A keyword classification module is also employed to identify meaningful expressions related to the type and location of fluorescein leakages to further facilitate the segmentation. Experimental results obtained in an in-house fundus fluorescein angiography database demonstrate the superiority of our method. We show how erroneous segmentation masks can be improved using FLeak-Seg, its advantages in the context of limited samples, and its behavior on segmenting different types of fluorescein leakages.

A multi-scale semantic attention representation for multi-label image recognition with graph networks

Multi-label image recognition is a basic and challenging task in computer vision and multimedia fields. Graph Convolutional Networks (GCNs) are often used to learn the multi-label semantic features and multi-label dependency. Although the label semantic features in GCNs can learn the global image visual representation well, they are rarely used on the local image regions. Therefore, we try to use GCNs to learn global and local features at the same time, and make a balance between them. In this paper, we give a multi-scale semantic attention model MS-SGA-GCN including three main modules (i.e., MS, SGA and GCN) for multi-label image recognition. The Multi-Scale module (MS) utilizes feature maps of different sizes to obtain global features and have strong generalization capabilities. Semantic Guide Attention module (SGA) applies the label embeddings learned by GCNs to guide the generation of the cross-modality class-specific attention maps, which can discover the locations of semantically related regions for each label. Experiments show that our model on two datasets MS-COCO and PASCAL VOC2007 separately achieves the classification accuracy by 83.4% and 94.2%, which has a competitive advantage over other mainstream models.

AGRFNet: Two-stage cross-modal and multi-level attention gated recurrent fusion network for RGB-D saliency detection

RGB-D saliency detection aims to identify the most attractive objects in a pair of color and depth images. However, most existing models adopt classic U-Net framework which progressively decodes two-stream features. In this paper, we decode the cross-modal and multi-level features in a unified unit, named Attention Gated Recurrent Unit (AGRU). It can reduce the influence of low-quality depth image, and retain more semantic features in the progressive fusion process. Specifically, the features of different modalities and different levels are organized as the sequential input, recurrently fed into AGRU which consists of reset gate, update gate and memory unit to be selectively fused and adaptively memorized based on attention mechanism. Further, two-stage AGRU serves as the decoder of RGB-D salient object detection network, named AGRFNet. Due to the recurrent nature, it achieves the best performance with the little parameters. In order to further improve the performance, three auxiliary modules are designed to better fuse semantic information, refine the features of the shallow layer and enhance the local detail. Extensive experiments on seven widely used benchmark datasets demonstrate that AGRFNet performs favorably against 18 state-of-the-art RGB-D SOD approaches.

Automatic Speech Fluency Evaluation Method Based on Multimodality for Putonghua Proficiency Test Propositional Speaking

The Putonghua Proficiency Test (Putonghua Shuiping Ceshi, PSC) is a speaking test in China. The propositional speaking section in PSC focuses on the ability of speakers to express ideas fluently and accurately without textual reference. However, unlike other sections of the PSC, propositional speaking is still scored manually, which can result in inefficiency, high costs, and subjectivity. To address these issues, an automatic speech fluency evaluation method based on multimodality is proposed. First, different neural networks are used to extract unimodal features. Then, cross-modal attention is applied to achieve multimodal fusion. Finally, fluency evaluation results are obtained using self-attention to reinforce high-contributing information. The proposed method achieves 81.67[Formula: see text] accuracy on a self-built dataset, demonstrating that combining textual and acoustic features provides complementary information to improve automatic speech fluency evaluation accuracy.

A Systematic Review on Serious Games in Attention Rehabilitation and Their Effects.

Attention is a basic and main mental task and can play an important role in the functioning of other brain abilities such as intelligence, memory, learning, and perception, and its deficit occurs in 80% of patients with traumatic brain injury. The use of game-based tools for rehabilitation is rapidly expanding. Cognitive rehabilitation via video games is an emerging hot topic in cognitive science. Serious games serve a specific purpose in addition to entertainment. They can be more engaging than exercises since they replace reward and motivation systems with real-world motivations as a complement for rehabilitation activities. This study was aimed at identifying and categorizing serious computer games used for attention rehabilitation and evaluating their effects. Six electronic databases (Scopus, PubMed, ISI, Embase, IEEE, and Cochrane) were searched in August 2021. The search strategy consisted of three main concepts of “serious game”, “cognitive deficits”, and “cognitive rehabilitation”. The inclusion criteria were (1) journal articles, (2) English language, (3) being published in the last 10 years, (4) human participants, and (5) game-based intervention. In the 30 included studies, 22 unique games were utilized for attention rehabilitation. Lumosity (20%), Brain Age (Dr. Kawashima's Brain Training) (10%), and MoHRS (6.66%) were the most common games among the studies. There were (57%) casual, (23%) action, (10%) simulation, and (10%) multiple genres. Of the 47 tools used in the studies, 5 utilized cross-modal oddball attention tasks, 4 utilized game performance, 3 utilized the paced auditory serial additional test (PASAT), and the rest employed other tools. A total of 73 outcome measures were related to attention, 42 measures did not have significant results, 30 were significantly improved, 1 was significantly deteriorated, and 4 articles did not have any specific measures for attention evaluation. Thus, the results revealed the positive effect of serious games on attention. However, issues such as absence of scientific teams, the variety of the disorders that cause defects, the variety of criteria, differences in measurements, lack of long-term follow-up, insufficient RCT studies, and small sample sizes should be considered when designing, developing, and using game-based systems to prevent bias.

Cross-modal distillation for RGB-depth person re-identification

Person re-identification is a key challenge for surveillance across multiple sensors. Prompted by the advent of powerful deep learning models for visual recognition, and inexpensive RGB-D cameras and sensor-rich mobile robotic platforms, e.g. self-driving vehicles, we investigate the relatively unexplored problem of cross-modal re-identification of persons between RGB (color) and depth images. The considerable divergence in data distributions across different sensor modalities introduces additional challenges to the typical difficulties like distinct viewpoints, occlusions, and pose and illumination variation. While some work has investigated re-identification across RGB and infrared, we take inspiration from successes in transfer learning from RGB to depth in object detection tasks. Our main contribution is a novel method for cross-modal distillation for robust person re-identification, which learns a shared feature representation space of person’s appearance in both RGB and depth images. In addition, we propose a cross-modal attention mechanism where the gating signal from one modality can dynamically activate the most discriminant CNN filters of the other modality. The proposed distillation method is compared to conventional and deep learning approaches proposed for other cross-domain re-identification tasks. Results obtained on the public BIWI and RobotPKU datasets indicate that the proposed method can significantly outperform the state-of-the-art approaches by up to 16.1% in mean Average Precision (mAP), demonstrating the benefit of the distillation paradigm. The experimental results also indicate that using cross-modal attention allows to improve recognition accuracy considerably with respect to the proposed distillation method and relevant state-of-the-art approaches.11Code: https://github.com/frhf/cross-modal-distillation-reidentification.

An Approach for Devising Stenography Application Using Cross Modal Attention

This paper focuses on providing a solution to the direct conversion of speech to shorthand. Since shorthand is not understood by many but is used for writing quick transcripts, a product is developed that converts the speech to its appropriate Gregg shorthand. A website that will be used as a front end, will use a speech-to-text API to record the speech in real-time. The converted text will then be fed into a text-to-image retrieval model that derives its corresponding Gregg shorthand for the text. The text will then be displayed to the user in real-time. By achieving this, the model reduces the need to depend upon stenographers for transcribing scripts. The resulting model achieves a good result.

A Crossmodal Multiscale Fusion Network for Semantic Segmentation of Remote Sensing Data

Driven by the rapid development of Earth observation sensors, semantic segmentation using multimodal fusion of remote sensing data has drawn substantial research attention in recent years. However, existing multimodal fusion methods based on convolutional neural networks cannot capture long-range dependencies across multiscale feature maps of remote sensing data in different modalities. To circumvent this problem, this work proposes a crossmodal multiscale fusion network (CMFNet) by exploiting the transformer architecture. In contrast to the conventional early, late, or hybrid fusion networks, the proposed CMFNet fuses information of different modalities at multiple scales using the cross-attention mechanism. More specifically, the CMFNet utilizes a novel cross-modal attention architecture to fuse multiscale convolutional feature maps of optical remote sensing images and digital surface model data through a crossmodal multiscale transformer (CMTrans) and a multiscale context augmented transformer (MCATrans). The CMTrans can effectively model long-range dependencies across multiscale feature maps derived from multimodal data, while the MCATrans can learn discriminative integrated representations for semantic segmentation. Extensive experiments on two large-scale fine-resolution remote sensing datasets, namely ISPRS Vaihingen and Potsdam, confirm the excellent performance of the proposed CMFNet as compared to other multimodal fusion methods.

The Weighted Cross-Modal Attention Mechanism With Sentiment Prediction Auxiliary Task for Multimodal Sentiment Analysis

Human brain extracts the spatial and temporal semantic information by processing the multi-modalities, which has contextually meaningful for perceiving and understanding the emotional state of an individual. However, there are two main challenges in modeling multimodal sequences: 1) the different sampling rates from multimodal data make the cross-modal interactions very difficult; 2) how to efficiently fuse unimodal representations and effectively capture relationships among multimodal data. In this paper, we design the weighted cross-modal attention mechanism, which not only captures the temporal correlation information and the spatial dependence information of each modality, but also dynamically adjusts the weight of each modality across different time steps. And the unimodal subtasks are led for assisting the representation learning of specific modality to jointly train the multimodal tasks and unimodal subtasks to explore the complementary relationships of each modality. Our model gets a new state-of-the-art record on the CMU-MOSI dataset and brings noticeable performance improvements on all the metrics. For the CMU-MOSEI dataset, the F1 score of the binary classification, the 7-class task, and the regression task of our model are still the highest among all models and the proposed model is only lower than the multimodal split attention fusion (MSAF) model with aligned data on the accuracy of the binary classification, showing the great performance of the suggested method.

Fine-Grained Question-Level Deception Detection via Graph-Based Learning and Cross-Modal Fusion

Automated deception detection has been found as a vital and concerned task, capable of assisting human users to assess truthfulness and detect deceptive behaviors in several situations (e.g., medical, legal, as well as occupational domains). As contact-free video cameras and data analysis techniques are leaping forward, leveraging one’s visual, acoustic and textual information captured in a video can be cost-effective for deception detection. In this study, we aim at a fine-grained question-level deception detection task, focusing on detecting whether a subject lies or not in answering each question rather than providing an overall “lie or not” judgement. A Graph-based Cross-modal Fusion Model (GCFM) is presented to learn the inherent associations among the subject’s reactions to different questions, plus a novel cross-modal attention mechanism to enhance the model’s learning capability. As revealed by the experimental results on the two datasets, the proposed graph-based GCFM outperformed eight other methods, and its two alternatives (clustering K-means based and sequential learning LSTM based methods), achieving accuracy 88.14% and 86.91% on the two datasets, respectively. Besides, through association learning, GCFM could increase the accuracy by 1.87% and 4.33% on the two datasets, respectively. Furthermore, its cross-modal attention mechanism led to the improvement of accuracy by 2.44% and 2.95% on the two datasets, respectively.

Language Reinforced Superposition Multimodal Fusion for Sentiment Analysis

Nowadays, BERT has been effective in context extraction and widely utilized in multimodal fusion. However, many multimodal learning systems with BERT consider different modalities equally important, which ignores the huge capability difference between the pre-trained models and others. Moreover, equal multimodal fusion might introduce noise and limit the performance of these systems. To this end, we propose a superposition multimodal fusion framework to strengthen the importance of language modality with nonverbal information transmission. With the superposition strategy, language information will fuse with nonverbal information at different stages, which allows the model to fully learn language modality information. Besides, to ensure that language information is the main content of the fusion, we propose a comparison loss to help the reinforced cross-modal attention module better transfer nonverbal information to language features. Extensive experiments are conducted to compare with baselines and the results demonstrate that our method achieves the superior performance.

A Novel Attention-Based Multi-Modal Modeling Technique on Mixed Type Data for Improving TFT-LCD Repair Process

In Thin-Film Transistor Liquid-Crystal Display (TFT-LCD) manufacturing, conducting a machine learning based system with multiple data types has become actively desired to solve complicated problems. This paper proposes a multi-modal learning approach: <i>TabVisionNet</i>, which is modeled by utilizing the information from both tabular data and image data. A novel attention mechanism called <i>Sequential Decision Attention</i> was integrated into the multi-modal modeling framework that improves the comprehension of the information from two modalities. This cross-modal attention mechanism can capture the complex relationship between modalities then gain better generalization and faster convergence in the training process. Conducting an experiment, the performance of our novel approach was significantly better than single-modal and other multi-modal learning approaches in our real case scenario.