Co-attention Network Research Articles

CAAN: Context-Aware attention network for visual question answering

Understanding multimodal information is the key to visual question answering (VQA) tasks. Most existing approaches use attention mechanisms to acquire fine-grained information understanding. However, these approaches with merely attention mechanisms do not solve the potential understanding bias problem. Hence, this paper introduces contextual information into VQA for the first time and presents a context-aware attention network (CAAN) to tackle the case. By improving the modular co-attention network (MCAN) framework, CAAN’s main work includes: designing a novel absolute position calculation method based on the coordinates of each image region in the image and the image’s actual size, the position information of all image regions are integrated as contextual information to enhance the visual representation; based on the question itself, several internal contextual information representations are introduced to participate in the modeling of the question words, solving the understanding bias caused by the similarity of the question. Additionally, we also designed two models of different scales, namely CAAN-base and CAAN-large, to explore the effect of the field of view on interaction. Finally, extensive experimental results show that CAAN significantly outperforms MCAN and achieves comparable or even better performance than other state-of-the-art approaches, proving our method can tackle the understanding bias.

SPCA-Net: a based on spatial position relationship co-attention network for visual question answering

SPCA-Net: a based on spatial position relationship co-attention network for visual question answering

Dynamic Co-Attention Networks for multi-horizon forecasting in multivariate time series

Although attention-based encoder–decoder models achieve encouraging performance in multivariate time series multi-horizon forecasting, two key limitations exist in current methods: (i) These methods merely focus on the spatial or temporal correlations among series, and ignore harmful interactions between non-predictive and target variables. (ii) Time series forecasting benefits significantly from knowledge of the recent past, but existing approaches only consider the encoder and neglect the previous decoder units. To tackle these challenges, we propose a novel prediction framework, termed Dynamic Co-Attention Networks (DCAN). For the first issue, we propose a two-stage variables embedding network as DCAN’s underlying structure to better capture the potential semantics that integrates target and non-predictive variables. The first stage selects the most important non-predictive variables and generates a spatial vector, and the second stage trades off how much information the potential semantics consider from the target variable and spatial vector. To alleviate the second issue, we design a dynamic attention synergetic network (DASN) as the superstructure of the proposed model, including a contextual attention and a filtration gate. A contextual attention expands the observation horizon for current prediction by continuously retrieving previous decoder states, and a filtration gate module selectively incorporates previous decoder information into the current decoder unit, thus improving the credibility and accuracy of the model decision-making. Experiments on three real-life datasets demonstrate that DCAN outperforms different types of state-of-the-art baselines.

A Feature Sparse Co-Attention Network for Visual internet of things (VIoT) sensing

Visual internet of things (VIoT) is a cross-modal learning task that requires a simultaneous understanding of image and question text information. The attention mechanism simulates human vision to retain the important part of the information and discard the unimportant part. However, since both important and unimportant features participate in the weighted summation, there is no true retention and discarding of information features. Furthermore, in many VIoT solutions, the utilization of the attention mechanism does not achieve the purpose of improving VIoT performance by aligning the key regions of image and question. To solve the problems mentioned above, this paper proposes an attention mechanism that sparse features to achieve attention with true retention and discard functions. At the same time, a feature sparse co-attention network is constructed to align the key regions of vision and text. The network is composed of image self-attention unit, question self-attention unit, and guiding attention unit. Each self-attention unit has a feature sparse function. These units are cascaded in depth to form a hierarchical structure that, as a whole, realizes co-attention. Several experiments conducted on the VQA-v2 dataset show that the proposed method outperforms the latest methods.

Mutual Information-Based Graph Co-Attention Networks for Multimodal Prior-Guided Magnetic Resonance Imaging Segmentation

Multimodal magnetic resonance imaging (MRI) provides complementary information about targets, and the segmentation of multimodal MRI is widely used as an essential preprocessing step for initial diagnosis, stage differentiation, and post-treatment efficacy evaluation in clinical situations. For the main modality or each of the modalities, it is important to enhance the visual information by modeling the connection and effectively fusing the features among them. However, the existing methods for multimodal segmentation have a drawback; they coincidentally drop information of individual modality during the fusion process. Recently, graph learning-based methods have been applied in segmentation, and these methods have achieved considerable improvements by modeling the relationships across feature regions and reasoning using global information. In this paper, we propose a graph learning-based approach to efficiently extract modality-specific features and establish regional correspondence effectively among all modalities. In detail, after projecting features into a graph domain and employing graph convolution to propagate information across all regions for learning global modality-specific features, we propose a mutual information-based graph co-attention module to learn the weight coefficients of one bipartite graph constructed by the fully connected graphs having different modalities in the graph domain and by selectively fusing the node features. Based on the deformation diagram between the spatial-graph space and our proposed graph co-attention module, we present a multimodal prior-guided segmentation framework, which uses two strategies for two clinical situations: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Modality-Specific Learning Strategy</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Co-Modality Learning Strategy</i> . Besides, the improved <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Co-Modality Learning Strategy</i> is used with trainable weights in the multi-task loss for the optimization of the proposed framework. We validated our proposed modules and frameworks on two multimodal MRI datasets: our private liver lesion dataset and a public prostate zone dataset. Our experimental results on both datasets prove the superiority of our proposed approaches.

Modeling user micro-behaviors and original interest via Adaptive Multi-Attention Network for session-based recommendation

Session-based recommendation aims to recommend the next item that matches the user’s preferences based on short-term limited session information. It does not need user’s account or other privacy information and can be applied widely on most e-commerce platforms. Many efforts have made great achievements. However, the following two insights are often neglected. First, a user’s micro-behaviors, such as item browsing, item clicking, item collecting, item carting and purchasing, offer detailed information and can help infer the user’s intention. Second, the first item of a session usually implies the user’s original interest. These insights motivate us to propose a novel model named AMAN (Adaptive Multi-Attention Network) which considers both user micro-behaviors and original interest for session-based recommendation. AMAN contains three key components: Directed Graph Attention Network (DGAN) learns item representation from item-level and adaptively captures items’ associations in the same micro-behavior sequence; Transformer-based Operation-level Attention Network (TOAN) learns item representation from operation-level and adaptively captures items’ associations across different micro-behavior sequences; Micro-behavior Co-Attention Network (MBCAN) learns micro-behavior sequence representation and adaptively captures the co-dependence among different micro-behavior sequences from item-level, and explicitly models the first item that reflects the user’s original interest. Experimental results on Yoochoose, Taobao14 and Taobao15 benchmark datasets show that AMAN outperforms the state-of-the-art baselines, justifying the effectiveness of incorporating user micro-behaviors and original interest for session-based recommendation.

Aspect-based sentiment analysis with component focusing multi-head co-attention networks

User-generated content based on customer opinions and experience has become a rich source of valuable information for enterprises. The purpose of aspect-based sentiment analysis is to predict the sentiment polarity of specific targets from user-generated content. This study proposes a component focusing multi-head co-attention network model which contains three modules: extended context, component focusing, and multi-headed co-attention, designed to improve upon problems encountered in the past. The extended context module improves the ability of bidirectional encoder representations from transformers to handle aspect-based sentiment analysis tasks, and the component focusing module improves the weighting of adjectives and adverbs, to alleviate the problem of average pooling, which treats every word as an equally important term. The multi-head co-attention network is applied to learn the important words in a multi-word target before acquiring the context representation and performs the attention mechanism on the sequence data. The performance of the proposed model is evaluated in extensive experiments on publicly available datasets. The results show that the performance of the proposed model is better than that of the recent state-of-the-art models.

Attention Over Self-Attention: Intention-Aware Re-Ranking With Dynamic Transformer Encoders for Recommendation

Re-ranking models refine the item recommendation list generated by the prior global ranking model with intra-item relationships. However, most existing re-ranking solutions refine recommendation list based on the implicit feedback with a shared re-ranking model, which regrettably ignore the intra-item relationships under diverse user intentions. In this paper, we propose a novel Intention-aware Re-ranking Model with Dynamic Transformer Encoder (RAISE), aiming to perform user-specific prediction for each target user based on her intentions. Specifically, we first propose to mine latent user intentions from text reviews with an intention discovering module (IDM). By differentiating the importance of review information with a co-attention network, the latent user intention can be explicitly modeled for each user-item pair. We then introduce a dynamic transformer encoder (DTE) to capture user-specific intra-item relationships among item candidates by seamlessly accommodating the learnt latent user intentions via IDM. As such, RAISE is able to perform user-specific prediction without increasing the depth (number of blocks) and width (number of heads) of the prediction model. Empirical study on four public datasets shows the superiority of our proposed RAISE, with up to 13.95%, 12.30%, and 13.03% relative improvements evaluated by Precision, MAP, and NDCG respectively.

Cross-Modal Relation-Aware Networks for Fake News Detection

With the speedy development of communication Internet and the widespread use of social multimedia, so many creators have published posts on social multimedia platforms that fake news detection has already been a challenging task. Although some works use deep learning methods to capture visual and textual information of posts, most existing methods cannot explicitly model the binary relations among image regions or text tokens to mine the global relation information in a modality deeply such as image or text. Moreover, they cannot fully exploit the supplementary cross-modal information, including image and text relations, to supplement and enrich each modality. In order to address these problems, in this paper, we propose an innovative end-to-end Cross-modal Relation-aware Networks (CRAN), which exploits jointly models the visual and textual information with their corresponding relations in a unified framework. (1) To capture the global structural relations in a modality, we design a global relation-aware network to explicitly model the relation-aware semantics of the fragment features in the target modality from a global scope perspective. (2) To effectively fuse cross-modal information, we propose a cross-modal co-attention network module for multi-modal information fusion, which utilizes the intra-modality relationships and inter-modality relationship jointly among image regions and textual words to replenish and heighten each other. Extensive experiments on two public real-world datasets demonstrate the superior performance of CRAN compared with other state-of-the-art baseline algorithms.

Graph Co-Attentive Session-based Recommendation

Session-based recommendation aims to generate recommendations merely based on the ongoing session, which is a challenging task. Previous methods mainly focus on modeling the sequential signals or the transition relations between items in the current session using RNNs or GNNs to identify user’s intent for recommendation. Such models generally ignore the dynamic connections between the local and global item transition patterns, although the global information is taken into consideration by exploiting the global-level pair-wise item transitions. Moreover, existing methods that mainly adopt the cross-entropy loss with softmax generally face a serious over-fitting problem, harming the recommendation accuracy. Thus, in this article, we propose a Graph Co-Attentive Recommendation Machine (GCARM) for session-based recommendation. In detail, we first design a Graph Co-Attention Network (GCAT) to consider the dynamic correlations between the local and global neighbors of each node during the information propagation. Then, the item-level dynamic connections between the output of the local and global graphs are modeled to generate the final item representations. After that, we produce the prediction scores and design a Max Cross-Entropy (MCE) loss to prevent over-fitting. Extensive experiments are conducted on three benchmark datasets, i.e., Diginetica, Gowalla, and Yoochoose. The experimental results show that GCARM can achieve the state-of-the-art performance in terms of Recall and MRR, especially on boosting the ranking of the target item.

CANet: Co-attention network for RGB-D semantic segmentation

Incorporating the depth (D) information to RGB images has proven the effectiveness and robustness in semantic segmentation. However, the fusion between them is not trivial due to their inherent physical meaning discrepancy, in which RGB represents RGB information but D depth information. In this paper, we propose a co-attention network (CANet) to build sound interaction between RGB and depth features. The key part in the CANet is the co-attention fusion part. It includes three modules. Specifically, the position and channel co-attention fusion modules adaptively fuse RGB and depth features in spatial and channel dimensions. An additional fusion co-attention module further integrates the outputs of the position and channel co-attention fusion modules to obtain a more representative feature which is used for the final semantic segmentation. Extensive experiments witness the effectiveness of the CANet in fusing RGB and depth features, achieving state-of-the-art performance on two challenging RGB-D semantic segmentation datasets, i.e., NYUDv2 and SUN-RGBD.

Bi-Directional Co-Attention Network for Image Captioning

Image Captioning, which automatically describes an image with natural language, is regarded as a fundamental challenge in computer vision. In recent years, significant advance has been made in image captioning through improving attention mechanism. However, most existing methods construct attention mechanisms based on singular visual features, such as patch features or object features, which limits the accuracy of generated captions. In this article, we propose a Bidirectional Co-Attention Network (BCAN) that combines multiple visual features to provide information from different aspects. Different features are associated with predicting different words, and there are a priori relations between these multiple visual features. Based on this, we further propose a bottom-up and top-down bi-directional co-attention mechanism to extract discriminative attention information. Furthermore, most existing methods do not exploit an effective multimodal integration strategy, generally using addition or concatenation to combine features. To solve this problem, we adopt the Multivariate Residual Module (MRM) to integrate multimodal attention features. Meanwhile, we further propose a Vertical MRM to integrate features of the same category, and a Horizontal MRM to combine features of the different categories, which can balance the contribution of the bottom-up co-attention and the top-down co-attention. In contrast to the existing methods, the BCAN is able to obtain complementary information from multiple visual features via the bi-directional co-attention strategy, and integrate multimodal information via the improved multivariate residual strategy. We conduct a series of experiments on two benchmark datasets (MSCOCO and Flickr30k), and the results indicate that the proposed BCAN achieves the superior performance.

Co-attention network with label embedding for text classification

Most existing methods for text classification focus on extracting a highly discriminative text representation, which, however, is typically computationally inefficient. To alleviate this issue, label embedding frameworks are proposed to adopt the label-to-text attention that directly uses label information to construct the text representation for more efficient text classification. Although these label embedding methods have achieved promising results, there is still much space for exploring how to use the label information more effectively. In this paper, we seek to exploit the label information by further constructing the text-attended label representation with text-to-label attention. To this end, we propose a Co-attention Network with Label Embedding (CNLE) that jointly encodes the text and labels into their mutually attended representations. In this way, the model is able to attend to the relevant parts of both. Experiments show that our approach achieves competitive results compared with previous state-of-the-art methods on 7 multi-class classification benchmarks and 2 multi-label classification benchmarks.

Collaborative Co-Attention Network for Session-Based Recommendation

Session-based recommendation aims to model a user’s intent and predict an item that the user may interact with in the next step based on an ongoing session. Existing session-based recommender systems mainly aim to model the sequential signals based on Recurrent Neural Network (RNN) structures or the item transition relations between items with Graph Neural Network (GNN) based frameworks to identify a user’s intent for recommendation. However, in real scenarios, there may be strong sequential signals existing in users’ adjacent behaviors or multi-step transition relations among different items. Thus, either RNN- or GNN-based methods can only capture limited information for modeling complex user behavior patterns. RNNs pay attention to the sequential relations among consecutive items, while GNNs focus on structural information, i.e., how to enrich the item embedding with its adjacent items. In this paper, we propose a Collaborative Co-attention Network for Session-based Recommendation (CCN-SR) to incorporate both sequential and structural information, as well as capture the co-relations between them for obtaining an accurate session representation. To be specific, we first model the ongoing session with an RNN structure to capture the sequential information among items. Meanwhile, we also construct a session graph to learn the item representations with a GNN structure. Then, we design a co-attention network upon these two structures to capture the mutual information between them. The designed co-attention network can enrich the representation of each node in the session with both sequential and structural information, and thus generate a more comprehensive representation for each session. Extensive experiments are conducted on two public e-commerce datasets, and the results demonstrate that our proposed model outperforms state-of-the-art baseline model for session based recommendation in terms of both Recall and MRR. We also investigate different combination strategies and the experimental results verify the effectiveness of our proposed co-attention mechanism. Besides, our CCN-SR model achieves better performance than baseline models with different session lengths.

Dual self-attention with co-attention networks for visual question answering

Visual Question Answering (VQA) as an important task in understanding vision and language has been proposed and aroused wide interests. In previous VQA methods, Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) are generally used to extract visual and textual features respectively, and then the correlation between these two features is explored to infer the answer. However, CNN mainly focuses on extracting local spatial information and RNN pays more attention on exploiting sequential architecture and long-range dependencies. It is difficult for them to integrate the local features with their global dependencies to learn more effective representations of the image and question. To address this problem, we propose a novel model, i.e., Dual Self-Attention with Co-Attention networks (DSACA), for VQA. It aims to model the internal dependencies of both the spatial and sequential structure respectively by using the newly proposed self-attention mechanism. Specifically, DSACA mainly contains three submodules. The visual self-attention module selectively aggregates the visual features at each region by a weighted sum of the features at all positions. The textual self-attention module automatically emphasizes the interdependent word features by integrating associated features among the sentence words. Besides, the visual-textual co-attention module explores the close correlation between visual and textual features learned from self-attention modules. The three modules are integrated into an end-to-end framework to infer the answer. Extensive experiments performed on three generally used VQA datasets confirm the favorable performance of DSACA compared with state-of-the-art methods.

Multimodal feature-wise co-attention method for visual question answering

VQA attracts lots of researchers in recent years. It could be potentially applied to the remote consultation of COVID-19. Attention mechanisms provide an effective way of utilizing visual and question information selectively in visual question and answering (VQA). The attention methods of existing VQA models generally focus on spatial dimension. In other words, the attention is modeled as spatial probabilities that re-weights the image region or word token features. However, feature-wise attention cannot be ignored, as image and question representations are organized in both spatial and feature-wise modes. Taking the question “What is the color of the woman’s hair” for example, identifying the hair color attribute feature is as important as focusing on the hair region. In this paper, we propose a novel neural network module named “multimodal feature-wise attention module” (MulFA) to model the feature-wise attention. Extensive experiments show that MulFA is capable of filtering representations for feature refinement and leads to improved performance. By introducing MulFA modules, we construct an effective union feature-wise and spatial co-attention network (UFSCAN) model for VQA. Our evaluation on two large-scale VQA datasets, VQA 1.0 and VQA 2.0, shows that UFSCAN achieves performance competitive with state-of-the-art models.

Co-attention networks based on aspect and context for aspect-level sentiment analysis

Aspect-level sentiment analysis aims to predict the sentiment orientation of a specific aspect in one context. Recent studies have achieved great success in modeling aspect and context by applying long–short term memory networks (LSTMs) and an attention mechanism. However, the semantic correlations between each word of the aspect and context fail to be noticed, which decreases the effectiveness of feature representations. Given this problem, in order to comprehensively analyze semantic correlations from the perspective of the word level and feature level, a co-attention mechanism is proposed to capture the interactions between aspect and context, which interactively concentrates the semantic influences on context and aspect to generate greater informative representation. Specifically, a co-attention mechanism consists of the 1-pair hop mechanism and an interactive mechanism, in which the 1-pair hop mechanism pays more attention to the important word’s aspect or context and the interactive mechanism highlights the significant feature of the aspect or context by calculating the interactive attention matrix from the perspective of the feature level. In addition, considering that one context contains more than one aspect, the novel loss function is designed to fully employ the attention weights of different aspects on every word in the given context. Extensive compared experiments are conducted based on the GloVe and BERT pre-trained models, and the results show that the proposed method can achieve state-of-the-art performance on the Restaurant and Twitter datasets. Furthermore, ablation studies are designed to validate the necessity and importance of the 1-pair hop mechanism and interactive mechanism.

Multi-level attentive deep user-item representation learning for recommendation system

With the development of e-commerce platforms, user reviews have become a vital source of information to address the sparsity problems for enhancing the predictive performance of the recommendation systems (RSs). However, the traditional methods of the RSs used to model user/item latent features based on static vectors in an independent manner without considering the dynamic nature of the user-item interactions which potentially affect the accuracy of the recommendation process. Thus, this paper proposes a RS model that exploits neural attention techniques to learn user/item representations by jointly considering the fine-grained semantic information for the user-item pairs. The proposed model utilizes both review-based and interaction-specific features for the user/item reviews to learn heterogeneous user/item representations. First, a BiLSTM sequence encoder is used to learn the contextual information of words, and a Co-attention network is then designed to jointly capture the most relevant semantic information of reviews for the user-item pair. To better capture user/item latent factors comprehensively, interaction-specific features based on the rating scores are further integrated with the review-specific latent features via a shared hidden layer. Finally, an attentive factorization machine (FM) is then applied on the shared hidden layer of the integrated user/item features for the final prediction. We carry out a series of experiments using real-world datasets and the results demonstrate that our proposed method is better than the baseline approaches in terms of both rating prediction and ranking performance.

Cross-modality co-attention networks for visual question answering

Cross-modality co-attention networks for visual question answering

Graph-CAT: Graph Co-Attention Networks via local and global attribute augmentations

Graph neural networks have achieved tremendous success in semi-supervised node classification. In this paper, we firstly analyse the propagation strategies in two milestone methods, Graph Convolutional Network (GCN) and Graph Attention Network (GAT), to reveal their underlying philosophies. According to our analysis, the propagations in GAT can be interpreted as learnable and asymmetric local attribute augmentations, while that of GCN can be interpreted as fixed and symmetric local attribute smoothing. Unfortunately, the local attribute augmentations in GAT is not adequate in certain circumstances, because the nodes tend to possess similar attributes in local neighbourhoods. With a toy experiment, we manage to demonstrate the necessity to incorporate global information. Therefore, we propose a novel Graph Co-ATtention Network (Graph-CAT), which performs both the local and global attribute augmentations based on two different yet complementary attention schemes. Extensive experiments in both the transductive and inductive tasks demonstrate the superiority of our Graph-CAT compared to the state-of-the-art methods.