Multi-attention Mechanism Research Articles

Gated multi-attention representation in reinforcement learning

Deep reinforcement learning (DRL) has achieved great success in recent years by combining the feature extraction power of deep learning and the decision power of reinforcement learning techniques. In the literature, Convolutional Neural Networks (CNN) is usually used as the feature extraction method and recent studies have shown that the performances of the DRL algorithms can be greatly improved with the utilization of the attention mechanism, where the raw attentions are directly used for the decision-making. However, as is well-known, reinforcement learning is a trial-and-error process and it is almost impossible to learn an optimal policy in the beginning of the learning, especially in environments with sparse rewards, which in turn will cause the raw attention-based models can only remember and utilize the attention information indiscriminately for different areas and may focus on some task-irrelevant regions, but the focusing on such task-irrelevant regions is usually helpless and ineffective for the agent to find the optimal policy. To address this issue, we propose a gated multi-attention mechanism, which is then combined with the Deep Q-learning network (GMAQN). The gated multi-attention representation module (GMA) in GMAQN can effectively eliminate task-irrelevant attention information in the early phase of the trial-and-error process and improve the stability of the model. The proposed method has been demonstrated on the challenging domain of classic Atari 2600 games and experimental results show that compared with the baselines, our method can achieve better performance in terms of both the scores and the effect of focusing in the key regions.

Atomtransmachine: An atomic feature representation model for machine learning

In this study, a self-monitoring model is proposed to extract the atomic characteristics of the main group elements and transition metals from several molecular structures. Different from previous studies, we use a spatial convolution layer to extract the spatial features of atoms and a multi-attention mechanism to screen their important features in forming new crystal structures. Extensive numerical analyses show that the features extracted using the proposed model are effective and can improve the efficiency of machine learning algorithms.

Human Action Recognition Method Based on Multi-Attention Mechanism and Spatiotemporal Graph Convolution Networks

<p indent="0mm">Human action recognition has become a challenging task in computer vision because it is difficult to combine spatiotemporal information. A multi-attention spatiotemporal graph convolution network is proposed. The core idea is to construct a connected graph according to the time series information and natural connection of human skeleton, and use the spatiotemporal graph convolution network with multi-attention mechanism to automatically learn spatial and temporal features and optimize the connected graph to realize prediction. Graph attention module is introduced, the topological structure of the graph constructed by the model will be optimized with the process of network training after initialization, then the topological structure which is more suitable for expressing human actions will be obtained. In addition, the channel attention module is added to make the network pay more attention to the important channel information, so as to extract the features of describing actions more effectively. A large number of experiments are carried out on the recognized large datasets: NTU-RGDB and Kinectics, which show that the method has higher recognition accuracy.

Interpretable MOOC recommendation: a multi-attention network for personalized learning behavior analysis

PurposeCourse recommendations are important for improving learner satisfaction and reducing dropout rates on massive open online course (MOOC) platforms. This study aims to propose an interpretable method of analyzing students' learning behaviors and recommending MOOCs by integrating multiple data sources.Design/methodology/approachThe study proposes a deep learning method of recommending MOOCs to students based on a multi-attention mechanism comprising learning records attention, word-level review attention, sentence-level review attention and course description attention. The proposed model is validated using real-world data consisting of the learning records of 6,628 students for 1,789 courses and 65,155 reviews.FindingsThe main contribution of this study is its exploration of multiple unstructured information using the proposed multi-attention network model. It provides an interpretable strategy for analyzing students' learning behaviors and conducting personalized MOOC recommendations.Practical implicationsThe findings suggest that MOOC platforms must fully utilize the information implied in course reviews to extract personalized learning preferences.Originality/valueThis study is the first attempt to recommend MOOCs by exploring students' preferences in course reviews. The proposed multi-attention mechanism improves the interpretability of MOOC recommendations.

Emotion Recognition Model Based on Multimodal Decision Fusion

In the process of human social activities and daily communication, speech, text and facial expressions are considered as the main channels to convey human emotions. In this paper, a fusion method of multi-modal emotion recognition based on speech, text and motion is proposed. In the speech emotion recognition (SER), a depth wavefield extrapolation - improved wave physics model (DWE-WPM) is designed. In order to simulate the information mining process of LSTM, a user-defined feature extraction scheme is used to reconstruct the wave and inject it into DWE-WPM. In the text emotion recognition (TER), the transformer model with multi attention mechanism is used to recognize the text emotion combined. In the motion emotion recognition (MER), the sequential features of facial expression and hand action are extracted in groups. Combined with the bidirectional three-layer LSTM model with attention mechanism, a joint model of four channels is designed. Experimental results show that the proposed method has high recognition accuracy in multi-modal, and the accuracy is improved by 9% in the interactive emotional dynamic motion capture (IEMOCAP) corpus.

Emotion-cause span extraction: a new task to emotion cause identification in texts

Emotional cause identification (ECI) is an important task for emotion analysis, aiming to identify the causes behind a certain emotion expressed in the text. Most of the previous studies are restricted to the clause-level binary classification, which have ignored an important fact that not all words in the clause are useful cause information for people to express emotions. In this work, we propose a new task: emotion-cause span extraction (ECSE), which is capable of obtaining more accurate and effective emotion causes. Inspired by recent advances in using joint learning approaches to ECI, we propose a novel joint learning framework for emotion-cause span extraction and span-based emotion classification so as to better address the ECSE task. Taken as the default backbone network, the Bidirectional Encoder Representations from Transformers (BERT) is used to encode multiple words and serve contextualized token representations. Furthermore, we also propose a multi-attention mechanism with emotional context awareness and a relative position learning mechanism on word-level, which is able to further capture the mutual interactions between the emotion clauses and candidate spans. According to the experimental results on a benchmark emotion cause corpus, it proves the reliability of the ECSE task and the effectiveness of our approach. In addition, through an in-depth analysis of traditional ECI task by converting ECSE into the clause-level binary classification task, we achieve the best performance among the systems in comparison, which further demonstrates the feasibility of the new ECSE task.

PMAN: Progressive Multi-Attention Network for Human Pose Transfer

This paper presents a novel approach for human pose transfer, progressive multi-attention network (PMAN), which generates a new human image by transferring the pose of a given person to a target pose. The network gradually updates the pose feature and the image feature through a series of multi-attention transfer blocks (MATBs). Each MATB consists of two attention mechanisms: pose-conditioned batch normalization (PCBN) and cooperative attention mechanism (CAM). Specifically, in low-level feature space, the PCBN layer with pose information is used to replace the BN layer of the image channel to realize the preliminary guidance of pose to image. The CAM is implemented as two gated mechanisms in high-level feature space, which reveals the essence of human pose transfer, that is, mutual guidance and dynamic control between pose and image. Gated memory writing is used to calculate the pixel-wise weight of the pose by using global image information to guide the update of the pose. Gated response utilizes an adaptive gating mechanism to dynamically control the pose information flow so as to update the image. A large number of subjective and objective experiments on DeepFashion and Market-1501 demonstrate the superiority of our method. The proposed multi-attention mechanism is well adapted to the human pose transfer task and provides a possible new idea for other cross-domain generation tasks.

Residual networks with multi-attention mechanism for hyperspectral image classification

Residual network which can effectively overcome gradient disappearance in convolutional neural networks has been successfully applied to hyperspectral classification. Yet simply accumulating residual units does not improve the model performance. Besides, the lack of training samples seriously restricts the hyperspectral classification accuracy. In attempt to solve the abovementioned problems, residual networks with multi-attention mechanism (RNMA) is proposed. The spectral-spatial attention module (SSAM) in accordance with the structure characteristics of hyperspectral images is introduced into the RNMA method. The SSAM is embedded after the input which can strengthen bands relevant to ground object and spatial salient regions of ground object while weaken less informative features. Meanwhile, channel attention module is introduced into residual network unit to form residual network unit with channel attention module (RUCA). Based on RUCA, the features of the spectral dimension and the spatial dimension are sequentially extracted. Besides, sample augmentation method with spectral and spatial constraints is applied to expand sample size. Experiments results on three hyperspectral datasets show that the RNMA model outperforms than other methods in classification accuracy. The RNMA model achieved superior classification result compared with other state-of-the-art methods in the small training sample size.

Sentence Representation Method Based on Multi-Layer Semantic Network

With the development of artificial intelligence, more and more people hope that computers can understand human language through natural language technology, learn to think like human beings, and finally replace human beings to complete the highly difficult tasks with cognitive ability. As the key technology of natural language understanding, sentence representation reasoning technology mainly focuses on the sentence representation method and the reasoning model. Although the performance has been improved, there are still some problems such as incomplete sentence semantic expression, lack of depth of reasoning model, and lack of interpretability of the reasoning process. In this paper, a multi-layer semantic representation network is designed for sentence representation. The multi-attention mechanism obtains the semantic information of different levels of a sentence. The word order information of the sentence is also integrated by adding the relative position mask between words to reduce the uncertainty caused by word order. Finally, the method is verified on the task of text implication recognition and emotion classification. The experimental results show that the multi-layer semantic representation network can promote sentence representation’s accuracy and comprehensiveness.

Multi-Attention Generative Adversarial Network for Multivariate Time Series Prediction

Multivariate Time series data play important roles in our daily life. How to use these data in the process of prediction is a highly attractive study for many researchers. To achieve this goal, in this paper, we present a novel multivariate time series prediction method based on multi-attention generative adversarial network. This method includes three phases to explore multivariate time series prediction. Firstly, the encoder stage consists of two modules, from which the input-attention and self-attention can encode the exogenous sequence into latent space. Secondly, the decoder stage consists of the temporal-convolution-attention module, which can extract long-term temporal patterns. To solve the problem of low accuracy in long-term prediction, inspired by the weight clipping method, we design an improved discrimination network finally. The experiment results indicate that multi-attention mechanism is useful and the discrimination network can improve the performance in multivariate time series prediction. We also tested extensive empirical studies with five real world datasets (NASDAQ100, SML2010, Energy, EEG and Air Quality) demonstrate the effectiveness and robustness of our proposed approach.

Application of Multiattention Mechanism in Power System Branch Parameter Identification

Maintaining accuracy and robustness has always been an unsolved problem in the task of power grid branch parameter identification. Therefore, many researchers have participated in the research of branch parameter identification. The existing methods of power grid branch parameter identification suffer from two limitations. (1) Traditional methods only use manual experience or instruments to complete parameter identification of single branch characteristics, but they are only used to identify a single target and cannot make full use of the historical information of power grid data. (2) Deep learning methods can complete model training through historical data, but these methods cannot consider the constraints of power grid topological structure, which is equivalent to identifying connected power grid branches separately. To overcome these limitations, we propose a novel multitask Graph Transformer Network (GTN), which combines a graph neural network and a multiattention mechanism to construct our model. Specifically, we input the global features and topology information of branch nodes into our GTN model. In the process of parameter identification, the multihead attention mechanism is used to fuse the branch feature information of different subspaces, which highlights the importance of different branches and enhances the ability of local feature extraction. Finally, the fitting and prediction of each branch feature are completed through the decoding layer. The experiment shows that our proposed GTN is superior to other machine learning methods and deep learning methods and can still realize accurate branch parameter identification under various noise conditions.

MACRO: Multi-Attention Convolutional Recurrent Model for Subject-Independent ERP Detection

Due to the low signal-to-noise ratio, limited training samples, and large inter-subject variabilities in electroencephalogram (EEG) signals, developing a subject-independent brain-computer interface (BCI) system used for new users without any calibration is still challenging. In this letter, we propose a novel Multi-Attention Convolutional Recurrent mOdel (MACRO) for EEG-based event-related potential (ERP) detection in the subject-independent scenario. Specifically, the convolutional recurrent network is designed to capture the spatial-temporal features, while the multi-attention mechanism is integrated to focus on the most discriminative channels and temporal periods of EEG signals. Comprehensive experiments conducted on a benchmark dataset for RSVP-based BCIs show that our method achieves the best performance compared with the five state-of-the-art baseline methods. This result indicates that our method is able to extract the underlying subject-invariant EEG features and generalize to unseen subjects. Finally, the ablation studies verify the effectiveness of the designed multi-attention mechanism in MACRO for EEG-based ERP detection.

Aspect-level sentiment analysis using context and aspect memory network

With the popularity of social networks, sentiment analysis has become one of the hottest topics in natural language processing (NLP). As the development of research on the fine-grained sentiment analysis, more and more researchers pay attention to aspect-level sentiment analysis. It aims to identify the same or different sentiment polarity in different aspects of the context. In this paper, a context and aspect memory network (CAMN) method is proposed to solve the problem of aspect level sentiment analysis. In this method, deep memory network, bi-directional long short-term memory network and multi-attention mechanism are introduced to better capture the sentiment features in short texts. It includes two strategies: one is to use the self-attention mechanism (i.e., CAMN-SA) to calculate the context relevance; the other is to use the encoder-decoder attention mechanism (i.e., CAMN-ED) to calculate the context and aspect relevance. In order to verify the function of each component in the proposed method, and to test the effect of different hops on the memory network, we conduct many experiments on three real-world datasets to compare the baseline models with our proposed method. Experimental results show that our proposed method can achieve better performance than the baseline models.

Multi-Attention Mechanism Medical Image Segmentation Combined with Word Embedding Technology

In order to solve the problems of low gray scale contrast and blurred organ boundaries in some medical images, we proposed a joint algorithm of Multi-Attention Parallel CNNs and Independent Recurrent Neural Networks (MACIR) with word embedding technique combined. First, the word embedding technique is used to map the sparse spatial relation matrix into a real dense vector, which is combined with gray scale and edge matrix as input features. The multi -attention mechanism is used to add weight information to capture the importance of each feature more sensitive. Then, the Parallel Convolutional Neural Networks are used to fully exploit the deep semantic information, and IndRNN is introduced to avoid the loss of pixel hierarchy information and realize the integration of information flow. Finally, the Softmax classifier is used to complete the medical image segmentation task. Experiments showed that word embedding MACIR algorithm could effectively improve the segmentation performance of medical images on the data sets of lung X-ray and cervical CT images.

Visual Sentiment Prediction with Attribute Augmentation and Multi-attention Mechanism

Recently, many methods that exploit attention mechanism to discover the relevant local regions via visual attributes, have demonstrated promising performance in visual sentiment prediction. In these methods, accurate detection of visual attributes is of vital importance to identify the sentiment relevant regions, which is crucial for successful assessment of visual sentiment. However, existing work merely utilize basic strategies on convolutional neural network for visual attribute detection and fail to obtain satisfactory results due to the semantic gap between visual features and subjective attributes. Moreover, it is difficult for existing attention models to localize subtle sentiment relevant regions, especially when the performance of attribute detection is relatively poor. To address these problems, we first design a multi-task learning based approach for visual attribute detection. By augmenting the attributes with sentiments supervision, the semantic gap can be effectively reduced. We then develop a multi-attention model for jointly discovering and localizing multiple relevant local regions given predicted attributes. The classifier built on top of these regions achieves a significant improvement in visual sentiment prediction. Experimental results demonstrate the superiority of our method against previous approaches.

Multi-Attention-Based Capsule Network for Uyghur Personal Pronouns Resolution

Anaphora resolution of Uyghur is a challenging task because of complex language structure and limited corpus. We propose a multi-attention based capsule network model for Uyghur personal pronouns resolution, which can obtain the multi-layer and implicit semantic information effectively. Independently recurrent neural network (IndRNN) is applied in this model to achieve the interdependent features with long distance. Moreover, the capsule network can extract richer textual information to improve expression ability. Compared with the single attention-based model which combines Long Short-Term Memory (LSTM), the multi-attention based capsule network can capture multi-layer semantic information through a multi-attention mechanism without using any external parsing results. Experimental results on Uyghur dataset show that our approach surpasses the state-of-the-art models and gets the highest F-score of 83.85%. Meanwhile, our experimental results demonstrate the proposed method can effectively improve the performance of Uyghur personal pronouns resolution.

Improved Relativistic Cycle-Consistent GAN With Dilated Residual Network and Multi-Attention for Speech Enhancement

Generative adversarial networks (GANs) have been increasingly used as feature mapping functions in speech enhancement, in which the noisy speech features are transformed to the clean ones through the generators. This article proposes a novel speech enhancement model based on a cycle-consistent relativistic GAN with Dilated Residual Networks and a Multi-attention mechanism. Using the adversarial loss, improved cycle-consistency losses, and an identity-mapping loss, a noisy-to-clean generator G and an inverse clean-to-noisy generator F simultaneously learn the forward and backward mappings between the source and target domains. To guarantee the stability of the training process, we replace vanilla GAN loss with relativistic average GAN loss and use spectral normalization in discriminators so that they conform to Lipschitz continuity. Furthermore, we employ two attention-based components as multi-attention mechanism to reduce importing signal distortion: attention U-net gates and dilated residual self-attention blocks. By employing these components, our proposed generators can capture long-term inner dependencies between elements of speech features and further preserve linguistic information. Experimental results on a public dataset indicate that the proposed model achieves state-of-the-art speech enhancement performance, especially in reducing speech distortion and improving signal overall quality. Compared with the representative GAN-based approaches, the proposed method significantly achieves the best performance in terms of STOI, CSIG, COVL, and CBAK objective metrics. Moreover, we demonstrate the contribution of each proposed component including relativistic average loss, attention U-net gate, self-attention layers, spectral normalization, and dilation operation by ten comparison systems.

CephaNN: A Multi-Head Attention Network for Cephalometric Landmark Detection

Cephalometric landmark detection is a crucial step in orthodontic and orthognathic treatments. To detect cephalometric landmarks accurately, we propose a novel multi-head attention neural network (CephaNN). CephaNN is an end-to-end network based on the heatmaps of annotated landmarks, and it consists of two parts, the multi-head part and the attention part. In the multi-head part, we adopt multi-head subnets to gain comprehensive knowledge of various subspaces of a cephalogram. The intermediate supervision is applied to accelerate the convergence. Based on the feature maps learned from the multi-head Part, the attention part applies the multi-attention mechanism to obtain a refined detection. For solving the class imbalance problem, we propose a region enhancing (RE) loss, to enhance the efficient regions on the regressed heatmaps. Experiments in the benchmark dataset demonstrate that CephaNN is state-of-the-art with the detection accuracy of 87.61% in the clinically accepted 2.0-mm range. Furthermore, CephaNN is efficient in classifying the anatomical types and robust in a real application on a 75-landmark dataset.

Hybrid Deep Neural Networks for Friend Recommendations in Edge Computing Environment

With the rising popularity of social networks and service recommendations, research on new methods of friend recommendation have become a key topic, especially when based on quality-driven resource processing in an edge computing environment. Traditional methods seldom systematically combine static attributes (e.g., interests, geographical locations, and common friends), dynamic behaviors (e.g., liking, making comments, forwarding and @), and network structures (e.g., social ties) to recommend a new friend to a target user. Meanwhile, with the advent of deep learning, it has become more challenging to integrate these features into a deep neural network framework for friend recommendation. For example, how do we optimally make use of these features to form a united framework and what type of deep neural network architecture should be introduced into a novel recommendation method in an edge computing environment? In this paper, we propose DFRec++, a hybrid deep neural network framework combining attribute attention and network embeddings to make social friend recommendations with the help of both interactive semantics and contextual enhancement. More specifically, we first utilize the latent dirichlet allocation (LDA) topic model to generate common interest topics between users and compute the similarity of the explicit static attribute vector representation of topics, locations, and common friends. Then we feed dynamic behavior attributes into a convolutional neural network (CNN) to obtain the implicit vector representation of the interactions and context between two users. Subsequently, a multi-attention mechanism is designed to further improve the deep vector representation of the attribute information. Next, the LINE-based network embeddings algorithm is applied to embed the network structure into a low-dimensional vector. Finally, the attribute attention vector and the network embeddings are concatenated to form a deep feature representation, which is subsequently fed to a fully connected neural network (FCNN) to capture the probability of friendship between two users. The output of FCNN indicates the probability of two users becoming friends. We conducted experiments on a real-world Weibo dataset and the results show that DFRec++ outperforms several existing methods.

Multi-Attention Generative Adversarial Network for image captioning

Recently, it has been shown that generative-adversarial-nets (GANs) can be directly utilized as an extension of traditional reinforcement-learning in image captioning tasks. However, the GANs-based methods generate captions as a function of only local points in the feature map without capturing non-local information. In this paper, a Multi-Attention mechanism is first proposed by utilizing both of the local and non-local evidence for more effective feature representation and reasoning in image captioning. Based on the mechanism, a Multi-Attention Generative Adversarial Image Captioning Network (MAGAN) is also proposed which contains a Multi-Attention generator and a Multi-Attention discriminator. The proposed generator is designed to generate more accurate sentences, while the proposed discriminator is employed to determine whether generated sentences are human described or machine generated. Extensive experiments are conducted to validate the proposed framework on MSCOCO benchmark dataset, and it achieves very competitive results evaluated by the evaluation server of MS COCO captioning challenge.