Hierarchical Attention Mechanism Research Articles

A unified hierarchical attention framework for sequential recommendation by fusing long and short-term preferences

Sequential recommendation becomes a critical task in many application scenarios, since people’s online activities are increasing. In order to predict the next item that users may be interested, it is necessary to take both general and dynamic preferences of the user into account. Existing approaches typically integrate the user–item or item–item feature interactions directly without considering the dynamic changes of the user’s long-term and short-term preferences, which also limits the capability of the model. To address these issues, we propose a novel unified framework for sequential recommendation task, modeling users’ long and short-term sequential behaviors at each time step and capturing item-to-item dependencies in higher-order by hierarchical attention mechanism. The proposed model considers the dynamic long and short-term user preferences simultaneously, and a joint learning mechanism is introduced to fuse them for better recommendation. We extensively evaluate our model with several state-of-the-art methods by different validation metrics on three real-world datasets. The experimental results demonstrate the significant improvement of our approach over other compared models.

An Attention-Based Multi-Representational Fusion Method for Social-Media-Based Text Classification

There exist various text-classification tasks using user-generated contents (UGC) on social media in the big data era. In view of advantages and disadvantages of feature-engineering-based machine-learning models and deep-learning models, we argue that fusing handcrafted-text representation via feature engineering and data-driven deep-text representations extracted by performing deep-learning methods is conducive to enhancing text-classification capability. Given the characteristics of different deep neural networks, their complementary effect needs to be investigated. Moreover, contributions of these representations need to be adaptively learned when it comes to addressing different tasks or predicting different samples. Therefore, in this paper, we propose a novel fused deep-neural-network architecture with a hierarchical attention mechanism for text classification with social media data. Specifically, in the context that handcraft features are available, we employ the attention mechanism to adaptively fuse totally data-driven-text representation and handcrafted representation. For the generation of the data-driven-text representation, we propose a data-driven encoder that fuses text representations derived from three deep-learning methods with the attention mechanism, to adaptively select discriminative representation and explore their complementary effect. To verify the effectiveness of our approach, we performed two text-classification tasks, i.e., identifying adverse drug reaction (ADR)-relevant tweets from social media and identifying comparative-relevant reviews from an E-commerce platform. Experimental results demonstrate that our approach outperforms other baselines.

Strategies for Improving Text Reading Ability Based on Human-Computer Interaction in Artificial Intelligence.

In order to improve text reading ability, a human-computer interaction method based on artificial intelligence (AI) human-computer interaction is proposed. Firstly, the design of the AI human-computer interaction model is constructed, which includes the Stanford Question Answering Dataset (SQuAD) and the designed baseline model. There are three components: the coding layer is based on a cyclic neural network (recurrent neural network [RNN] encoder layer), which aims to encode the problem and text into a hidden state; the interaction layer is used to integrate problems and text representation; the output layer connects two independent soft Max layers after a fully connected layer, one is used to obtain the starting position of the answer in the text and the other is used to obtain the ending position. In the interaction layer of the model, this manuscript uses hierarchical attention and aggregation mechanism to improve text coding. The traditional model interaction layer has a simple structure, which leads to weak relevance between text and problems, and poor understanding ability of the model. Finally, the self-attention model is used to further enhance the feature representation of text. The experimental results show that the improved model in this manuscript is compared with the public AI human-computer interaction reading comprehension model. According to the data in the table, the accuracy of the model in this manuscript is better than that of the baseline model, in which the exact match (EM) value is increased by 1.4% and the F1 value is increased by 2.7%. However, compared with improvement point 2, the EM and F1 values of the model have decreased by 0.7%. It shows that the output layer has a certain impact on the effect of the model, and the improvement and optimization of the output layer can also improve the performance of the model. It is proved that AI human-computer interaction can effectively improve text reading ability.

Uncertainty-aware hierarchical segment-channel attention mechanism for reliable and interpretable multichannel signal classification

Multichannel signal data analysis has been crucial in various industrial applications, such as human activity recognition, vehicle failure predictions, and manufacturing equipment monitoring. Recently, deep neural networks have come into use for multichannel signal data because of their ability to automatically extract useful features from complex multichannel signals. However, deep neural networks are black-box models whose internal working mechanisms cannot be put in a form readily understood by humans. To address this issue, we have proposed an uncertainty-aware hierarchical segment-channel attention model that consists of a time segment and channel level attentions. The hierarchical attention mechanism enables a neural network to identify important time segments and channels critical for prediction, making the model explainable. In addition, the model uses variational inferences to provide uncertainty information that yields a confidence interval that can be easily explained. We conducted experiments on simulated and real-world datasets to demonstrate the usefulness and applicability of our method. The results confirm that our method can attend to important time segments and sensors while achieving better classification performance.

Aspect-level sentiment analysis for based on joint aspect and position hierarchy attention mechanism network

The purpose of aspect-level sentiment analysis is to identify the contextual sentence expressions given by sentiment for some aspects. For previous works, many scholars have proved the importance of the interaction between aspects and contexts. However, most existing methods ignore or do not specifically capture the position information of the aspect targets in the sentence. Thus, we propose an aspect-level sentiment analysis based on joint aspect and position hierarchy attention mechanism network. At the same time, the model adopts a joint approach to make the model of the aspect features and the position features. On the one hand, this method clearly captures the interaction between aspect words and context when inputting word vector information. On the other hand, this method can enhance the importance of position information in the sentence and boost the information retrieval ability of the model. Additionally, the model utilizes a hierarchical attention mechanism to extract feature information and to differentiate sentiment towards, which is similar to filtering useless information again. Experiment on the SemEval 2014 dataset represent that our model achieves better performance on aspect-level sentiment classification.

Dual Co-Attention-Based Multi-Feature Fusion Method for Rumor Detection

Social media has become more popular these days due to widely used instant messaging. Nevertheless, rumor propagation on social media has become an increasingly important issue. The purpose of this study is to investigate the impact of various features in social media on rumor detection, propose a dual co-attention-based multi-feature fusion method for rumor detection, and explore the detection capability of the proposed method in early rumor detection tasks. The proposed BERT-based Dual Co-attention Neural Network (BDCoNN) method for rumor detection, which uses BERT for word embedding. It simultaneously integrates features from three sources: publishing user profiles, source tweets, and comments. In the BDCoNN method, user discrete features and identity descriptors in user profiles are extracted using a one-dimensional convolutional neural network (CNN) and TextCNN, respectively. The bidirectional gate recurrent unit network (BiGRU) with a hierarchical attention mechanism is used to learn the hidden layer representation of tweet sequence and comment sequence. A dual collaborative attention mechanism is used to explore the correlation among publishing user profiles, tweet content, and comments. Then the feature vector is fed into classifier to identify the implicit differences between rumor spreaders and non-rumor spreaders. In this study, we conducted several experiments on the Weibo and CED datasets collected from microblog. The results show that the proposed method achieves the state-of-the-art performance compared with baseline methods, which is 5.2% and 5% higher than the dEFEND. The F1 value is increased by 4.4% and 4%, respectively. In addition, this paper conducts research on early rumor detection tasks, which verifies the proposed method detects rumors more quickly and accurately than competitors.

A hierarchical conditional random field-based attention mechanism approach for gastric histopathology image classification

In the Gastric Histopathology Image Classification (GHIC) tasks, which are usually weakly supervised learning missions, there is inevitably redundant information in the images. Therefore, designing networks that can focus on distinguishing features has become a popular research topic. In this paper, to accomplish the tasks of GHIC superiorly and assist pathologists in clinical diagnosis, an intelligent Hierarchical Conditional Random Field based Attention Mechanism (HCRF-AM) model is proposed. The HCRF-AM model consists of an Attention Mechanism (AM) module and an Image Classification (IC) module. In the AM module, an HCRF model is built to extract attention regions. In the IC module, a Convolutional Neural Network (CNN) model is trained with the attention regions selected, and then an algorithm called Classification Probability-based Ensemble Learning is applied to obtain the image-level results from the patch-level output of the CNN. In the experiment, a classification specificity of 96.67% is achieved on a gastric histopathology dataset with 700 images. Our HCRF-AM model demonstrates high classification performance and shows its effectiveness and future potential in the GHIC field. In addition, the AM module and transfer learning technique allow the network to generalize well to other types of image data except histopathology images, and we obtain 95.5% and 95.8% accuracies on IG02 and Oxford-IIIT Pet Datasets.

Deep learning-assisted analysis of automobiles handling performances

Abstract The luxury car market has demanding product development standards aimed at providing state-of-the-art features in the automotive domain. Handling performance is amongst the most important properties that must be assessed when developing a new car model. In this work, we analyse the problem of predicting subjective evaluations of automobiles handling performances from objective records of driving sessions. A record is a multi-dimensional time series describing the temporal evolution of the mechanical state of an automobile. A categorical variable quantifies the evaluations of handling properties. We describe an original deep learning system, featuring a denoising autoencoder and hierarchical attention mechanisms, that we designed to solve this task. Attention mechanisms intrinsically compute probability distributions over their inputs’ components. Combining this feature with the saliency maps technique, our system can compute heatmaps that provide a visual aid to identify the physical events conditioning its predictions.

Orientational Distribution Learning With Hierarchical Spatial Attention for Open Set Recognition.

Open set recognition (OSR) aims to correctly recognize the known classes and reject the unknown classes for increasing the reliability of the recognition system. The distance-based loss is often employed in deep neural networks-based OSR methods to constrain the latent representation of known classes. However, the optimization is usually conducted using the nondirectional euclidean distance in a single feature space without considering the potential impact of spatial distribution. To address this problem, we propose orientational distribution learning (ODL) with hierarchical spatial attention for OSR. In ODL, the spatial distribution of feature representation is optimized orientationally to increase the discriminability of decision boundaries for open set recognition. Then, a hierarchical spatial attention mechanism is proposed to assist ODL to capture the global distribution dependencies in the feature space based on spatial relationships. Moreover, a composite feature space is constructed to integrate the features from different layers and different mapping approaches, and it can well enrich the representation information. Finally, a decision-level fusion method is developed to combine the composite feature space and the naive feature space for producing a more comprehensive classification result. The effectiveness of ODL has been demonstrated on various benchmark datasets, and ODL achieves state-of-the-art performance.

Label Oriented Hierarchical Attention Neural Network for Short Text Classification

Text classification task is a very common task in natural language processing. The conventional text classification task model often uses word bag model or representation model, but the existing model usually deals with long text classification task, which is not suitable for short text classification. Short text features are relatively fewer and often need more sophisticated feature extraction, and the role of keywords play great importance roles in the classification of short text. In this paper, we propose a label-oriented hierarchical attention mechanism network for short text classification. The model achieves better results on public data sets of Tiao and Weibo, compared with convolutional neural network CNN, GATE control unit neural network GRU, gate control unit neural network fusion convolutional neural network GRU-CNN and translation model Transfomer. It is proved that the model has good performance. Our model has two significant advantages: (1) it is a hierarchical structure consisting of two levels of attention mechanisms, which facilitates interpretation and analysis; (2) Compared with other architectures, this model can be used to extend multi-label text classification tasks. In addition, it can be used for long-term importance analysis in many industrial scenarios.

Neural Network With Hierarchical Attention Mechanism for Contextual Topic Dialogue Generation

The encoder-decoder model has achieved remarkable results in natural language generation. However, in the dialogue generation work, we often ignore the influence of the dialogue context information and topic information in the generation, resulting in the generated replies not close to the context or lack of topic information leads to general responses. In this work, we study the generation of multi-turn dialogues based on a large corpus and take advantage of the context information and topic information of the conversation in the process of dialogue generation to generate more coherent context-sensitive responses. We improve upon existing models and attention mechanisms and propose a new hierarchical model to better solve the problem of dialogue context (the HAT model). This method enables the model to obtain more contextual information when processing and improves the ability of the model in terms of contextual relevance to produce high-quality responses. In addition, to address the absence of topics in the responses, we pre-train the LDA(Latent Dirichlet Allocation) topic model to extract topic words of the dialogue content and retain as much topic information of dialogue as possible. Our model is extensively tested in several corpora, and the experiments illustrate that our model is superior to most hierarchical and non-hierarchical models with respect to multiple evaluation metrics.

Modeling Health Stage Development of Patients with Dynamic Attributed Graphs in Online Health Communities

In this paper, we propose a novel DynAttGraph2Seq framework to model complex dynamic transitions of an individual user's activities and the textual information of the posts over time in online health forums and learning how these correspond to his/her health stage. To achieve this, we first formulate the transition of user activities as a dynamic attributed graph with multi-attributed nodes that evolves over time, then formalize the health stage inference task as a dynamic attributed graph to sequence learning problem. Our proposed model consists of a novel dynamic graph encoder along with a two-level sequential encoder to capture the semantic features from user posts and an interpretable sequence decoder that learn the mapping between a sequence of time-evolving user activity graphs as well as user posts to a sequence of target health stages. We go on to propose new dynamic graph regularization and dynamic graph hierarchical attention mechanisms to facilitate the necessary multi-level interpretability. A comprehensive experimental analysis of its use for a health stage prediction task demonstrates both the effectiveness and the interpretability of the proposed models.

THAN: Multimodal Transportation Recommendation With Heterogeneous Graph Attention Networks

Multi-modal transportation recommendation plays an important role in navigation applications. It aims to recommend a travel plan with various transport modes, such as bus, metro, taxi, bicycle, and a hybrid. Analysis of real-world large-scale navigation data shows that the correlation between the data can be represented by a graph containing different types of nodes and edges. As an emerging technology, graph neural networks (GNN) have shown powerful capabilities in representing graph data. However, existing solutions based on GNN only consider converting heterogeneous graph data into homogeneous graph data, ignoring the effects of different types of nodes and edges. In addition, those methods usually face the over-smoothing problem, which reduces the accuracy of recommendation. To this end, we propose a multi-modal <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <i>T</i> </b> ransportation recommendation algorithm with <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <i>H</i> </b> eterogeneous graph <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <i>A</i> </b> ttention <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <i>N</i> </b> etworks (THAN) based on carefully constructed heterogeneous graphs. We first design a novel graph embedding method to represent the correlation between the origin and the destination, as well as the correlation between origin-destination (OD) pairs and users. Next, a heterogeneous graph from large-scale data is built to describe the relationship between users, OD pairs, and transport modes. Then, we design a hierarchical attention mechanism with residual blocks to generate node embedding in terms of homogeneity and heterogeneity. Finally, a fusion neural layer is designed to fuse embeddings from different views and predict the proper transport mode for users. Extensive experimental results on a large-scale real-world dataset demonstrate that the performance of THAN outperforms five baselines.

HAM-Net: Predictive Business Process Monitoring with a hierarchical attention mechanism

One of the essential tasks in Business Process Management (BPM) is Predictive Business Process Monitoring. This task aims to predict the behavior of an ongoing process based on the historical data stored in event logs. Since feed-forward neural networks do not consider the order of events for the prediction, they may not be helpful in predictive process monitoring. Recent research shows that using Recurrent Neural Networks such as LSTM and GRU may not be also helpful in predictive process monitoring. Because these networks use only the last hidden state as the context vector, and may lose some of past information, especially in long sequences. In addition, many existing approaches just use the activity name of each event as the representative of that event. In this context, they may ignore other events’ attributes in generating the feature vector. Some works have utilized these attributes simply by concatenating all of them together. While we need to use all event attributes to predict the next activity, it should be noted that not all of them are equally important. In this paper, we use two layers of attention mechanism on top of LSTM: (i) at the attribute level, to determine which attributes have more importance; and (ii) at the event level, to identify important events in predicting the next activity. Experimental evaluation of the real-world event logs showed that the use of hierarchical attention mechanisms in the proposed approach could effectively predict the next activity of an ongoing process.

Improved strategies of relation extraction based on graph convolutional model on tree structure for web information processing

In the Industry 4.0/5.0 era, information integration is employed to fuse information from different companies to facilitate interoperation. However, information extraction is an important preprocessing phase that must be performed prior to integrating data from different contexts. Relation extraction, which is an element of information extraction, is typically the basis of many upper-level applications, e.g., information visualization and inference. Some current models may not fully consider the complementary effect of information at different levels of granularity featured by different neural networks. In this paper, two improvement relation extraction strategies based on the graph convolutional model on tree structure (GCNTree) are proposed. The first strategy integrates a hierarchical attention mechanism and correlation analysis between subjects and objects to generate sentence and entity vectors, respectively. The second strategy merges a named-entity recognition subnetwork with GCNTree to realize joint learning of relation and entity extraction. Experimental results demonstrate that the proposed strategies are comparable to state-of-the-art methods.11Our code will be available at https://github.com/yangshuodelove/.

Multi actor hierarchical attention critic with RNN-based feature extraction

Deep reinforcement learning has made significant progress in multi-agent tasks in recent years. However, most previous studies focus on solving fully cooperative tasks, which do not perform well in mixed tasks. In mixed tasks, the agent needs to comprehensively consider the information provided by its friends and enemies to learn its strategy, and its strategy is sensitive to the received information. Additionally, the input space of the critic network increases rapidly with the number of agents in the actor-critic framework. It’s of great necessity to efficiently learn information representation to obtain important features. To this end, we present an approach that conducts information representation with attention mechanism. Our approach adopts the framework of centralized training and decentralized execution. We apply the multi-head hierarchical attention mechanism to centrally computed critics, so critics can process the received information more accurately and assist actors in choosing better actions. The hierarchical attention critic adopts a bi-level attention structure which is composed of the agent-level and the group-level. They are designed to assign different weights to friends’ and enemies’ information and then summarize them at each timestep. It achieves high efficiency and scalability in mixed tasks. Furthermore, we use the feature extraction based on the recurrent neural network to encode the state-action sequence information of each agent. Experimental results show that our approach is not only applicable to cooperative environments but also better in mixed environments, especially in the predator-prey task, the reward obtained by our method is twice that of the baselines.

TAGAT: Type-Aware Graph Attention neTworks for reasoning over knowledge graphs

With the rapid development of knowledge graphs (KGs), various AI-related applications have been affected positively. However, even though some KGs are relatively large, they still suffer from incompleteness. This has mighty promoted the development of reasoning over KGs to complement them. However, most existing reasoning methods only focus on semantics in the KG, ignoring potential or valuable information hidden in it, the most typical ones are neighborhood information and type information. This limits not only the reasoning performance but also the interpretability of the embedding space. To this end, we propose a Type-Aware Graph Attention neTworks (TAGAT) for the reasoning task over KGs. Except for combining type-related information during the embedding process, TAGAT further adopts a hierarchical attention mechanism to realize the perception of type and neighborhood information meticulously. For each entity, different attention levels respectively focus on considering the contribution of different relations, different types under each relation and different entity of each type under each relation. Moreover, the embedding space of TAGAT is constrained by the type information and naturally has better type-related interpretability, which greatly complements the defects of existing KGC models. Extensive experiments validate the high reasoning performance and the relatively ideal interpretability of our model.

Hierarchical Attention Link Prediction Neural Network

In this paper, a novel end-to-end neural link prediction model, named Hierarchical Attention Link Prediction Neural Network (HalpNet), is proposed. HalpNet comprehensively explores neighborhood information, which has proved important for link prediction, via the core component, i.e., hierarchical attention mechanism. The proposed hierarchical attention mechanism consists of two neural attention layers, modeling crucial structure information at node level and subgraph level, respectively. At node level, a structure-preserving attention is developed to preserve structure features of each node in the neighborhood subgraph. Based on the latent node features, at subgraph level, a structure-aggregating attention is designed to learn how important that each node in the subgraph is for the linkage of the target node pair and aggregate node features with learned attentions as a comprehensive subgraph representation. Given this expressive representation of neighborhood subgraph, HalpNet is able to predict link score of target node pair effectively. We evaluate HalpNet on 8 benchmark datasets against 14 popular and state-of-the-art approaches. The experimental results demonstrate its significant superiority and wide applicability on link prediction problem.

A Pronunciation Prior Assisted Vowel Reduction Detection Framework with Multi-Stream Attention Method

Vowel reduction is a common pronunciation phenomenon in stress-timed languages like English. Native speakers tend to weaken unstressed vowels into a schwa-like sound. It is an essential factor that makes the accent of language learners sound unnatural. To improve vowel reduction detection in a phoneme recognition framework, we propose an end-to-end vowel reduction detection method that introduces pronunciation prior knowledge as auxiliary information. In particular, we have designed two methods for automatically generating pronunciation prior sequences from reference texts and have implemented a main and auxiliary encoder structure that uses hierarchical attention mechanisms to utilize the pronunciation prior information and acoustic information dynamically. In addition, we also propose a method to realize the feature enhancement after encoding by using the attention mechanism between different streams to obtain expanded multi-streams. Compared with the HMM-DNN hybrid method and the general end-to-end method, the average F1 score of our approach for the two types of vowel reduction detection increased by 8.8% and 6.9%, respectively. The overall phoneme recognition rate increased by 5.8% and 5.0%, respectively. The experimental part further analyzes why the pronunciation prior knowledge auxiliary input is effective and the impact of different pronunciation prior knowledge types on performance.

Hierarchical temporal–spatial preference modeling for user consumption location prediction in Geo-Social Networks

Predicting where people will consume in the future is of great significance for promoting local business. Although the prevalence of Geo-Social Networks (GSNs) has provided sufficient and desirable geo-tagged data for user mobility modeling, most studies attempt to directly fit user’s preference toward locations through exploring the complex interaction between user,location pairs, which is usually hard to incorporate temporal–spatial context and side information. Moreover, the availability of multi-modal data associated with both user and location in GSNs has not yet been comprehensively leveraged. In view of the above-mentioned situations, in this article, we propose a two-stage framework composed of a Temporal Base Model (TBM) and a Location Prediction Model (LPM) to accomplish the task of user consumption location prediction at a given time in the future. In the first stage, based on user sentimental textual reviews, we leverage the hierarchical attention mechanism to capture time-sensitive user latent preference. In the second stage, we fuse the multifaceted context to derive the user’s consumption probability toward different locations at the given time. We conduct extensive experiments over three real-world GSN datasets to verify the performance of the proposed approach. The experimental results encouragingly demonstrate the effectiveness of the two-stage framework, which outperforms multiple baselines in terms of different evaluation metrics such as accuracy, average percentile rank (APR) and coverage ratio.