Two-level Attention Mechanism Research Articles

Chinese text dual attention network for aspect-level sentiment classification.

English text has a clear and compact subject structure, which makes it easy to find dependency relationships between words. However, Chinese text often conveys information using situational settings, which results in loose sentence structures, and even most Chinese comments and experimental summary texts lack subjects. This makes it challenging to determine the dependency relationship between words in Chinese text, especially in aspect-level sentiment recognition. To solve this problem faced by Chinese text in the field of sentiment recognition, a Chinese text dual attention network for aspect-level sentiment recognition is proposed. First, Chinese syntactic dependency is proposed, and sentiment dictionary is introduced to quickly and accurately extract aspect-level sentiment words, opinion extraction and classification of sentimental trends in text. Additionally, in order to extract context-level features, the CNN-BILSTM model and position coding are also introduced. Finally, to better extract fine-grained aspect-level sentiment, a two-level attention mechanism is used. Compared with ten advanced baseline models, the model's capabilities are being further optimized for better performance, with Accuracy of 0.9180, 0.9080 and 0.8380 respectively. This method is being demonstrated by a vast array of experiments to achieve higher performance in aspect-level sentiment recognition in less time, and ablation experiments demonstrate the importance of each module of the model.

Leveraging a Two-Level Attention Mechanism for Deep Face Recognition with Siamese One-Shot Learning

Discriminative feature embedding is used for largescale facial recognition. Many image-based facial recognition networks use CNNs like ResNets and VGG-nets. Humans prioritise different elements, but CNNs treat all facial pictures equally. NLP and computer vision use attention to learn the most important part of an input signal. The inter-channel and inter-spatial attention mechanism is used to assess face image component significance in this study. Channel scalars are calculated using Global Average Pooling in face recognition channel attention. A recent study found that GAP encodes low-frequency channel information first. We compressed channels using discrete cosine transform (DCT) instead of scalar representation to evaluate information at frequencies other than the lowest frequency for the channel attention mechanism. Later layers can acquire the feature map after spatial attention. Channel and spatial attention increase CNN facial recognition feature extraction. Channel-only, spatial-only, parallel, sequential, or channel-after-spatial attention blocks exist. Current face recognition attention approaches may be outperformed on public datasets (Labelled Faces in the Wild).

Heterogeneous graph attention network with motif clique

Heterogeneous information networks (HINs) have been proven to be powerful in modeling various real-world networks, such as academic networks and social media networks. By distinguishing the types of relationships and nodes, HINs obtain the capability to model multifarious data, while bringing challenges to data mining and analysis. Specifically, how to comprehensively mine the semantic information in HINs remains an open question. Although recently studied network schemas such as meta-paths or meta-graphs have achieved empirical success, they still have limitations in modeling the complex underlying relations to facilitate the graph neural networks, e.g., different schemas may overlap in semantics and lead to redundant computations. To address this issue, we leverage a recently introduced higher-order network schema known as the motif clique (abbreviated as m-clique). This schema offers greater expressiveness, enabling us to effectively construct the semantic neighborhood of nodes. We further propose a novel heterogeneous graph attention network with m-cliques, named HAMC, which employs a two-level attention mechanism (node-level and semantic-level) to learn node representations. The two-level attention measures the importance of neighbors and m-cliques for each node, respectively. Extensive experiments demonstrate that the proposed HAMC outperforms the state-of-the-art methods in many heterogeneous network analytic tasks such as node classification and clustering. Our code is available at https://github.com/wcx21/HAMC-Heterogeneous-Graph-Attention-Network-with-Motif-Clique.

Learning implicit sentiments in Alzheimer's disease recognition with contextual attention features.

Alzheimer's disease (AD) is difficult to diagnose on the basis of language because of the implicit emotion of transcripts, which is defined as a supervised fuzzy implicit emotion classification at the document level. Recent neural network-based approaches have not paid attention to the implicit sentiments entailed in AD transcripts. A two-level attention mechanism is proposed to detect deep semantic information toward words and sentences, which enables it to attend to more words and fewer sentences differentially when constructing document representation. Specifically, a document vector was built by progressively aggregating important words into sentence vectors and important sentences into document vectors. Experimental results showed that our method achieved the best accuracy of 91.6% on annotated public Pitt corpora, which validates its effectiveness in learning implicit sentiment representation for our model. The proposed model can qualitatively select informative words and sentences using attention layers, and this method also provides good inspiration for AD diagnosis based on implicit sentiment transcripts.

Dual-Branch Network Fused With Two-Level Attention Mechanism for Clothes-Changing Person Re-Identification

Clothes-changing person re-identification is a hot topic in the current academic circles. Most of the current methods assume that the clothes of a person will not change in a short period of time, but they are not applicable when people change clothes. Based on this situation, this paper proposes a dual-branch network for clothes-changing person re-identification that integrates a two-level attention mechanism and captures and aggregates fine-grained person semantic information in channels and spaces through a two-level attention mechanism and suppresses the sensitivity of the network to clothing features by training the clothing classification branch. The method does not use auxiliary means such as human skeletons, and the complexity of the model is greatly reduced compared with most methods. This paper conducts experiments on the popular clothes-changing person re-identification dataset PRCC and a very large-scale cross-spatial-temporal dataset (LaST). The experimental results show that the method in this paper is more advanced than the existing methods.

Heterogeneous Feature Based Time Series Classification With Attention Mechanism

Time series classification (TSC) problem has been a significantly attractive research problem for decades. A large number of models with various types of features have been proposed. However, with the rapid development of new applications, like IoT and intelligent manufacturing, the time series data from different industries and applications are constantly emerging. To classify these data accurately, data scientists face the challenges of 1) how to select the optimal features and classification models and 2) how to interpret the results. To tackle these two challenges, in this paper, we propose a heterogeneous feature ensemble network, named FEnet. Multiple features, including time-domain, frequency-domain, and so on, are combined to build the model so that it can deal with the diversity of the data characteristics. Furthermore, to improve the interpretability, we propose a two-level attention mechanism. Finally, we propose two model optimization strategies to enhance classification accuracy and efficiency. Extensive experiments are conducted on real datasets and the results verify the accuracy, operation efficiency, and interpretability of FEnet.

FIRE: knowledge-enhanced recommendation with feature interaction and intent-aware attention networks.

To solve the information overload issue and enhance the user experience of various web applications, recommender systems aim to better model user interests and preferences. Knowledge Graphs (KGs), consisting of real-world objective facts and fruitful entities, play a vital role in recommender systems. Recently, a technological trend has been to develop end-to-end Graph Neural Networks (GNNs)-based knowledge-aware recommendation (a.k.a., Knowledge Graph Recommendation, KGR) models. Unfortunately, current GNNs-based KGR approaches focus on how to capture high-order feature information on KGs while neglecting the following two crucial limitations: 1) The explicitly high-order feature interaction and fusion mechanism and 2) The valid user intent modelling mechanism. As such, these issues lead to insufficient user/item representation learning capability and unsatisfactory KGR performance. In this work, we present a novel Knowledge-enhanced Re commendation with F eature I nteraction and Intent-aware Attention Networks (FIRE) to address the latent intent modelling and high-order feature interaction deficiencies ignored by existing KGR methods. Based on the prototype user/item representation learning leveraging the GNNs-based approach, our model offers the following major improvements: One is the innovative use of Convolutional Neural Networks (CNNs) that perform vertical convolutional (a.k.a., bit-level convolutional) and horizontal convolutional (a.k.a., vector-level convolutional) processes to model multi-granular high-order feature interactions to enhance item-side representation learning. Another is to model users' latent intent factors by utilizing a two-level attention mechanism (i.e., node- and intent-level attention mechanism) to enhance user-side representation learning. Extensive experiments on three KGs domain public datasets demonstrate that our method outperforms the existing state-of-the-art baseline. Last but not least, numerous ablation- and model studies demystify the working mechanism and elucidate the plausibility of the proposed model.

DeepMPM: a mortality risk prediction model using longitudinal EHR data

BackgroundAccurate precision approaches have far not been developed for modeling mortality risk in intensive care unit (ICU) patients. Conventional mortality risk prediction methods can hardly extract the information in longitudinal electronic medical records (EHRs) effectively, since they simply aggregate the heterogeneous variables in EHRs, ignoring the complex relationship and interactions between variables and the time dependence in longitudinal records. Recently deep learning approaches have been widely used in modeling longitudinal EHR data. However, most existing deep learning-based risk prediction approaches only use the information of a single disease, neglecting the interactions between multiple diseases and different conditions.ResultsIn this paper, we address this unmet need by leveraging disease and treatment information in EHRs to develop a mortality risk prediction model based on deep learning (DeepMPM). DeepMPM utilizes a two-level attention mechanism, i.e. visit-level and variable-level attention, to derive the representation of patient risk status from patient’s multiple longitudinal medical records. Benefiting from using EHR of patients with multiple diseases and different conditions, DeepMPM can achieve state-of-the-art performances in mortality risk prediction.ConclusionsExperiment results on MIMIC III database demonstrates that with the disease and treatment information DeepMPM can achieve a good performance in terms of Area Under ROC Curve (0.85). Moreover, DeepMPM can successfully model the complex interactions between diseases to achieve better representation learning of disease and treatment than other deep learning approaches, so as to improve the accuracy of mortality prediction. A case study also shows that DeepMPM offers the potential to provide users with insights into feature correlation in data as well as model behavior for each prediction.

Enabling Timely Medical Intervention by Exploring Health-Related Multivariate Time Series with a Hybrid Attentive Model

Modern healthcare practice, especially in intensive care units, produces a vast amount of multivariate time series of health-related data, e.g., multi-lead electrocardiogram (ECG), pulse waveform, blood pressure waveform and so on. As a result, timely and accurate prediction of medical intervention (e.g., intravenous injection) becomes possible, by exploring such semantic-rich time series. Existing works mainly focused on onset prediction at the granularity of hours that was not suitable for medication intervention in emergency medicine. This research proposes a Multi-Variable Hybrid Attentive Model (MVHA) to predict the impending need of medical intervention, by jointly mining multiple time series. Specifically, a two-level attention mechanism is designed to capture the pattern of fluctuations and trends of different time series. This work applied MVHA to the prediction of the impending intravenous injection need of critical patients at the intensive care units. Experiments on the MIMIC Waveform Database demonstrated that the proposed model achieves a prediction accuracy of 0.8475 and an ROC-AUC of 0.8318, which significantly outperforms baseline models.

Enhancing the Interpretability of Deep Models in Healthcare Through Attention: Application to Glucose Forecasting for Diabetic People

The adoption of deep learning in healthcare is hindered by their “black box” nature. In this paper, we explore the RETAIN architecture for the task of glucose forecasting for diabetic people. By using a two-level attention mechanism, the recurrent-neural-network-based RETAIN model is interpretable. We evaluate the RETAIN model on the type-2 IDIAB and the type-1 OhioT1DM datasets by comparing its statistical and clinical performances against two deep models and three models based on decision trees. We show that the RETAIN model offers a very good compromise between accuracy and interpretability, being almost as accurate as the LSTM and FCN models while remaining interpretable. We show the usefulness of its interpretable nature by analyzing the contribution of each variable to the final prediction. It revealed that signal values older than 1[Formula: see text]h are not used by the RETAIN model for 30[Formula: see text]min ahead of time prediction of glucose. Also, we show how the RETAIN model changes its behavior upon the arrival of an event such as carbohydrate intakes or insulin infusions. In particular, it showed that the patient’s state before the event is particularly important for the prediction. Overall the RETAIN model, thanks to its interpretability, seems to be a very promising model for regression or classification tasks in healthcare.

Seatbelt detection in road surveillance images based on improved dense residual network with two-level attention mechanism

Seatbelt detection is an important topic in intelligent transport systems. The accuracy of seatbelt detection in traffic video surveillance is affected by many factors, such as complex road environments, lighting, weather, direction of the camera, etc., which bring difficulties for traditional image processing methods. We propose a seatbelt detection method based on classification using convolutional neural networks. Dense residual blocks are used to avoid gradient dispersion and information loss during the network training. A two-level attention mechanism is also introduced to further improve the network performance by simultaneously modeling the correlation among different feature channels using channel attention mechanism, and correlation among different pixels within the feature map using pixel attention mechanism. To make the whole system complete, we added the driver area localization module, which is accomplished using the YOLOV3 model. Small images concentrated on the driver location, which are obtained through the localization operation from larger images, are passed into the classification network for seatbelt detection. We compared the proposed method with several other methods. Comparative results show that the proposed method has higher accuracy and is more robust for seatbelt detection for complex environments.

$A^3$-FKG: Attentive Attribute-Aware Fashion Knowledge Graph for Outfit Preference Prediction

With the booming development of the online fashion industry, effective personalized recommender systems have become indispensable for the convenience they brought to the customers and the profits to the e-commercial platforms. Estimating the user’s preference towards the outfit is at the core of a personalized recommendation system. Existing works on fashion recommendation are largely centering on modelling the clothing compatibility without considering the user factor or characterizing the user’s preference over the single item. However, how to effectively model the outfits with either few or even none interactions, is yet under-explored. In this paper, we address the task of personalized outfit preference prediction via a novel <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A</b> ttentive <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A</b> ttribute- <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A</b> ware <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</b> ashion <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</b> nowledge <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">G</b> raph ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$A^3$</tex-math></inline-formula> -FKG), which is incorporated to build the association between different outfits with both outfit- and item- level attributes. Additionally, a two-level attention mechanism is developed to capture the user’s preference: 1) User-specific relation-aware attention layer, which captures the user’s fine-grained preferences with different focus on relations for learning outfit representation; 2) Target-aware attention layer, which characterizes the user’s latent diverse interests from his/her behavior sequences for learning user representation. Extensive experiments conducted on a large-scale fashion outfit dataset demonstrate significant improvements over other methods, which verify the excellence of our proposed framework.

Hierarchical Attentive Transaction Embedding With Intra- and Inter-Transaction Dependencies for Next-Item Recommendation

A transaction-based recommender system (TBRS) aims to predict the next item by modeling dependencies in transactional data. Generally, two kinds of dependencies considered are intra-transaction dependence and inter-transaction dependence. Most existing TBRSs recommend next item by only modeling the intra-transaction dependence within the current transaction while ignoring inter-transaction dependence with recent transactions that may also affect the next item. However, as not all recent transactions are relevant to the current and next items, the relevant ones should be identified and prioritized. In this article, we propose a novel hierarchical attentive transaction embedding (HATE) model to tackle these issues. Specifically, a two-level attention mechanism integrates both item embedding and transaction embedding to build an attentive context representation that incorporates both intra- and inter-transaction dependencies. With the learned context representation, HATE then recommends the next item. Experimental evaluations on two real-world transaction datasets show that HATE significantly outperforms the state-of-the-art methods in terms of recommendation accuracy.

Driver Drowsiness Recognition via 3D Conditional GAN and Two-Level Attention Bi-LSTM

Driver drowsiness has currently been a severe issue threatening road safety, hence it is vital to develop an effective drowsiness recognition algorithm to avoid traffic accidents. However, recognizing drowsiness is still very challenging, due to the large intra-class variations in facial expression, head pose and illumination condition. In this paper, a new deep learning framework based on the hybrid of 3D conditional generative adversarial network and two-level attention bidirectional long short-term memory network (3DcGAN-TLABiLSTM) has been proposed for robust driver drowsiness recognition. Aiming at extracting short-term spatial-temporal features with abundant drowsiness-related information, we design a 3D encoder-decoder generator with the condition of auxiliary information to generate high-quality fake image sequences and devise a 3D discriminator to learn drowsiness-related representation from spatial-temporal domain. In addition, for long-term spatial-temporal fusion, we investigate the use of two-level attention mechanism to guide the bidirectional long short-term memory learn the saliency of short-term memory information and long-term temporal information. For experiment, we evaluate our 3DcGAN-TLABiLSTM framework on a public NTHU-DDD dataset. Experimental results show that the proposed approach achieves higher precision of drowsiness recognition compared to the state-of-the-art.

Relational Graph Neural Network with Hierarchical Attention for Knowledge Graph Completion

The rapid proliferation of knowledge graphs (KGs) has changed the paradigm for various AI-related applications. Despite their large sizes, modern KGs are far from complete and comprehensive. This has motivated the research in knowledge graph completion (KGC), which aims to infer missing values in incomplete knowledge triples. However, most existing KGC models treat the triples in KGs independently without leveraging the inherent and valuable information from the local neighborhood surrounding an entity. To this end, we propose a Relational Graph neural network with Hierarchical ATtention (RGHAT) for the KGC task. The proposed model is equipped with a two-level attention mechanism: (i) the first level is the relation-level attention, which is inspired by the intuition that different relations have different weights for indicating an entity; (ii) the second level is the entity-level attention, which enables our model to highlight the importance of different neighboring entities under the same relation. The hierarchical attention mechanism makes our model more effective to utilize the neighborhood information of an entity. Finally, we extensively validate the superiority of RGHAT against various state-of-the-art baselines.

Research on relation extraction of named entity on social media in smart cities

Social media make significant contribution to the evolution of smart cities. The key issue of smart cities is to develop a series of automatic methods to support smart applications. As one of the basic techniques of smart cities, the task of relation extraction of named entities on social media provides an indispensable means to construct and expand knowledge map and contributes to the utilization of information resources in smart cities. What’s more, it is conducive to improve the efficiency of network supervision. This paper proposes an automatic method to extract entity relations via deep learning techniques on a two-level neural network named Bi-CLSTM. The research conquers some challenges of relation extraction on social media. To extract entity relations on conversation scenarios, Bi-CLSTM represents texts with the strategy of “word embedding + position embedding + shortest dependency path” and extracts relations via a hybrid model of LSTM and PCNN. The nodes and networks of Bi-CLSTM are designed to adapt to the scenarios of conversation and over-sentence. To reduce the dependency on training data, distant supervised strategy is employed and a two-level attention mechanism is used to prevent noise signals. Experiments are carried out on Sina Microblog corpus, and the results show that Bi-CLSTM model makes outstanding performance.