Two-level Attention Research Articles

Multi-task learning for spatial events prediction from social data

Multi-task learning is becoming more popular and is being applied in a variety of applications. It improves the accuracy of prediction by simultaneously learning related tasks and saves cost through shared structures. In particular, the prediction of event type from social data is also an area where multi-task learning can be utilized. In this paper, we present a novel deep learning framework called Spatial Events Prediction (SEP) based on multi-task learning to predict the types of events that happen at a specific location from social data. The proposed model focuses on predicting the attribute types of an event, which is referred to as subtypes. Specifically, an event type-specific attention mechanism is introduced to extract the representations of social data and to identify their important components. The proposed attention mechanism is based on a two-level attention, which measures the importance of words and sentences to the subtypes of an event. We also propose a representation sharing method using semantic and spatial relationships between locations to alleviate the sparsity and incompleteness of data. The proposed representation sharing preserves the spatial heterogeneity between locations and significantly improves the accuracy of the overall framework. Experiments with real-world datasets confirm the effectiveness and efficiency of the proposed method.

Enhancing the Interpretability of Deep Models in Healthcare Through Attention: Application to Glucose Forecasting for Diabetic People

The adoption of deep learning in healthcare is hindered by their “black box” nature. In this paper, we explore the RETAIN architecture for the task of glucose forecasting for diabetic people. By using a two-level attention mechanism, the recurrent-neural-network-based RETAIN model is interpretable. We evaluate the RETAIN model on the type-2 IDIAB and the type-1 OhioT1DM datasets by comparing its statistical and clinical performances against two deep models and three models based on decision trees. We show that the RETAIN model offers a very good compromise between accuracy and interpretability, being almost as accurate as the LSTM and FCN models while remaining interpretable. We show the usefulness of its interpretable nature by analyzing the contribution of each variable to the final prediction. It revealed that signal values older than 1[Formula: see text]h are not used by the RETAIN model for 30[Formula: see text]min ahead of time prediction of glucose. Also, we show how the RETAIN model changes its behavior upon the arrival of an event such as carbohydrate intakes or insulin infusions. In particular, it showed that the patient’s state before the event is particularly important for the prediction. Overall the RETAIN model, thanks to its interpretability, seems to be a very promising model for regression or classification tasks in healthcare.

Seatbelt detection in road surveillance images based on improved dense residual network with two-level attention mechanism

Seatbelt detection is an important topic in intelligent transport systems. The accuracy of seatbelt detection in traffic video surveillance is affected by many factors, such as complex road environments, lighting, weather, direction of the camera, etc., which bring difficulties for traditional image processing methods. We propose a seatbelt detection method based on classification using convolutional neural networks. Dense residual blocks are used to avoid gradient dispersion and information loss during the network training. A two-level attention mechanism is also introduced to further improve the network performance by simultaneously modeling the correlation among different feature channels using channel attention mechanism, and correlation among different pixels within the feature map using pixel attention mechanism. To make the whole system complete, we added the driver area localization module, which is accomplished using the YOLOV3 model. Small images concentrated on the driver location, which are obtained through the localization operation from larger images, are passed into the classification network for seatbelt detection. We compared the proposed method with several other methods. Comparative results show that the proposed method has higher accuracy and is more robust for seatbelt detection for complex environments.

Two-Level Attention Model of Representation Learning for Fraud Detection

Fraud detection has attracted significant attention in financial institutions, especially utilizing some artificial intelligent methods to automatically detect fraudulent transactions. With the study and application of intelligent fraud detection technology, scholars found that the representation learning method can reveal more information about fraud patterns, which is also crucial for detection task. Therefore, in this work, we present a novel method for detecting fraud transactions by combining two modules learning hidden information at different levels of data in a unified framework. To address and explore the deep representation of features of transaction behaviors, we propose a two-level attention model to capture them by integrating two data embeddings at the data sample level and the feature level. In particular, the sample-level attention model captures the detailed information more centrally that is difficult to determine; the feature-level attention model extends the information of feature dependences. We further combine them to train a final fraud detection model. Extensive experiments are conducted using a data set provided by a financial company in China and several public financial data sets. The results confirm the effectiveness of our proposed method in detecting fraudulent transactions compared with other state-of-the-art methods.

$A^3$-FKG: Attentive Attribute-Aware Fashion Knowledge Graph for Outfit Preference Prediction

With the booming development of the online fashion industry, effective personalized recommender systems have become indispensable for the convenience they brought to the customers and the profits to the e-commercial platforms. Estimating the user’s preference towards the outfit is at the core of a personalized recommendation system. Existing works on fashion recommendation are largely centering on modelling the clothing compatibility without considering the user factor or characterizing the user’s preference over the single item. However, how to effectively model the outfits with either few or even none interactions, is yet under-explored. In this paper, we address the task of personalized outfit preference prediction via a novel <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A</b> ttentive <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A</b> ttribute- <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A</b> ware <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</b> ashion <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</b> nowledge <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">G</b> raph ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$A^3$</tex-math></inline-formula> -FKG), which is incorporated to build the association between different outfits with both outfit- and item- level attributes. Additionally, a two-level attention mechanism is developed to capture the user’s preference: 1) User-specific relation-aware attention layer, which captures the user’s fine-grained preferences with different focus on relations for learning outfit representation; 2) Target-aware attention layer, which characterizes the user’s latent diverse interests from his/her behavior sequences for learning user representation. Extensive experiments conducted on a large-scale fashion outfit dataset demonstrate significant improvements over other methods, which verify the excellence of our proposed framework.

Hierarchical Attentive Transaction Embedding With Intra- and Inter-Transaction Dependencies for Next-Item Recommendation

A transaction-based recommender system (TBRS) aims to predict the next item by modeling dependencies in transactional data. Generally, two kinds of dependencies considered are intra-transaction dependence and inter-transaction dependence. Most existing TBRSs recommend next item by only modeling the intra-transaction dependence within the current transaction while ignoring inter-transaction dependence with recent transactions that may also affect the next item. However, as not all recent transactions are relevant to the current and next items, the relevant ones should be identified and prioritized. In this article, we propose a novel hierarchical attentive transaction embedding (HATE) model to tackle these issues. Specifically, a two-level attention mechanism integrates both item embedding and transaction embedding to build an attentive context representation that incorporates both intra- and inter-transaction dependencies. With the learned context representation, HATE then recommends the next item. Experimental evaluations on two real-world transaction datasets show that HATE significantly outperforms the state-of-the-art methods in terms of recommendation accuracy.

Driver Drowsiness Recognition via 3D Conditional GAN and Two-Level Attention Bi-LSTM

Driver drowsiness has currently been a severe issue threatening road safety, hence it is vital to develop an effective drowsiness recognition algorithm to avoid traffic accidents. However, recognizing drowsiness is still very challenging, due to the large intra-class variations in facial expression, head pose and illumination condition. In this paper, a new deep learning framework based on the hybrid of 3D conditional generative adversarial network and two-level attention bidirectional long short-term memory network (3DcGAN-TLABiLSTM) has been proposed for robust driver drowsiness recognition. Aiming at extracting short-term spatial-temporal features with abundant drowsiness-related information, we design a 3D encoder-decoder generator with the condition of auxiliary information to generate high-quality fake image sequences and devise a 3D discriminator to learn drowsiness-related representation from spatial-temporal domain. In addition, for long-term spatial-temporal fusion, we investigate the use of two-level attention mechanism to guide the bidirectional long short-term memory learn the saliency of short-term memory information and long-term temporal information. For experiment, we evaluate our 3DcGAN-TLABiLSTM framework on a public NTHU-DDD dataset. Experimental results show that the proposed approach achieves higher precision of drowsiness recognition compared to the state-of-the-art.

Relational Graph Neural Network with Hierarchical Attention for Knowledge Graph Completion

The rapid proliferation of knowledge graphs (KGs) has changed the paradigm for various AI-related applications. Despite their large sizes, modern KGs are far from complete and comprehensive. This has motivated the research in knowledge graph completion (KGC), which aims to infer missing values in incomplete knowledge triples. However, most existing KGC models treat the triples in KGs independently without leveraging the inherent and valuable information from the local neighborhood surrounding an entity. To this end, we propose a Relational Graph neural network with Hierarchical ATtention (RGHAT) for the KGC task. The proposed model is equipped with a two-level attention mechanism: (i) the first level is the relation-level attention, which is inspired by the intuition that different relations have different weights for indicating an entity; (ii) the second level is the entity-level attention, which enables our model to highlight the importance of different neighboring entities under the same relation. The hierarchical attention mechanism makes our model more effective to utilize the neighborhood information of an entity. Finally, we extensively validate the superiority of RGHAT against various state-of-the-art baselines.

Research on relation extraction of named entity on social media in smart cities

Social media make significant contribution to the evolution of smart cities. The key issue of smart cities is to develop a series of automatic methods to support smart applications. As one of the basic techniques of smart cities, the task of relation extraction of named entities on social media provides an indispensable means to construct and expand knowledge map and contributes to the utilization of information resources in smart cities. What’s more, it is conducive to improve the efficiency of network supervision. This paper proposes an automatic method to extract entity relations via deep learning techniques on a two-level neural network named Bi-CLSTM. The research conquers some challenges of relation extraction on social media. To extract entity relations on conversation scenarios, Bi-CLSTM represents texts with the strategy of “word embedding + position embedding + shortest dependency path” and extracts relations via a hybrid model of LSTM and PCNN. The nodes and networks of Bi-CLSTM are designed to adapt to the scenarios of conversation and over-sentence. To reduce the dependency on training data, distant supervised strategy is employed and a two-level attention mechanism is used to prevent noise signals. Experiments are carried out on Sina Microblog corpus, and the results show that Bi-CLSTM model makes outstanding performance.

Two-level attention and score consistency network for plant segmentation

Semantic segmentation of plant images can provide valuable information for plant phenotypic studies. However, challenges exist when facing complex background interference and uneven illumination. In this study, we propose a Two-level Attention and Score Consistency based Network (TASCN) for semantic segmentation of plant. Our TASCN includes a two-level attention sub-module (TAM) and a multi-scale feature fusion sub-module (FFM). The TAM combines a top-down semantic attention gate (Inter-layer attention, level 1) and a self-attention mechanism (Intra-layer attention, level 2) to highlight the salient features and suppress the irrelevant or interference information. The FFM fuses the multi-scale features from the TAM to obtain the final segmentation score-map. The score-map is further restricted with the score consistency loss to promote the semantic consistency and reduces the impact of uneven illumination. Experiments on rice dataset from the Chinese spacelab TG-2 demonstrate the impressing segmentation performance of our TASCN, especially for complex background interference and uneven illumination. In addition, both the quantitative evaluation (Mean IoU) and qualitative visual results show that our TASCN outperforms the state-of-the-art methods, including FCN, U-Net, SegNet, PSPNet, DeepLabv3+ and DANet. In conclusion, a novel TASCN is proposed to meet the increasing need for fine plant parsing in plant research.

Two-level attention with two-stage multi-task learning for facial emotion recognition

Compared with facial emotion estimation on categorical model, dimensional emotion estimation can describe numerous emotions more accurately. Most prior works of dimensional emotion estimation only considered laboratory data and used video, speech or other multi-modal features. Compared with other modal data, static images has superiorities of accessibility, which is more conducive to the emotion estimation in real world. In this paper, a two-level attention with two-stage multi-task learning (2Att-2Mt) framework is proposed for facial emotion estimation on only static images. Firstly, the features of corresponding region (position level features) are extracted and enhanced automatically by first-level attention mechanism. Then, we utilize Bi-directional Recurrent Neural Network (Bi-RNN) with self-attention (second-level attention) to make full use of the relationship features of different layers (layer-level features) adaptively. And then, we propose a two-stage multi-task learning structure, which exploits categorical representations to ameliorate the dimensional representations and estimate valence and arousal simultaneously in view of the inherent complexity of dimensional representations and correlation of the two targets. The quantitative results conducted on AffectNet dataset show significant advancement on Concordance Correlation Coefficient(CCC) and Root Mean Square Error (RMSE), illustrating the superiority of the proposed framework. Besides, extensive comparative experiments have also fully demonstrated the effectiveness of different components (2Att and 2Mt) in our framework.

Two-Level Attentions and Grouping Attention Convolutional Network for Fine-Grained Image Classification

The focus of fine-grained image classification tasks is to ignore interference information and grasp local features. This challenge is what the visual attention mechanism excels at. Firstly, we have constructed a two-level attention convolutional network, which characterizes the object-level attention and the pixel-level attention. Then, we combine the two kinds of attention through a second-order response transform algorithm. Furthermore, we propose a clustering-based grouping attention model, which implies the part-level attention. The grouping attention method is to stretch all the semantic features, in a deeper convolution layer of the network, into vectors. These vectors are clustered by a vector dot product, and each category represents a special semantic. The grouping attention algorithm implements the functions of group convolution and feature clustering, which can greatly reduce the network parameters and improve the recognition rate and interpretability of the network. Finally, the low-level visual features and high-level semantic information are merged by a multi-level feature fusion method to accurately classify fine-grained images. We have achieved good results without using pre-training networks and fine-tuning techniques.

Two-Level Attention Network With Multi-Grain Ranking Loss for Vehicle Re-Identification.

Vehicle re-identification (re-ID) aims to identify the same vehicle across multiple non-overlapping cameras, which is rather a challenging task. On the one hand, subtle changes in viewpoint and illumination condition can make the same vehicle look much different. On the other hand, different vehicles, even different vehicle models, may look quite similar. In this paper, we propose a novel Two-level Attention network supervised by a Multi-grain Ranking loss (TAMR) to learn an efficient feature embedding for the vehicle re-ID task. The two-level attention network consisting of hard part-level attention and soft pixel-level attention can adaptively extract discriminative features from the visual appearance of vehicles. The former one is designed to localize the salient vehicle parts, such as windscreen and car head. The latter one gives an additional attention refinement at pixel level to focus on the distinctive characteristics within each part. In addition, we present a multi-grain ranking loss to further enhance the discriminative ability of learned features. We creatively take the multi-grain relationship between vehicles into consideration. Thus, not only the discrimination between different vehicles but also the distinction between different vehicle models is constrained. Finally, the proposed network can learn a feature space, where both intra-class compactness and inter-class discrimination are well guaranteed. Extensive experiments demonstrate the effectiveness of our approach and we achieve state-of-the-art results on two challenging datasets, including VehicleID and Vehicle-1M.

Two-Level Attention Model Based Video Action Recognition Network

The complex environment background, lighting conditions, and other action-irrelevant visual information in the video frame bring a lot of redundancy and noise to the action spatial features, which seriously affects the accuracy of action recognition. Aiming at this point, we propose a recurrent region attention cell to capture the action-relevant regional visual information in the spatial feature, and according to the temporal sequential natures of the video, on the basis of the recurrent region attention cell, a Recurrent Region Attention model (RRA) is proposed. The recurrent region attention cell in the RRA iterates according to the temporal sequence of the video, so that the attention performance of the RRA is gradually improved. Secondly, we propose a Video Frame Attention model (VFA) that can highlight the more important frames in the whole action video sequence, so as to reduce the interference caused by the similarity between the heterogeneous action video sequences. Finally, we propose an end-to-end trainable network: Two-level Attention Model based video action recognition network (TAMNet). We experimented on two video action recognition benchmark datasets: UCF101 and HMDB51. Experiments show that our end-to-end TAMNet network can reliably focus on the more important video frames in the video sequence, and effectively capture the action-relevant regional visual information in the spatial features of each frame of the video sequence. Inspired by the two-stream structure, we construct a two-modalities TAMNet network. In the same training conditions, the two-modalities TAMNet network achieved optimal performance on both datasets.

Dual-level attention-aware network for temporal emotion segmentation

Human emotions are known to always have four phases in the temporal domain: neutral, onset, apex, and offset. This has been demonstrated to be of great benefit for emotion recognition. Therefore, temporal segmentation has attracted considerable research interest. Although state-of-the-art techniques use recurrent neural networks to highly increase the performance, they ignore the relevance of each frame (time step) of a video, and they do not consider the changing contribution of different features when fusing them. We propose a framework called dual-level attention-aware bidirectional grated recurrent unit, which integrates ideas from attention models to discover the most important frames and features for improving temporal segmentation. Specifically, it applies attention mechanisms at two levels: frame and feature. A significant advantage is that the two-level attention weights provide a meaningful value to depict the importance of each frame and feature. The experiments demonstrated that the proposed framework outperforms state-of-the-art methods.