Hierarchical Attention Network Research Articles

Using attention methods to predict judicial outcomes

The prediction of legal judgments is one of the most recognized fields in Natural Language Processing, Artificial Intelligence, and Law combined. By legal prediction, we mean intelligent systems capable of predicting specific judicial characteristics such as the judicial outcome, the judicial class, and the prediction of a particular case. In this study, we used an artificial intelligence classifier to predict the decisions of Brazilian courts. To this end, we developed a text crawler to extract data from official Brazilian electronic legal systems, consisting of two datasets of cases of second-degree murder and active corruption. We applied various classifiers, such as Support Vector Machines, Neural Networks, and others, to predict judicial outcomes by analyzing text features from the dataset. Our research demonstrated that Regression Trees, Gated Recurring Units, and Hierarchical Attention Networks tended to have higher metrics across our datasets. As the final goal, we searched the weights of one of the algorithms, Hierarchical Attention Networks, to find samples of the words that might be used to acquit or convict defendants based on their relevance to the algorithm.

SuHAN: Substructural hierarchical attention network for molecular representation

Recently, molecular representation and property exploration, with the combination of neural network, play a critical role in the field of drug design and discovery for assisting in drug related research. However, previous research in molecular representation relies heavily on artificial extraction of features based on biological experiments which may result in a manually introduced noise of molecular information with high cost in time and money. In this paper, a novel method named Substructural Hierarchical Attention Network (SuHAN) is proposed to discover inherent characteristics of molecules for representation learning. Specifically, SuHAN is composed of the cascaded layer: atom-level layer and substructure-level layer. Molecule in the SMILES format is divided into several substructural fragments by predefined partition rules, and then they are fed into atom-level layer and substructure-level layer successively to obtain feature representation from different perspective: atomic view and substructural view. In this way, the prominent structural features that may be omitted in global extraction are excavated from a fine-grained viewpoint and fused to reconstruct representative pattern in an overall view. Experiments on biophysics and physiology datasets demonstrate that our model is competitive with a significant improvement of both accuracy and stability in performance. We confirmed that the substructural segments and progressive hierarchical networks lead to an effective molecular representation for downstream tasks. These results provide a novel perspective about reconstructing overall pattern through local prominent structure.

TERTIAN: Clinical Endpoint Prediction in ICU via Time-Aware Transformer-Based Hierarchical Attention Network.

Accurately predicting the clinical endpoint in ICU based on the patient's electronic medical records (EMRs) is essential for the timely treatment of critically ill patients and allocation of medical resources. However, the patient's EMRs usually consist of a large amount of heterogeneous multivariate time series data such as laboratory tests and vital signs, which are produced irregularly. Most existing methods fail to effectively model the time irregularity inherent in longitudinal patient medical records and capture the interrelationships among different types of data. To tackle these limitations, we propose a novel time-aware transformer-based hierarchical attention network (TERTIAN) for clinical endpoint prediction. In this model, a time-aware transformer is introduced to learn the personalized irregular temporal patterns of medical events, and a hierarchical attention mechanism is deployed to get the accurate patient fusion representation by comprehensively mining the interactions and correlations among multiple types of medical data. We evaluate our model on the MIMIC-III dataset and MIMIC-IV dataset for the task of mortality prediction, and the results show that TERTIAN achieves higher performance than state-of-the-art approaches.

HMIAN: A Hierarchical Mapping and Interactive Attention Data Fusion Network for Traffic Forecasting

With the development of intelligent transportation system (ITS), the vital technology of ITS, short-term traffic forecasting, gains increasing attention. However, the existing prediction models ignore the impact of urban functional zones (FZs) on traffic data, resulting in inaccurate extractions of dynamic spatial relationships from network. Furthermore, how to calculate the influence of external factors, such as weather and holidays on traffic is an unsolved problem. This article proposes a spatio-temporal hierarchical mapping and interactive attention network (HMIAN), which extracts the spatial features from traffic network by constructing FZs, and designs an effective external factors fusion method. HMIAN uses the hierarchical mapping structure to aggregate the roads into FZs, calculate the interaction between FZs and feed this information back to the spatial features. And the interactive attention mechanism is utilized to fuse the traffic data with external factors effectively, and extracts temporal features. In addition, some experiments were carried out on three real traffic data sets. First, experiment results show the better prediction performance of the proposed model compared with other existing methods in a complex traffic network. Second, the longitudinal comparison experiment verifies that the hierarchical mapping structure is effective in extracting spatial features in a complex road network. Finally, the influence of different external factors and fusion methods on traffic prediction are compared, which provides a consult for subsequent research on the influence of external factors.

WaterHRNet: A multibranch hierarchical attentive network for water body extraction with remote sensing images

• Multi-resolution feature extractor for strong-semantic feature encoding. • Dual-attention module for semantic-relevant feature promotion. • Semantic enhancement principle for target-oriented feature representation. • Hierarchical attentive network for high-quality water body extraction. Water is a kind of vital natural resource, which acts as the lifeblood of the ecosystem and the energy source for the living and production activities of humans. Regularly mapping the conditions of water resources and taking effective measures to prevent them from pollutions and shortages are very important and necessary to maintain the sustainability of the ecosystem. As a preliminary step for image-based water resource analysis, the complete recognition and accurate extraction of water bodies are important prerequisites in many applications. Nevertheless, due to the issues of topology diversities, appearance variabilities, and land cover interferences, there is still a large gap to achieve the human-level water bodies interpretation quality. This paper presents a hierarchical attentive high-resolution network, abbreviated as WaterHRNet, for extracting water bodies from remote sensing imagery. First, by building a multibranch high-resolution feature extractor integrated with global feature semantics aggregation, the WaterHRNet behaves laudably to supply high-quality, strong-semantic feature representations. Furthermore, by inlaying an effective feature attention scheme with the comprehensive exploitation of both the spatial and channel feature significances, the WaterHRNet is forced to strengthen the semantic-determinate, task-aware feature encodings. In addition, by designing a hierarchical processing principle with the progressive enhancement of category-attentive feature semantics, the WaterHRNet performs effectively to export semantic-discriminative, target-oriented feature representations for precise water body segmentation. The WaterHRNet is elaborately verified both quantitatively and qualitatively on three remote sensing datasets. Evaluation results show that the WaterHRNet achieves an average precision of 98.44%, average recall of 97.84%, average IoU of 96.35%, and average F 1 -score of 98.14%. Comparative analyses also demonstrate the superior performance and excellent feasibility of the WaterHRNet in segmenting water bodies.

A Context Awareness Hierarchical Attention Network for Next POI Recommendation in IoT Environment

The rapid increase in the number of sensors in the Internet of things (IoT) environment has resulted in the continuous generation of massive and rich data in Location-Based Social Networks (LBSN). In LBSN, the next point-of-interest (POI) recommendation has become an important task, which provides the best POI recommendation according to the user’s recent check-in sequences. However, all existing methods for the next POI recommendation only focus on modeling the correlation between POIs based on users’ check-in sequences but ignore the significant fact that the next POI recommendation is a time-subtle recommendation task. In view of the fact that the attention mechanism does not comprehensively consider the influence of the user’s trajectory sequences, time information, social relations and geographic information of Point-of-Interest (POI) in the next POI recommendation field, a Context Geographical-Temporal-Social Awareness Hierarchical Attention Network (CGTS-HAN) model is proposed. The model extracts context information from the user’s trajectory sequences and designs a Geographical-Temporal-Social attention network and a common attention network for learning dynamic user preferences. In particular, a bidirectional LSTM model is used to capture the temporal influence between POIs in a user’s check-in trajectory. Moreover, In the context interaction layer, a feedforward neural network is introduced to capture the interaction between users and context information, which can connect multiple context factors with users. Then an embedded layer is added after the interaction layer, and three types of vectors are established for each POI to represent its sign-in trend so as to solve the heterogeneity problem between context factors. Finally reconstructs the objective function and learns model parameters through a negative sampling algorithm. The experimental results on Foursquare and Yelp real datasets show that the AUC, precision and recall of CGTS-HAN are better than the comparison models, which proves the effectiveness and superiority of CGTS-HAN.

BHGAttN: A Feature-Enhanced Hierarchical Graph Attention Network for Sentiment Analysis.

Recently, with the rise of deep learning, text classification techniques have developed rapidly. However, the existing work usually takes the entire text as the modeling object and pays less attention to the hierarchical structure within the text, ignoring the internal connection between the upper and lower sentences. To address these issues, this paper proposes a Bert-based hierarchical graph attention network model (BHGAttN) based on a large-scale pretrained model and graph attention network to model the hierarchical relationship of texts. During modeling, the semantic features are enhanced by the output of the intermediate layer of BERT, and the multilevel hierarchical graph network corresponding to each layer of BERT is constructed by using the dependencies between the whole sentence and the subsentence. This model pays attention to the layer-by-layer semantic information and the hierarchical relationship within the text. The experimental results show that the BHGAttN model exhibits significant competitive advantages compared with the current state-of-the-art baseline models.

Knowledge-aware hierarchical attention network for recommendation

Knowledge graphs have been introduced into recommender systems due to the rich connectivity information. Many knowledge-aware recommendation methods use graph neural networks (GNNs) to capture the high-order structural and semantic information of knowledge graphs. However, previous GNN-based methods have the following limitations: (1) they fail to make full use of the neighborhood information of entities and (2) they ignore the importance of user interaction sequences on reflecting user preferences. As such, these models are insufficient for generating accurate representations of users and items. In this study, we propose a Knowledge-aware Hierarchical Attention Network (KHAN) to provide better recommendation. Specifically, the proposed model mainly consists of an item encoder and a user encoder. The item encoder is equipped with a hierarchical attention network, which is used to generate entity (item) representations by carefully aggregating neighborhood information of entities. The user encoder is also designed to learn more informative user representations from user interaction sequences using multi-head self-attention. The learned user representations are then combined with user representations introduced in the item encoder through a gating mechanism to generate the final user representations. Extensive experiments on two real-world datasets about movie and restaurant recommendation demonstrate the effectiveness of our model.

Describe like a pathologist: Glomerular immunofluorescence image caption based on hierarchical feature fusion attention network

Clinically, the histopathological assessment of glomerular immunofluorescence (IF) images is widely-used and indispensable for diagnosis of most chronic kidney diseases, however the histopathological assessment always requires human’s judgements and suffers from the inter-observer variability. In this paper, firstly, a hierarchical feature fusion attention network (HFANet) is proposed to enhance the feature extraction capability for glomerular IF images, which concatenates the weighted feature maps extracted from different depths. Secondly, a proposed intensity equalization (IE) algorithm improves the qualities of images and generated IF descriptions by adjusting all images to a uniform and appropriate intensity level. Thirdly, an encoder–decoder architecture with the HFANet as the core is utilized to generate descriptions that emulate the reading habits of clinicians. At last, we construct a large dataset containing 11,506 glomerular IF images and their corresponding IF descriptions to support the model training. In the testing phase, the proposed framework achieves the BLEU-1, BLEU-2, BLEU-3, BLEU-4, ROUGE-L, and CIDEr of 0.704, 0.563, 0.444, 0.387, 0.760, and 3.594, respectively. Extensive comparative experiments and ablation studies reveal the effectiveness of proposed modules and the generalization performance of the framework for various cases and IF descriptive indicators.

Lung Lobe Segmentation and Feature Extraction-Based Hierarchical Attention Network for COVID-19 Prediction from Chest X-Ray Images

AbstractCoronavirus disease 2019 (COVID-19) is a rising respiratory sickness. It causes harsh pneumonia and is considered to cover higher collisions in the healthcare domain. The diagnosis at an early stage is more complex to get accurate treatment for reducing the stress in the clinical sector. Chest X-ray scan is the standard imaging diagnosis test employed for pneumonia disease. Automatic detection of COVID-19 helps to control the community outbreak but tracing this viral infection through X-ray results in a challenging task in the medical community. To automatically detect the viral disease in order to reduce the mortality rate, an effective COVID-19 detection method is modelled in this research by the proposed manta-ray multi-verse optimization-based hierarchical attention network (MRMVO-based HAN) classifier. Accordingly, the MRMVO is the incorporation of manta-ray foraging optimization and multi-verse optimizer. Based on the segmented lung lobes, the features are acquired from segmented regions in such a way that the process of COVID-19 detection mechanism is carried out with the features acquired from interested lobe regions. The proposed method has good performance with the measures, such as accuracy, true positive rate and true negative rate with the values of 93.367, 89.921 and 95.071%.

TDO-Spider Taylor ChOA: An Optimized Deep-Learning-Based Sentiment Classification and Review Rating Prediction

Modern review websites, namely Yelp and Amazon, permit the users to post online reviews for numerous businesses, services and products. Currently, online reviewing is an imperative task in the manipulation of shopping decisions produced by customers. These reviews afford consumers experience and information regarding the superiority of the product. The prevalent method of strengthening online review evolution is the performance of Sentiment Classification, which is an attractive domain in industrial and academic research. The review helps various domains, and it is problematic to collect interpreted training data. In this paper, an effectual Review Rating Prediction and Sentiment Classification was developed. Here, a Gated Recurrent Unit (GRU) was employed for the Sentiment Classification process, whereas a Hierarchical Attention Network (HAN) was applied for Review Rating Prediction. The significant features, such as statistical, SentiWordNet and classification features, were extracted for the Sentiment Classification and Review Rating Prediction process. Moreover, the GRU was trained by the designed TD-Spider Taylor ChOA approach, and the HAN was trained by the designed Jaya-TDO approach. The experimental results show that the proposed Jaya-TDO technique attained a better performance of 0.9425, 0.9654 and 0.9538, and that TD-Spider Taylor ChOA achieved 0.9524, 0.9698 and 0.9588 in terms of the precision, recall and F-measure.

Joint user profiling with hierarchical attention networks

Joint user profiling with hierarchical attention networks

OpExHAN: opinion extraction using hierarchical attention network from unstructured reviews.

In recent decades, sellers and merchants have asked their customers to share their opinion on the products at online marketplace. Analysis of the massive amounts of reviews for a potential customer is an immense challenge, to decide whether to purchase a product or not. In this paper, a hierarchical attention network-based framework is presented to resolve this challenge. In this proposed framework, the Amazon’s Smartphone review dataset is preprocessed using NLP approaches and then applied Glove embedding to extract word vector representation of reviews which identifies contextual information of words. These word vectors are fed into the hierarchical attention network, which produces vectors at the word and sentence levels. Bi-GRU model encodes the words and sentences into hidden vectors. Finally, reviews are classified into five opinion classes like extremely positive, positive, extremely negative, negative and neutral. Furthermore, to perform the experiments with the proposed method, the dataset is divided into three parts such as 80% train, 10% validation and 10% test. Experiments reveal that the proposed framework outperforms baseline methods in terms of accuracy, precision and recall. The OpExHAN model achieved admirable results like 94.6% accuracy, 91% both precision and recall after a lot of hyper-parameter experimentation.

A Comprehensive Study of Learning Approaches for Author Gender Identification

In recent years, author gender identification is an important yet challenging task in the fields of information retrieval and computational linguistics. In this paper, different learning approaches are presented to address the problem of author gender identification for Turkish articles. First, several classification algorithms are applied to the list of representations based on different paradigms: fixed-length vector representations such as Stylometric Features (SF), Bag-of-Words (BoW) and distributed word/document embeddings such as Word2vec, fastText and Doc2vec. Secondly, deep learning architectures, Convolution Neural Network (CNN), Recurrent Neural Network (RNN), special kinds of RNN such as Long-Short Term Memory (LSTM) and Gated Recurrent Unit (GRU), C-RNN, Bidirectional LSTM (bi-LSTM), Bidirectional GRU (bi-GRU), Hierarchical Attention Networks and Multi-head Attention (MHA) are designated and their comparable performances are evaluated. We conducted a variety of experiments and achieved outstanding empirical results. To conclude, ML algorithms with BoW have promising results. fast-Text is also probably suitable between embedding models. This comprehensive study contributes to literature utilizing different learning approaches based on several ways of representations. It is also first important attempt to identify author gender applying SF, fastText and DNN architectures to the Turkish language.

Identifying the Perceived Severity of Patient-Generated Telemedical Queries Regarding COVID: Developing and Evaluating a Transfer Learning-Based Solution.

BackgroundTriage of textual telemedical queries is a safety-critical task for medical service providers with limited remote health resources. The prioritization of patient queries containing medically severe text is necessary to optimize resource usage and provide care to those with time-sensitive needs.ObjectiveWe aim to evaluate the effectiveness of transfer learning solutions on the task of telemedical triage and provide a thorough error analysis, identifying telemedical queries that challenge state-of-the-art natural language processing (NLP) systems. Additionally, we aim to provide a publicly available telemedical query data set with labels for severity classification for telemedical triage of respiratory issues.MethodsWe annotated 573 medical queries from 3 online health platforms: HealthTap, HealthcareMagic, and iCliniq. We then evaluated 6 transfer learning solutions utilizing various text-embedding strategies. Specifically, we first established a baseline using a lexical classification model with term frequency–inverse document frequency (TF-IDF) features. Next, we investigated the effectiveness of global vectors for text representation (GloVe), a pretrained word-embedding method. We evaluated the performance of GloVe embeddings in the context of support vector machines (SVMs), bidirectional long short-term memory (bi-LSTM) networks, and hierarchical attention networks (HANs). Finally, we evaluated the performance of contextual text embeddings using transformer-based architectures. Specifically, we evaluated bidirectional encoder representation from transformers (BERT), Bio+Clinical-BERT, and Sentence-BERT (SBERT) on the telemedical triage task.ResultsWe found that a simple lexical model achieved a mean F1 score of 0.865 (SD 0.048) on the telemedical triage task. GloVe-based models using SVMs, HANs, and bi-LSTMs achieved a 0.8-, 1.5-, and 2.1-point increase in the F1 score, respectively. Transformer-based models, such as BERT, Bio+Clinical-BERT, and SBERT, achieved a mean F1 score of 0.914 (SD 0.034), 0.904 (SD 0.041), and 0.917 (SD 0.037), respectively. The highest-performing model, SBERT, provided a statistically significant improvement compared to all GloVe-based and lexical baselines. However, no statistical significance was found when comparing transformer-based models. Furthermore, our error analysis revealed highly challenging query types, including those with complex negations, temporal relationships, and patient intents.ConclusionsWe showed that state-of-the-art transfer learning techniques work well on the telemedical triage task, providing significant performance increase over lexical models. Additionally, we released a public telemedical triage data set using labeled questions from online medical question-and-answer (Q&A) platforms. Our analysis highlights various avenues for future works that explicitly model such query challenges.

Hierarchical Attention Network of Multi-Modal Biometric for a Secure Cloud-Based User Verification

The rise of the Internet has created a need to secure verification for accessing the certain information. The cloud based biometric system can provide a convenient way to realize it. In recent years, various Artificial Intelligence models have been researched for single b iometric but rare for multi-modal biometric. This article presents new a multi-model approach as Hierarchical Attention Network (HAN) which is based on four user features i.e palm image, facial image, audio signature, and digital signature. Through the layer-by-layer filtering and improved attention mechanism, HAN can provide a higher verification accuracy, which can provide a more robust and secure user verification service. At the same time, this work also implemented the proposed model on a Cloud-based service with android application. After the evaluation, the HAN method we proposed has been proved to be better than the previous models.

Which structure of academic articles do referees pay more attention to?: perspective of peer review and full-text of academic articles

PurposeThe purpose of this paper is to explore which structures of academic articles referees would pay more attention to, what specific content referees focus on, and whether the distribution of PRC is related to the citations.Design/methodology/approachFirstly, utilizing the feature words of section title and hierarchical attention network model (HAN) to identify the academic article structures. Secondly, analyzing the distribution of PRC in different structures according to the position information extracted by rules in PRC. Thirdly, analyzing the distribution of feature words of PRC extracted by the Chi-square test and TF-IDF in different structures. Finally, four correlation analysis methods are used to analyze whether the distribution of PRC in different structures is correlated to the citations.FindingsThe count of PRC distributed in Materials and Methods and Results section is significantly more than that in the structure of Introduction and Discussion, indicating that referees pay more attention to the Material and Methods and Results. The distribution of feature words of PRC in different structures is obviously different, which can reflect the content of referees' concern. There is no correlation between the distribution of PRC in different structures and the citations.Research limitations/implicationsDue to the differences in the way referees write peer review reports, the rules used to extract position information cannot cover all PRC.Originality/valueThe paper finds a pattern in the distribution of PRC in different academic article structures proving the long-term empirical understanding. It also provides insight into academic article writing: researchers should ensure the scientificity of methods and the reliability of results when writing academic article to obtain a high degree of recognition from referees.

Automatic Pavement Defect Detection and Classification Using RGB-Thermal Images Based on Hierarchical Residual Attention Network.

A convolutional neural network based on an improved residual structure is proposed to implement a lightweight classification model for the recognition of complex pavement conditions, which uses RGB-thermal as input and embeds an attention module to adjust the spatial, as well as channel, information of the images. The best prediction accuracy of the proposed model is 98.88%, while the RGB-thermal is used as input and an attention mechanism is used. The attention mechanism increases the attention to detail of the image and regulates the use of image channels, which enhances the final performance of the model. It is also compared with state-of-the-art (SOTA) deep learning models, indicating our model has fewer parameters, shorter training time, and higher recognition accuracy compared to existing image classification models. A visualization method incorporating gradient-weighted class activation mapping (Grad-CAM) is proposed to analyze the classification results, comparing the data the model learns from the images under different input data.

Depression Detection on Social Media Twitter Using Hierarchical Attention Network Method

Mental illness, including depression, is not a mild condition that only some mentally weak people experience. Technology is developing so rapidly, especially communication technology through social media. Twitter is a very popular social media today. Users can easily quickly and simply communicate all the feelings they are experiencing through tweets, which allows us to find information about emotional feelings to the level of user depression. Auto-mated analysis of social media has the potential to provide a method for early detection. This study aims to predict early signs of depression using data from social media Twitter. The method used in this research is classification by analyzing social media sentiment using the Hierarchical Attention Network. Classification using the Hierarchical Attention Network method was chosen because the method showed outstanding results for classifying texts in previous studies. The classification model in this study that represents the best accuracy, 74%, was performed by applying the Hierarchical Attention Network.

A multi-branch hierarchical attention network for medical target segmentation

Medical imaging techniques have been widely used in modern clinical disease diagnosis and treatment programs. The captured medical images can well reflect the conditions of the human body tissues, which are significantly helpful to the doctors to determine the existence or the severity of the disease. In this paper, we develop a hierarchical attentive high-resolution convolutional network (AttHRNet) for segmenting targets of interest from medical images aiming to improve the automated processing standard and the intelligent interpretation quality of the medical images. The AttHRNet is an improved version of the high-resolution network (HRNet) structure with three novel modules. First, built with an improved HRNet structure assisted by a multiscale context augmentation (MSCA) module as the feature extraction backbone, the AttHRNet can produce a set of high-quality, strong-semantic feature maps at different resolutions. The MSCA module functions to reduce the information loss during feature downsampling. Second, designed with an effective feature attention principle, the feature encoding quality in each branch can be significantly promoted by concentrating on the informative and salient feature encodings across both channels and spatial locations. Furthermore, formulated with a hierarchical segmentation scheme, the output feature maps can be further augmented by including the semantic-level category exploitation (SLCE) module with a global perspective. The SLCE module allows the information from lower resolution segmentations to inform higher resolution segmentations. Through quantitative examinations, visual verifications, and comparative evaluations on four medical image datasets, we convince the promising applicability and competitive superiority of the AttHRNet in medical target segmentation issues.