Hierarchical Attention Network Research Articles

Privacy-preserving for assembly deviation prediction in a machine learning model of hydraulic equipment under value chain collaboration

Hydraulic equipment, as a typical mechanical product, has been wildly used in various fields. Accurate acquisition and secure transmission of assembly deviation data are the most critical issues for hydraulic equipment manufacturer in the PLM-oriented value chain collaboration. Existing deviation prediction methods are mainly used for assembly quality control, which concentrate in the product design and assembly stage. However, the actual assembly deviations generated in the service stage can be used to guide the equipment maintenance and tolerance design. In this paper, a high-fidelity prediction and privacy-preserving method is proposed based on the observable assembly deviations. A hierarchical graph attention network (HGAT) is established to predict the assembly feature deviations. The hierarchical generalized representation and differential privacy reconstruction techniques are also introduced to generate the graph attention network model for assembly deviation privacy-preserving. A derivation gradient matrix is established to calculate the defined modified necessary index of assembly parts. Two privacy-preserving strategies are designed to protect the assembly privacy of node representation and adjacent relationship. The effectiveness and superiority of the proposed method are demonstrated by a case study with a four-column hydraulic press.

Hierarchical attention network for multivariate time series long-term forecasting.

Multivariate time series long-term forecasting has always been the subject of research in various fields such as economics, finance, and traffic. In recent years, attention-based recurrent neural networks (RNNs) have received attention due to their ability of reducing error accumulation. However, the existing attention-based RNNs fail to eliminate the negative influence of irrelevant factors on prediction, and ignore the conflict between exogenous factors and target factor. To tackle these problems, we propose a novel Hierarchical Attention Network (HANet) for multivariate time series long-term forecasting. At first, HANet designs a factor-aware attention network (FAN) and uses it as the first component of the encoder. FAN weakens the negative impact of irrelevant exogenous factors on predictions by assigning small weights to them. Then HANet proposes a multi-modal fusion network (MFN) as the second component of the encoder. MFN employs a specially designed multi-modal fusion gate to adaptively select how much information about the expression of current time come from target and exogenous factors. Experiments on two real-world datasets reveal that HANet not only outperforms state-of-the-art methods, but also provides interpretability for prediction.

Automatic personality prediction: an enhanced method using ensemble modeling

Human personality is significantly represented by those words which he/she uses in his/her speech or writing. As a consequence of spreading the information infrastructures (specifically the Internet and social media), human communications have reformed notably from face to face communication. Generally, Automatic Personality Prediction (or Perception) (APP) is the automated forecasting of the personality on different types of human generated/exchanged contents (like text, speech, image, video, etc.). The major objective of this study is to enhance the accuracy of APP from the text. To this end, we suggest five new APP methods including term frequency vector-based, ontology-based, enriched ontology-based, latent semantic analysis (LSA)-based, and deep learning-based (BiLSTM) methods. These methods as the base ones, contribute to each other to enhance the APP accuracy through ensemble modeling (stacking) based on a hierarchical attention network (HAN) as the meta-model. The results show that ensemble modeling enhances the accuracy of APP.

MBHAN: Motif-Based Heterogeneous Graph Attention Network

Graph neural networks are graph-based deep learning technologies that have attracted significant attention from researchers because of their powerful performance. Heterogeneous graph-based graph neural networks focus on the heterogeneity of the nodes and links in a graph. This is more effective at preserving semantic knowledge when representing data interactions in real-world graph structures. Unfortunately, most heterogeneous graph neural networks tend to transform heterogeneous graphs into homogeneous graphs when using meta-paths for representation learning. This paper therefore presents a novel motif-based hierarchical heterogeneous graph attention network algorithm, MBHAN, that addresses this problem by incorporating a hierarchical dual attention mechanism at the node-level and motif-level. Node-level attention aims to learn the importance between a node and its neighboring nodes within its corresponding motif. Motif-level attention is capable of learning the importance of different motifs in the heterogeneous graph. In view of the different vector space features of different types of nodes in heterogeneous graphs, MBHAN also aggregates the features of different types of nodes, so that they can jointly participate in downstream tasks after passing through segregated independent shallow neural networks. MBHAN’s superior network representation learning capability has been validated by extensive experiments on two real-world datasets.

Hierarchical graph attention network with pseudo-metapath for skeleton-based action recognition

Skeleton-based action recognition has gained significant attention in computer vision. Most state-of-the-art (SOTA) approaches view the skeleton as a homogeneous graph. Unlike those approaches, this paper shows that methods in the heterogeneous graph manner can also achieve competitive performance. In this paper, a logical heterogeneous skeleton graph is built under the assumption of the heterogeneity of joints and bones at different positions, and the action recognition task is formulated as message aggregation and prediction on this heterogeneous graph. Specifically, a novel semantic concept named pseudo-metapath is introduced to represent dependencies between joints, based on which a hierarchical graph attention network with the joint-level attention and the semantic-level attention modules is proposed to capture richer skeleton features. The joint-level attention module intends to get the local difference among the joints within each pseudo-metapath, while the semantic-level attention module is capable of learning the global importance of different pseudo-metapaths. Extensive experiments on the NTU-RGB + D 60, NTU-RGB + D 120 and the SYSU datasets, validate that our model can attain comparable performance to the SOTA methods with 15x fewer input frames, 26.3x less FLOPs and 2.8x less parameters.

Big data analytics in health care by data mining and classification techniques

Big data is the compilation of enormous data that arrives from diverse sources for instance online transaction details, social media, sensor data, etc. Such assortment of enormous data developed is tough to evaluate by conventional processing relevance’s. By the development and upcoming latent in the healthcare business field, it is essential to analyze an​ enormous noisy data to get significant information. In healthcare system, the aim of this work is to evaluate the medical database of diabetes patients by a mixture of innovative hierarchical decision attention network, association rules (AR) and multiclass outlier classification with MapReduce framework. The association rule apriori algorithm in a MapReduce framework considers health data to create regulations. This is employed to discover the association among disease and their signs. This examination is made by means of UCI machine learning datasets of diabetes containing 50 attributes. The results of the proposed algorithm are offered by parameters for instance precision, accuracy, recall, and F-score.

Gender Bias in Occupation Classification from the New York Times Obituaries

Technological developments such as artificial intelligence can strengthen social prejudices prevailing in society, regardless of the developer's intention. Therefore, researchers should be aware of the ethical issues that may arise from a developed product/solution. In this study, we investigate the effect of gender bias on occupational classification. For this purpose, a new dataset was created by collecting obituaries from the New York Times website and is provided in two different versions: With and without gender indicators. Category distributions from this dataset show that gender and occupation variables have dependence. Thus, gender affects occupation classification. To test the effect, we perform occupation classification using SVM (Support Vector Machine), HAN (Hierarchical Attention Network), and DistilBERT-based classifiers. Moreover, to get further insights into the relationship of gender and occupation in classification problems, a multi-tasking model in which occupation and gender are learned together is evaluated. Experimental results reveal that there is a gender bias in job classification.

Investigating Contextual Influence in Document-Level Translation

Current state-of-the-art neural machine translation (NMT) architectures usually do not take document-level context into account. However, the document-level context of a source sentence to be translated could encode valuable information to guide the MT model to generate a better translation. In recent times, MT researchers have turned their focus to this line of MT research. As an example, hierarchical attention network (HAN) models use document-level context for translation prediction. In this work, we studied translations produced by the HAN-based MT systems. We examined how contextual information improves translation in document-level NMT. More specifically, we investigated why context-aware models such as HAN perform better than vanilla baseline NMT systems that do not take context into account. We considered Hindi-to-English, Spanish-to-English and Chinese-to-English for our investigation. We experimented with the formation of conditional context (i.e., neighbouring sentences) of the source sentences to be translated in HAN to predict their target translations. Interestingly, we observed that the quality of the target translations of specific source sentences highly relates to the context in which the source sentences appear. Based on their sensitivity to context, we classify our test set sentences into three categories, i.e., context-sensitive, context-insensitive and normal. We believe that this categorization may change the way in which context is utilized in document-level translation.

Contextual Features and Optimal Hierarchical Attention Networks for Sentiment Classification Under Data Streaming Environment

Contextual Features and Optimal Hierarchical Attention Networks for Sentiment Classification Under Data Streaming Environment

Multi-failure detection using device hierarchical attention network

With rapid developments in the information industry, data centers have become increasingly important for collecting and storing data. The devices in data centers are not only connected to external machines to provide a variety of services, but they also store vast amounts of data, as device failures in data centers can result in fatal and heavy economic damage. Various methods have been studied in recent years to effectively predict failures in connected devices. However, in data center-scale systems, there is a problem of low frequency of failure when predicting the failure for each device. In addition, there are complex failures that may occur within the data center owing to a mix of devices and systems, and it is difficult to determine the cause of failure in such cases. In this study, we present a device hierarchical attention network (DHAN) methodology that can predict all device failures by simultaneously using existing device information regarding the devices in the data center. Because the devices in the data center could potentially affect each other, this information regarding the device is used in a composite manner. When using information from a single device, it was observed that failure could be predicted more effectively compared to the results obtained from failure prediction. In addition, by extracting attention information from the DHAN model, we identified a device that plays an important role in predicting the failure of a particular device. Thereafter, we utilized it to cluster and reconstruct the DHAN model and identify the results of predicting failures more effectively. Based on the results presented herein, it is expected that the proposed system can be stably maintained and repaired by identifying the potential impact of the devices.

Scalable and Transferable Reinforcement Learning for Multi-Agent Mixed Cooperative-Competitive Environments Based on Hierarchical Graph Attention.

Most previous studies on multi-agent systems aim to coordinate agents to achieve a common goal, but the lack of scalability and transferability prevents them from being applied to large-scale multi-agent tasks. To deal with these limitations, we propose a deep reinforcement learning (DRL) based multi-agent coordination control method for mixed cooperative–competitive environments. To improve scalability and transferability when applying in large-scale multi-agent systems, we construct inter-agent communication and use hierarchical graph attention networks (HGAT) to process the local observations of agents and received messages from neighbors. We also adopt the gated recurrent units (GRU) to address the partial observability issue by recording historical information. The simulation results based on a cooperative task and a competitive task not only show the superiority of our method, but also indicate the scalability and transferability of our method in various scale tasks.

Multitasking Learning Model Based on Hierarchical Attention Network for Arabic Sentiment Analysis Classification

Limited approaches have been applied to Arabic sentiment analysis for a five-point classification problem. These approaches are based on single task learning with a handcrafted feature, which does not provide robust sentence representation. Recently, hierarchical attention networks have performed outstandingly well. However, when training such models as single-task learning, these models do not exhibit superior performance and robust latent feature representation in the case of a small amount of data, specifically on the Arabic language, which is considered a low-resource language. Moreover, these models are based on single task learning and do not consider the related tasks, such as ternary and binary tasks (cross-task transfer). Centered on these shortcomings, we regard five ternary tasks as relative. We propose a multitask learning model based on hierarchical attention network (MTLHAN) to learn the best sentence representation and model generalization, with shared word encoder and attention network across both tasks, by training three-polarity and five-polarity Arabic sentiment analysis tasks alternately and jointly. Experimental results showed outstanding performance of the proposed model, with high accuracy of 83.98%, 87.68%, and 84.59 on LABR, HARD, and BRAD datasets, respectively, and a minimum macro mean absolute error of 0.632% on the Arabic tweets dataset for five-point Arabic sentiment classification problem.

Drug resistance prediction and resistance genes identification in Mycobacterium tuberculosis based on a hierarchical attentive neural network utilizing genome-wide variants.

Prediction of antimicrobial resistance based on whole-genome sequencing data has attracted greater attention due to its rapidity and convenience. Numerous machine learning-based studies have used genetic variants to predict drug resistance in Mycobacterium tuberculosis (MTB), assuming that variants are homogeneous, and most of these studies, however, have ignored the essential correlation between variants and corresponding genes when encoding variants, and used a limited number of variants as prediction input. In this study, taking advantage of genome-wide variants for drug-resistance prediction and inspired by natural language processing, we summarize drug resistance prediction into document classification, in which variants are considered as words, mutated genes in an isolate as sentences, and an isolate as a document. We propose a novel hierarchical attentive neural network model (HANN) that helps discover drug resistance-related genes and variants and acquire more interpretable biological results. It captures the interaction among variants in a mutated gene as well as among mutated genes in an isolate. Our results show that for the four first-line drugs of isoniazid (INH), rifampicin (RIF), ethambutol (EMB) and pyrazinamide (PZA), the HANN achieves the optimal area under the ROC curve of 97.90, 99.05, 96.44 and 95.14% and the optimal sensitivity of 94.63, 96.31, 92.56 and 87.05%, respectively. In addition, without any domain knowledge, the model identifies drug resistance-related genes and variants consistent with those confirmed by previous studies, and more importantly, it discovers one more potential drug-resistance-related gene.

Spider Taylor-ChOA: Optimized Deep Learning Based Sentiment Classification for Review Rating Prediction

The prediction of review rating is an imperative sentiment assessment task that aims to discover the intensity of users’ sentiment toward a target product from several reviews. This paper devises a technique based on sentiment classification for predicting the review rating. Here, the review data are taken from the database. The significant features, such as SentiWordNet-based statistical features, term frequency–inverse document frequency (TF-IDF), number of capitalized words, numerical words, punctuation marks, elongated words, hashtags, emoticons, and number of sentences are mined in feature extraction. The features are mined for sentiment classification, which is performed by random multimodal deep learning (RMDL). The training of RMDL is done using the proposed Spider Taylor-ChOA, which is devised by combining spider monkey optimization (SMO) and Taylor-based chimp optimization algorithm (Taylor-ChOA). Concurrently, the features are considered input for the review rating prediction, which determines positive and negative reviews using the hierarchical attention network (HAN), and training is done using proposed Spider Taylor-ChOA. The proposed Spider Taylor-ChOA-based RMDL performed best with the highest precision of 94.1%, recall of 96.5%, and highest F-measure of 95.3%. The proposed spider Taylor-ChOA-based HAN performed best with the highest precision of 93.1%, recall of 95.4% and highest F-measure of 94.3%.

Firefly Competitive Swarm Optimization Based Hierarchical Attention Network for Lung Cancer Detection

Lung cancer is a severe disease, which causes high deaths in the world. Earlier discovery of lung cancer is useful to enhance the rate of survival in patients. Computed Tomography (CT) is utilized for determining the tumor and identifying the cancer level in the body. However, the issues of CT images cause less tumor visibility areas and unconstructive rates in tumor regions. This paper devises an optimization-driven technique for classifying lung cancer. The CT image is utilized for determining the position of the tumor. Here, the CT image undergoes segmentation, which is performed using the DeepJoint model. Furthermore, the feature extraction is carried out, wherein features such as local ternary pattern-based features, Histogram of Gradients (HoG) features, and statistical features, like variance, mean, kurtosis, energy, entropy, and skewness. The categorization of lung cancer is performed using Hierarchical Attention Network (HAN). The training of HAN is carried out using proposed Firefly Competitive Swarm Optimization (FCSO), which is devised by combining firefly algorithm (FA), and Competitive Swarm Optimization (CSO). The proposed FCSO-based HAN provided effective performance with high accuracy of 91.3%, sensitivity of 88%, and specificity of 89.1%.

A deep learning method DCWR with HANet for stock market prediction using news articles

Stock movement prediction is a challenging problem to analyze in both academic and financial research areas. The advancement of deep learning (DL) techniques has grasped the attention of researchers to employ them for predicting the stock market’s future trends. Few frameworks can understand the financial terms in literature, and the volatile nature of stock markets further complicates this process. This paper has tried to overcome the existing challenges by introducing a DL-based framework using financial news articles to forecast the stock market. After performing preprocessing step, the deep contextualized word representation (DCWR) approach is applied for feature extraction. In the next step, the independent component analysis (ICA) method is used for feature reduction. Finally, the resultant features train the hierarchical attention networks (HANet) classifier to predict the stock movements. The proposed scheme is evaluated over the 7 years of data from a publicly available dataset gathered from the Reuter’s website and attained an average prediction accuracy of 92.5% which shows our framework’s robustness.

Hierarchical graph attention network for miRNA-disease association prediction

Many biological studies show that the mutation and abnormal expression of microRNAs (miRNAs) could cause a variety of diseases. As an important biomarker for disease diagnosis, miRNA is helpful to understand pathogenesis, and could promote the identification, diagnosis and treatment of diseases. However, the pathogenic mechanism how miRNAs affect these diseases has not been fully understood. Therefore, predicting the potential miRNA-disease associations is of great importance for the development of clinical medicine and drug research. In this study, we proposed a novel deep learning model based on hierarchical graph attention network for predicting miRNA-disease associations (HGANMDA). Firstly, we constructed a miRNA-disease-lncRNA heterogeneous graph based on known miRNA-disease associations, miRNA-lncRNA associations and disease-lncRNA associations. Secondly, the node-layer attention was applied to learn the importance of neighbor nodes based on different meta-paths. Thirdly, the semantic-layer attention was applied to learn the importance of different meta-paths. Finally, a bilinear decoder was employed to reconstruct the connections between miRNAs and diseases. The extensive experimental results indicated that our model achieved good performance and satisfactory results in predicting miRNA-disease associations.

Identification of Encrypted Traffic Through Attention Mechanism Based Long Short Term Memory

Network traffic classification has become an important part of network management, which is beneficial for achieving intelligent network operation and maintenance, enhancing the network quality of service (QoS), and for network security. Given the rapid development of various applications and protocols, more and more encrypted traffic has emerged in networks. Traditional traffic classification methods exhibited the unsatisfied performance since the encrypted traffic is no longer in plain text. In this work, we modeled the time-series network traffic by the recurrent neural network (RNN). Moreover, the attention mechanism was introduced for assisting network traffic classification in the form of the following two models, the attention aided long short term memory (LSTM) as well as the hierarchical attention network (HAN). Finally, relying on the ISCX VPN-NonVPN dataset, extensive experiments were conducted, showing that the proposed methods achieved 91.2 percent in accuracy while the highest accuracy of other methods was 89.8 percent relying on the same dataset.

EmmDocClassifier: Efficient Multimodal Document Image Classifier for Scarce Data

Document classification is one of the most critical steps in the document analysis pipeline. There are two types of approaches for document classification, known as image-based and multimodal approaches. Image-based document classification approaches are solely based on the inherent visual cues of the document images. In contrast, the multimodal approach co-learns the visual and textual features, and it has proved to be more effective. Nonetheless, these approaches require a huge amount of data. This paper presents a novel approach for document classification that works with a small amount of data and outperforms other approaches. The proposed approach incorporates a hierarchical attention network (HAN) for the textual stream and the EfficientNet-B0 for the image stream. The hierarchical attention network in the textual stream uses dynamic word embedding through fine-tuned BERT. HAN incorporates both the word level and sentence level features. While earlier approaches rely on training on a large corpus (RVL-CDIP), we show that our approach works with a small amount of data (Tobacco-3482). To this end, we trained the neural network at Tobacco-3482 from scratch. Therefore, we outperform the state-of-the-art by obtaining an accuracy of 90.3%. This results in a relative error reduction rate of 7.9%.

Recipe Recommendation With Hierarchical Graph Attention Network.

Recipe recommendation systems play an important role in helping people find recipes that are of their interest and fit their eating habits. Unlike what has been developed for recommending recipes using content-based or collaborative filtering approaches, the relational information among users, recipes, and food items is less explored. In this paper, we leverage the relational information into recipe recommendation and propose a graph learning approach to solve it. In particular, we propose HGAT, a novel hierarchical graph attention network for recipe recommendation. The proposed model can capture user history behavior, recipe content, and relational information through several neural network modules, including type-specific transformation, node-level attention, and relation-level attention. We further introduce a ranking-based objective function to optimize the model. Thorough experiments demonstrate that HGAT outperforms numerous baseline methods.