Hierarchical Attention Mechanism Research Articles

Optimizing cardiovascular image segmentation through integrated hierarchical features and attention mechanisms.

Cardiovascular diseases are the top cause of death in China. Manual segmentation of cardiovascular images, prone to errors, demands an automated, rapid, and precise solution for clinical diagnosis. The paper highlights deep learning in automatic cardiovascular image segmentation, efficiently identifying pixel regions of interest for auxiliary diagnosis and research in cardiovascular diseases. In our study, we introduce innovative Region Weighted Fusion (RWF) and Shape Feature Refinement (SFR) modules, utilizing polarized self-attention for significant performance improvement in multiscale feature integration and shape fine-tuning. The RWF module includes reshaping, weight computation, and feature fusion, enhancing high-resolution attention computation and reducing information loss. Model optimization through loss functions offers a more reliable solution for cardiovascular medical image processing. Our method excels in segmentation accuracy, emphasizing the vital role of the RWF module. It demonstrates outstanding performance in cardiovascular image segmentation, potentially raising clinical practice standards. Our method ensures reliable medical image processing, guiding cardiovascular segmentation for future advancements in practical healthcare and contributing scientifically to enhanced disease diagnosis and treatment.

Dependency-position relation graph convolutional network with hierarchical attention mechanism for relation extraction

Dependency-position relation graph convolutional network with hierarchical attention mechanism for relation extraction

Neural network approaches for rumor stance detection: Simulating complex rumor propagation systems

SummaryThis research introduces a comprehensive suite of neural network models designed to tackle the challenging task of rumor stance detection within the framework of simulating complex rumor propagation systems. Our objective centers on accurately modeling the intricate structures of rumor dialogues and propagation patterns to identify user stances—whether they are in support, denial, questioning, or commenting on rumors. Unlike conventional methods that rely on simplistic keyword targeting and fail in the nuanced context of social networks, our models delve into the complexities of dialogue and propagation structures, offering a more precise and insightful analysis of rumor dynamics. In addressing the simulation and modeling of complex systems, our approach specifically focuses on the elaborate interaction networks that underpin rumor spread and reception. While our methodology does not directly engage with brain‐like computing paradigms, it reflects a similar level of sophistication in handling layered and complex information flows, analogous to cognitive processes in understanding and interpreting human communications. Employing a hierarchical attention mechanism, our models adeptly parse through multitiered dialogue sequences, effectively distinguishing between various indicators of user stances. This allows for a nuanced and detailed representation of the rumor ecosystem, significantly enhancing the accuracy of stance detection. Through rigorous testing on diverse datasets, our approach has demonstrated superior performance over existing models, thereby establishing a new benchmark in the field.

Hierarchical spatial attention-based cross-scale detection network for Digital Works Supervision System (DWSS)

Vision-based on-site detection is a crucial component of the Digital Works Supervision System (DWSS). However, accurate detecting cross-scale objects in complicated construction sites remains a challenge. In this research, a hierarchical spatial attention-based cross-scale detection network is designed to address these challenges and provide accurate on-site information for the DWSS. Firstly, a convolutional neural network is constructed to extract multi-scale feature maps with different levels of information. Furthermore, a hierarchical spatial attention mechanism is proposed to facilitate information propagation and complement between multi-scale feature maps. Secondly, the cascade detection mechanism is designed to improve the detection performance of cross-scale objects. Large-scale feature maps with more detailed information are used for the detection of small-scale objects such as workers. Small-scale feature maps with semantic features are used for the detection of large-scale objects such as construction machines. Finally, the on-site information is automatically extracted from detection results and converted into suitable data formats to generate on-site reports for the DWSS. Experiments demonstrate that the proposed method can achieve SOTA detection performance on both MOCS and SODA dataset, especially for the small-scale objects and complicated construction scenarios.

HCA-YOLO: a non-salient object detection method based on hierarchical attention mechanism

HCA-YOLO: a non-salient object detection method based on hierarchical attention mechanism

Citation Forecasting with Multi-Context Attention-Aided Dependency Modeling

Forecasting citations of scientific patents and publications is a crucial task for understanding the evolution and development of technological domains and for foresight into emerging technologies. By construing citations as a time series, the task can be cast into the domain of temporal point processes. Most existing work on forecasting with temporal point processes, both conventional and neural network-based, only performs single-step forecasting. In citation forecasting, however, the more salient goal is n -step forecasting: predicting the arrival of the next n citations. In this article, we propose Dynamic Multi-Context Attention Networks (DMA-Nets), a novel deep learning sequence-to-sequence (Seq2Seq) model with a novel hierarchical dynamic attention mechanism for long-term citation forecasting. Extensive experiments on two real-world datasets demonstrate that the proposed model learns better representations of conditional dependencies over historical sequences compared to state-of-the-art counterparts and thus achieves significant performance for citation predictions.

Hierarchical and dynamic graph attention network for drug-disease association prediction.

In the realm of biomedicine, the prediction of associations between drugs and diseases holds significant importance. Yet, conventional wet lab experiments often fall short of meeting the stringent demands for prediction accuracy and efficiency. Many prior studies have predominantly focused on drug and disease similarities to predict drug-disease associations, but overlooking the crucial interactions between drugs and diseases that are essential for enhancing prediction accuracy. Hence, in this paper, a resilient and effective model named Hierarchical and Dynamic Graph Attention Network (HDGAT) has been proposed to predict drug-disease associations. Firstly, it establishes a heterogeneous graph by leveraging the interplay of drug and disease similarities and associations. Subsequently, it harnesses the capabilities of graph convolutional networks and bidirectional long short-term memory networks (Bi-LSTM) to aggregate node-level information within the heterogeneous graph comprehensively. Furthermore, it incorporates a hierarchical attention mechanism between convolutional layers and a dynamic attention mechanism between nodes to learn embeddings for drugs and diseases. The hierarchical attention mechanism assigns varying weights to embeddings learned from different convolutional layers, and the dynamic attention mechanism efficiently prioritizes inter-node information by allocating each node with varying rankings of attention coefficients for neighbour nodes. Moreover, it employs residual connections to alleviate the over-smoothing issue in graph convolution operations. The latent drug-disease associations are quantified through the fusion of these embeddings ultimately. By conducting 5-fold cross-validation, HDGAT's performance surpasses the performance of existing state-of-the-art models across various evaluation metrics, which substantiates the exceptional efficacy of HDGAT in predicting drug-disease associations.

Infrared and visible image fusion method based on hierarchical attention mechanism

Infrared and visible image fusion method based on hierarchical attention mechanism

Abstractive text summarization model combining a hierarchical attention mechanism and multiobjective reinforcement learning

Text summarization research is significant and challenging in the domain of natural language processing. Abstractive text summarization mainly uses the encoder-decoder framework, wherein the encoder component does not have a sufficient semantic comprehension of the input text, and there are exposure biases and semantic inconsistencies between the reference and generated summaries during the training process. We propose an improved encoder-decoder model that incorporates a hierarchical attention mechanism and multiobjective reinforcement learning. The encoder introduces a multihead self-attention mechanism to allow for the acquisition of more comprehensive semantic information from multiple angles and dimensions, while the decoder introduces a pointer-generator network to solve the out-of-vocabulary problem. Multiobjective reinforcement learning methods are constructed throughout the training process to optimize the model in terms of addressing exposure bias, maintaining semantic consistency, and enhancing readability. The results of the comparative experiments demonstrate that the proposed model significantly improved in terms of the ROUGE evaluation metric, and the generated summaries were semantically similar to the reference summaries.

Attention-based Temporal Graph Representation Learning for EEG-based Emotion Recognition.

Due to the objectivity of emotional expression in the central nervous system, EEG-based emotion recognition can effectively reflect humans' internal emotional states. In recent years, convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have made significant strides in extracting local features and temporal dependencies from EEG signals. However, CNNs ignore spatial distribution information from EEG electrodes; moreover, RNNs may encounter issues such as exploding/vanishing gradients and high time consumption. To address these limitations, we propose an attention-based temporal graph representation network (ATGRNet) for EEG-based emotion recognition. Firstly, a hierarchical attention mechanism is introduced to integrate feature representations from both frequency bands and channels ordered by priority in EEG signals. Second, a graph convolutional neural network with top-k operation is utilized to capture internal relationships between EEG electrodes under different emotion patterns. Next, a residual-based graph readout mechanism is applied to accumulate the EEG feature node-level representations into graph-level representations. Finally, the obtained graph-level representations are fed into a temporal convolutional network (TCN) to extract the temporal dependencies between EEG frames. We evaluated our proposed ATGRNet on the SEED, DEAP and FACED datasets. The experimental findings show that the proposed ATGRNet surpasses the state-of-the-art graph-based mehtods for EEG-based emotion recognition.

Coarse- and Fine-Grained Fusion Hierarchical Network for Hole Filling in View Synthesis.

Depth image-based rendering (DIBR) techniques play an essential role in free-viewpoint videos (FVVs), which generate the virtual views from a reference 2D texture video and its associated depth information. However, the background regions occluded by the foreground in the reference view will be exposed in the synthesized view, resulting in obvious irregular holes in the synthesized view. To this end, this paper proposes a novel coarse and fine-grained fusion hierarchical network (CFFHNet) for hole filling, which fills the irregular holes produced by view synthesis using the spatial contextual correlations between the visible and hole regions. CFFHNet adopts recurrent calculation to learn the spatial contextual correlation, while the hierarchical structure and attention mechanism are introduced to guide the fine-grained fusion of cross-scale contextual features. To promote texture generation while maintaining fidelity, we equip CFFHNet with a two-stage framework involving an inference sub-network to generate the coarse synthetic result and a refinement sub-network for refinement. Meanwhile, to make the learned hole-filling model better adaptable and robust to the "foreground penetration" distortion, we trained CFFHNet by generating a batch of training samples by adding irregular holes to the foreground and background connection regions of high-quality images. Extensive experiments show the superiority of our CFFHNet over the current state-of-the-art DIBR methods. The source code will be available at https://github.com/wgc-vsfm/view-synthesis-CFFHNet.

Health-aware food recommendation system with dual attention in heterogeneous graphs

Recommender systems (RS) have been increasingly applied to food and health. However, challenges still remain, including the effective incorporation of heterogeneous information and the discovery of meaningful relationships among entities in the context of food and health recommendations. To address these challenges, we propose a novel framework, the Health-aware Food Recommendation System with Dual Attention in Heterogeneous Graphs (HFRS-DA), for unsupervised representation learning on heterogeneous graph-structured data. HFRS-DA utilizes an attention technique to reconstruct node features and edges and employs a dual hierarchical attention mechanism for enhanced unsupervised learning of attributed graph representations. HFRS-DA addresses the challenge of effectively leveraging the heterogeneous information in the graph and discovering meaningful semantic relationships between entities. The framework analyses recipe components and their neighbours in the heterogeneous graph and can discover popular and healthy recipes, thereby promoting healthy eating habits. We compare HFRS-DA using the Allrecipes dataset and find that it outperforms all the related methods from the literature. Our study demonstrates that HFRS-DA enhances the unsupervised learning of attributed graph representations, which is important in scenarios where labelled data is scarce or unavailable. HFRS-DA can generate node embeddings for unused data effectively, enabling both inductive and transductive learning.

SAM-Net: Self-Attention based Feature Matching with Spatial Transformers and Knowledge Distillation

In this research paper, we introduce a novel approach to enhance the performance of 2D feature matching and pose estimation through the integration of a hierarchical attention mechanism and knowledge distillation. Our proposed hierarchical attention mechanism operates at multiple scales, enabling both global context awareness and precise matching of 2D features, which is crucial for various computer vision tasks. To further improve our model’s performance, we incorporate insights from an existing model PixLoc (Sarlin et al., 2021) through knowledge distillation, effectively acquiring its behavior and capabilities by ignoring dynamic objects. SAM-Net outperforms state-of-the-art methods, validated on both indoor and outdoor public datasets. For the indoor dataset, our approach achieves remarkable AUC (5°/10°/20°) scores of 55.31/71.70/83.37. Similarly, for the outdoor dataset, we demonstrate outstanding AUC values of 26.01/46.44/63.61. Furthermore, SAM-Net achieves top ranking among published methods in two public visual localization benchmarks, highlighting the real benefits of the proposed method. The code and test suite can be accessed at link.11https://benjaminkelenyi.github.io/samnet/.

Fault detection in complex mechatronic systems by a hierarchical graph convolution attention network based on causal paths

Fault detection in mechatronic systems is crucial for their maintainability and safety. However, the systems monitoring variables are often abundant, with intricate connections. It is difficult to characterize their relationships and to extract effective features. In this paper, a hierarchical graph convolution attention network based on causal paths (HGCAN) is developed to improve the performance and interpretability of data-driven fault detection models in complex mechatronic systems. A hybrid causal discovery algorithm is introduced to discover the inherit causality among monitoring variables. The causal paths that sequentially connect the cause-effect variables serve as the reception fields to extract features using multiscale convolution. Different levels of the features are aggregated based on a hierarchical attention mechanism, which assigns adaptive weights considering the varied feature importance. To verify the effectiveness of the proposed method, a dataset of real high-speed train braking systems is considered. Experimental results demonstrate promising performance improvement of the proposed method, and analysis on interpretability indicates its potential to facilitate practical decision-making.

An adaptive structural dominant failure modes searching method based on graph neural network

Dominant failure modes (DFMs) of structural systems are integral to life prediction and reliability assessment. However, the computational efficiency of existing DFMs searching methods is constrained by neglecting the structural non-Euclidean properties. To break free from this shackle, this paper proposes a DFMs searching algorithm based on the graph neural network (GNN). The proposed algorithm can adaptively identify graph samples representing DFMs via completing graph classification. First, the target structural system is converted into an undirected graph to preserve its topological features. Second, a hierarchical graph attention mechanism is developed to establish the mapping relationship between structure intrinsic properties and DFMs. Finally, two adaptive sample selection strategies are devised to iteratively search DFMs and supplement graph datasets. In order to reduce the number of reliability analyses, the algorithm will terminate prematurely when unable to identify new DFMs. A 2D truss and a 3D frame are selected to test the computational efficiency and stability of the algorithm. The search results indicate that, despite providing different initial training sets, this GNN-based algorithm still converges to DFMs consistent with the result of Monte Carlo Simulation (MCS). Compared to the genetic algorithm (GA) and the β-unzipping method, the proposed algorithm exhibits higher computational efficiency.

The Source Code Comment Generation Based on Deep Reinforcement Learning and Hierarchical Attention

Code summarization provides the main aim described in natural language of the given function, it can benefit many tasks in software engineering. Due to the special grammar and syntax structure of programming languages and various shortcomings of different deep neural networks, the accuracy of existing code summarization approaches is not good enough. This work proposes to adopt the hierarchical attention mechanism to enable the code summarization framework to translate three representations of source code to the hidden spce and then it injects them into a deep reinforcement learning model to enhance the performance of code summarization. We conduct a few of experiments, and the results of which prove that the proposed approaches can obtain better accuracy compared with the baseline approaches.

A hierarchical attention detector for bearing surface defect detection

Multi-scale defect detection on bearing surfaces is a challenging task due to the varying sizes and shapes of the defects and character-induced noise backgrounds. It has been proven that introducing attention mechanism in convolutional neural networks can often enhance feature extraction performance. However, most existing detectors use attention mechanisms at only one specific level of abstraction, which limits their capability to extract all defect features. In response to this issue, inspired by natural language processing, we present a novel visual hierarchical attention detector for multi-scale defect location and classification. The detector leverages texture, semantic, and instance features of defects through a hierarchical attention mechanism, enabling multi-scale defect detection in bearing images with complex backgrounds. First, a multi-attentional backbone uses spatial attention mechanisms to extract the spatial and semantic information of defects. Second, a semantic alignment and channel attention based feature pyramid network aligns the semantic information of the extracted features and concentrates on the important layers of the feature pyramid. Third, a semantic guided anchoring based region proposal network uses instance-level attention to refine the anchors. Furthermore, the distance-intersection over union loss optimizes the detector to locate defects. The experiments on an industrial bearing surface defect image dataset demonstrated that the proposed method outperformed the baseline by 6.03 % and 8.76 % in terms of mean average precision (mAP) and mean average recall (mAR), achieving 0.7838 mAP and 0.9588 mAR, respectively.

Adaptive Neighbor Graph Aggregated Graph Attention Network for Heterogeneous Graph Embedding

Graph attention network can generate effective feature embedding by specifying different weights to different nodes. The key of the research on heterogeneous graph embedding is the way to combine its rich structural information with semantic relations to aggregate the neighborhood information. Most of the existing heterogeneous graph representation learning methods guide the selection of neighbors by defining various meta-paths on heterogeneous graphs. However, these models only consider the information contained in the nodes under different paths and ignore the potential semantic relationships of nodes in different neighbor graph structures, which leads to the underutilization of graph structure information. In this article, we propose a novel adaptive framework named Neighbor Graph Aggregated Graph Attention Network (NGGAN) to fully exploit graph topological details in heterogeneous graph, and aggregates their information to obtain an effective embedding. The key idea is to use different levels of sampling methods to define neighborhood, and use neighbor graphs to represent the complex structural interaction between nodes. In this way, the high-order relationship between nodes and the latent semantics of neighbor graphs can be fully explored. Afterward, a hierarchical attention mechanism is applied to adaptively learn the importance of different objects, including node information, path information, and neighbor graph information. Multiple downstream tasks are performed on four real-world heterogeneous graph datasets, and the experimental results demonstrate the effectiveness of NGGAN.

Deep learning model with hierarchical attention mechanism for sentiment classification of Vietnamese comments

Deep learning model with hierarchical attention mechanism for sentiment classification of Vietnamese comments

Asset pricing via fused deep learning with visual clues

Asset pricing via machine learning provides a promising way to capture price trends by fusing heterogeneous market factors to analyze their joint impact on stock movements rather than relying on statistical and econometric models in finance to explore the causality between a market indicator and stock returns. However, the fusion nature of machine learning also hides the way to unveil the internal mechanism of stock movements. In this study, a deep learning framework with visual clues is presented to unveil the entangled factors and their function on stock movements. In particular, a context-aware hierarchical attention mechanism (CHARM) is first proposed to encode unstructured textual media information to trace the literal power of news on such media-aware stock movements. The encoded media and other structured market factors are further fused via tensor-based learning to infer and visualize their interactions on stock fluctuations. Last, a pre-estimating method for locating turning points as trading clues is utilized to improve the efficiency of each investment opportunity. Experiments conducted in real securities markets demonstrate the proposed framework not only improve the readability of the investing strategies, but also enhance the predictive accuracy and the investing returns comparing with the state-of-the-art models, including AZFinText, TeSIA, eLSTM, CMT, MAC and SA-DLSTM.