Hierarchical Attention Mechanism Research Articles

Deep Reinforcement Learning Recommendation System Algorithm Based on Multi-Level Attention Mechanisms

Traditional recommendation systems, which rely on static user profiles and historical interaction data, frequently face difficulties in adapting to the rapid changes in user preferences that are typical of dynamic environments. In contrast, recommendation algorithms based on deep reinforcement learning are capable of dynamically adjusting their strategies to accommodate real-time fluctuations in user preferences. However, current deep reinforcement learning recommendation algorithms encounter several challenges, including the oversight of item features associated with high long-term rewards that reflect users’ enduring interests, as well as a lack of significant relevance between user attributes and item characteristics. This leads to an inadequate extraction of personalized information. To address these issues, this study presents a novel recommendation system known as the Multi-Level Hierarchical Attention Mechanism Deep Reinforcement Recommendation (MHDRR), which is fundamentally grounded in a multi-layer attention mechanism. This mechanism consists of a local attention layer, a global attention layer, and a Transformer layer, allowing for a detailed analysis of individual attributes and interactions within short-term preferred items, while also exploring users’ long-term interests. This methodology promotes a comprehensive understanding of users’ immediate and enduring preferences, thereby improving the overall effectiveness of the system over time. Experimental results obtained from three publicly available datasets validate the effectiveness of the proposed model.

Enhancing human behavior recognition with dynamic graph convolutional networks and multi-scale position attention

PurposeHuman behavior recognition poses a pivotal challenge in intelligent computing and cybernetics, significantly impacting engineering and management systems. With the rapid advancement of autonomous systems and intelligent manufacturing, there is an increasing demand for precise and efficient human behavior recognition technologies. However, traditional methods often suffer from insufficient accuracy and limited generalization ability when dealing with complex and diverse human actions. Therefore, this study aims to enhance the precision of human behavior recognition by proposing an innovative framework, dynamic graph convolutional networks with multi-scale position attention (DGCN-MPA) to sup.Design/methodology/approachThe primary applications are in autonomous systems and intelligent manufacturing. The main objective of this study is to develop an efficient human behavior recognition framework that leverages advanced techniques to improve the prediction and interpretation of human actions. This framework aims to address the shortcomings of existing methods in handling the complexity and variability of human actions, providing more reliable and precise solutions for practical applications. The proposed DGCN-MPA framework integrates the strengths of convolutional neural networks and graph-based models. It innovatively incorporates wavelet packet transform to extract time-frequency characteristics and a MPA module to enhance the representation of skeletal node positions. The core innovation lies in the fusion of dynamic graph convolution with hierarchical attention mechanisms, which selectively attend to relevant features and spatial relationships, adjusting their importance across scales to address the variability in human actions.FindingsTo validate the effectiveness of the DGCN-MPA framework, rigorous evaluations were conducted on benchmark datasets such as NTU-RGB + D and Kinetics-Skeleton. The results demonstrate that the framework achieves an F1 score of 62.18% and an accuracy of 75.93% on NTU-RGB + D and an F1 score of 69.34% and an accuracy of 76.86% on Kinetics-Skeleton, outperforming existing models. These findings underscore the framework’s capability to capture complex behavior patterns with high precision.Originality/valueBy introducing a dynamic graph convolutional approach combined with multi-scale position attention mechanisms, this study represents a significant advancement in human behavior recognition technologies. The innovative design and superior performance of the DGCN-MPA framework contribute to its potential for real-world applications, particularly in integrating behavior recognition into engineering and autonomous systems. In the future, this framework has the potential to further propel the development of intelligent computing, cybernetics and related fields.

HCA-RFLA: A SAR Remote Sensing Ship Detection Based on Hierarchical Collaborative Attention Method and Gaussian Receptive Field-Driven Label Assignment Strategy

Ensuring safety at sea has become a primary focus of marine monitoring, driving the increasing adoption of ship detection technology in the maritime industry. Detecting small ship targets in SAR images presents challenges, as they occupy only a small portion of the image and exhibit subtle features, reducing detection efficiency. To address these challenges, we propose the HCA-RFLA algorithm for ship detection in SAR remote sensing. To better capture small targets, we design a hierarchical collaborative attention (HCA) mechanism that enhances feature representation by integrating multi-level features with contextual information. Additionally, due to the scarcity of positive samples for small targets under IoU and center sampling strategies, we propose a label assignment strategy based on Gaussian receptive fields, known as RFLA. RFLA assigns positive samples to small targets based on the Gaussian distribution between feature points and ground truth, increasing the model’s sensitivity to small samples. The HCA-RFLA was experimentally validated using the SSDD, HRSID, and SSD datasets. Compared to other state-of-the-art methods, HCA-RFLA improves detection accuracy by 6.2%, 4.4%, and 3.6%, respectively. These results demonstrate that HCA-RFLA outperforms existing algorithms in SAR remote sensing ship detection.

Deep Learning-Enabled De-Noising of Fiber Bragg Grating-Based Glucose Sensor: Improving Sensing Accuracy of Experimental Data

This paper outlines the successful utilization of deep learning (DL) techniques to elevate data quality for assessing Au-TFBG (tilted fiber Bragg grating) sensor performance. Our approach involves a well-structured DL-assisted framework integrating a hierarchical composite attention mechanism. In order to mitigate high variability in experimental data, we initially employ seasonal decomposition using moving averages (SDMA) statistical models to filter out redundant data points. Subsequently, sequential DL models extrapolate the normalized transmittance (Tn) vs. wavelength spectra, which showcases promising results through our SpecExLSTM model. Furthermore, we introduce the AttentiveSpecExLSTM model, integrating a composite attention mechanism to improve Tn sequence prediction accuracy. Evaluation metrics demonstrate its superior performance, including a root mean square error of 1.73 ± 0.05, a mean absolute error of 1.20 ± 0.04, and a symmetric mean absolute percentage error of 2.22 ± 0.05, among others. Additionally, our novel minima difference (Min. Dif.) metric achieves a value of 1.08 ± 0.46, quantifying wavelength for the global minima within the Tn sequence. The composite attention mechanism in the AttentiveSpecExLSTM adeptly captures both high-level and low-level dependencies, refining the model’s comprehension and guiding informed decisions. Hierarchical dot and additive attention within this model enable nuanced attention refinement across model layers; dot attention focuses on high-level dependencies, while additive attention fine-tunes its focus on low-level dependencies within the sequence. This innovative strategy enables accurate estimation of the spectral width (full-width half maxima) of the Tn curve, surpassing raw data’s capabilities. These findings significantly contribute to data quality enhancement and sensor performance analysis. Insights from this study hold promise for future sensor applications, enhancing sensitivity and accuracy by improving experimental data quality and sensor performance assessment.

Fine-grained Abnormality Detection and Natural Language Description of Medical CT Images Using Large Language Models

Medical report generation demands accurate abnormality detection and precise description generation from CT images. While large language models have shown promising results in natural language processing tasks, their application in medical imaging analysis faces challenges due to the complexity of fine-grained feature detection and the requirement for domain-specific knowledge. This paper presents a novel framework integrating large language models with specialized medical image processing techniques for fine-grained abnormality detection and natural language description generation. Our approach incorporates a multi-modal knowledge enhancement module and a hierarchical attention mechanism to bridge the gap between visual understanding and textual description. The framework employs an adapter-based architecture for efficient domain adaptation and introduces a medical knowledge-enhanced loss function to improve description accuracy. Experimental results on three public datasets demonstrate the effectiveness of our approach, achieving 94.6% detection accuracy and a BLEU-4 score of 0.421 for description generation, surpassing current state-of-the-art methods. The system shows particular strength in handling subtle abnormalities, with a 91.2% average precision in fine-grained detection tasks. Comprehensive ablation studies validate the contribution of each component, while qualitative analysis demonstrates the clinical relevance of generated descriptions. The proposed framework represents a significant advancement in automated medical image analysis, offering potential benefits for clinical workflow optimization and diagnostic support.

Hierarchical Attention and Contextual Embedding for Robust Sarcasm Detection

Objective: The study aims to develop a technique for sarcasm detection in social media text to enrich the sentiment analysis. Methods: The methodology used in the present research is as follows: emotion-embedded vectors for capturing the emotional content of the text, dynamic contextual modulation for the adaption of the model to the given context, and Hierarchical attention mechanism for segmentation of the text at the different level of abstraction. Findings: Evaluation with the test set proved that the proposed model achieved accuracy rates 89% higher than the benchmark models. Including these several complicated methods helped achieve greater accuracy and F1 measures, contributing to sarcasm detection efficiency. Novelty: This approach ensures that vital sarcastic alerts are detected, providing a better text analysis and increasing sentiment analysis's accuracy and robustness. Keywords: Sarcasm detection, Sentiment analysis, Natural language processing, Deep learning, Opinion mining

SCKansformer: Fine-Grained Classification of Bone Marrow Cells via Kansformer Backbone and Hierarchical Attention Mechanisms.

The incidence and mortality rates of malignant tumors, such as acute leukemia, have risen significantly. Clinically, hospitals rely on cytological examination of peripheral blood and bone marrow smears to diagnose malignant tumors, with accurate blood cell counting being crucial. Existing automated methods face challenges such as low feature expression capability, poor interpretability, and redundant feature extraction when processing highdimensional microimage data. We propose a novel finegrained classification model, SCKansformer, for bone marrow blood cells, which addresses these challenges and enhances classification accuracy and efficiency. The model integrates the Kansformer Encoder, SCConv Encoder, and Global-Local Attention Encoder. The Kansformer Encoder replaces the traditional MLP layer with the KAN, improving nonlinear feature representation and interpretability. The SCConv Encoder, with its Spatial and Channel Reconstruction Units, enhances feature representation and reduces redundancy. The Global-Local Attention Encoder combines Multi-head Self-Attention with a Local Part module to capture both global and local features. We validated our model using the Bone Marrow Blood Cell FineGrained Classification Dataset (BMCD-FGCD), comprising over 10,000 samples and nearly 40 classifications, developed with a partner hospital. Comparative experiments on our private dataset, as well as the publicly available PBC and ALL-IDB datasets, demonstrate that SCKansformer outperforms both typical and advanced microcell classification methods across all datasets.

Object Detection in Satellite Images with Hierarchical Collective Hunter Algorithm Optimized Ensemble Classifier

Object detection in remote sensing images plays an essential role in computer vision task, which significantly identifies the position and class of the located objects with arbitrary orientation information. Recently, object detection plays an important role in various applications such as Surveillance and security in airports, organizations, and so on. However, target tracking is a very challenging task in natural images to attain appropriate detection results together with they obtain certain issues such as limited availability of high-resolute images, background complexity, and so on. To avoid these limitations, developed a Hierarchical collective Hunter algorithm with a Deep convolutional Neural network and a Bidirectional long short-term classifier (2HC-DBTM) model, which improved the interaction range of feature extraction by reducing the background complexity. Additional attributes mechanism such as the Hierarchical Collective Hunter (HCH) algorithm and hybrid attention mechanism are integrated with the proposed model for improve the ability and performance with minimum computational complexity and achieves better detection accuracy. The experimental results obtained in the evaluation of the model are achieved with high robustness and efficiency when compared with other conventional methods. The achieved performance evaluation of the model for evaluation metric F1-score is 97.71%, precision is 97.16%, recall is 98.27% and accuracy is 98.05%.

TABLE: Time-aware Balanced Multi-view Learning for stock ranking

Stock ranking is a significant and challenging problem. In recent years, the use of multi-view data, such as price and tweet, for stock ranking has gained considerable attention in the research field. Most existing methods are performed in (some of) the 3 steps: 1) view-specific representation learning; 2) cross-view representation interaction; 3) multi-view representation fusion. Although these methods make breakthroughs in stock ranking, they often treat all views equally. This neglects the unbalanced phenomenon in multi-view stock data, i.e., the dimension of the text view may be extremely big compared with those of other views; the price view exhibits standard and high-quality data, whereas the text view contains noise and has irregular time intervals. To solve this, we propose a Time-Aware Balanced multi-view LEarning (TABLE) method. TABLE method consists of a view-specific learning stage and a multi-view fusion stage. In the first stage, we aim to improve the quality of the low-quality text view. We achieve this by attenuating the negative impact of irrelevant texts using a hierarchical temporal attention mechanism that captures text correlations. Additionally, we explicitly model the time irregularities between sequential texts. In the fusion stage, we address the dimensions unbalance problem by establishing a multi-view decision fusion paradigm by weighted averaging the view-specific stock predictions. These weights are dynamic and determined based on the quality discrepancy between the views. Finally, we obtain the optimal stock ranking list by optimizing the point-wise regression loss and the ranking-aware loss. We empirically compare TABLE method with state-of-the-art baselines using the publicly available dataset, S&P500. The experimental results demonstrate that TABLE method outperforms the baseline methods in terms of accuracy and investment revenue.

An improved infrared simulation method based on generative adversarial networks

Infrared simulation technology has great application value in the fields of UAV target simulation and infrared signal processing. However, traditional infrared simulation technology has the disadvantages of poor scene adaptability, low fidelity of infrared simulation images and long training time. This paper proposes an infrared image simulation algorithm based on improved Generative Adversarial Networks(GAN) to generate multi-scene infrared images, which could provide a reliable expansion of infrared database. First, we construct a GAN model by introducing a multi-resolution hierarchical structure and attention mechanism, which can effectively enhance the model’s adaptability to different scenes during the simulation of generating infrared images. Secondly, the SSIM loss term and L1 loss term are introduced during the training process to form a composite loss function, which prompts the proposed GAN to generate high-fidelity infrared simulation images. Finally, while maintaining the quality of simulated infrared images, a lightweight model is designed through model compression to significantly reduce the number of model parameters.The traditional discriminator based on the UNet-self attention is replaced by a discriminator based on attention mechanisms and adaptive max-pooling structure. The proposed GAN is trained and tested on three public datasets. Experimental results demonstrate that the proposed method improves structural similarity by 11% compared to Pix2Pix and achieves a Peak Signal-to-Noise Ratio (PSNR) of 25.8, which is 3.9 higher than InfraGAN. At the same time, compared with InfraGAN, the number of model parameters is reduced by 68% , which makes the method efficiently obtain simulated infrared images corresponding to given visible images.

Automated crack localization for road safety using contextual u-net with spatial-channel feature integration

Accurate and timely crack localization is crucial for road safety and maintenance, but image processing and hand-crafted feature engineering methods, often fail to distinguish cracks from background noise under diverse lighting and surface conditions. This paper proposes a framework utilizing contextual U-Net deep learning model to automatically localize cracks in road images. The framework design considers crack localization as a task of pixel-level segmenting, and analyzing each pixel in a road image to determine if it belongs to a crack or not. The proposed U-Net model uses a robust EfficientNet encoder to capture crucial details (spatial features) and overall patterns (channel-wise features) within the road image. This balanced approach helps the model learn effectively from both individual elements and the context of the images, leading to improved crack detection. A customized hierarchical attention mechanism is designed to make U-Net model contextually adaptive to focus on relevant features at different scales and resolutions for accurately localizing road cracks that can vary widely in size and shape. The model's effectiveness is demonstrated through extensive evaluations on the benchmarked and custom-made datasets.

Optimizing cardiovascular image segmentation through integrated hierarchical features and attention mechanisms.

Cardiovascular diseases are the top cause of death in China. Manual segmentation of cardiovascular images, prone to errors, demands an automated, rapid, and precise solution for clinical diagnosis. The paper highlights deep learning in automatic cardiovascular image segmentation, efficiently identifying pixel regions of interest for auxiliary diagnosis and research in cardiovascular diseases. In our study, we introduce innovative Region Weighted Fusion (RWF) and Shape Feature Refinement (SFR) modules, utilizing polarized self-attention for significant performance improvement in multiscale feature integration and shape fine-tuning. The RWF module includes reshaping, weight computation, and feature fusion, enhancing high-resolution attention computation and reducing information loss. Model optimization through loss functions offers a more reliable solution for cardiovascular medical image processing. Our method excels in segmentation accuracy, emphasizing the vital role of the RWF module. It demonstrates outstanding performance in cardiovascular image segmentation, potentially raising clinical practice standards. Our method ensures reliable medical image processing, guiding cardiovascular segmentation for future advancements in practical healthcare and contributing scientifically to enhanced disease diagnosis and treatment.

Dependency-position relation graph convolutional network with hierarchical attention mechanism for relation extraction

Dependency-position relation graph convolutional network with hierarchical attention mechanism for relation extraction

Neural network approaches for rumor stance detection: Simulating complex rumor propagation systems

SummaryThis research introduces a comprehensive suite of neural network models designed to tackle the challenging task of rumor stance detection within the framework of simulating complex rumor propagation systems. Our objective centers on accurately modeling the intricate structures of rumor dialogues and propagation patterns to identify user stances—whether they are in support, denial, questioning, or commenting on rumors. Unlike conventional methods that rely on simplistic keyword targeting and fail in the nuanced context of social networks, our models delve into the complexities of dialogue and propagation structures, offering a more precise and insightful analysis of rumor dynamics. In addressing the simulation and modeling of complex systems, our approach specifically focuses on the elaborate interaction networks that underpin rumor spread and reception. While our methodology does not directly engage with brain‐like computing paradigms, it reflects a similar level of sophistication in handling layered and complex information flows, analogous to cognitive processes in understanding and interpreting human communications. Employing a hierarchical attention mechanism, our models adeptly parse through multitiered dialogue sequences, effectively distinguishing between various indicators of user stances. This allows for a nuanced and detailed representation of the rumor ecosystem, significantly enhancing the accuracy of stance detection. Through rigorous testing on diverse datasets, our approach has demonstrated superior performance over existing models, thereby establishing a new benchmark in the field.

Hierarchical spatial attention-based cross-scale detection network for Digital Works Supervision System (DWSS)

Vision-based on-site detection is a crucial component of the Digital Works Supervision System (DWSS). However, accurate detecting cross-scale objects in complicated construction sites remains a challenge. In this research, a hierarchical spatial attention-based cross-scale detection network is designed to address these challenges and provide accurate on-site information for the DWSS. Firstly, a convolutional neural network is constructed to extract multi-scale feature maps with different levels of information. Furthermore, a hierarchical spatial attention mechanism is proposed to facilitate information propagation and complement between multi-scale feature maps. Secondly, the cascade detection mechanism is designed to improve the detection performance of cross-scale objects. Large-scale feature maps with more detailed information are used for the detection of small-scale objects such as workers. Small-scale feature maps with semantic features are used for the detection of large-scale objects such as construction machines. Finally, the on-site information is automatically extracted from detection results and converted into suitable data formats to generate on-site reports for the DWSS. Experiments demonstrate that the proposed method can achieve SOTA detection performance on both MOCS and SODA dataset, especially for the small-scale objects and complicated construction scenarios.

HCA-YOLO: a non-salient object detection method based on hierarchical attention mechanism

HCA-YOLO: a non-salient object detection method based on hierarchical attention mechanism

Citation Forecasting with Multi-Context Attention-Aided Dependency Modeling

Forecasting citations of scientific patents and publications is a crucial task for understanding the evolution and development of technological domains and for foresight into emerging technologies. By construing citations as a time series, the task can be cast into the domain of temporal point processes. Most existing work on forecasting with temporal point processes, both conventional and neural network-based, only performs single-step forecasting. In citation forecasting, however, the more salient goal is n -step forecasting: predicting the arrival of the next n citations. In this article, we propose Dynamic Multi-Context Attention Networks (DMA-Nets), a novel deep learning sequence-to-sequence (Seq2Seq) model with a novel hierarchical dynamic attention mechanism for long-term citation forecasting. Extensive experiments on two real-world datasets demonstrate that the proposed model learns better representations of conditional dependencies over historical sequences compared to state-of-the-art counterparts and thus achieves significant performance for citation predictions.

Hierarchical and dynamic graph attention network for drug-disease association prediction.

In the realm of biomedicine, the prediction of associations between drugs and diseases holds significant importance. Yet, conventional wet lab experiments often fall short of meeting the stringent demands for prediction accuracy and efficiency. Many prior studies have predominantly focused on drug and disease similarities to predict drug-disease associations, but overlooking the crucial interactions between drugs and diseases that are essential for enhancing prediction accuracy. Hence, in this paper, a resilient and effective model named Hierarchical and Dynamic Graph Attention Network (HDGAT) has been proposed to predict drug-disease associations. Firstly, it establishes a heterogeneous graph by leveraging the interplay of drug and disease similarities and associations. Subsequently, it harnesses the capabilities of graph convolutional networks and bidirectional long short-term memory networks (Bi-LSTM) to aggregate node-level information within the heterogeneous graph comprehensively. Furthermore, it incorporates a hierarchical attention mechanism between convolutional layers and a dynamic attention mechanism between nodes to learn embeddings for drugs and diseases. The hierarchical attention mechanism assigns varying weights to embeddings learned from different convolutional layers, and the dynamic attention mechanism efficiently prioritizes inter-node information by allocating each node with varying rankings of attention coefficients for neighbour nodes. Moreover, it employs residual connections to alleviate the over-smoothing issue in graph convolution operations. The latent drug-disease associations are quantified through the fusion of these embeddings ultimately. By conducting 5-fold cross-validation, HDGAT's performance surpasses the performance of existing state-of-the-art models across various evaluation metrics, which substantiates the exceptional efficacy of HDGAT in predicting drug-disease associations.

Infrared and visible image fusion method based on hierarchical attention mechanism

Infrared and visible image fusion method based on hierarchical attention mechanism

Abstractive text summarization model combining a hierarchical attention mechanism and multiobjective reinforcement learning

Text summarization research is significant and challenging in the domain of natural language processing. Abstractive text summarization mainly uses the encoder-decoder framework, wherein the encoder component does not have a sufficient semantic comprehension of the input text, and there are exposure biases and semantic inconsistencies between the reference and generated summaries during the training process. We propose an improved encoder-decoder model that incorporates a hierarchical attention mechanism and multiobjective reinforcement learning. The encoder introduces a multihead self-attention mechanism to allow for the acquisition of more comprehensive semantic information from multiple angles and dimensions, while the decoder introduces a pointer-generator network to solve the out-of-vocabulary problem. Multiobjective reinforcement learning methods are constructed throughout the training process to optimize the model in terms of addressing exposure bias, maintaining semantic consistency, and enhancing readability. The results of the comparative experiments demonstrate that the proposed model significantly improved in terms of the ROUGE evaluation metric, and the generated summaries were semantically similar to the reference summaries.