Attention Mechanism Research Articles

Sentiment Analysis for E-commerce Product Reviews by Deep Learning Model of Bert- BiGRU-Softmax

The sentiment analysis has the key problem for the e-commerce product quality management, which customers know their people’s attitudes on interested products by e-commerce reviews. Meanwhile, manufacturers are able to learn the public sentiment on their products being sold in E-commerce platforms. This paper proposed the deep learning models of Bert-BiGRU-Softmax, which uses the input layer of Sentiment Bert model to extract multi-dimensional of E-commerce reviews, the hidden layer of Bidirectional GRU model to obtain semantic codes and calculate representing weights of reviews, the Softmax with attention mechanism as the output layer to classify the sentiment tendency. We conduct experiments on a large-scale dataset involving over 500 thousand reviews compared with different learning models. The experimental results show that the proposed models reach the high accuracy 95.5% on the E-commerce reviews, and the outperforms of RNN, BiGRU, Bert-BiLSTM in terms of accuracy and loss.

A Generalized Attention Mechanism to Enhance the Accuracy Performance of Neural Networks.

In many modern machine learning (ML) models, attention mechanisms (AMs) play a crucial role in processing data and identifying significant parts of the inputs, whether these are text or images. This selective focus enables subsequent stages of the model to achieve improved classification performance. Traditionally, AMs are applied as a preprocessing substructure before a neural network, such as in encoder/decoder architectures. In this paper, we extend the application of AMs to intermediate stages of data propagation within ML models. Specifically, we propose a generalized attention mechanism (GAM), which can be integrated before each layer of a neural network for classification tasks. The proposed GAM allows for at each layer/step of the ML architecture identification of the most relevant sections of the intermediate results. Our experimental results demonstrate that incorporating the proposed GAM into various ML models consistently enhances the accuracy of these models. This improvement is achieved with only a marginal increase in the number of parameters, which does not significantly affect the training time.

A novel heat load prediction model of district heating system based on hybrid whale optimization algorithm (WOA) and CNN-LSTM with attention mechanism

Machine learning models, particularly long short-term memory (LSTM) networks, have been extensively employed for heat load prediction in district heating systems (DHS). Nevertheless, the over-reliance on default hyperparameter settings in most methods hinders further enhancement of prediction accuracy. A novel load prediction model is presented, which integrates the whale optimization algorithm (WOA) to refine the hyperparameters of an LSTM model bolstered by an attention mechanism (ATT) and convolutional neural network (CNN). Three hybrid models (WOA-CNN-ATT-LSTM, PSO-CNN-ATT-LSTM and GA-CNN-ATT-LSTM) are constructed by comparing WOA with particle swarm optimization (PSO) and genetic algorithm (GA). The proposed hybrid models are evaluated against traditional LSTM models using an 1100-hour dataset from a real DHS. The outcomes reveal that the WOA-CNN-ATT-LSTM model surpasses both the PSO-CNN-ATT-LSTM and GA-CNN-ATT-LSTM models in heat load prediction accuracy, achieving improvements of 1.9% and 3.2% respectively, and attaining the highest prediction accuracy (R2=0.9962, MSE=0.0001, MAE=0.0082). Moreover, the WOA-CNN-ATT-LSTM model demonstrates superior performance across various time scales (half-day, one-day, three-days, and one-week), highlighting its robustness in heat load prediction. This novel model adaptively adjusts its hyperparameters to identify the optimal configuration, thereby significantly augmenting the overall predictive capabilities of the model.

A deep learning method based on CNN-BiGRU and attention mechanism for proton exchange membrane fuel cell performance degradation prediction

The performance of proton exchange membrane fuel cells (PEMFCs) will gradually deteriorate during long-term operation. Accurate performance degradation prediction is crucial for extending the lifespan and improve the durability of fuel cells. This paper proposes a deep learning method (CNN-BiGRU-AM) that incorporates convolutional neural network (CNN), bidirectional gated recurrent unit (BiGRU) and attention mechanism (AM) for fuel cell degradation prediction. In the proposed method, CNN extracts complex features from the input data through convolutional operations, BiGRU models temporal information by considering both forward and reverse directions of the input sequence, and attention mechanism highlights key information in the input data through weight allocation. The proposed method is validated using long-term experimental data from fuel cells under steady-state and quasi-dynamic conditions. The results indicate that the absolute error of the proposed method is less than 1.2mV for 97.94% of the data samples under steady-state conditions and less than 1.2mV for 94.82% of the data samples under quasi-dynamic conditions. The prediction accuracy and stability of the proposed method are significantly improved compared to other deep learning prediction methods.

Transliteration and translation of Hindi language using integrated domain-based Auto-encoder

The main objective of translation is to translate words' meanings from one language to another; in contrast, transliteration does not translate any contextual meanings between languages. Transliteration, as opposed to translation, just considers the individual letters that make up each word. In this paper an Integrated deep neural network transliteration and translation model (NNTT) based autoencoder model is developed. The model is segmented into transliteration model and translation model; the transliteration involves the process of converting text from one script to another evaluated on the Dakshina dataset wherein Hindi typically uses a sequence-to-sequence model with an attention mechanism, the translation model is trained to translate text from one language to another. Translation models regularly use a sequence-to-sequence model performed on the WAT (Workshop on Asian Translation) 2021 dataset with an attention mechanism, similar to the one used in the transliteration model for Hindi. The proposed NNTT model merges the in-domain and out-domain frameworks to develop a training framework so that the information is transferred between the domains. The results evaluated show that the proposed model works effectively in comparison with the existing system for the Hindi language.

Research on deformation prediction of CFETR multi-purpose overloaded robot based on real-time structural simulator

The CFETR Multi-Purpose Overload Robot (CMOR) is a key subsystem of the China Fusion Engineering Test Reactor (CFETR), which can perform maintenance tasks of the internal components. However, the large slenderness ratio of the structure results in low control accuracy of the CMOR end-effector. This paper proposes a CMOR deformation prediction method based on the real-time structural simulator. The overall deformation model of CMOR is analyzed based on the single joint deformation mechanism. Based on the principle of layered control, the control framework of the CMOR structural simulator is constructed, and the deformation data of CMOR in different positions are calculated based on the finite element method. A hybrid neural network containing a multilayer perceptron, transformer, and attention mechanism is designed to train the CMOR deformation prediction model. The training results show that the deformation prediction model converges quickly and fits the deformation of the CMOR structure well with small prediction errors. Finally, the real-time structure simulator is developed based on the deformation prediction model, and the deformation of CMOR is reconstructed with the update frequency of 2 Hz and the absolute error at any point within ±10 mm, which verifies the correctness of the CMOR structural deformation prediction method.

Heterogeneous Graph Neural Network with Hierarchical Attention for Group-Aware Paper Recommendation in Scientific Social Networks

In recent years, the academic groups established in Scientific Social Networks (SSNs) have not only facilitated collaboration among researchers but also enriched the relations in SSNs, providing valuable information for paper recommendation tasks. However, existing paper recommendation methods rarely consider group information and they fail to fully leverage the group information due to the heterogeneous and complex relations between researchers, papers, and groups. In this paper, a heterogeneous graph neural network with hierarchical attention, named HHA-GPR, is proposed for group-aware paper recommendation. Firstly, a heterogeneous graph is constructed based on the interactions of researchers, papers, and groups in SSNs. Secondly, a random walk-based sampling strategy is utilized to sample highly correlated heterogeneous neighbors for researchers and papers. Thirdly, a hierarchical attention network with intra-type and inter-type attention mechanisms is designed to aggregate the sampled neighbors and comprehensively model the complex relations among the heterogeneous neighbors. More specifically, an intra-type attention mechanism is introduced to aggregate the neighbors of the same type, and an inter-type attention mechanism is employed to combine the embeddings of different types to form the ultimate node embedding. Extensive experiments are conducted on the real-world CiteULike and AMiner datasets, and the experimental results demonstrate that our proposed method outperforms other benchmark methods with an average improvement of 5.3% in Precision, 5.6% in Recall, and 5.1% in Normalized Discounted Cumulative Gain (NDCG) across both datasets.

Robust and energy-efficient RPL optimization algorithm with scalable deep reinforcement learning for IIoT

The increasing complexity and quantity of the Industrial Internet of Things (IIoT) pose new challenges to the traditional routing protocol for low-power and lossy networks (RPL) in terms of dynamic management, data transmission reliability, and energy efficiency optimization. This paper proposes a scalable deep reinforcement learning (DRL) algorithm with a multi-attention actor double critic model for routing optimization (MADC) to meet the requirements of IIoT for efficient and intelligent routing decisions while improving data transmission reliability and energy efficiency. Specifically, MADC employs the centralized training and decentralized execution (CTDE) learning paradigm to decouple the model’s training and inference tasks, which reduces the difficulty and computational cost of model learning and improves the training efficiency. In addition, a lightweight actor network based on multi-scale convolutional attention mechanism is designed in MADC, which can provide intelligent and real-time decision-making capabilities for resource-constrained nodes with low computational and storage complexities. Moreover, a scalable critic network utilizing multiple attention mechanisms is proposed. It is not only suitable for dynamic and changing network environments but also can more comprehensively and accurately evaluate local observation states, providing more accurate and efficient guidance for model optimization. Furthermore, MADC incorporates a double critic network architecture to mitigate potential overestimation issues during training, thereby ensuring the model’s robustness and reliability. Simulation results demonstrate that MADC outperforms existing RPL optimization algorithms in terms of energy efficiency, data transmission reliability, and adaptability.

Enhancing text classification through novel deep learning sequential attention fusion architecture

<span lang="EN-US">Text classification is a pivotal task within natural language processing (NLP), aimed at assigning semantic labels to text sequences. Traditional methods of text representation often fall short in capturing intricacies in contextual information, relying heavily on manual feature extraction. To overcome these limitations, this research work presents the sequential attention fusion architecture (SAFA) to enhance the features extraction. SAFA combines deep long sort-term memory (LSTM) and multi-head attention mechanism (MHAM). This model efficiently preserves data, even for longer phrases, while enhancing local attribute understanding. Additionally, we introduce a unique attention mechanism that optimizes data preservation, a crucial element in text classification. The paper also outlines a comprehensive framework, incorporating convolutional layers and pooling techniques, designed to improve feature representation and enhance classification accuracy. The model's effectiveness is demonstrated through 2-dimensional convolution processes and advanced pooling, significantly improving prediction accuracy. This research not only contributes to the development of more accurate text classification models but also underscores the growing importance of NLP techniques.</span>

Enhancing performance of end-to-end communication system using Attention Mechanism-based Sparse Autoencoder over Rayleigh fading channel

Deep learning has revolutionized communication systems by introducing innovative approaches to address channel impairments through end-to-end models. Autoencoders, a type of deep learning architecture, are adept at learning compact data representations. However, conventional autoencoders in end-to-end models can suffer from overfitting, which limits their effectiveness in noisy communication environments. To address this issue, we propose a Sparse Autoencoder-based (SAE) model that enforces sparsity and promotes the extraction of robust features. Despite its effectiveness, the SAE model may still lack the ability to focus on the most relevant features of the input data. To overcome this limitation, we further introduce an Attention Mechanism-based Sparse Autoencoder (ASA) model. This model integrates the feature extraction capabilities of a sparse autoencoder with an attention mechanism that selectively highlights informative features of the signal. Through simulations, we demonstrate that both proposed models significantly improve M-PSK and M-QAM communication system performance. When trained at 7 dB, both proposed models exhibit significant performance improvements at higher testing average SNRs. Our results show that the SAE model outperforms the conventional Maximum Likelihood Detection (MLD) model and baseline autoencoder systems but suffers from error floor issues. The SAE model suffers from an error floor at average SNRs beyond 16 dB for BPSK and 14 dB for higher-order modulation schemes. As the value of M increases, the performance gap between the MLD and the proposed SAE model narrows. The ASA model, however, effectively mitigates the error floor observed in the SAE model for all values of M and across all modulation schemes. This research highlights the benefits of integrating an attention mechanism with SAE, resulting in enhanced robustness and reliability in communication systems characterized by improved accuracy and reduced error rates.

Deep hashing and attention mechanism-based image retrieval of osteosarcoma scans for diagnosis of bone cancer

BackgroundDue to its intricate nature and substantial data size, microscopic image data of osteosarcoma often present a significant obstacle to the effectiveness of conventional image retrieval methods. Therefore, this study investigates a new approach for medical image retrieval using advanced deep hashing techniques and attention mechanisms to address these challenges more effectively. MethodThe proposed algorithm significantly improves osteosarcoma cell microscopic image retrieval efficiency and accuracy using deep hashing and attention mechanisms. Image preprocessing includes adaptive histogram equalization and dataset augmentation to enhance quality and diversity. Feature extraction employs the WRN-AM model to map high-dimensional features to a low-dimensional hash code space, improving retrieval efficiency. Finally, similarity matching via Hamming distance allows rapid and precise identification of similar images. ResultsThe study shows notable advancements: the WRN-AM model achieves 93.2% classification accuracy and 97.09% mAP using 64-bit hash codes. These findings underscore the technique’s effective performance in extracting and categorizing diverse microscopic cell data efficiently and reliably. ConclusionsThis innovative approach provides a robust solution for retrieving and classifying microscopic data of osteosarcoma cells and other cell types, speeding up clinical diagnosis and medical research. It facilitates quicker access and analysis of patient image data, enhancing diagnostic precision and treatment planning for healthcare professionals. Concurrently, it supports researchers in leveraging medical image data more efficiently, fostering progress and innovation in the medical field.

Artificial Intelligence in Sign Language Recognition: A Comprehensive Bibliometric and Visual Analysis

Sign language recognition (SLR) plays a crucial role in bridging the communication gap between individuals with hearing impairments and the auditory communities. This study explores the use of artificial intelligence (AI) in SLR through a comprehensive bibliometric analysis of 2,720 articles published from 1988 to 2024. Utilizing tools like VOSviewer and CiteSpace, the research uncovers the landscape of publication outputs, influential articles, leading authors, as well as the intellectual framework of current topics and emerging trends. The findings indicate that since the inception of SLR research in 1988, there has been a rapid expansion in the field, particularly from 2004 onwards. China and India lead in research productivity. Keyword and co-citation analyses highlight that Hidden Markov Model, Kinect, and Deep Learning have been focal points at various stages of SLR development, while transfer learning, Bidirectional Long Short-Term Memory, attention mechanisms, and Transformer models represent recent emerging trends. This research offers valuable insights for scholars and practitioners interested in AI-based SLR.

Latent Semantic and Disentangled Attention.

Sequential learning using transformer has achieved state-of-the-art performance in natural language tasks and many others. The key to this success is the multi-head self attention which encodes and gathers the features from individual tokens of an input sequence. The mapping or decoding is performed to produce an output sequence via cross attention. There are threefold weaknesses by using such an attention framework. First, since the attention would mix up the features of different tokens in input and output sequences, it is likely that redundant information exists in sequence data representation. Second, the patterns of attention weights among different heads tend to be similar. The model capacity is bounded. Third, the robustness in an encoder-decoder network against the model uncertainty is disregarded. To handle these weaknesses, this paper presents a Bayesian semantic and disentangled mask attention to learn latent disentanglement in multi-head attention where the redundant features in transformer are compensated with the latent topic information. The attention weights are filtered by a mask which is optimized through semantic clustering. This attention mechanism is implemented according to Bayesian learning for clustered disentanglement. The experiments on machine translation and speech recognition show the merit of Bayesian clustered disentanglement for mask attention.

Improving Long-Term Flood Forecasting Accuracy Using Ensemble Deep Learning Models and an Attention Mechanism

Improving Long-Term Flood Forecasting Accuracy Using Ensemble Deep Learning Models and an Attention Mechanism

Lateralized Alpha Oscillatory Activity in the Inferior Parietal Lobule to the Right Hemisphere during Left-Side Visual Stimulation

Understanding the neural mechanisms underlying spatial attention is crucial for unraveling the pathogenesis of unilateral spatial neglect (USN). However, the neural link between spatial attention and USN remains unclear. Thus, the neural mechanisms of spatial attention in the left and right hemispheres were compared. Twenty healthy volunteers participated in a hand mental rotation task in which they determined whether images depicted left or right hands. The hand images were randomly displayed in the upper, lower, left, and right directions, centered around a fixation point. The laterality index for the alpha oscillatory activity was determined to assess the lateralization of neural activity during visual stimulation. Our results revealed a significant shift in alpha oscillatory neural activity in the inferior parietal lobule (IPL) towards the right hemisphere when visual stimulation occurred on the left side. In contrast, no significant oscillatory shift in the alpha band towards the left hemisphere was observed in the IPL when the visual stimulus was presented on the right side. These findings indicate that the spatial attention on the left side depends on oscillatory alpha activity in the right IPL, whereas that on the right side doesn't depend on either hemispheric alpha activity. These results provide valuable insights into the neural mechanisms of hemispatial neglect.

Deep Learning for Real-Time P-Wave Detection: A Case Study in Indonesia's Earthquake Early Warning System

Detecting seismic events in real-time for prompt alerts and responses is a challenging task that requires accurately capturing P-wave arrivals. This task becomes even more challenging in regions like Indonesia, where widely spaced seismic stations exist. The wide station spacing makes associating the seismic signals with specific even more difficult. This paper proposes a novel deep learning-based model with three convolutional layers, enriched with dual attention mechanisms—Squeeze, Excitation, and Transformer Encoder (CNN-SE-T) —to refine feature extraction and improve detection sensitivity. We have integrated several post-processing techniques to further bolster the model's robustness against noise. We conducted comprehensive evaluations of our method using three diverse datasets: local earthquake data from East Java, the publicly available Seismic Waveform Data (STEAD), and a continuous waveform dataset spanning 12 hours from multiple Indonesian seismic stations. The performance of the CNN-SE-T P-wave detection model yielded exceptionally high F1 scores of 99.10% for East Java, 92.64% for STEAD, and 80% for the 12-hour continuous waveforms across Indonesia's network, demonstrating the model's effectiveness and potential for real-world application in earthquake early warning systems.

Deep Learning Based Method for Dynamic Tracking of Bubbles in Microfluidic Gas-Driven Water Experiments

Using microfluidic chips for gas displacement experiments is an important approach for studying CO2 displacement mechanisms and gas-liquid exchange processes. Analyzing the micro-scale distribution of gas and water during gas displacement enables the optimization of injection strategies, offering valuable guidance for enhancing oil and gas recovery rates. Traditional image processing techniques often face challenges in effectively handle the complex gas-liquid flow dynamics and irregular bubble shapes encountered in gas displacement experiments. To address this challenge, this paper proposes a gas bubble detection and tracking method for gas-driven water microfluidic experiments based on domain adaptation in deep learning and improved YOLOv8. Using the CycleGAN style transfer network to generate numerous simulated bubble training samples significantly reduces the manual labeling workload for bubbles. Additionally, the SE attention mechanism is integrated into the YOLOv8 backbone network to enhance the feature extraction capability of bubbles, and the Fast-PConv Head structure is incorporated into the detection head to enhance detection efficiency. Experimental results demonstrate that using just 20 annotated experimental images, the bubble detection accuracy of the proposed method can reach 94%. Lastly, DeepSORT is employed to track bubbles and visualize the dynamic trends of bubble changes in the form of visual charts. The deep learning approach proposed in this paper provides a new perspective for intelligent analysis of bubbles in microfluidic chip experiments, and future work aiming to its application to the complex multiphase flow field in porous media.

Multi 2D-CNN-based model for short-term PV power forecast embedded with Laplacian Attention

Amid the bloom of Renewable energy (RE) integrated into the grid, an accurate Photovoltaic(PV) power forecast is considered to be a crucial task in maintaining the reliability and stability of the power systems since this technology strongly depends on various external factors, causing the fluctuation in the output power. However, the poor quality of input data, which is very common in practical circumstances owing to the low-cost measurement and data acquisition devices, poses an enormous challenge for the predictive model to deeply extract the spatial and temporal correlation of the input data. This study proposes a Multi Two-Dimensional Convolutional Neural Network (2D-CNN) for short-term PV power forecast embedded with Laplacian Attention mechanism. By viewing the input sequences in a 2D form, the input map is constructed, and the interconnected feature among variables can be captured by convolution operation. Moreover, with the multiple CNN layers working in parallel architecture, different representations hidden inside the input map can be detected, enabling the proposed model to bring out promising performance across forecast time-step without modifying its initial parameters. In order to reduce the decay impact of irrelevant variables existing inside the input data, the Laplacian Attention mechanism is employed. The Attention matrix is dynamically modified during the training process to produce an accurate attention matrix, which represents the correlation between variables. Therefore, the model is able to focus on informative features and ignore negative ones. The experiments conducted on two different datasets with opposite characteristics provide deep insights into the strength of the proposed model over the baseline model, which strongly demonstrates the efficiency of the proposed model, especially when dealing with datasets bearing tough characteristics.

Efficient Neural Collaborative Search for Pickup and Delivery Problems.

In this paper, we introduce Neural Collaborative Search (NCS), a novel learning-based framework for efficiently solving pickup and delivery problems (PDPs). NCS pioneers the collaboration between the latest prevalent neural construction and neural improvement models, establishing a collaborative framework where an improvement model iteratively refines solutions initiated by a construction model. Our NCS collaboratively trains the two models via reinforcement learning with an effective shared-critic mechanism. In addition, the construction model enhances the improvement model with high-quality initial solutions via curriculum learning, while the improvement model accelerates the convergence of the construction model through imitation learning. Besides the new framework design, we also propose the efficient Neural Neighborhood Search (N2S), an efficient improvement model employed within the NCS framework. N2S exploits a tailored Markov decision process formulation and two customized decoders for removing and then reinserting a pair of pickup-delivery nodes, thereby learning a ruin-repair search process for addressing the precedence constraints in PDPs efficiently. To balance the computation cost between encoders and decoders, N2S streamlines the existing encoder design through a light Synthesis Attention mechanism that allows the vanilla self-attention to synthesize various features regarding a route solution. Moreover, a diversity enhancement scheme is further leveraged to ameliorate the performance during the inference of N2S. Our NCS and N2S are both generic, and extensive experiments on two canonical PDP variants show that they can produce state-of-the-art results among existing neural methods. Remarkably, our NCS and N2S could surpass the well-known LKH3 solver especially on the more constrained PDP variant.

Graph Multi-Convolution and Attention Pooling for Graph Classification.

Many studies have achieved excellent performance in analyzing graph-structured data. However, learning graph-level representations for graph classification is still a challenging task. Existing graph classification methods usually pay less attention to the fusion of node features and ignore the effects of different-hop neighborhoods on nodes in the graph convolution process. Moreover, they discard some nodes directly during the graph pooling process, resulting in the loss of graph information. To tackle these issues, we propose a new Graph Multi-Convolution and Attention Pooling based graph classification method (GMCAP). Specifically, the designed Graph Multi-Convolution (GMConv) layer explicitly fuses node features learned from different perspectives. The proposed weight-based aggregation module combines the outputs of all GMConv layers, for adaptively exploiting the information over different-hop neighborhoods to generate informative node representations. Furthermore, the designed Local information and Global Attention based Pooling (LGAPool) utilizes the local information of a graph to select several important nodes and aggregates the information of unselected nodes to the selected ones by a global attention mechanism when reconstructing a pooled graph, thus effectively reducing the loss of graph information. Extensive experiments show that GMCAP outperforms the state-of-the-art methods on graph classification tasks, demonstrating that GMCAP can learn graph-level representations effectively.