Attention Model Research Articles

Hybrid LSTM–Attention and CNN Model for Enhanced Speech Emotion Recognition

Emotion recognition is crucial for enhancing human–machine interactions by establishing a foundation for AI systems that integrate cognitive and emotional understanding, bridging the gap between machine functions and human emotions. Even though deep learning algorithms are actively used in this field, the study of sequence modeling that accounts for the shifts in emotions over time has not been thoroughly explored. In this research, we present a comprehensive speech emotion-recognition framework that amalgamates the ZCR, RMS, and MFCC feature sets. Our approach employs both CNN and LSTM networks, complemented by an attention model, for enhanced emotion prediction. Specifically, the LSTM model addresses the challenges of long-term dependencies, enabling the system to factor in historical emotional experiences alongside current ones. We also incorporate the psychological “peak–end rule”, suggesting that preceding emotional states significantly influence the present emotion. The CNN plays a pivotal role in restructuring input dimensions, facilitating nuanced feature processing. We rigorously evaluated the proposed model utilizing two distinct datasets, namely TESS and RAVDESS. The empirical outcomes highlighted the model’s superior performance, with accuracy rates reaching 99.8% for TESS and 95.7% for RAVDESS. These results are a notable advancement, showcasing our system’s precision and innovative contributions to emotion recognition.

Object-based feedback attention in convolutional neural networks improves tumour detection in digital pathology

Human visual attention allows prior knowledge or expectations to influence visual processing, allocating limited computational resources to only that part of the image that are likely to behaviourally important. Here, we present an image recognition system based on biological vision that guides attention to more informative locations within a larger parent image, using a sequence of saccade-like motions. We demonstrate that at the end of the saccade sequence the system has an improved classification ability compared to the convolutional neural network (CNN) that represents the feedforward part of the model. Feedback activations highlight salient image features supporting the explainability of the classification. Our attention model deviates substantially from more common feedforward attention mechanisms, which linearly reweight part of the input. This model uses several passes of feedforward and backward activation, which interact non-linearly. We apply our feedback architecture to histopathology patch images, demonstrating a 3.5% improvement in accuracy (p < 0.001) when retrospectively processing 59,057 9-class patches from 689 colorectal cancer WSIs. In the saccade implementation, overall agreement between expert-labelled patches and model prediction reached 93.23% for tumour tissue, surpassing inter-pathologist agreement. Our method is adaptable to other areas of science which rely on the analysis of extremely large-scale images.

Temporal mechanisms underlying visual processing bias in peri-hand space.

The immediate space surrounding the hands has often been termed the peri-hand space (PHS), and is characterized by a smaller reaction time (RT), better detection, and enhanced accuracy for stimuli presented in this space, relative to those stimuli presented beyond this space. Such behavioral changes have been explained in terms of a biased allocation of cognitive resources such as perception, attention, and memory, for the efficient processing of information presented in the PHS. However, in two experiments, the current study shows that these cognitive biases seem to have an underlying temporal basis. The first experiment requires participants to perform a temporal bisection task, whereas the second experiment requires them to perform a verbal estimation task when stimuli are presented either near the hands or relatively far. Results from both experiments give evidence for slowing down of temporal mechanisms in the PHS - reflected in the form of temporal dilation for stimuli presented in the PHS relative to those presented further away. The slowing down of time in the PHS seems crucial in giving sufficient temporal allowance for the allocation of cognitive resources to prioritize the processing of information in the PHS. The findings are in line with the early anticipatory mechanisms associated with the PHS and seem to be driven by the switch/gate mechanism, and not the pacemaker component of the attentional gate model of time perception. Thus, the current study tries to integrate the theories of time perception with the peripersonal space literature.

A novel spatio-temporal attention mechanism model for car-following in autonomous driving

Car following is critical for the overall safety, efficiency, and smooth operation of autonomous vehicles in traffic. However, existing car-following model primarily focuses on local feature extraction, overlooking the spatial–temporal relationships between vehicles and key information within the time series. This limitation negatively impacts prediction accuracy and generalization capabilities. To address these issues, a novel Spatio-Temporal Car-Following model based on Graph Convolution Network and Attention mechanism is proposed to enhance the safety, efficiency, and comfort of autonomous driving systems. Firstly, the spatial–temporal structure of the car-following model is constructed using the GCN network. This approach efficiently captures the topological relationships between vehicles. Secondly, we interleave spatial and temporal blocks to extract feature information from both spatial and temporal dimensions, which allows us to perceive spatial interactions and temporal motion patterns of the car. Additionally, a self-attention module assigns different attention weights to the various neighbors of a node in the car-following model, which can facilitate the capture of diverse aspects of node relationships and enhance expressive power. Finally, a Multi-Layer Perceptron (MLP) layer predicts the future behaviors of following vehicles. The proposed car-following model (CF) was trained and evaluated using five real-world datasets: HighD, SPMD, Waymo, NGSIM, and Lyft. The results indicate that our approach surpasses current models in terms of Mean Squared Error (MSE) for spacing, while also maintaining a zero collision rate across every dataset. These findings indicate that the Spatio-Temporal Attention Model (GSTAM-CF) proposed in this paper enhances safety, efficiency, and operational comfort compared to existing models.

Short-term forecasting of dissolved oxygen based on spatial-temporal attention mechanism and kernel-based loss function

Dissolved Oxygen (DO) is a fundamental indicator of the health of aquatic ecosystems. This study introduces a novel spatial-temporal attention model combined with Kernel Mean Squared Error loss function (STA-KMSE) for forecasting short-term DO. An attention-based Convolutional Neural Network (A-CNN) is used to incorporate the influence of upstream stations in the spatial module, a factor often overlooked in prior studies. The temporal module captures the influence of both previous hours and days using attention-based Long Short-Term Memory (A-LSTM), offering a more comprehensive temporal context. Our approach addresses the issue of outlier sensitivity and non-linear error patterns by utilizing the KMSE loss function, which improves the model's robustness and more effectively captures intricate variations in DO levels compared to the standard Mean Squared Error (MSE) commonly used in water quality modelling. The model is trained on an extensive dataset consisting of various water quality parameters collected from twenty locations along the Ganga River, its tributary, and nallahs spanning from 1 April 2017 to 30 April 2021. In terms of RMSE, STA-MSE outperforms A-CNN by 62.669 % and A-LSTM by 7.419 %. Compared to STA-MSE, the STA-KMSE shows an average reduction in RMSE by 3.067 % for one-step, 3.197 % for two-step, and 3.428 % for three-step forecasting. Experiments indicate better performance for river locations than for nallahs. SHapley Additive exPlanation (SHAP) analysis highlights pH and temperature as crucial factors for estimating DO in rivers, while level, temperature, conductivity, total suspended solids, pH, chlorine, and chemical oxygen demand are significant for nallahs.

Time-frequency attention mechanism-based model for enhancing wind speed prediction accuracy

Wind energy constitutes a substantial renewable energy source, and its efficient exploitation depends significantly on precise wind speed forecasting. However, achieving accurate predictions in this area is challenging. Taking into account the consistent physical information contained within the time and frequency domain of a time signal, a novel time–frequency attention block and model is proposed. The pivotal component, denoted as the time–frequency attention (TFA) block, normalizes the time series of wind speed, utilizes the discrete cosine transform to obtain frequency domain information, and subsequently maps the time–frequency data to a high-dimensional space, employing an attention mechanism for adaptive fusion. After residual connections and multi-layer stacking, the predictive values are obtained by connecting with fully connected layers. Experimental results demonstrate that, compared to state-of-the-art models such as the Transformer and TimesNet, the TFA model exhibits favorable predictive performance. While hyperparameters do exert some influence on the predictive performance of the TFA model, its overall performance remains stable and consistently optimal. Regarding the operational mechanism of the model, visualization of the intermediate outputs confirms the TFA model’s exceptional ability to capture temporal and frequency domain fluctuations, as well as mean features

Federated inception-multi-head attention models for cyber-attacks detection

&lt;span lang="EN-US"&gt;With the proliferation of internet of things (IoT) devices, ensuring the security of these interconnected systems has become a critical concern. Cyberattacks targeting IoT devices pose significant threats to individuals and organizations due to the generation of vast amounts of data across many connected devices, which traditional centralized methods cannot solve. Federated learning (FL) could be a promising solution to mitigate privacy concerns associated with centralized approaches and address cybersecurity concerns. This paper uses FL and deep learning (DL) approaches to cybersecurity in IoT applications. The goal of cyber security is achieved by forming a federation of acquired and shared models at the head of the various participants. We use inception time and multi-head attention (CNN) algorithm based on FL to detect cyber-attacks and avoid data privacy leaks under two distribution modes, namely IID and Non-IID. In contrast, the FedAvg and FedMA algorithms aggregate local model updates. A global model is produced after several communication rounds between the IoT devices and the model parameter server. Cyber threats are simulated using edge-IIoT datasets. Experiment results show that the federated inception model's best global accuracy was 93, 91%, and 93, 49% using multi-head attention.&lt;/span&gt;

Abnormality-aware bone fracture detection and classification using the triple context attention model

&lt;span lang="EN-US"&gt;In this study, a novel approach is introduced for fracture detection in bone x-ray images, introducing the triple context attention model (TCAN) that combines concentrated extensive convolutional segments with an attention mechanism to enhance positional data. The TCAN model significantly improves fracture recognition accuracy while reducing model complexity. Leveraging a diverse dataset, consistently achieving high accuracy levels across various body parts. By addressing, mislabelling issues, and employing a visual attention network (VAN), to refine the model's performance. The TCAN model emerges as a robust, computationally efficient solution, offering a remarkable average accuracy of 97.86%. This study contributes valuable advancements to medical imaging and diagnostics, providing a highly effective tool for skeletal fracture detection.&lt;/span&gt;

Deep learning-based attention models for sarcasm detection in text

Deep learning-based attention models for sarcasm detection in text

Enhanced scene text recognition using deep learning based hybrid attention recognition network

&lt;span lang="EN-US"&gt;The technique of automatically recognizing and transforming text that is present in pictures or scenes into machine-readable text is known as scene text recognition. It facilitates applications like content extraction, translation, and text analysis in real-world visual data by enabling computers to comprehend and extract textual information from images, videos, or documents. Scene text recognition is essential for many applications, such as language translation and content extraction from photographs. The hybrid attention recognition network (HARN), unique technology presented in this research, is intended to greatly improve efficiency and accuracy of text recognition in complicated scene situations. HARN makes use of cutting-edge elements including alignment-free sequence-to-sequence (AFS) module, creative attention mechanisms, and hybrid architecture that blends attention models with convolutional neural networks (CNNs). Thanks to its novel attention processes, HARN is capable of comprehending wide range of scene text components by capturing both local and global context information. Through faster network convergence, shorter training times, and better utilization of computing resources, the suggested technique raises bar for state-of-the-art. HARN’s versatility makes it a good choice for range of scene text recognition applications, including multilingual text analysis and data extraction. Extensive tests are conducted to assess the effectiveness of HARN approach and demonstrate it is ability to greatly influence real-world applications where accurate and efficient text recognition is essential.&lt;/span&gt;

The Application and Effects of Augmented Reality in Geometry Learning

With the development of new technology, augmented reality (AR) technology is revolutionizing the educational landscape, particularly in geometry. This innovative approach deepens their comprehension of geometric principles and stimulates their motivation to learn. This paper aims to provide a comprehensive introduction to ARs application in the educational field, especially the topic of geometry, and evaluate its effectiveness in improving users in two aspects: motivation and academic results. The study applies the Attention, Relevance, Confidence, and Satisfaction (ARCS) Model to assess the impact of AR in education compared to traditional teaching. It finds that AR boosts student motivation, especially for those with lower academic performance. While AR enhances learning experiences, its effect on improving grades is not immediate. The method that is mainly utilized in this study is reviewing the literature, summarizing, and analyzing critical information from them. And it turns out that short-term grade improvements are not significantly observable, AR technology offers long-term benefits by enhancing engagement and conceptual comprehension through interactive and spatial visualizations.

Research on Non-Intrusive Load Disaggregation Technology Based on VMD–Nyströmformer–BiTCN

Non-intrusive load disaggregation is a technique that monitors the total electrical load of an entire building or household. It uses a single power metering device to measure the total load. Then, it employs algorithms to break it down into the individual usage of different electrical devices. To address issues in load disaggregation models such as long training times, feature interference caused by the activation of other loads, and accuracy deficiencies caused by behavioral interference from users’ electricity usage habits, this paper proposes a VMD–Nyströmformer–BiTCN network architecture. The variational mode decomposition (VMD) filters the raw power data, reducing errors caused by noise and enhancing the accuracy of decomposing the load. A deep learning network utilizes a modified attention model, Nyströmformer, to reduce feature entanglement and accuracy degradation caused by habitual behavior interference during load disaggregation, while ensuring precise accuracy and improving network operational speed. The training network uses a bidirectional temporal convolutional network (BiTCN) and incorporates a residual network to expand the receptive field, allowing it to receive longer load sequence data and acquire more effective load information, thereby improving the disaggregation effectiveness for target appliances.

Combination of a Rabbit Optimization Algorithm and a Deep-Learning-Based Convolutional Neural Network–Long Short-Term Memory–Attention Model for Arc Sag Prediction of Transmission Lines

Arc droop presents significant challenges in power system management due to its inherent complexity and dynamic nature. To address these challenges in predicting arc sag for transmission lines, this paper proposes an innovative time–series prediction model, AROA-CNN-LSTM-Attention(AROA-CLA). The model aims to enhance arc sag prediction by integrating a convolutional neural network (CNN), a long short-term memory network (LSTM), and an attention mechanism, while also utilizing, for the first time, the adaptive rabbit optimization algorithm (AROA) for CLA parameter tuning. This combination improves both the prediction performance and the generalization capability of the model. By effectively leveraging historical data and exhibiting superior time–series processing capabilities, the AROA-CLA model demonstrates excellent prediction accuracy and stability across different time scales. Experimental results show that, compared to traditional and other modern optimization models, AROA-CLA achieves significant improvements in RMSE, MAE, MedAE, and R2 metrics, particularly in reducing errors, accelerating convergence, and enhancing robustness. These findings confirm the effectiveness and applicability of the AROA-CLA model in arc droop prediction, offering novel approaches for transmission line monitoring and intelligent power system management.

Advanced Lung Disease Detection: CBAM-Augmented, Lightweight EfficientNetB2 with Visual Insights.

This paper presents a multichannel deep-learning method for detecting lung diseases using chest X-ray images. Using EfficientNetB0 through EfficientNetB7 pretrained models, the methodology offers improved performance in classifying COVID-19, viral pneumonia, and normal chest Xrays. The EfficientNetB2 model was customized by incorporating Squeeze-and-Excitation (SE) blocks and the Convolutional Block Attention Module (CBAM) to improve the model's attention mechanisms. Additional convolutional layers were added for improved feature extraction, and multiscale feature fusion was implemented to capture features at different scales. In this study, 99.3% of the unseen chest X-ray images were identified using the proposed model. It demonstrated superior performance, surpassing existing techniques and highlighting its robustness and generalizability on unseen data samples. Moreover, visualization techniques were used to inspect the intermediate layers of the model, providing deeper insights into its processing and interpretation of medical images. The proposed method offers healthcare radiologists a valuable tool for rapid and accurate point-of-care diagnoses.

Energy Consumption Prediction for Drilling Pumps Based on a Long Short-Term Memory Attention Method

In the context of carbon neutrality and emission reduction goals, energy consumption optimization in the oil and gas industry is crucial for reducing carbon emissions and improving energy efficiency. As a key component in drilling operations, optimizing the energy consumption of drilling pumps has significant potential for energy savings. However, due to the complex and variable geological conditions, diverse operational parameters, and inherent nonlinear relationships in the drilling process, accurately predicting energy consumption presents considerable challenges. This study proposes a novel Long Short-Term Memory Attention model for precise prediction of drilling pump energy consumption. By integrating Long Short-Term Memory (LSTM) networks with the Attention mechanism, the model effectively captures complex nonlinear relationships and long-term dependencies in energy consumption data. Comparative experiments with traditional LSTM and Convolutional Neural Network (CNN) models demonstrate that the LSTM-Attention model outperforms these models across multiple evaluation metrics, significantly reducing prediction errors and enhancing robustness and adaptability. The proposed model achieved Mean Absolute Error (MAE) values ranging from 5.19 to 10.20 and R2 values close to one (0.95 to 0.98) in four test scenarios, demonstrating excellent predictive performance under complex conditions. The high-precision prediction of drilling pump energy consumption based on this method can support energy optimization and provide guidance for field operations.

Dual attention model with reinforcement learning for classification of histology whole-slide images

Digital whole slide images (WSIs) are generally captured at microscopic resolution and encompass extensive spatial data (several billions of pixels per image). Directly feeding these images to deep learning models is computationally intractable due to memory constraints, while downsampling the WSIs risks incurring information loss. Alternatively, splitting the WSIs into smaller patches (or tiles) may result in a loss of important contextual information. In this paper, we propose a novel dual attention approach, consisting of two main components, both inspired by the visual examination process of a pathologist: The first soft attention model processes a low magnification view of the WSI to identify relevant regions of interest (ROIs), followed by a custom sampling method to extract diverse and spatially distinct image tiles from the selected ROIs. The second component, the hard attention classification model further extracts a sequence of multi-resolution glimpses from each tile for classification. Since hard attention is non-differentiable, we train this component using reinforcement learning to predict the location of the glimpses. This approach allows the model to focus on essential regions instead of processing the entire tile, thereby aligning with a pathologist’s way of diagnosis. The two components are trained in an end-to-end fashion using a joint loss function to demonstrate the efficacy of the model. The proposed model was evaluated on two WSI-level classification problems: Human epidermal growth factor receptor 2 (HER2) scoring on breast cancer histology images and prediction of Intact/Loss status of two Mismatch Repair (MMR) biomarkers from colorectal cancer histology images. We show that the proposed model achieves performance better than or comparable to the state-of-the-art methods while processing less than 10% of the WSI at the highest magnification and reducing the time required to infer the WSI-level label by more than 75%. The code is available at github.

A lightweight convolutional neural network super-resolution method to improve the quality of target image in space measurement tasks

With the rapid advancement of space technology, space target observation images have become an essential tool for the precise measurement and shape analysis of spacecraft. However, due to the challenging conditions of the space environment, these images often suffer from blurring and distortion, which hampers the effectiveness of spacecraft observation and measurement missions. Although recent progress has been made in super-resolution reconstruction techniques, the limited processing capacity of on-board equipment prevents the direct deployment of these high-complexity methods. In this paper, we propose an efficient and lightweight super-resolution reconstruction algorithm called the Pyramid Frequency-Aware Network for space target observation images. Specifically, we use a divide-and-conquer strategy to separately process low-frequency and high-frequency features, ensuring high-quality feature extraction while reducing the number of parameters. To further improve the model’s ability to capture edge and detailed texture information, we introduce a pyramidal wavelet decomposition and a multi-scale large separable kernel attention model. For high-frequency information, we design an enhanced fusion convolution block that facilitates multi-scale feature extraction and channel mixing. Furthermore, we have established a dataset of space target observation images, which can serve as a valuable reference for future studies on the reconstruction of such images. Extensive experimental results demonstrate that our Pyramid Frequency-Aware Network achieves an excellent balance between peak signal-to-noise ratio, structural similarity index, number of parameters, flops, and running time, both on public standard datasets and our self-built space target observation image dataset. Additionally, the network is lightweight enough to be deployed on resource-constrained equipment, such as satellites.

Remembering the blues: negative emotion during encoding improve memory recall in major depressive Disorder

ABSTRACT Substantial research indicates that individuals with Major Depressive Disorder (MDD) remember more negative information compared to neutral and positive information. This phenomenon is commonly attributed to attentional biases toward negative over neutral and positive information. A recent attentional resources model suggests that in MDD, negative cues not only capture attention, but also lead to deeper processing of subsequent information, irrespective of its content. This study aimed to replicate findings supporting this attentional resources model and go beyond it by investigating the effect of negative cues on encoding and retrieval processes. Forty-one participants with MDD and no comorbid diagnoses, and 42 healthy-controls completed the emotional recall task with negative or positive videos presented during encoding and retrieval stages of a neutral word-list memory test. During encoding, only the MDD group exhibited a difference between negative and positive videos, such that for negative videos memory recall was improved and for positive words it was reduced. Emotional videos had no effect when presented during retrieval. These results suggest that in MDD, encountering emotional cues not only biases retrieval processes toward recalling more negative content, but rather fundamentally alters the depth of information processing, while not leading to a broad-spectrum recruitment of cognitive resources.

Attention model of EEG signals based on reinforcement learning.

Applying convolutional neural networks to a large number of EEG signal samples is computationally expensive because the computational complexity is linearly proportional to the number of dimensions of the EEG signal. We propose a new Gated Recurrent Unit (GRU) network model based on reinforcement learning, which considers the implementation of attention mechanisms in Electroencephalogram (EEG) signal processing scenarios as a reinforcement learning problem. The model can adaptively select target regions or position sequences from inputs and effectively extract information from EEG signals of different resolutions at multiple scales. Just as convolutional neural networks benefit from translation invariance, our proposed network also has a certain degree of translation invariance, making its computational complexity independent of the EEG signal dimension, thus maintaining a lower learning cost. Although the introduction of reinforcement learning makes the model non differentiable, we use policy gradient methods to achieve end-to-end learning of the model. We evaluated our proposed model on publicly available EEG dataset (BCI Competition IV-2a). The proposed model outperforms the current state-of-the-art techniques in the BCI Competition IV- 2a dataset with an accuracy of 86.78 and 71.54% for the subject-dependent and subject-independent modes, respectively. In the field of EEG signal processing, attention models that combine reinforcement learning principles can focus on key features, automatically filter out noise and redundant data, and improve the accuracy of signal decoding.

The Development of Intercultural Competence for Teachers of English as a Foreign Language through an Instructional Design Project

Most teaching pedagogies in English as a Foreign Language (EFL) classes prioritize linguistic skills development over exploring how cultural factors shape language interpretation. To address the gap, this study developed and evaluated an instructional design project using the attention, relevance, confidence, and satisfaction (ARCS) motivational design model combined with critical pedagogy through computer-mediated communication (CMC) tools. The purpose of this research is to enhance EFL teachers’ intercultural competence. Multimedia resources were used to capture participants’ attention, and news articles were selected to connect to their lived experiences. Online forums were used to build their confidence, and intercultural activities provided satisfaction through practical engagement. Critical literacy pedagogy guided the design of questions to encourage EFL teachers to explore assumptions, challenge perspectives, and critically analyze language. Data from 16 EFL teachers’ questionnaires, online comments, and interviews indicated that task attractiveness and the online environment motivated EFL teachers to develop critical literacy skills. Future studies could explore additional strategies for integrating intercultural competence into EFL teaching and examine how cultural dynamics within groups influence online communication.