Attention Model Research Articles

TDFFM: Transformer and Deep Forest Fusion Model for Predicting Coronavirus 3C-Like Protease Cleavage Sites.

COVID-19, caused by the highly contagious SARS-CoV-2 virus, is distinguished by its positive-sense, single-stranded RNA genome. A thorough understanding of SARS-CoV-2 pathogenesis is crucial for halting its proliferation. Notably, the 3C-like protease of the coronavirus (denoted as 3CLpro) is instrumental in the viral replication process. Precise delineation of 3CLpro cleavage sites is imperative for elucidating the transmission dynamics of SARS-CoV-2. While machine learning tools have been deployed to identify potential 3CLpro cleavage sites, these existing methods often fall short in terms of accuracy. To improve the performances of these predictions, we propose a novel analytical framework, the Transformer and Deep Forest Fusion Model (TDFFM). Within TDFFM, we utilize the AAindex and the BLOSUM62 matrix to encode protein sequences. These encoded features are subsequently input into two distinct components: a Deep Forest, which is an effective decision tree ensemble methodology, and a Transformer equipped with a Multi-Level Attention Model (TMLAM). The integration of the attention mechanism allows our model to more accurately identify positive samples, thus enhancing the overall predictive performance. Evaluation on a test set demonstrates that our TDFFM achieves an accuracy of 0.955, an AUC of 0.980, and an F1-score of 0.367, substantiating the model's superior prediction capabilities.

Prediction of mine water quality by the Seq2Seq model based on attention mechanism

In recent years, as China's industrialization level has advanced, the issue of environmental pollution, particularly mine water pollution, has become increasingly severe. Water quality prediction is a fundamental aspect of water resource protection and a critical approach to addressing the water resource crisis. For improvement in water quality prediction, this research first analyzes the characteristics of mine water quality changes and provides a brief overview of water quality prediction. Subsequently, the Long Short-Term Memory and Sequence to Sequence (Seq2Seq) models, derived from Artificial Neural Networks, are introduced. The Seq2Seq water quality prediction model is implemented, incorporating the attention mechanism. Experimental validation confirms the effectiveness of the proposed model. The results demonstrate that the attention mechanism-based Seq2Seq model accurately predicts parameters such as pH value, Dissolved Oxygen, ammonia nitrogen, and Chemical Oxygen Demand, exhibiting a high degree of consistency with actual results. They play a vital role in assessing the health of the water and its ability to support aquatic life. The change of these indicators can reflect the degree and type of water pollution. Moreover, the Seq2Seq + attention model stands out with the lowest predicted Root Mean Square Error of 0.309. Notably, in comparison to the traditional Seq2Seq model, the incorporation of attention mechanisms in the Seq2Seq model results in a substantial 2.94 reduction in Mean Absolute Error. This research on the Seq2Seq water quality prediction model with attention mechanism provides valuable insights and references for future endeavors in water quality prediction.

Context enhanced multi-level attention model for session-based recommendation

Context enhanced multi-level attention model for session-based recommendation

GSTGM: Graph, spatial–temporal attention and generative based model for pedestrian multi-path prediction

Pedestrian trajectory prediction in urban environments has emerged as a critical research area with extensive applications across various domains. Accurate prediction of pedestrian trajectories is essential for the safe navigation of autonomous vehicles and robots in pedestrian-populated environments. Effective prediction models must capture both the spatial interactions among pedestrians and the temporal dependencies governing their movements. Existing research primarily focuses on forecasting a single trajectory per pedestrian, limiting its applicability in real-world scenarios characterised by diverse and unpredictable pedestrian behaviours. To address these challenges, this paper introduces the Graph Convolutional Network, Spatial–Temporal Attention, and Generative Model (GSTGM) for pedestrian trajectory prediction. GSTGM employs a spatiotemporal graph convolutional network to effectively capture complex interactions between pedestrians and their environment. Additionally, it integrates a spatial–temporal attention mechanism to prioritise relevant information during the prediction process. By incorporating a time-dependent prior within the latent space and utilising a computationally efficient generative model, GSTGM facilitates the generation of diverse and realistic future trajectories. The effectiveness of GSTGM is validated through experiments on real-world scenario datasets. Compared to the state-of-the-art models on benchmark datasets such as ETH/UCY, GSTGM demonstrates superior performance in accurately predicting multiple potential trajectories for individual pedestrians. This superiority is measured using metrics such as Final Displacement Error (FDE) and Average Displacement Error (ADE). Moreover, GSTGM achieves these results with significantly faster processing speeds.

Association between visual attention and parent-reported social problems in pediatric cancer survivors

ABSTRACT Social deficits in pediatric cancer survivors have been well documented and have been linked to neurocognitive late effects, particularly attention deficits. The purpose of this study was to examine the association between components of attention, both parent-report and a performance-based measure, with parent-reported social functioning in survivors of pediatric cancer. The study included data from outpatient neuropsychological evaluations of 76 survivors of pediatric cancer. The Integrated Visual and Auditory (IVA) continuous performance test was used to evaluate the components of attention with variables corresponding to Posner’s model of attention. Assessment of inattention and peer problems were derived from the Conner’s Parent Rating Scale, third edition. Parent-reported attention symptoms were significantly associated with worse auditory prudence (r = −0.382, p < .001) and visual consistency (r = −0.234, p = .046) and higher parent-reported peer problems (r = 0.302, p = .008). Auditory attention domains were not significantly predictive of peer problems (p > .05). When controlling for FSIQ and type of cancer, worse visual consistency (p = .029) and visual speed (p = .036) were significantly associated with more peer problems. Parent-reported inattention was associated with only some domains of performance-based attention, highlighting the importance of using multi-modal assessments of attention. This study also revealed an important relationship between visual attention and peer problems.

Characterizing the concentration of ethanol-water solutions by oblique-incidence reflectivity difference combined with deep learning algorithms

The detection of ethanol–water solution concentration plays an important role in industries, medical care, food and other aspects, which has attracted much attention. In this paper, a 632.8 nm laser combined with the oblique-incidence reflectivity difference (OIRD) method was used to obtain a signal linearly related to the solution concentration and containing the information of the dielectric constant of the solution. Combined with a variety of deep learning algorithms, ethanol–water solutions with a volume concentration of 0–95 % are detected. Among them, the prediction accuracy of the MLP, CNN, LSTM, CNN + BiLSTM + Attention models were 93.65 %, 96.54 %, 97.12 %, 99.23 %, respectively. The experimental results indicate that the OIRD method can achieve rapid, non-destructive, accurate and reliable detection of ethanol–water solutions.

Heat of the Moment: How Temperature Influences the Search and Purchase of Energy-Using Appliances

Decisions with significant and long-lasting consequences can be influenced by conditions at the moment of choice, such as weather. Using administrative data from an online retailer, we examine whether temperature and other weather variables affect the search and purchase of energy-using durables, namely, air conditioners (ACs) and dryers. We observe more sales of ACs on hot days and fewer sales of dryers on hot, windy days. We find no impact for appliances whose usefulness is not affected by the weather. For AC, weather-induced searches and purchases are in lower-efficiency energy classes. Product search data allow us to look into the process leading up to purchase. Prospective AC buyers search less intensively when the temperature is higher, and the opposite holds for buyers of dryers when temperature and wind speed increase. Models of memory and attention can explain these behavioral patterns. Understanding these dynamics is important for designing adaptation and mitigation policies, given the energy needs of cooling technologies and their increased demand and usefulness in a rapidly warming world.

Foreign object detection in urban rail transit based on deep differentiation segmentation neural network

With the increasing scale of urban rail transit, foreign object intrusion has become a significant operational safety hazard in urban rail transit. Although the laser-based automatic foreign object detection system has advantages such as long-distance detection and insensitivity to light changes, it has drawbacks such as large blind spots and low visualization. In response to the problems existing in laser detection systems, we proposed a novel video-based deep differentiation segmentation neural network for foreign object detection. Firstly, the foreign object detection is transformed into a binary classification problem, and the foreign object is determined as the image's foreground using image segmentation principles. Secondly, build a deep segmentation network based on deep convolution. Finally, perform morphological operations and threshold judgment on the foreground segmentation image to filter out the final detection results. To improve the detection effect, we reduced the impact of airflow disturbance by sampling and calculating the average background image. At the same time, the channel attention model and spatial attention model are added to the deep differentiation neural network. Collecting real data on subway platforms for experiments shows that the proposed method has a detection accuracy of 95.8 %, which is superior to traditional detection methods and recent image segmentation neural networks.

Detection of Cyber-Attacks in a Discrete Event System Based on Deep Learning

This paper addresses the problem of cyber-attack detection in a discrete event system by proposing a novel model. The model utilizes graph convolutional networks to extract spatial features from event sequences. Subsequently, it employs gated recurrent units to re-extract spatio-temporal features from these spatial features. The obtained spatio-temporal features are then fed into an attention model. This approach enables the model to learn the importance of different event sequences, ensuring that it is sufficiently general for identifying cyber-attacks, obviating the need to specify attack types. Compared with traditional methods that rely on synchronous product computations to synthesize diagnosers, our deep learning-based model circumvents state explosion problems. Our method facilitates real-time and efficient cyber-attack detection, eliminating the necessity to specifically identify system states or distinguish attack types, thereby significantly simplifying the diagnostic process. Additionally, we set an adjustable probability threshold to determine whether an event sequence has been compromised, allowing for customization to meet diverse requirements. Experimental results demonstrate that the proposed method performs well in cyber-attack detection, achieving over 99.9% accuracy at a 1% threshold and a weighted F1-score of 0.8126, validating its superior performance.

DaSAM: Disease and Spatial Attention Module-Based Explainable Model for Brain Tumor Detection

Brain tumors are the result of irregular development of cells. It is a major cause of adult demise worldwide. Several deaths can be avoided with early brain tumor detection. Magnetic resonance imaging (MRI) for earlier brain tumor diagnosis may improve the chance of survival for patients. The most common method of diagnosing brain tumors is MRI. The improved visibility of malignancies in MRI makes therapy easier. The diagnosis and treatment of brain cancers depend on their identification and treatment. Numerous deep learning models are proposed over the last decade including Alexnet, VGG, Inception, ResNet, DenseNet, etc. All these models are trained on a huge dataset, ImageNet. These general models have many parameters, which become irrelevant when implementing these models for a specific problem. This study uses a custom deep-learning model for the classification of brain MRIs. The proposed Disease and Spatial Attention Model (DaSAM) has two modules; (a) the Disease Attention Module (DAM), to distinguish between disease and non-disease regions of an image, and (b) the Spatial Attention Module (SAM), to extract important features. The experiments of the proposed model are conducted on two multi-class datasets that are publicly available, the Figshare and Kaggle datasets, where it achieves precision values of 99% and 96%, respectively. The proposed model is also tested using cross-dataset validation, where it achieved 85% accuracy when trained on the Figshare dataset and validated on the Kaggle dataset. The incorporation of DAM and SAM modules enabled the functionality of feature mapping, which proved to be useful for the highlighting of important features during the decision-making process of the model.

Let the Ultrasound Games Begin: Developing Knowledge and Clinical Skills With Gamification.

Point of Care Ultrasonography (POCUS) is customary practice in the clinical setting. Formal training ensures competent skills using POCUS but is lacking. Preparing nurse practitioners in POCUS with gamification is an innovative educational tool. The Attention, Relevance, Confidence, and Satisfaction (ARCS) model was used to implement POCUS. Learners participated in a bracket-style elimination tournament. Learners (N = 109) in the adult-gerontology acute care nurse practitioner track participated. Eighty-eight Bachelor of Science in Nursing (BSN) to Master of Science in Nursing and 21 BSN to Doctor of Nursing Practice learners. Extremely satisfied with the experience were 72.36%, 67.11% were somewhat comfortable, 10.53% were extremely comfortable, and 11.84% were somewhat uncomfortable using POCUS. Gamification for POCUS training is a practical tool to develop knowledge and comfort using ultrasound. Adult learners retain information in various modalities and gamification offers them a multimodal learning opportunity that increases competency. [J Nurs Educ. 2024;63(X):XXX-XXX.].

A Semi-Supervised Lie Detection Algorithm Based on Integrating Multiple Speech Emotional Features

When people tell lies, they often exhibit tension and emotional fluctuations, reflecting a complex psychological state. However, the scarcity of labeled data in datasets and the complexity of deception information pose significant challenges in extracting effective lie features, which severely restrict the accuracy of lie detection systems. To address this, this paper proposes a semi-supervised lie detection algorithm based on integrating multiple speech emotional features. Firstly, Long Short-Term Memory (LSTM) and Auto Encoder (AE) network process log Mel spectrogram features and acoustic statistical features, respectively, to capture the contextual links between similar features. Secondly, the joint attention model is used to learn the complementary relationship among different features to obtain feature representations with richer details. Lastly, the model combines the unsupervised loss Local Maximum Mean Discrepancy (LMMD) and supervised loss Jefferys multi-loss optimization to enhance the classification performance. Experimental results show that the algorithm proposed in this paper achieves better performance.

A Study of Improved Two-Stage Dual-Conv Coordinate Attention Model for Sound Event Detection and Localization.

Sound Event Detection and Localization (SELD) is a comprehensive task that aims to solve the subtasks of Sound Event Detection (SED) and Sound Source Localization (SSL) simultaneously. The task of SELD lies in the need to solve both sound recognition and spatial localization problems, and different categories of sound events may overlap in time and space, making it more difficult for the model to distinguish between different events occurring at the same time and to locate the sound source. In this study, the Dual-conv Coordinate Attention Module (DCAM) combines dual convolutional blocks and Coordinate Attention, and based on this, the network architecture based on the two-stage strategy is improved to form the SELD-oriented Two-Stage Dual-conv Coordinate Attention Model (TDCAM) for SELD. TDCAM draws on the concepts of Visual Geometry Group (VGG) networks and Coordinate Attention to effectively capture critical local information by focusing on the coordinate space information of the feature map and dealing with the relationship between the feature map channels to enhance the feature selection capability of the model. To address the limitation of a single-layer Bi-directional Gated Recurrent Unit (Bi-GRU) in the two-stage network in terms of timing processing, we add to the structure of the two-layer Bi-GRU and introduce the data enhancement techniques of the frequency mask and time mask to improve the modeling and generalization ability of the model for timing features. Through experimental validation on the TAU Spatial Sound Events 2019 development dataset, our approach significantly improves the performance of SELD compared to the two-stage network baseline model. Furthermore, the effectiveness of DCAM and the two-layer Bi-GRU structure is confirmed by performing ablation experiments.

An efficient colorectal cancer detection network using atrous convolution with coordinate attention transformer and histopathological images

The second most common type of malignant tumor worldwide is colorectal cancer. Histopathology image analysis offers crucial data for the clinical diagnosis of colorectal cancer. Currently, deep learning techniques are applied to enhance cancer classification and tumor localization in histopathological image analysis. Moreover, traditional deep learning techniques might loss integrated information in the image while evaluating thousands of patches recovered from whole slide images (WSIs). This research proposes a novel colorectal cancer detection network (CCDNet) that combines coordinate attention transformer with atrous convolution. CCDNet first denoises the input histopathological image using a Wiener based Midpoint weighted non-local means filter (WMW-NLM) for guaranteeing precise diagnoses and maintain image features. Also, a novel atrous convolution with coordinate attention transformer (AConvCAT) is introduced, which successfully combines the advantages of two networks to classify colorectal tissue at various scales by capturing local and global information. Further, coordinate attention model is integrated with a Cross-shaped window (CrSWin) transformer for capturing tiny changes in colorectal tissue from multiple angles. The proposed CCDNet achieved accuracy rates of 98.61% and 98.96%, on the colorectal histological image and NCT-CRC-HE-100 K datasets correspondingly. The comparison analysis demonstrates that the suggested framework performed better than the most advanced methods already in use. In hospitals, clinicians can use the proposed CCDNet to verify the diagnosis.

A hybrid neural combinatorial optimization framework assisted by automated algorithm design

In recent years, the application of Neural Combinatorial Optimization (NCO) techniques in Combinatorial Optimization (CO) has emerged as a popular and promising research direction. Currently, there are mainly two types of NCO, namely, the Constructive Neural Combinatorial Optimization (CNCO) and the Perturbative Neural Combinatorial Optimization (PNCO). The CNCO generally trains an encoder-decoder model via supervised learning to construct solutions from scratch. It exhibits high speed in construction process, however, it lacks the ability for sustained optimization due to the one-shot mapping, which bounds its potential for application. Instead, the PNCO generally trains neural network models via deep reinforcement learning (DRL) to intelligently select appropriate human-designed heuristics to improve existing solutions. It can achieve high-quality solutions but at the cost of high computational demand. To leverage the strengths of both approaches, we propose to hybrid the CNCO and PNCO by designing a hybrid framework comprising two stages, in which the CNCO is the first stage and the PNCO is the second. Specifically, in the first stage, we utilize the attention model to generate preliminary solutions for given CO instances. In the second stage, we employ DRL to intelligently select and combine appropriate algorithmic components from improvement pool, perturbation pool, and prediction pool to continuously optimize the obtained solutions. Experimental results on synthetic and real Capacitated Vehicle Routing Problems (CVRPs) and Traveling Salesman Problems(TSPs) demonstrate the effectiveness of the proposed hybrid framework with the assistance of automated algorithm design.

Wind power forecasting using a GRU attention model for efficient energy management systems

Wind power forecasting using a GRU attention model for efficient energy management systems

Top-Down Priors Disambiguate Target and Distractor Features in Simulated Covert Visual Search.

Several models of visual search consider visual attention as part of a perceptual inference process, in which top-down priors disambiguate bottom-up sensory information. Many of these models have focused on gaze behavior, but there are relatively fewer models of covert spatial attention, in which attention is directed to a peripheral location in visual space without a shift in gaze direction. Here, we propose a biologically plausible model of covert attention during visual search that helps to bridge the gap between Bayesian modeling and neurophysiological modeling by using (1) top-down priors over target features that are acquired through Hebbian learning, and (2) spatial resampling of modeled cortical receptive fields to enhance local spatial resolution of image representations for downstream target classification. By training a simple generative model using a Hebbian update rule, top-down priors for target features naturally emerge without the need for hand-tuned or predetermined priors. Furthermore, the implementation of covert spatial attention in our model is based on a known neurobiological mechanism, providing a plausible process through which Bayesian priors could locally enhance the spatial resolution of image representations. We validate this model during simulated visual search for handwritten digits among nondigit distractors, demonstrating that top-down priors improve accuracy for estimation of target location and classification, relative to bottom-up signals alone. Our results support previous reports in the literature that demonstrated beneficial effects of top-down priors on visual search performance, while extending this literature to incorporate known neural mechanisms of covert spatial attention.

Attentional dynamics of evidence accumulation explain why more numerate people make better decisions under risk

In decisions under risk, more numerate people are typically more likely to choose the option with the highest expected value (EV) than less numerate ones. Prior research indicates that this finding cannot be explained by differences in the reliance on explicit EV calculation. The current work uses the attentional Drift Diffusion Model as a unified computational framework to formalize three candidate mechanisms of pre-decisional information search and processing—namely, attention allocation, amount of deliberation, and distorted processing of value—which may differ between more and less numerate people and explain differences in decision quality. Computational modeling of an eye-tracking experiment on risky choice demonstrates that numeracy is linked to how people allocate their attention across the options, how much evidence they require before committing to a choice, and also how strongly they distort currently non-attended information during preference formation. Together, especially the latter two mechanisms largely mediate the effect of numeracy on decision quality. Overall, the current work disentangles and quantifies latent aspects of the dynamics of preference formation, explicates how their interplay may give rise to manifest differences in decision quality, and thereby provides a fully formalized, mechanistic explanation for the link between numeracy and decision quality in risky choice.

Multi-Task Multi-Objective Evolutionary Search Based on Deep Reinforcement Learning for Multi-Objective Vehicle Routing Problems with Time Windows

The vehicle routing problem with time windows (VRPTW) is a widely studied combinatorial optimization problem in supply chains and logistics within the last decade. Recent research has explored the potential of deep reinforcement learning (DRL) as a promising solution for the VRPTW. However, the challenge of addressing the VRPTW with many conflicting objectives (MOVRPTW) still remains for DRL. The MOVRPTW considers five conflicting objectives simultaneously: minimizing the number of vehicles required, the total travel distance, the travel time of the longest route, the total waiting time for early arrivals, and the total delay time for late arrivals. To tackle the MOVRPTW, this study introduces the MTMO/DRP-AT, a multi-task multi-objective evolutionary search algorithm, by making full use of both DRL and the multitasking mechanism. In the MTMO/DRL-AT, a two-objective MOVRPTW is constructed as an assisted task, with the objectives being to minimize the total travel distance and the travel time of the longest route. Both the main task and the assisted task are simultaneously solved in a multitasking scenario. Each task is decomposed into scalar optimization subproblems, which are then solved by an attention model trained using DRL. The outputs of these trained models serve as the initial solutions for the MTMO/DRL-AT. Subsequently, the proposed algorithm incorporates knowledge transfer and multiple local search operators to further enhance the quality of these promising solutions. The simulation results on real-world benchmarks highlight the superior performance of the MTMO/DRL-AT compared to several other algorithms in solving the MOVRPTW.

Employing a Computational Model to Reveal Stimulus-Driven Influences in Interfaces

It is challenging to design complex interfaces that support efficient visual searches. It requires understanding how numerous sources of information contribute to attentional selection, along with an understanding of the tools and tasks used to study visual search. This work is intended to examine visual salience, a pre-attentive process often overlooked in traditional design approaches. Although the importance of visual salience in task-driven searches has been debated, recent evidence suggests this stimulus-driven property contributes to the guidance of attention. Computational modeling has proven effective in predicting the influence of salience (e.g., N-SEEV and Attentional Priority models). With advances in this area of research, continuing examination of the applications and limitations of computational salience models is warranted. Therefore, the present work also examines the predictive power of computational models in displays like web pages and mobile interfaces. Emphasis is placed on the role salience should play in optimizing interface search efficiency.