Attention Network Research Articles

Device placement using Laplacian PCA and graph attention networks

Abstract The exponential growth in data and parameters in modern neural networks has created the need to distribute these models across multiple devices for efficient training, resulting in the device placement problem. Existing graph encoding approaches for device placement suffer from low efficiency when searching for optimal parallel strategies, primarily due to suboptimal positional information retrieval. To address these challenges, we propose the Laplacian Principal Component Analysis-graph attention networks (LPCA-GAT) model. Firstly, we employ GAT to capture complex relationships between nodes and generate node encodings. Secondly, we leverage LPCA on the graph Laplacian matrix to extract crucial low-dimensional positional information. Finally, by integrating these two components, we obtain the final node encodings. This enhances the representation capability of nodes within the graph, enabling efficient device placement. The experimental results demonstrate that LPCA-GAT achieves superior device placement results, specifically accelerating execution and computation time by 13.24% and 96.38%, respectively, leading to significant improvements in both operational efficiency and performance.

A comparative analysis of face and object perception in 2D laboratory and virtual reality settings: insights from induced oscillatory responses.

In psychophysiological research, the use of Virtual Reality (VR) for stimulus presentation allows for the investigation of how perceptual processing adapts to varying degrees of realism. Previous time-domain studies have shown that perceptual processing involves modality-specific neural mechanisms, as evidenced by distinct stimulus-locked components. Analyzing induced oscillations across different frequency bands can provide further insights into neural processes that are not strictly phase-locked to stimulus onset. This study uses a simple perceptual paradigm presenting images of faces and cars on both a standard 2D monitor and in an immersive VR environment. To investigate potential modality-dependent differences in attention, cognitive load, and task-related post-movement processing, the induced alpha, theta and beta band responses are compared between the two modalities. No evidence was found for differences in stimulus-dependent attention or task-related post-movement processing between the 2D conditions and the realistic virtual conditions in electrode space, as posterior alpha suppression and re-synchronization of centro-parietal beta did not differ between conditions. However, source analysis revealed differences in the attention networks engaged during 2D and 3D perception. Midfrontal theta was significantly stronger in laboratory conditions, indicating higher cognitive load than in the VR environment. Exploratory analysis of posterior theta showed stronger responses in VR, possibly reflecting the processing of depth information provided only by the 3D material. In addition, the theta response seems to be generated by distinct neuronal sources under realistic virtual conditions indicating enhanced involvement of semantic information processing and social cognition.

Graph neural network-based anomaly detection for human cyber physical systems

ABSTRACT Human Cyber Physical Systems (HCPS) encompass humans, networks, and physical devices, characterized by intricate interactions and interdependencies. The complexity of HCPS networks makes them vulnerable to network issues and cyberattacks, potentially resulting in network paralysis and significantly impacting the reliable operation of HCPS. Therefore, precise anomaly detection is imperative for HCPS. To tackle this issue, we integrate HCPS with the Software Defined Networking (SDN) infrastructure layer and propose an anomaly detection strategy based on Graph Neural Networks (GNN). Our approach utilizes Graph Sample and Aggregates (GraphSAGE) to capture adjacent feature information of nodes, revealing hidden relationships within the graph. These embeddings are then combined with Graph Attention Networks (GAT), which calculate attention coefficients between a node and its neighboring nodes, representing the importance of neighboring nodes to the central node. Finally, a fully connected layer is employed to obtain anomaly node labels. To assess the performance of our method, we conduct experiments using real datasets and compare them with other advanced anomaly detection methods in terms of accuracy, precision, recall, F1 score, and area under the ROC curve (AUC). Our method accurately detects anomalous nodes, offering a viable solution for practical applications.

Knowledge Graph Completion for High-Speed Railway Turnout Switch Machine Maintenance Based on the Multi-Level KBGC Model

The incompleteness of the existing knowledge graphs in the railway domain creates information gaps, impacting their quality and effectiveness in the operation and maintenance of high-speed railway turnout switch machines. To address this, we propose a multi-layer model (KBGC) that combines KG-BERT, graph attention network (GAT), and Convolutional Embedding Network (ConvE) for knowledge graph completion in railway maintenance. KG-BERT fine-tunes a pre-trained BERT model to extract deep semantic features from entities and relationships, converting them into graph structures. GAT captures key structural relationships between nodes using an attention mechanism, producing enriched semantic and structural embeddings. Finally, ConvE reshapes and convolves these embeddings to learn complex entity interactions, enabling accurate link prediction. Extensive experiments on the HRTOM dataset, containing triplet data from high-speed railway turnout switch machines, demonstrate the model’s effectiveness, achieving an MRR of 50.8% and a Hits@10 of 60.7%. These findings show that the KBGC model significantly improves knowledge graph completion, aiding railway maintenance personnel in decision making and preventive maintenance, and providing new tools for railway maintenance applications.

Cognition and Ventral Attention Network Connectivity: Associations With Treatment Response in Posttraumatic Stress Disorder.

Posttraumatic stress disorder (PTSD) is a highly heterogeneous disorder, which makes it difficult to link clinical phenotypes with biomarkers to improve treatment outcomes. Findings from previous studies suggest that cognitive measures such as verbal memory or attention paired with within-ventral attention network (VAN) or salience network resting-state functional connectivity may predict treatment response among individuals with PTSD. In a sample comprising 20 individuals with PTSD and 10 healthy control group individuals, the investigators subtyped individuals by using both discriminant function analysis and standardized norms for a single measure of memory and neuropsychological batteries of memory, attention, and executive functioning; attempted to replicate previous findings of lower within-VAN connectivity among individuals with cognitive impairment; and explored whether within-VAN connectivity paired with cognitive impairment predicted treatment outcomes. PTSD patients with cognitive impairment (defined by using a discriminant function analysis with verbal memory performance) had greater within-VAN resting-state functional connectivity compared with control group individuals and cognitively intact PTSD patients at a level that fell short of statistical significance (F=3.41; df=2, 21; ηp2=0.237). The interaction between verbal memory performance and within-VAN connectivity also predicted treatment-related change in PTSD symptoms at a level that also fell short of statistical significance (β=-0.442). These findings somewhat support the clinical utility of identifying cognitive phenotypes within PTSD (by using discriminant function analysis and verbal memory performance) to predict treatment outcomes.

The distinct functional brain network and its association with psychotic symptom severity in men with methamphetamine-associated psychosis

BackgroundIndividuals using methamphetamine (METH) may experience psychosis, which usually requires aggressive treatment. Studies of the neural correlates of METH-associated psychosis (MAP) have focused predominantly on the default mode network (DMN) and cognitive control networks. We hypothesize that METH use alters global functional connections in resting-state brain networks and that certain cross-network connections could be associated with psychosis.MethodsWe recruited 24 healthy controls (CRL) and 54 men with METH use disorder (MUD) who were then divided into 25 without psychosis (MNP) and 29 with MAP. Psychotic symptom severity was assessed using the Positive and Negative Syndrome Scale (PANSS), evaluating (1) large-scale alterations in regional-wise resting-state functional connectivity (rsFC) across 11 brain networks and (2) associations between rsFC and psychotic symptom severity.ResultsThe MUD group exhibited greater rsFC between the salience network (SN)-DMN, and subcortical network (SCN)-DMN compared to the CRL group. The MAP group exhibited decreased rsFC in the sensory/somatomotor network (SMN)-dorsal attention network (DAN), SMN-ventral attention network (VAN), SMN-SN, and SMN-auditory network (AN), whereas the MNP group exhibited increased rsFC in the SMN-DMN and the frontoparietal network (FPN)-DMN compared to CRL. Additionally, the MAP group exhibited decreased rsFC strength between the SMN-DMN, SMN-AN, SMN-FPN, and DMN-VAN compared to the MNP group. Furthermore, across the entire MUD group, the PANSS-Positive subscale was negatively correlated with the DMN-FPN and FPN-SMN, while the PANSS-Negative subscale was negatively correlated with the DMN-AN and SMN-SMN.ConclusionMUD is associated with altered global functional connectivity. In addition, the MAP group exhibits a different brain functional network compared to the MNP group.

Multi-Source Fusion Deformation-Monitoring Accuracy Calibration Method Based on a Normal Distribution Transform–Convolutional Neural Network–Self Attention Network

Multi-Source Fusion Deformation-Monitoring Accuracy Calibration Method Based on a Normal Distribution Transform–Convolutional Neural Network–Self Attention Network

Multi-Label Image Recognition Based on Attention and Multi-Scale Dynamic Graph Convolutional Network

Multi-Label Image Recognition Based on Attention and Multi-Scale Dynamic Graph Convolutional Network

Deep Learning for Epileptic Seizure Detection Using a Causal-Spatio-Temporal Model Based on Transfer Entropy.

Drug-resistant epilepsy is frequent, persistent, and brings a heavy economic burden to patients and their families. Traditional epilepsy detection methods ignore the causal relationship of seizures and focus on a single time or spatial dimension, and the effect varies greatly in different patients. Therefore, it is necessary to research accurate automatic detection technology of epilepsy in different patients. We propose a causal-spatio-temporal graph attention network (CSTGAT), which uses transfer entropy (TE) to construct a causal graph between multiple channels, combining graph attention network (GAT) and bi-directional long short-term memory (BiLSTM) to capture temporal dynamic correlation and spatial topological structure information. The accuracy, specificity, and sensitivity of the SWEZ dataset were 97.24%, 97.92%, and 98.11%. The accuracy of the private dataset reached 98.55%. The effectiveness of each module was proven through ablation experiments and the impact of different network construction methods was compared. The experimental results indicate that the causal relationship network constructed by TE could accurately capture the information flow of epileptic seizures, and GAT and BiLSTM could capture spatiotemporal dynamic correlations. This model accurately captures causal relationships and spatiotemporal correlations on two datasets, and it overcomes the variability of epileptic seizures in different patients, which may contribute to clinical surgical planning.

A Novel Two-Channel Classification Approach Using Graph Attention Network with K-Nearest Neighbor

A Novel Two-Channel Classification Approach Using Graph Attention Network with K-Nearest Neighbor

Ensemble graph neural networks for fake news detection using user engagement and text features

In this digital world with massive information sharing, spreading misinformation is relatively easy, but its consequences are really enormous. Detecting fake news is a critical challenge in today's digital age, where misinformation can spread rapidly and influence public opinion. This paper proposes an ensemble model (EGNN) that leverages Graph Neural Networks (GNNs) and text features to enhance the accuracy of fake news detection. The model is designed to operate effectively even when labeled data is scarce by utilizing a substantial amount of unlabeled data. We employ multiple GNNs, including a standard GNN, a Graph Attention Network (GAT), and a Bidirectional GCN (BiGCN), to capture user engagement patterns and social context. Additionally, text embeddings are extracted using spaCy, BERT, and a combination of spaCy with user profile information. These embeddings are then classified using a neural network, and the final detection is performed using ensemble techniques such as Majority Voting, Weighted Average, and Stacking. To address class imbalance, we incorporate focal loss alongside the traditional binary cross-entropy loss. Experimental results demonstrate that our proposed EGNN model significantly improves the detection of fake news, providing a robust solution for this pervasive problem.

TRRHA: A two-stream re-parameterized refocusing hybrid attention network for synthesized view quality enhancement

In multi-view video systems, the decoded texture video and its corresponding depth video are utilized to synthesize virtual views from different perspectives using the depth-image-based rendering (DIBR) technology in 3D-high efficiency video coding (3D-HEVC). However, the distortion of the compressed multi-view video and the disocclusion problem in DIBR can easily cause obvious holes and cracks in the synthesized views, degrading the visual quality of the synthesized views. To address this problem, a novel two-stream re-parameterized refocusing hybrid attention (TRRHA) network is proposed to significantly improve the quality of synthesized views. Firstly, a global multi-scale residual information stream is applied to extract the global context information by using refocusing attention module (RAM), and the RAM can detect the contextual feature and adaptively learn channel and spatial attention feature to selectively focus on different areas. Secondly, a local feature pyramid attention information stream is used to fully capture complex local texture details by using re-parameterized refocusing attention module (RRAM). The RRAM can effectively capture multi-scale texture details with different receptive fields, and adaptively adjust channel and spatial weights to adapt to information transformation at different sizes and levels. Finally, an efficient feature fusion module is proposed to effectively fuse the extracted global and local information streams. Extensive experimental results show that the proposed TRRHA achieves significantly better performance than the state-of-the-art methods. The source code will be available at https://github.com/647-bei/TRRHA.

Plant lncRNA-miRNA Interaction Prediction Based on Counterfactual Heterogeneous Graph Attention Network.

Identifying interactions between long non-coding RNAs (lncRNAs) and microRNAs (miRNAs) provides a new perspective for understanding regulatory relationships in plant life processes. Recently, computational methods based on graph neural networks (GNNs) have been widely employed to predict lncRNA-miRNA interactions (LMIs), which compensate for the inadequacy of biological experiments. However, the low-semantic and noise of graph limit the performance of existing GNN-based methods. In this paper, we develop a novel Counterfactual Heterogeneous Graph Attention Network (CFHAN) to improve the robustness to against the noise and the prediction of plant LMIs. Firstly, we construct a real-world based lncRNA-miRNA (L-M) heterogeneous network. Secondly, CFHAN utilizes the node-level attention, the semantic-level attention, and the counterfactual links to enhance the node embeddings learning. Finally, these embeddings are used as inputs for Multilayer Perceptron (MLP) to predict the interactions between lncRNAs and miRNAs. Evaluating our method on a benchmark dataset of plant LMIs, CFHAN outperforms five state-of-the-art methods, and achieves an average AUC and average ACC of 0.9953 and 0.9733, respectively. This demonstrates CFHAN's ability to predict plant LMIs and exhibits promising cross-species prediction ability, offering valuable insights for experimental LMI researches.

Dual syntax aware graph attention networks with prompt for aspect-based sentiment analysis

Aspect-based sentiment analysis (ABSA) is a challenging task due to the presence of multiple aspect words with different sentiment polarities in a sentence. Recently, pre-trained language models like BERT have been widely used as context encoders in ABSA. Graph neural networks have also been employed to extract syntactic and semantic information from sentence parsing trees, resulting in superior results. However, dependency trees may establish irrelevant dependencies for sentences with irregular syntax and complex structures. Additionally, previous methods have not fully utilized recent developments in pre-trained language models. Therefore, we propose a Dual Syntax aware Graph attention networks with Prompt (DSGP) model to address these issues. Our model utilizes prompt templates to maximize the potential of pre-trained models and masked vector outputs of templates as supplementary aspect feature representations. We also leverage both dependency trees and constituent trees with graph attention networks to extract different types of syntactic information. The dependency tree captures syntactic correlation between words, while the constituent tree provides a high-level formation of the sentence. Finally, the output from the prompt and parsing trees is fused and fed into a standard classifier. Experimental results on four public datasets demonstrate the competitive performance of our model.

High-precision and lightweight small-target detection algorithm for low-cost edge intelligence

The proliferation of edge devices driven by advancements in Internet of Things (IoT) technology has intensified the challenge of achieving high-precision small target detection, as it demands extensive computational resources. This amplifies the conflict between the need for precise detection and the requirement for cost-efficiency across numerous edge devices. To solve this problem, this paper introduces an enhanced target detection algorithm, MSGD-YOLO, built upon YOLOv8. The Faster Implementation of CSP Bottleneck with 2 convolutions (C2f) module is enhanced through the integration of the Ghost module and dynamic convolution, resulting in a more lightweight architecture while enhancing feature generation. Additionally, Spatial Pyramid Pooling with Enhanced Local Attention Network (SPPELAN) replaces Spatial Pyramid Pooling Fast (SPPF) to expand the receptive field, optimizing multi-level feature aggregation for improved performance. Furthermore, a novel Multi-Scale Ghost Convolution (MSGConv) and Multi-Scale Generalized Feature Pyramid Network (MSGPFN) are introduced to enhance feature fusion and integrate multi-scale information. Finally, four optimized dynamic convolutional detection heads are employed to capture target features more accurately and improve small target detection precision. Evaluation on the VisDrone2019 dataset shows that compared with YOLOv8-n, MSGD-YOLO improves mAP@50 and mAP@50–95 by 14.1% and 11.2%, respectively. In addition, the model not only achieves a 16.1% reduction in parameters but also attains a processing speed of 24.6 Frames Per Second (FPS) on embedded devices, thereby fulfilling real-time detection requirements.

DLRA-Net: Deep Local Residual Attention Network with Contextual Refinement for Spectral Super-Resolution

DLRA-Net: Deep Local Residual Attention Network with Contextual Refinement for Spectral Super-Resolution

Aerodynamic Performance Prediction for Wide-Incidence Turbines Using Graph Neural Network Models Driven by Small-Scale Experimental Data

Abstract To rapidly and accurately predict turbine rotor blade losses within a wide range of incidences (−50–30 deg), graph neural networks (GNNs) are utilized to predict the aerodynamic parameters of two-dimensional turbine blades based on a small-scale experimental dataset. By comparing the backpropagation neural network (BPnn) model and computational fluid dynamics (CFD) results, it is demonstrated that GNNs with appropriately designed graph structures can accurately and quickly predict high-fidelity aerodynamic parameters based on limited experimental data. Unlike traditional data-driven modeling approaches, two innovative methods for improving blade profiles into graph structures are proposed. Relatively few input features are used to comprehensively and effectively represent the turbine blade profile by applying blade profile features in the GNN. The research findings indicate that due to the graph structure, which divides the turbine blade profile into five nodes based on five key points, coupled with high-fidelity experimental data and the unique weight updating mechanism of the graph attention network (GAT) model, the GAT-5 model exhibits the best performance among the studied models. Additionally, when assessing unknown validation blade profiles, the GAT-5 model maintains an absolute error below 6% at an incidence angle of 30 deg compared to the experimental results.

Locus coeruleus co-activation patterns at rest show higher state persistence in patients with dissociative seizures: A Pilot Study.

Dissociative seizures are paroxysmal disruptions of awareness and behavioral control in the context of affective arousal. Alterations in stress-related endocrine function have been demonstrated, but the timescale of dissociation suggests that the central locus coeruleus (LC) noradrenergic system is likely pivotal. Here, we investigate whether LC activation at rest is associated with altered brain network dynamics. A preliminary co-activation pattern (CAP) analysis of resting-state functional magnetic resonance imaging (fMRI) in 14 patients with dissociative seizures and 14 healthy controls was performed by using the LC as a seeding region. The red nucleus served as a control condition. Entry rates, durations, and state transition probabilities of identified CAPs were calculated. Analyses were corrected for demographic, technical, and clinical confounders including depression and anxiety. Three LC-related CAPs were identified, with the dominant two showing inverse activations and deactivations of the default mode network and the attention networks, respectively. Analysis of transition probabilities between and within the three CAPs revealed higher state persistence in patients compared to healthy controls for both CAP2LC (Cohen's d = -0.55; p = 0.01) and CAP3LC (Cohen's d = -0.57; p = 0.01). The control analysis using the red nucleus as a seed yielded similar CAPs, but no significant between-group differences in transition probabilities. Higher state persistence of LC-CAPs in patients with dissociative seizures generates the novel hypothesis that arousal-related impairments of network switching might be a candidate neural mechanism of dissociation. Dissociative seizures often arise during high affective arousal. The locus coeruleus is a brain structure involved in managing such acute arousal states. We investigated whether the activity of the locus coeruleus correlates with activity in other regions of the brain (which we refer to as "brain states"), and whether those brain states were different between patients with dissociative seizures and healthy controls. We found that patients tended to stay in certain locus coeruleus-dependent brain states instead of switching between them. This might be related to the loss of awareness and disruptions of brain functions ("dissociation") that patients experience during seizures.

Dietary patterns related to attention and physiological function in high-altitude migrants.

High altitude exposure negatively affects human attentional function. However, no studies have explored the regulation of attentional and physiological functions from a dietary perspective. A total of 116 Han Chinese students from Tibet University who were born and raised in a plain area and had been living in Tibet for > 2 years were recruited. All participants were male migrants. A food frequency questionnaire, complete blood count, and attention network test were performed on the participants. Pearson's correlation was applied to assess the reliability and validity of the food frequency questionnaire. Principal component analysis was utilized to extract dietary patterns. A linear mixed model was employed to account for individual differences. The results showed that the five main dietary patterns were coarse grain, alcohol, meat, protein, and snacking dietary patterns. Furthermore, individuals who adhered to the coarse grain dietary pattern and had high mean corpuscular hemoglobin showed better attentional performance. Individuals with high alcohol consumption and systemic immune-inflammation index levels exhibited worse attentional performance. These findings imply that high-altitude migrants should include more coarse grains in their daily diet and avoid excessive alcohol consumption to improve attention.

Deep learning models for the prediction of acute postoperative pain in PACU for video‐assisted thoracoscopic surgery

BackgroundPostoperative pain is a prevalent symptom experienced by patients undergoing surgical procedures. This study aims to develop deep learning algorithms for predicting acute postoperative pain using both essential patient details and real-time vital sign data during surgery.MethodsThrough a retrospective observational approach, we utilized Graph Attention Networks (GAT) and graph Transformer Networks (GTN) deep learning algorithms to construct the DoseFormer model while incorporating an attention mechanism. This model employed patient information and intraoperative vital signs obtained during Video-assisted thoracoscopic surgery (VATS) surgery to anticipate postoperative pain. By categorizing the static and dynamic data, the DoseFormer model performed binary classification to predict the likelihood of postoperative acute pain.ResultsA total of 1758 patients were initially included, with 1552 patients after data cleaning. These patients were then divided into training set (n = 931) and testing set (n = 621). In the testing set, the DoseFormer model exhibited significantly higher AUROC (0.98) compared to classical machine learning algorithms. Furthermore, the DoseFormer model displayed a significantly higher F1 value (0.85) in comparison to other classical machine learning algorithms. Notably, the attending anesthesiologists' F1 values (attending: 0.49, fellow: 0.43, Resident: 0.16) were significantly lower than those of the DoseFormer model in predicting acute postoperative pain.ConclusionsDeep learning model can predict postoperative acute pain events based on patients' basic information and intraoperative vital signs.