Attention-based Model Research Articles

A resource-aware workload scheduling method for unbalanced GEMMs on GPUs

Abstract GEMM (General Matrix Multiplication) serves as a fundamental operator for deep learning computations. Especially in attention-based deep learning models, such as Bert, GPT, and SAM, the sizes of matrices involved in GEMMs exhibit an unbalanced distribution due to the variable input, resulting in the low utilization of hardware resources. To address the issue, this paper proposes inserting a novel GEMM processing layer into the deep learning inference stack and using an adaptive load balancing method to partition and schedule GEMM computation tasks. The method is implemented with hardware runtime resource information, such as the occupancy of computing units, etc. Experiment results show the remarkable performance of our method in unbalanced input GEMM scenarios, achieving an average performance improvement of 2.3x. The method also performs well in attention-based models (GPT-2 and SAM), achieving an average inference speed improvement of 1.1x. These findings highlight the effectiveness of resource-aware algorithm optimization, especially for computation task scheduling.

Interpretable multi-step hybrid deep learning model for karst spring discharge prediction: Integrating temporal fusion transformers with ensemble empirical mode decomposition

Karst groundwater is a critical freshwater resource for numerous regions worldwide. Monitoring and predicting karst spring discharge is essential for effective groundwater management and the preservation of karst ecosystems. However, the high heterogeneity and karstification pose significant challenges to physics-based models in providing robust predictions of karst spring discharge. In this study, an interpretable multi-step hybrid deep learning model called selective EEMD-TFT is proposed, which adaptively integrates temporal fusion transformers (TFT) with ensemble empirical mode decomposition (EEMD) for predicting karst spring discharge. The selective EEMD-TFT hybrid model leverages the strengths of both EEMD and TFT techniques to learn inherent patterns and temporal dynamics from nonlinear and nonstationary signals, eliminate redundant components, and emphasize useful characteristics of input variables, leading to the improvement of prediction performance and efficiency. It consists of two stages: in the first stage, the daily precipitation data is decomposed into multiple intrinsic mode functions using EEMD to extract valuable information from nonlinear and nonstationary signals. All decomposed components, temperature and categorical date features are then fed into the TFT model, which is an attention-based deep learning model that combines high-performance multi-horizon prediction and interpretable insights into temporal dynamics. The importance of input variables will be quantified and ranked. In the second stage, the decomposed precipitation components with high importance are selected to serve as the TFT model’s input features along with temperature and categorical date variables for the final prediction. Results indicate that the selective EEMD-TFT model outperforms other sequence-to-sequence deep learning models, such as LSTM and single TFT models, delivering reliable and robust prediction performance. Notably, it maintains more consistent prediction performance at longer forecast horizons compared to other sequence-to-sequence models, highlighting its capacity to learn complex patterns from the input data and efficiently extract valuable information for karst spring prediction. An interpretable analysis of the selective EEMD-TFT model is conducted to gain insights into relationships among various hydrological processes and analyze temporal patterns.

Deep attention for enhanced OCT image analysis in clinical retinal diagnosis

AbstractRetinal illnesses such as age-related macular degeneration (AMD) and diabetic maculopathy pose serious risks to vision in the developed world. The diagnosis and assessment of these disorders have undergone revolutionary change with the development of optical coherence tomography (OCT). This study proposes a novel method for improving clinical precision in retinal disease diagnosis by utilizing the strength of Attention-Based DenseNet, a deep learning architecture with attention processes. For model building and evaluation, a dataset of 84495 high-resolution OCT images divided into NORMAL, CNV, DME, and DRUSEN classes was used. Data augmentation techniques were employed to enhance the model's robustness. The Attention-Based DenseNet model achieved a validation accuracy of 0.9167 with a batch size of 32 and 50 training epochs. This discovery presents a promising route for more precise and speedy identification of retinal illnesses, ultimately enhancing patient care and outcomes in clinical settings by integrating cutting-edge technology with powerful neural network architectures.

Deep Learning-Based Integration of Morphology and Immunophenotype Predicts Molecular Genetic Subtypes of Multiple Myeloma

Abstract Multiple myeloma, the second most common hematologic malignancy, is characterized by recurrent molecular genetic variants which portend important prognostic and therapeutic implications. These variants, which include chromosomal aneuploidies, translocations, and other structural changes, are currently identified by performing karyotyping and fluorescence in situ hybridization analysis on bone marrow biopsy samples. Comprehensive variant testing for full characterization of multiple myeloma cases can take days to weeks, is costly, and is not available to all patients. Past studies have explored to what extent cellular morphologic information from bone marrow aspirates or immunophenotypic information from flow cytometry can predict these genetic variants via the use of machine learning algorithms. However, the integration of multiple hematopathology data modalities using machine learning towards improving the diagnosis and characterization of hematologic neoplasms has yet to be explored. To this end, we developed a deep learning-based algorithm which, for individual patients, integrates tens-of-thousands of cell images from Wright-stained bone marrow aspirate smears with hundreds-of-thousands of events from myeloma flow cytometry panels to predict the presence or absence of particular molecular genetic variants among 85 cases of multiple myeloma. Attention-based multi-instance learning models were developed utilizing convolutional neural networks to analyze cell image data and multi-layer perceptrons to analyze flow cytometry data. Models trained solely on cell image data showed variable performance in predicting genetic subtypes of multiple myeloma cases: t(11;14), AUROC = 0.795; 1q+, AUROC = 0.743; del(17p), AUROC = 0.520. Similar models trained solely on flow cytometry data showed fair performance for all subtypes: t(11;14), AUROC = 0.761; 1q+, AUROC = 0.761; del(17p), AUROC = 0.787. However, models which combine features among high-attention events from both cell image and flow cytometry data yield significantly improved performance: t(11;14), AUROC = 0.892; 1q+, AUROC = 0.789; del(17p), AUROC = 0.876. Similarly, models trained to predict established prognostic classifications of multiple myeloma cases performed best when cell image and flow cytometry data is integrated: cell images alone, AUROC = 0.721; flow cytometry alone, AUROC = 0.827; both data combined, AUROC = 0.839. Additional improvements in prognostic classification were obtained when utilizing a Kronecker product to more completely utilize pairwise interactions between cell image and flow cytometry features: AUROC = 0.864. This study is, to our knowledge, the first to utilize deep learning to integrate multiple hematopathology data modalities towards automated diagnosis or subclassification of hematologic neoplasms in individual patients. The results of this study demonstrate the feasibility of accurately predicting molecular genetic subtypes of multiple myeloma cases solely from cell morphology and flow cytometric information. With further refinement, these promising models could be applied in low resource settings as well as research settings to better understand the biological relationship between plasma cell genetics with morphology and immunophenotype.

Multi-View Soft Attention-Based Model for the Classification of Lung Cancer-Associated Disabilities.

Background: The detection of lung nodules at their early stages may significantly enhance the survival rate and prevent progression to severe disability caused by advanced lung cancer, but it often requires manual and laborious efforts for radiologists, with limited success. To alleviate it, we propose a Multi-View Soft Attention-Based Convolutional Neural Network (MVSA-CNN) model for multi-class lung nodular classifications in three stages (benign, primary, and metastatic). Methods: Initially, patches from each nodule are extracted into three different views, each fed to our model to classify the malignancy. A dataset, namely the Lung Image Database Consortium Image Database Resource Initiative (LIDC-IDRI), is used for training and testing. The 10-fold cross-validation approach was used on the database to assess the model's performance. Results: The experimental results suggest that MVSA-CNN outperforms other competing methods with 97.10% accuracy, 96.31% sensitivity, and 97.45% specificity. Conclusions: We hope the highly predictive performance of MVSA-CNN in lung nodule classification from lung Computed Tomography (CT) scans may facilitate more reliable diagnosis, thereby improving outcomes for individuals with disabilities who may experience disparities in healthcare access and quality.

Modular Co-attention Networks in Nepali Visual Question Answering Systems

Visual question answering (VQA) has been regarded as a challenging task requiring a perfect blend of computer vision and natural language processing. As no dataset was available to train such a model for the Nepali language, a new dataset was developed during the research by translating the VQAv2 dataset. Then the dataset consisting of 202,577 images and 886,560 questions was used to train an attention-based VQA model. The dataset consists of yes/no, counting, and other questions with primarily one-word answers. Modular Co-attention Network (MCAN) was applied to the visual features extracted using the Faster RCNN framework and question embeddings extracted using the Nepali GloVe model. After co-attending the visual and language features for a few cascaded MCAN layers, the features are fused to train the whole network. During evaluation, an overall accuracy of 69.87% was obtained with 81.09% accuracy in yes/no type questions. The results surpassed the performance of models developed for Hindi and Bengali languages. Overall, novel research has been done in the Nepali Language VQA domain paving the way for further advancements.

An attention-based hybrid model for spatial and temporal sentiment analysis of COVID-19 related tweets in the contiguous United States

ABSTRACT Understanding the sentiments of social media posts can help health authorities respond to disease outbreaks, through a proxy measure of fear, confidence, and community compliance. Sentiment analysis identifies a pattern of emotion through the written word and assigns a positive, neutral, or negative value to it. As of February 2023, there were 677.7 million confirmed cases of coronavirus disease (COVID-19) and more than 6.7 million confirmed deaths. In this paper, around 170,000 COVID-19-related tweets were collected between September 2020 and January 2021 in the contiguous United States. Data preprocessing and exploratory investigation were completed for analysis of the collected dataset. Further, a novel and unified architecture called attention-based one-dimensional convolution with bidirectional long short-term memory layers (CNN-BiLSTM-ATT) is proposed to classify people’s sentiments as positive, neutral, and negative based on COVID-19-related tweets. In the CNN-BiLSTM-ATT model, the CNN layer can extract the low-level semantic features from textual data, and the BiLSTM layer can extract both the previous and future contextual representations. The attention module can improve the information focus from the outputted layer of the BiLSTM. The proposed method can extract both the local phrase representations and the global feature of sentences. Numerical experiments were conducted on COVID-19-related tweets using the proposed method and other baseline models to compare their performances. Our experimental results demonstrate that the CNN-BiLSTM-ATT model achieves an average accuracy of 95.16% and a macro-average F1-score of 95.12%, which outperforms the baseline models.

Multidimensional dynamic attention for multivariate time series forecasting

Attention-based models have been very effective in identifying important lagged variables for multivariate time series (MTS) forecasting applications. However, current attention-based models only provide static weights and do not consider the dynamic nature of predictions for multistep predictions of heterogeneous MTS. To address these limitations, this paper proposes a novel multidimensional dynamic attention (MDA) model for computing lagged variable importance. It incorporates a dynamic representation learner unit and considers multiple attention calculations to account for prediction dynamics, temporal information, and variable relations. Extensive experiments with both synthetic and real-world data demonstrate the effectiveness of the MDA model. It outperforms existing methods in the literature for sequence-to-sequence prediction of heterogeneous MTS in most cases and accurately identifies important features. MDA demonstrates enhancements up to 33% with real-world datasets. These findings demonstrate that the MDA model is a promising approach to MTS forecasting. The proposed attention mechanism can be utilised for other tasks related to MTS analysis beyond just forecasting, potentially enhancing the performance and interpretability of various MTS applications. The code is publicly available on GitHub, promoting widespread adoption and further research in this field [ Note: The GitHub link will be shared based on the paper’s acceptance].

Predicting temporal evacuation travel time in staircases between adjacent floors of super high-rise buildings by artificial neural networks

Predicting temporal evacuation travel time in staircases between adjacent floors is crucial for dynamic evacuation guidance to improve vertical evacuation efficiency during emergencies in super high-rise buildings. However, the prediction methods used in previous studies mainly focus on the total evacuation time in buildings and cannot predict the temporal evacuation travel time in every single part of the total evacuation route. This study proposes a novel method to predict temporal evacuation travel time in staircases between adjacent floors of super high-rise buildings by using artificial neural networks. Firstly, evacuation simulation data of a 71-storey office building are analyzed for feature engineering and dataset building. Secondly, three types of artificial neural networks, including convolutional neural networks, RNN-based model (the traditional recurrent neural networks, long-short term memory networks, and bidirectional long-short term memory networks), and Attention-based model (Attention model and Transformer model) are extended to predict the temporal evacuation travel time in staircases between adjacent floors based on the simulation data. Finally, the prediction results by different artificial neural networks are compared. The results show that the artificial neural networks can provide accurate prediction values for the temporal evacuation travel time in staircases between adjacent floors of super high-rise buildings.

An Automated Intrusion Detection System in IoT system using Attention based Deep Bidirectional Sparse Auto Encoder model

Nowadays, the Internet of Things (IoT) is a smart network connected to the Internet for transmitting gathered data with verified protocols. Attackers frequently use communication protocol defects as the basis for their attacks. Better protection measures are required since attacks affect the reputations of service providers. Both machine learning (ML) and deep learning (DL) methods have been developed in a number of research works to detect network intrusions. However, the system's security is limited by the rising number of new threats. Critical problems in IoT platforms, cyber-physical systems, wireless networks, and fog computing are caused by such attacks. The development of various cyber-security attacks reinforces the need for a strong intrusion detection system (IDS) in the IoT platform. The proposed study introduced a robust deep-feature learning mechanism for automatically detecting network intruders in the IoT platform. Initially, input data are gathered from the given dataset. Pre-processing helps reduce any noise in the data and improves the data quality using cleaning, outlier removal, and min-max normalization. The proposed Attention-based Deep Bidirectional Sparse Auto Encoder (AD-BiSA) model is the most important feature retrieved using the attention-based deep Bi-LSTM model. The different IoT network threats are categorized using a sparse Autoencoder approach. The chaotic Seagull Optimization (CSGO) algorithm decreases the loss and enhances the weight in the proposed DL technique. The UNSW NB15_IDS and NSL-KDD datasets achieve accuracy rates of 99.71% and 98.97%, respectively, for the proposed technique. The proposed method achieves better performance than existing approaches.

BrainQCNet: A Deep Learning attention-based model for the automated detection of artifacts in brain structural MRI scans

Abstract Analyses of structural MRI (sMRI) data depend on robust upstream data quality control (QC). It is also crucial that researchers seek to retain maximal amounts of data to ensure reproducible, generalizable models and to avoid wasted effort, including that of participants. The time-consuming and difficult task of manual QC evaluation has prompted the development of tools for the automatic assessment of brain sMRI scans. Existing tools have proved particularly valuable in this age of Big Data; as datasets continue to grow, reducing execution time for QC evaluation will be of considerable benefit. The development of Deep Learning (DL) models for artifact detection in structural MRI scans offers a promising avenue toward fast, accurate QC evaluation. In this study, we trained an interpretable Deep Learning model, ProtoPNet, to classify minimally preprocessed 2D slices of scans that had been manually annotated with a refined quality assessment (ABIDE 1; n = 980 scans). To evaluate the best model, we applied it to 2141 ABCD T1-weighted MRI scans for which gold-standard manual QC annotations were available. We obtained excellent accuracy: 82.4% for good quality scans (Pass), 91.4% for medium to low quality scans (Fail). Further validation using 799 T1w MRI scans from ABIDE 2 and 750 T1w MRI scans from ADHD-200 confirmed the reliability of our model. Accuracy was comparable to or exceeded that of existing ML models, with fast processing and prediction time (1 minute per scan, GPU machine, CUDA-compatible). Our attention model also performs better than traditional DL (i.e., convolutional neural network models) in detecting poor quality scans. To facilitate faster and more accurate QC prediction for the neuroimaging community, we have shared the model that returned the most reliable global quality scores as a BIDS-app (https://github.com/garciaml/BrainQCNet).

A Bi-LSTM and Attention Based Approach for Developing an Assamese AI Chatbot

Objectives: The main objective of this study is to develop, evaluate, and analyses a Bi-LSTM and attention-based model and decide whether it is the best deep learning hybrid model for developing the Assamese AI Conversational Agent. Methods: A dataset for the Assamese language was created for this research. A bi-LSTM and attention-based model was developed and trained with the dataset. We have compared the accuracy and performance of the proposed model with those of other deep learning models. The bag of words technique was used for feature extraction of text. For natural language processing, a custom stemmer, stop word functions and root word and stop word database were developed. The performance of the model is measured using metrics such as precision, recall, and F1-score. Findings: From the research study, we have found that the bi-LSTM and attention-based model performed better in terms of all the metrics compared to other models. This model was able to attain an accuracy of 89.99%. Novelty: This research is novel as it is the first attempt to develop an AI chatbot in Assamese based on a deep learning model. Assamese is a language that has not received much attention in the field of AI chatbots. This research has great significance for the society where Assamese is spoken, as it can provide a platform for communication, information, and education in their native language. The Assamese language is in its beginning stages on the digital platform and has limited research done in the field of AI chatbots. This research will contribute a lot to this research area and open a new door for researchers who are interested in this topic. It will also allow them to further explore, improve, and develop new ideas. Keywords: Deep Learning, Natural language processing, Chatbot, Assamese

A novel spatio-temporal attention-based bidirectional LSTM model for moisture content prediction in drying process

Accurate prediction of moisture content is significantly crucial for ensuring process stability and product quality in the cylinder drying process. However, the drying process exhibits complex spatio-temporal characteristics and strong interference, which make accurate prediction challenging for the deep learning approach. To address this issue, this article proposes a new spatio-temporal attention-based bidirectional long-short temporal memory network (STA-BiLSTM) model for accurate moisture content prediction. First, Maximum Relevance Minimum Redundancy (mRMR) is adopted to identify optimal features highly related to moisture content. Secondly, bidirectional long-short temporal memory (Bi-LSTM) network is utilized to extract temporal dependencies from the sequential data. Subsequently, spatio-temporal attention mechanisms are designed to adaptively focus on the most relevant features and timesteps, enhancing the model’s generalization ability. Finally, due to the harsh industrial environment, eXtreme Gradient Boosting (XGBoost) is adapted to improve generalizability and robustness. Extensive experiments on a real industrial dataset of the drying process demonstrate that the proposed STA-BiLSTM approach significantly outperforms alternative approaches for predicting moisture content, validating its effectiveness and superiority.

Correction to: Enhanced streamflow forecasting using attention-based neural network models: a comparative study in MOPEX basins

Correction to: Enhanced streamflow forecasting using attention-based neural network models: a comparative study in MOPEX basins

Quaternion attention-based JND model for macrophotography image watermarking

With the advancement of imaging technology, macrophotography images (MPIs) have become a popular research topic. Unlike natural images, MPIs often feature sharp foregrounds and blurred backgrounds, leading to distinct perceptual characteristics in estimation. As the number of MPIs grows rapidly, concerns over image quality and security increase. Robust watermarking techniques have been introduced to address these challenges. Just Noticeable Difference (JND) has been widely used in quantization-based watermarking frameworks. However, existing JND models handle each image area with a single-level perceptual attention. Visual attention in Quaternion Discrete Wavelet Transform (QDWT), which can reflect the Multi-level perceptual attention feature. Therefore, we propose a new method called Quaternion Attention-based Just Noticeable Difference model for MPIs Watemarking (QAJnd-MW) for watermarking MPIs. This method uses visual attention mechanisms, recognizing that the HVS is more sensitive to attention regions. We generate a masking effect in the JND field. The input image undergoes QDWT to explore multi-scale features. The multi-scale feature maps, with multi-directional luminance and multi-channel color, help create local and global attention maps, which are fused to form the final attention map. Specifically, considering both attention-based masking effects, the quaternion attention-guided JND model is designed for a robust MPI watermarking framework, aiming to further improve MPI security. Extensive experiments on the MP2020 and Blur Detection datasets show that the proposed model significantly improves robustness against JPEG compression attacks, reducing the bit error rate (BER) by up to 12%. Additionally, the model performs well against other attacks, such as those in online social networks, with lower BER than current state-of-the-art techniques.

A novel energy management of public charging stations using attention-based deep learning model

Electricity grids are complex systems that must balance the supply and demand of electricity in real-time. However, with the increasing adoption of electric vehicles (EVs), managing the grid’s stability has become more challenging. EV charging can cause spikes in electricity demand, leading to peak demand periods that strain the power grid’s infrastructure. With the help of load forecasting, this effect on the grid can be mitigated by predicting the charging demand of electric vehicles in advance. This will help utilities adjust their energy supply in real-time, ensuring enough energy is available to meet demand, and preventing overloads or under utilization of the grid. Moreover, the EV charging demand is influenced by a wide range of factors, including charging station locations, weather, and time of day. Therefore, advanced deep learning models are required to learn these complex relationships and identify patterns in EV charging demand, enabling utilities to make more informed decisions. In this research, an attention-based deep learning approach is proposed for more accurate prediction of EV load demand. This novel approach integrates attention mechanisms with traditional deep learning models like LSTM and GRU, allowing the model to dynamically weight the importance of different features and focus on the most relevant information. The outcomes are compared to conventional deep learning and machine learning algorithms. To test the efficacy of the proposed framework, an actual ACN dataset for public EV charging stations is utilized.

Continuously Learning Prediction Models for Smart Domestic Hot Water Management

Domestic hot water (DHW) consumption represents a significant portion of household energy usage, prompting the exploration of smart heat pump technology to efficiently meet DHW demands while minimizing energy waste. This paper proposes an innovative investigation of models using deep learning and continual learning algorithms to personalize DHW predictions of household occupants’ behavior. Such models, alongside a control system that decides when to heat, enable the development of a heat-pumped-based smart DHW production system, which can heat water only when needed and avoid energy loss due to the storage of hot water. Deep learning models, and attention-based models particularly, can be used to predict time series efficiently. However, they suffer from catastrophic forgetting, meaning that when they dynamically learn new patterns, older ones tend to be quickly forgotten. In this work, the continuous learning of DHW consumption prediction has been addressed by benchmarking proven continual learning methods on both real dwelling and synthetic DHW consumption data. Task-per-task analysis reveals, among the data from real dwellings that do not present explicit distribution changes, a gain compared to the non-evolutive model. Our experiment with synthetic data confirms that continual learning methods improve prediction performance.

Attention-Based Load Forecasting with Bidirectional Finetuning

Accurate load forecasting is essential for the efficient and reliable operation of power systems. Traditional models primarily utilize unidirectional data reading, capturing dependencies from past to future. This paper proposes a novel approach that enhances load forecasting accuracy by fine tuning an attention-based model with a bidirectional reading of time-series data. By incorporating both forward and backward temporal dependencies, the model gains a more comprehensive understanding of consumption patterns, leading to improved performance. We present a mathematical framework supporting this approach, demonstrating its potential to reduce forecasting errors and improve robustness. Experimental results on real-world load datasets indicate that our bidirectional model outperforms state-of-the-art conventional unidirectional models, providing a more reliable tool for short and medium-term load forecasting. This research highlights the importance of bidirectional context in time-series forecasting and its practical implications for grid stability, economic efficiency, and resource planning.

Siamese based few-shot learning lightweight transformer model for coagulant and disinfectant dosage simultaneous regulation

Deep learning (DL) has emerged as a transformative and promising approach to address inefficient resource management due to imprecise chemical dosing in conventional manual-dependent drinking water treatment. However, the scarcity of high-quality data and limited computing resources of embedded devices hinder the development and deployment of state-of-the-art (SOTA) DL models. Herein, a novel lightweight Transformer (LighT) model integrated with the Siamese structure was proposed for simultaneous regulation of coagulant dosage (FeCl3 and PAC) and disinfectant dosage (NaClO) and compared with other attention-based DL models. The results demonstrate that LighT showed robust few-shot learning effectiveness and outperformed other models in multi-output prediction (R2 values of 0.99 and RMSE of 0.20 mg/L, 0.16 mg/L and 0.45 mg/L, respectively). The LighT model also exhibited enhanced cost-effectiveness with a 79.74 % compression in model parameters and a 36.14 % decrease in inference time. The LighT-based chemical dosage prediction program was sustained for one month of operation, which can save chemical dosage by 15.81 % while maintaining satisfactory product water quality, indicating the potential of LighT for economizing chemical dosing strategy in drinking water treatment.

InfoFlowNet: A multi-head attention-based self-supervised learning model with surrogate approach for uncovering brain effective connectivity

Deciphering brain network topology can enhance the depth of neuroscientific knowledge and facilitate the development of neural engineering methods. Effective connectivity, which gauges the directional influences among brain regions, is pivotal for these studies. This research introduces InfoFlowNet, a novel self-supervised learning model designed to infer causal relationships in electroencephalogram (EEG) data, aiming at capturing the complex information flow among brain regions. Specifically, InfoFlowNet employs convolution operations and multi-head self-attention mechanisms with a masking feature, which ignores self-connections to focus on inter-regional dynamics. Additionally, a new causal magnitude estimation is introduced by assessing the impact of shuffled surrogate data on signal prediction to quantify causal influence. Experiments using synthetic data and real EEG datasets demonstrate that InfoFlowNet effectively uncovers time-varying causal relationships. Compared with the Granger causality model (GCM) and the temporal causal discovery framework (TCDF), InfoFlowNet shows superior sensitivity in detecting significant causal edges. For instance, in a psychomotor vigilance task, InfoFlowNet identified 8 out of 16 significant causal edges, significantly outperforming GCM and TCDF. Furthermore, InfoFlowNet successfully passes the whiteness test on the model's residuals, and advanced analyses illustrate the advantages of utilizing multi-head attention and masking mechanisms to capture effective connectivity. In sum, InfoFlowNet represents a significant advancement in the analysis of effective connectivity, offering a deeper understanding of brain network dynamics. The model's ability to discern complex causal interactions supports its potential application in broader neuroscientific research and applications.