Deep Attention Research Articles

DIGITAL PEDAGOGY AND CULTURAL EVOLUTION: INSIGHTS FROM THE DIGITAL BOOK SERIES IN EARLY CHILDHOOD EDUCATION AND SPEECH

This research investigates the role of digital pedagogy in the context of cultural evolution, with a focus on the use of digital book series in early childhood education and its impact on speaking abilities. The research method used is a reflection on literature which includes theoretical studies of digital pedagogy, culture and language development in early childhood, as well as a qualitative approach to analysing relevant data. Through a literature perspective, researchers identify main patterns in the use of digital pedagogy in the context of early childhood education and understand how digital books can influence language development. Conduct qualitative analysis of relevant empirical studies to gain a deeper understanding of the practical experience of using digital book series in early childhood learning. The results show that digital pedagogy can play a significant role in the evolution of educational culture, with digital book series being a potential tool for improving the speaking abilities of young children. Factors such as interactivity, visualisation and engaging narratives can provide a more engaging and effective experience for children's learning. But challenges such as technology accessibility and concerns about screen use also need to be seriously considered. This research provides important insights for educators, researchers, and policymakers into how digital pedagogy can influence educational culture and language development in early childhood. The implications of this research highlight the importance of thoughtful integration of digital technologies in child-focused educational approaches, with deep attention, cultural relevance, and sustainability of their application in diverse contexts.

Identification of congenital valvular murmurs in young patients using deep learning-based attention transformers and phonocardiograms.

One in every four newborns suffers from congenital heart disease (CHD) that causes defects in the heart structure. The current gold-standard assessment technique, echocardiography, causes delays in the diagnosis owing to the need for experts who vary markedly in their ability to detect and interpret pathological patterns. Moreover, echo is still causing cost difficulties for low-and middle-income countries. Here, we developed a deep learning based attention transformer model to automate the detection of heart murmurs caused by CHD at an early stage of life using cost-effective and widely available phonocardiography (PCG). PCG recordings were obtained from 942 young patients at four major auscultation locations, including the aortic valve (AV), mitral valve (MV), pulmonary valve (PV), and tricuspid valve (TV), and they were annotated by experts as absent, present, or unknown murmurs. A transformation to wavelet features was performed to reduce the dimensionality before the deep learning stage for inferring the medical condition. The performance was validated through 10-fold cross-validation and yielded an average accuracy and sensitivity of 90.23% and 72.41%, respectively. The accuracy of discriminating between murmurs' absence and presence reached 76.10% when evaluated on unseen data. The model had accuracies of 70%, 88%, and 86% in predicting murmur presence in infants, children, and adolescents, respectively. The interpretation of the model revealed proper discrimination between the learned attributes, and AV channel was found important (score > 0.75) for the murmur absence predictions while MV and TV were more important for murmur presence predictions. The findings potentiate deep learning as a powerful front-line tool for inferring CHD status in PCG recordings leveraging early detection of heart anomalies in young people. It is suggested as a tool that can be used independently from high-cost machinery or expert assessment. With additional validation on external datasets, more insights on the generalizability of deep learning tools could be obtained before being implemented in real-world clinical settings.

Improving multi-step prediction performance of multi-channel QoT over optical backbone networks: deep echo state attention network

Improving multi-step prediction performance of multi-channel QoT over optical backbone networks: deep echo state attention network

Deep Attentive Interest Collaborative Filtering for Recommender Systems

Collaborative filtering (CF) is a pivotal building block in commercial recommender systems due to its strength and utility in user interest modeling. Recently, many researchers have turned to deep learning as a way to capture richer collaborative signals from user-item feature interactions. However, most deep-based methods only consider nonlinear, high-order interactions while ignoring the explicit collaborative signals in low-order interactions. They also typically ignore the quality of the user and item profiles. These are cornerstones in item recommendation that, we argue, must be considered for high-quality recommendations. Hence, we propose Deep Attentive Interest Collaborative Filtering (DAICF) to overcome these limitations. DAICF profiles users based on their interactive items, i.e., user neighborhood information. Similarly, item profiles are based on users who had interacted with it, i.e., item neighborhood information. Given a user's profile varies over different items, DAICF accurately models his attentive interests based on the specific target item. Low-order collaborative signals are captured by a shallow component, and high-order collaborative signals are captured by a deep component. These two complementary collaborative signals are then fused to provide rich recommendations that cut through today's information overload. By designing a personalized feature extraction method based on bilateral neighborhood information to solve the data sparsity problem in recommender systems, DAICF can dynamically distinguish the importance of a user's historical interaction items for predicting user preferences for a specific target item. A set of experiments against four real-world datasets validate that DAICF outperforms the most recent state-of-the-art recommendation algorithms and justifies the effectiveness and interpretability of our method.

DRANetSplicer: A Splice Site Prediction Model Based on Deep Residual Attention Networks.

The precise identification of splice sites is essential for unraveling the structure and function of genes, constituting a pivotal step in the gene annotation process. In this study, we developed a novel deep learning model, DRANetSplicer, that integrates residual learning and attention mechanisms for enhanced accuracy in capturing the intricate features of splice sites. We constructed multiple datasets using the most recent versions of genomic data from three different organisms, Oryza sativa japonica, Arabidopsis thaliana and Homo sapiens. This approach allows us to train models with a richer set of high-quality data. DRANetSplicer outperformed benchmark methods on donor and acceptor splice site datasets, achieving an average accuracy of (96.57%, 95.82%) across the three organisms. Comparative analyses with benchmark methods, including SpliceFinder, Splice2Deep, Deep Splicer, EnsembleSplice, and DNABERT, revealed DRANetSplicer's superior predictive performance, resulting in at least a (4.2%, 11.6%) relative reduction in average error rate. We utilized the DRANetSplicer model trained on O. sativa japonica data to predict splice sites in A. thaliana, achieving accuracies for donor and acceptor sites of (94.89%, 94.25%). These results indicate that DRANetSplicer possesses excellent cross-organism predictive capabilities, with its performance in cross-organism predictions even surpassing that of benchmark methods in non-cross-organism predictions. Cross-organism validation showcased DRANetSplicer's excellence in predicting splice sites across similar organisms, supporting its applicability in gene annotation for understudied organisms. We employed multiple methods to visualize the decision-making process of the model. The visualization results indicate that DRANetSplicer can learn and interpret well-known biological features, further validating its overall performance. Our study systematically examined and confirmed the predictive ability of DRANetSplicer from various levels and perspectives, indicating that its practical application in gene annotation is justified.

KG-TREAT: Pre-training for Treatment Effect Estimation by Synergizing Patient Data with Knowledge Graphs.

Treatment effect estimation (TEE) is the task of determining the impact of various treatments on patient outcomes. Current TEE methods fall short due to reliance on limited labeled data and challenges posed by sparse and high-dimensional observational patient data. To address the challenges, we introduce a novel pre-training and fine-tuning framework, KG-TREAT, which synergizes large-scale observational patient data with biomedical knowledge graphs (KGs) to enhance TEE. Unlike previous approaches, KG-TREAT constructs dual-focus KGs and integrates a deep bi-level attention synergy method for in-depth information fusion, enabling distinct encoding of treatment-covariate and outcome-covariate relationships. KG-TREAT also incorporates two pre-training tasks to ensure a thorough grounding and contextualization of patient data and KGs. Evaluation on four downstream TEE tasks shows KG-TREAT's superiority over existing methods, with an average improvement of 7% in Area under the ROC Curve (AUC) and 9% in Influence Function-based Precision of Estimating Heterogeneous Effects (IF-PEHE). The effectiveness of our estimated treatment effects is further affirmed by alignment with established randomized clinical trial findings.

A deep multi-branch attention model for histopathological breast cancer image classification

Since the impressive superior performance demonstrated by deep learning methods is widely used in histopathological image analysis and diagnosis, existing work cannot fully extract the information in the breast cancer images due to the limited high resolution of histopathological images. In this study, we construct a novel intermediate layer structure that fully extracts feature information and name it DMBANet, which can extract as much feature information as possible from the input image by up-dimensioning the intermediate convolutional layers to improve the performance of the network. Furthermore, we employ the depth-separable convolution method on the Spindle Structure by decoupling the intermediate convolutional layers and convolving them separately, to significantly reduce the number of parameters and computation of the Spindle Structure and improve the overall network operation speed. We also design the Spindle Structure as a multi-branch model and add different attention mechanisms to different branches. Spindle Structure can effectively improve the performance of the network, the branches with added attention can extract richer and more focused feature information, and the branch with residual connections can minimize the degradation phenomenon in our network and speed up network optimization. The comprehensive experiment shows the superior performance of DMBANet compared to the state-of-the-art method, achieving about 98% classification accuracy, which is better than existing methods. The code is available at https://github.com/Nagi-Dr/DMBANet-main.

Networked explicit hybrid predictive control of an industrial benchmark process

Hybrid systems have acquired deep attention in research as many systems in industries and laboratories manifest hybrid dynamic behaviors. Optimal control design for hybrid systems exhibits challenges for controlling the interacting continuous and discrete dynamics. A double-tank hybrid system (DTHS) is proposed in this paper with a mixed-integer actuation mechanism. The proposed DTHS aims to control the liquid level of the two tanks simultaneously by a networked hybrid predictive controller. Hybrid frameworks are provided to investigate the system through reachability analysis. An explicit hybrid model predictive control (EHMPC) has been developed to control the levels of the DTHS. Hardware-In-the-Loop (HIL) simulation is performed for assessing the performance of the EHMPC with a hardware real-time networked controller. The networked real-time controller is constructed and implemented using the Node.js frameworks. The simulation and realtime results are presented and discussed to evaluate the hybrid controller performance.

Simple and deep graph attention networks

Graph Attention Networks (GATs) and Graph Convolutional Neural Networks (GCNs) are two state-of-the-art architectures in Graph Neural Networks (GNNs). It is well known that both models suffer from performance degradation when more GNN layers are stacked, and many works have been devoted to address this problem. We notice that main research efforts in the line focus on the GCN models, and their techniques cannot well fit the GAT models due to the inherent difference between these two architectures. In GAT, the attention mechanism is limited as it ignores the overwhelming propagation from certain nodes as the number of layers increases. To sufficiently utilize the expressive power of GAT, we propose a new version of GAT named Layer-wise Self-adaptive GAT (LSGAT), which can effectively alleviate the oversmoothing issue in deep GAT and is strictly more expressive than GAT. We redesign the attention coefficients computation mechanism adaptively adjusted by layer depth, which considers both immediate neighboring and non-adjacent nodes from a global view. The experimental evaluation confirms that LSGAT consistently achieves better results on node classification tasks over relevant counterparts.

Adapting face recognition to the masked world: leveraging deep attention networks with a custom dataset

In the era of COVID-19, past face recognition algorithms' performance is downgraded due to the partial occlusion of face mask. A new Indian face image dataset has been proposed titled, Handcrafted Indian Face (HIF) dataset, addressing the issues, viz. variegated illumination, pose, and partial occlusion conditions. It bridges the gap between the performance of the DL models used, pre and post COVID-19 effect. A novel idea for choosing the train-test sample has been presented in the paper, which improves the accuracy on existing state of art DL models. In this paper, a new DL architecture has been proposed named the InceptBlock Enhanced Attention Fusion Network (IBEAFNet) which consists of the combination of ECBAM (Enhanced Convolution Block Attention Module) and InceptionV3 architecture. The proposed architecture's attention layer placement allows it to suppress less relevant mask regions of face, while emphasizing on significant fine and coarse level features with reduced complexity. IBEAFNet is trained and tested on two existing datasets, viz. Casia & Yale (including simulated masked images) and the proposed HIF dataset. The performance of IBEAFNet is compared with the results fetched by changing the attention layers in IBEAFNet with the blocks of SENet and CBAM. IBEAFNet outperformed the state-of-art models with the accuracy of 91.00%, 89.5%, and 93.00% on CASIA, Yale, and HIF dataset, respectively.

Accurate detection of coronavirus cases using deep learning with attention mechanism and genetic algorithm

The novel coronavirus disease has caused severe threats to the daily life and health of people all over the world. Hence, early detection and timely treatment of this disease are significant to prevent the coronavirus's spread and ensure more effective patient care. This work adopted an integrated framework comprising deep learning and attention mechanism to provide a more effective and reliable diagnosis. This framework consists of two convolution neural network (CNN), a bidirectional LSTM, two fully-connected layers (FCL), and an attention mechanism. The main aim of the proposed framework is to reveal a promising approach based on deep learning for early and timely detection of coronavirus disease. For greater accuracy, the framework's hyperparameters are tuned by means of a genetic algorithm. The effectiveness of the proposed framework has been examined utilizing a public dataset including 18 different blood findings from Albert Einstein Israelita Hospital in Sao Paulo, Brazil. Additionally, within the experimental studies, the proposed framework is subjected to comparison with the state-of-the-art techniques, evaluated across various metrics. Based on the derived consequences, the proposed framework has yielded enhancements in accuracy, recall, precision, and F1-score, registering approximate improvements of 1.27%, 4.07%, 3.20%, and 2.88%, respectively, as measured against the second-best rates.

Lightweight Patch-Wise Casformer for dynamic scene deblurring

In dynamic scenes, motion blur can often occur, which is non-uniform and can be difficult to remove. Recently, the Transformer has shown excellent performance in various image-related tasks such as classification, recognition, and segmentation. Using a Transformer-based backbone network has also shown potential advantages in image deblurring. However, the computational complexity of Transformers increases quadratically with spatial resolution, making it difficult to apply to high-resolution images. To address the above issue, we propose a cascade Transformer (Casformer) that consists of two key modules: Deep Separable Attention (DSA) and Double-Flow Gate (DFG). Our approach effectively reduces computational complexity while suppressing blurry information. Additionally, we discovered an inconsistency between training and testing images during the image restoration process. We addressed this issue by experimentally verifying an inference aggregation method (IAM) that independently predicts patches during inference to address the problem of imbalanced information distribution. Experimental results demonstrate that our design performs well on GoPro and other datasets, e.g. 29.20 dB PSNR on RealBlur-J, exceeding the previous state-of-the-art (SOTA) 0.14 dB.

Deep Learning-Based Attention Mechanism for Automatic Drowsiness Detection Using EEG Signal

Deep Learning-Based Attention Mechanism for Automatic Drowsiness Detection Using EEG Signal

Deep Feature Extraction and Attention Fusion for Multimodal Emotion Recognition

Deep Feature Extraction and Attention Fusion for Multimodal Emotion Recognition

MSMVAN: Multi Step Multi Variate Deep Attention Network for Renewable Energy Forecast

MSMVAN: Multi Step Multi Variate Deep Attention Network for Renewable Energy Forecast

Deep exponential excitation networks: toward stronger attention mechanism for weak fault diagnosis

Considering that large mechanical equipment often has various excitation sources, the signals generated by these excitation sources are often not simply added or multiplied together, but nonlinearly mixed, which exhibit complex non-stationary characteristics, making classical algorithms difficult to extract fault features. Especially when faults just occur, the fault symptom is often weak and submerged by noise, resulting in low diagnosing accuracy. Accordingly, this article develops a new deep attention method, namely deep exponential excitation networks, which improves diagnosing performance by amplifying important information, that is, the discriminative information between normal and weak fault conditions. The developed method introduces an exponential function into the attention mechanism, yielding a stronger focus on important features. Here, our “stronger” means that the attention mechanism has larger weights, and wider weight ranges, which are achieved via three paradigms of exponential excitation blocks. Meanwhile, the weights are automatically learned in deep networks, which can adaptively amplify the information related to early weak faults according to different severities of faults. Finally, extensive experiments on the marine engine datasets containing different noises demonstrate the effectiveness of the developed method.

A short-term power prediction method for wind farm cluster based on the fusion of multi-source spatiotemporal feature information

In recent years, the installed capacity of wind power has rapidly increased. And the wind power prediction is the foundation for ensuring large-scale wind power grid connection. The current short-term prediction methods of wind farm cluster (WFC) are difficult to sufficiently extract spatiotemporal features to achieve high-precision prediction. The article proposes a short-term power prediction method for WFC based on deep attention embedded graph clustering-TimesNet (DAEGC-TimesNet). Firstly, the directed power curves of WFC are proposed to analyze wind data. Then, a graph attention network is constructed based on geographic location and numerical weather prediction (NWP) information to guide clustering algorithms to achieve effective cluster partitioning. Finally, the input of model is constructed based on the feature information from various sub-clusters of WFC and the prediction result is obtained through the TimesNet. The method is applied to WFCs in Inner Mongolia, Jilin province and Yunnan province of China. The result shows that the RMSE reduces 0.0155 and the MAE reduces 0.0156 as well as the coefficient of determination and accuracy rate are highest comparing with comparative algorithms averagely based on above three WFCs. The simulation results are superior to the comparison algorithms, which makes greater contributions to ensure regional power supply.

Vessel trajectory prediction based on spatio-temporal graph convolutional network for complex and crowded sea areas

In order to improve the navigation ability of vessels and ensure the safety of maritime traffic, vessel trajectory prediction plays a crucial role in the intelligent navigation and collision avoidance. Especially for complex and crowded waters, autonomous vessels must have high situational awareness to detect other vessels, and predict their future trajectories and assess collision risks. In this work, a deep attention-aware spatio-temporal graph convolutional network based on AIS data (DAA-SGCN) is proposed to predict the future trajectories of vessels. It mainly includes three modules: motion information encoding of vessel trajectories, the spatio-temporal feature extraction module and the trajectory prediction module. The LSTM is used to extract the motion features of vessels, the spatio-temporal graph is constructed based on deep attention mechanism. The spatial social interaction features are extracted by ST-GCN, and the temporal correlation are extracted by RT-CNN, to obtain the high-level spatio-temporal features of vessel trajectories. Then, the feature embedding is fed into the trajectory prediction module to predict the future trajectories of vessels. On the basis of a large number of experiments, the prediction performance of the DAA-SGCN compared to the optimal baseline model is improved by about 74% and 69% in ADE and FDE metrics.

G-DGANet: Gated deep graph attention network with reinforcement learning for solving traveling salesman problem

Combinatorial optimization problem (COP) is an NP-hard problem for which finding an optimal solution is difficult, especially as the problem size increases. The Traveling Salesman Problem (TSP), one of the COPs that can be formulated over a graph, is a well-researched area in operations research and computer science. Deep Reinforcement Learning (DRL) is now regarded as a promising approach for solving TSP and other NP hard problems. In this paper, we propose a novel Gated Deep Graph Attention Network (G-DGANet) which builds upon the existing Graph Neural Network (GNN) to solve TSP. The proposed G-DGANet uses gating mechanism between subsequent layers of the network to extract representations of nodes deeper in the network without loss in performance. G-DGANet also designs a novel aggregator to construct global graph embeddings from different embedding preferences. In addition, to effectively learn underlying structure of a graph, G-DGANet integrates node and edge information of the graph while updating node representations in the message passing mechanism. We used proximal policy optimization (PPO) to train G-DGANet on randomly generated instances. We conducted an experiment on randomly generated instances and on real-world road network data generated from digital maps to verify the performance of G-DGANet. The findings from experiments demonstrate that G-DGANet outperforms most traditional heuristics and existing DRL approaches, with high generalization abilities from random instance training to random instance testing and real-world road network instance testing.

Super-resolution reconstruction of underwater polarized images with a fused attention mechanism.

The polarization imaging technique leverages the disparity between target and background polarization information to mitigate the impact of backward scattered light, thereby enhancing image quality. However, the imaging model of this method exhibits limitations in extracting inter-image features, resulting in less-than-optimal outcomes in turbid underwater environments. In recent years, machine learning methodologies, particularly neural networks, have gained traction. These networks, renowned for their superior fitting capabilities, can effectively extract information from multiple images. The incorporation of an attention mechanism significantly augments the capacity of neural networks to extract inter-image correlation attributes, thereby mitigating the constraints of polarization imaging methods to a certain degree. To enhance the efficacy of polarization imaging in complex underwater environments, this paper introduces a super-resolution network with an integrated attention mechanism, termed as SRGAN-DP. This network is a fusion of an enhanced SRGAN network and the high-performance deep pyramidal split attention (DPSA) module, also proposed in this paper. SRGAN-DP is employed to perform high-resolution reconstruction of the underwater polarimetric image dataset, constructed specifically for this study. A comparative analysis with existing algorithms demonstrates that our proposed algorithm not only produces superior images but also exhibits robust performance in real-world environments.