Multi-head Attention Mechanism Research Articles

Research on Aspect-Level Sentiment Analysis Based on Adversarial Training and Dependency Parsing

Aspect-level sentiment analysis is used to predict the sentiment polarity of a specific aspect in a sentence. However, most current research cannot fully utilize semantic information, and the models lack robustness. Therefore, this article proposes a model for aspect-level sentiment analysis based on a combination of adversarial training and dependency syntax analysis. First, BERT is used to transform word vectors and construct adjacency matrices with dependency syntactic relationships to better extract semantic dependency relationships and features between sentence components. A multi-head attention mechanism is used to fuse the features of the two parts, simultaneously perform adversarial training on the BERT embedding layer to enhance model robustness, and, finally, to predict emotional polarity. The model was tested on the SemEval 2014 Task 4 dataset. The experimental results showed that, compared with the baseline model, the model achieved significant performance improvement after incorporating adversarial training and dependency syntax relationships.

Effective Video Summarization by Extracting Parameter-Free Motion Attention

Video summarization remains a challenging task despite increasing research efforts. Traditional methods focus solely on long-range temporal modeling of video frames, overlooking important local motion information that cannot be captured by frame-level video representations. In this article, we propose the Parameter-free Motion Attention Module (PMAM) to exploit the crucial motion clues potentially contained in adjacent video frames, using a multi-head attention architecture. The PMAM requires no additional training for model parameters, leading to an efficient and effective understanding of video dynamics. Moreover, we introduce the Multi-feature Motion Attention Network (MMAN), integrating the PMAM with local and global multi-head attention based on object-centric and scene-centric video representations. The synergistic combination of local motion information, extracted by the proposed PMAM, with long-range interactions modeled by the local and global multi-head attention mechanism, can significantly enhance the performance of video summarization. Extensive experimental results on the benchmark datasets, SumMe and TVSum, demonstrate that the proposed MMAN outperforms other state-of-the-art methods, resulting in remarkable performance gains.

A Transformer and LSTM-Based Approach for Blind Well Lithology Prediction

Petrographic prediction is crucial in identifying target areas and understanding reservoir lithology in oil and gas exploration. Traditional logging methods often rely on manual interpretation and experiential judgment, which can introduce subjectivity and constraints due to data quality and geological variability. To enhance the precision and efficacy of lithology prediction, this study employed a Savitzky–Golay filter with a symmetric window for anomaly data processing, coupled with a residual temporal convolutional network (ResTCN) model tasked with completing missing logging data segments. A comparative analysis against the support vector regression and random forest regression model revealed that the ResTCN achieves the smallest MAE, at 0.030, and the highest coefficient of determination, at 0.716, which are indicative of its proximity to the ground truth. These methodologies significantly enhance the quality of the training data. Subsequently, a Transformer–long short-term memory (T-LS) model was applied to identify and classify the lithology of unexplored wells. The input layer of the Transformer model follows an embedding-like principle for data preprocessing, while the encoding block encompasses multi-head attention, Add & Norm, and feedforward components, integrating the multi-head attention mechanism. The output layer interfaces with the LSTM layer through dropout. A performance evaluation of the T-LS model against established rocky prediction techniques such as logistic regression, k-nearest neighbor, and random forest demonstrated its superior identification and classification capabilities. Specifically, the T-LS model achieved a precision of 0.88 and a recall of 0.89 across nine distinct lithology features. A Shapley analysis of the T-LS model underscored the utility of amalgamating multiple logging data sources for lithology classification predictions. This advancement partially addresses the challenges associated with imprecise predictions and limited generalization abilities inherent in traditional machine learning and deep learning models applied to lithology identification, and it also helps to optimize oil and gas exploration and development strategies and improve the efficiency of resource extraction.

ADDGCN: A Novel Approach with Down-Sampling Dynamic Graph Convolution and Multi-Head Attention for Traffic Flow Forecasting

An essential component of autonomous transportation system management and decision-making is precise and real-time traffic flow forecast. Predicting future traffic conditionsis a difficult undertaking because of the intricate spatio-temporal relationships involved. Existing techniques often employ separate modules to model spatio-temporal features independently, thereby neglecting the temporally and spatially heterogeneous features among nodes. Simultaneously, many existing methods overlook the long-term relationships included in traffic data, subsequently impacting prediction accuracy. We introduce a novel method to traffic flow forecasting based on the combination of the feature-augmented down-sampling dynamic graph convolutional network and multi-head attention mechanism. Our method presents a feature augmentation mechanism to integrate traffic data features at different scales. The subsampled convolutional network enhances information interaction in spatio-temporal data, and the dynamic graph convolutional network utilizes the generated graph structure to better simulate the dynamic relationships between nodes, enhancing the model’s capacity for capturing spatial heterogeneity. Through the feature-enhanced subsampled dynamic graph convolutional network, the model can simultaneously capture spatio-temporal dependencies, and coupled with the process of multi-head temporal attention, it achieves long-term traffic flow forecasting. The findings demonstrate that the ADDGCN model demonstrates superior prediction capabilities on two real datasets (PEMS04 and PEMS08). Notably, for the PEMS04 dataset, compared to the best baseline, the performance of ADDGCN is improved by 2.46% in MAE and 2.90% in RMSE; for the PEMS08 dataset, compared to the best baseline, the ADDGCN performance is improved by 1.50% in RMSE, 3.46% in MAE, and 0.21% in MAPE, indicating our method’s superior performance.

Research on power generation prediction of hydropower in river basin based on multi-head attention graph convolutional neural network

For the power generation prediction of traditional hydropower stations, most of them only use time series prediction and neglect to study the spatial topological relationship of hydropower stations in the river basin, so that it is difficult to fully explore the characteristic relationship of space power stations. In this paper, a research method for power generation prediction of hydropower in river basin hydropower stations based on multi-head attention map convolutional neural network is proposed. This method establishes a first-level node neighborhood feature map based on the spatial geographic distribution information of hydropower stations in the basin, and uses the method of graph convolution to carry out node feature mining and feature learning, so as to transform the power generation capacity evaluation problem of the hydropower station in the basin into the node prediction problem in the graph, which is different from the global normalization rule. the multi-head attention mechanism introduced further improves the information aggregation quality of the graph node, and uses the historical temperature, power generation, electricity price, unit status and other data of each hydropower station in the basin for training. The reasoning results show that the proposed method achieves higher accuracy than other schemes, and the power prediction method is conducive to the formulation of power plans of hydropower stations in the basin, and can also play a positive role in guiding the site selection of hydropower plants.

A Novel Improved Variational Mode Decomposition-Temporal Convolutional Network-Gated Recurrent Unit with Multi-Head Attention Mechanism for Enhanced Photovoltaic Power Forecasting

Photovoltaic (PV) power forecasting plays a crucial role in optimizing renewable energy integration into the grid, necessitating accurate predictions to mitigate the inherent variability of solar energy generation. We propose a novel forecasting model that combines improved variational mode decomposition (IVMD) with the temporal convolutional network-gated recurrent unit (TCN-GRU) architecture, enriched with a multi-head attention mechanism. By focusing on four key environmental factors influencing PV output, the proposed IVMD-TCN-GRU framework targets a significant research gap in renewable energy forecasting methodologies. Initially, leveraging the sparrow search algorithm (SSA), we optimize the parameters of VMD, including the mode component K-value and penalty factor, based on the minimum envelope entropy principle. The optimized VMD then decomposes PV power, while the TCN-GRU model harnesses TCN’s proficiency in learning local temporal features and GRU’s capability in rapidly modeling sequence data, while leveraging multi-head attention to better utilize the global correlation information within sequence data. Through this design, the model adeptly captures the correlations within time series data, demonstrating superior performance in prediction tasks. Subsequently, the SSA is employed to optimize GRU parameters, and the decomposed PV power mode components and environmental feature attributes are inputted into the TCN-GRU neural network. This facilitates dynamic temporal modeling of multivariate feature sequences. Finally, the predicted values of each component are summed to realize PV power forecasting. Validation using real data from a PV station corroborates that the novel model demonstrates a substantial reduction in RMSE and MAE of up to 55.1% and 54.5%, respectively, particularly evident in instances of pronounced photovoltaic power fluctuations during inclement weather conditions. The proposed method exhibits marked improvements in accuracy compared to traditional PV power prediction methods, underscoring its significance in enhancing forecasting precision and ensuring the secure scheduling and stable operation of power systems.

Higher-order embedded learning for heterogeneous information networks and adaptive POI recommendation

Traditional POI recommendation results tend to be homogeneous due to data sparsity, and cannot adapt to the dynamic switching of user locations. Regarding the issues, we propose an adaptive Point of Interest (POI) recommendation algorithm that integrates the user social attributes and POI natural attributes. On the one hand, to fully utilize the complex node interactions and semantic information of heterogeneous networks, an effective higher-order embedding learning framework based on multi-head attention mechanism and multi-task shared features, called UI-HEHo, is presented. It divides hierarchical embedding learning into inter-order and intra-order aggregation to learn richer and more comprehensive representations. Based on UI-HEHo framework, a POI-user social attribute model is constructed to obtain the relational distance between interest points. On the other hand, a POI-natural attributes model based on user preference activity range prediction is constructed to further draw the influence between interest points. Furthermore, a tensor-based dynamic update scheme is designed to reduce the high cost of model updates. Finally, the Radviz graph is utilized to obtain the final recommendation results. Experimental results demonstrate our proposed approach significantly improves the POI recommendation performance, especially achieves 20.7% and 23.8% improvement on the Foursquare and Yelp datasets in terms of the newly defined diversity and novelty metrics.

Real-Time Detection Technology of Corn Kernel Breakage and Mildew Based on Improved YOLOv5s

In order to solve low recognition of corn kernel breakage degree and corn kernel mildew degree during corn kernel harvesting, this paper proposes a real-time detection method for corn kernel breakage and mildew based on improved YOlOv5s, which is referred to as the CST-YOLOv5s model algorithm in this paper. The method continuously obtains images through the discrete uniform sampling device of corn kernels and generates whole corn kernels, breakage corn kernels, and mildew corn kernel dataset samples. We aimed at the problems of high similarity of some corn kernel features in the acquired images and the low precision of corn kernel breakage and mildew recognition. Firstly, the CBAM attention mechanism is added to the backbone network of YOLOv5s to finely allocate and process the feature information, highlighting the features of corn breakage and mildew. Secondly, the pyramid pooling structure SPPCPSC, which integrates cross-stage local networks, is adopted to replace the SPPF in YOLOv5s. SPP and CPSC technologies are used to extract and fuse features of different scales, improving the precision of object detection. Finally, the original prediction head is converted into a transformer prediction head to explore the prediction potential with a multi-head attention mechanism. The experimental results show that the CST-YOLOv5s model has a significant improvement in the detection of corn kernel breakage and mildew. Compared with the original YOLOv5s model, the average precision (AP) of corn kernel breakage and mildew recognition increased by 5.2% and 7.1%, respectively, and the mean average precision (mAP) of all kinds of corn kernel recognition is 96.1%, and the frame rate is 36.7 FPS. Compared with YOLOv4-tiny, YOLOv6n, YOLOv7, YOLOv8s, and YOLOv9-E detection model algorithms, the CST-YOLOv5s model has better overall performance in terms of detection accuracy and speed. This study can provide a reference for real-time detection of breakage and mildew kernels during the harvesting process of corn kernels.

HANSynergy: Heterogeneous Graph Attention Network for Drug Synergy Prediction.

Drug synergy therapy is a promising strategy for cancer treatment. However, the extensive variety of available drugs and the time-intensive process of determining effective drug combinations through clinical trials pose significant challenges. It requires a reliable method for the rapid and precise selection of drug synergies. In response, various computational strategies have been developed for predicting drug synergies, yet the exploitation of heterogeneous biological network features remains underexplored. In this study, we construct a heterogeneous graph that encompasses diverse biological entities and interactions, utilizing rich data sets from sources, such as DrugCombDB, PubChem, UniProt, and cancer cell line encyclopedia (CCLE). We initialize node feature representations and introduce a novel virtual node to enhance drug representation. Our proposed method, the heterogeneous graph attention network for drug-drug synergy prediction (HANSynergy), has been experimentally validated to demonstrate that the heterogeneous graph attention network can extract key node features, efficiently harness the diversity of information, and further enhance network functionality through the incorporation of a multihead attention mechanism. In the comparative experiment, the highest accuracy (Acc) and area under the curve (AUC) are 0.877 and 0.947, respectively, in DrugCombDB_early data set, demonstrating the superiority of HANSynergy over the competing methods. Moreover, protein-protein interactions are important in understanding the mechanism of action of drugs. The heterogeneous attention mechanism facilitates protein-protein interaction analysis. By analyzing the changes of attention weight before and after heterogeneous network training, we investigated proteins that may be associated with drug combinations. Additionally, case studies align our findings with existing research, underscoring the potential of HANSynergy in drug synergy prediction. This advancement not only contributes to the burgeoning field of drug synergy prediction but also holds the potential to provide valuable insights and uncover new drug synergies for combating cancer.

Interpretable and explainable hybrid model for daily streamflow prediction based on multi-factor drivers.

Streamflow time series data typically exhibit nonlinear and nonstationary characteristics that complicate precise estimation. Recently, multifactorial machine learning (ML) models have been developed to enhance the performance of streamflow predictions. However, the lack of interpretability within these ML models raises concerns about their inner workings and reliability. This paper introduces an innovative hybrid architecture, the TCN-LSTM-Multihead-Attention model, which combines two layers of temporal convolutional networks (TCN) followed by one layer of long short-term memory (LSTM) units, integrated with a Multihead-Attention mechanism for predicting streamflow with streamflow causation-driven prediction samples (RCDP), employing local and global interpretability studies through Shapley values and partial dependency analysis. The find_peaks method was used to identify peak flow events in the test dataset, validating the model's generality and uncovering the physical causative patterns of streamflow. The results show that (1) compared to the LSTM model with the same hyperparameter settings, the proposed TCN-LSTM-Multihead-Attention hybrid model increased the R2 by 52.9%, 2.5%, 43.1%, and 10.7% respectively at four stations in the test set predictions using RCDP samples. Moreover, comparing the prediction results of the hybrid model under different samples in Hengshan station, the R2 for RCDP increased by 5.06% and 1.22% compared to streamflow autoregressive prediction samples (RAP) and meteorological-soil volumetric water content coupled autoregressive prediction samples (MCSAP) respectively. (2) Historical streamflow data from the preceding 3days predominantly influences predictions due to strong autocorrelation, with flow quantity (Q) typically emerging as the most significant feature alongside precipitation (P), surface soil moisture (SSM), and adjacent station flow data. (3) During periods of low and normal flow, historical data remains the most crucial factor; however, during flood periods, the roles of upstream inflow and precipitation become significantly more pronounced. This model facilitates the identification and quantification of various hydrodynamic impacts on flow predictions, including upstream flood propagation, precipitation, and soil moisture conditions. It also elucidates the model's nonlinear relationships and threshold responses, thereby enhancing the interpretability and reliability of streamflow predictions.

Reservoir fluid identification based on multi-head attention with UMAP

Gas logging curves obtained during drilling, along with conventional well logs, typically serve as foundational data for identifying reservoir fluids. Traditional methods for identifying layers containing oil or gas often suffer from limitations such as inadequate feature extraction, underutilization of differences in data points, and limited applicability. These limitations lead to reduced accuracy in predicting reservoir fluid segmentation. To address the challenge of spatiotemporal feature extraction in logging curves, this paper presents a fluid identification method based on deep learning networks and multivariate sequence classification. The method takes into account both the sequential variations in geological strata and the relationships among different curves for fluid prediction. Our proposed method utilizes a Transformer multi-head attention mechanism and follows a series of steps. First, an improved Symbolic Aggregate approXimation (SAX) is employed to assess the response characteristics of the logging curves to the target fluid and identify sensitive curves based on information similarity. Subsequently, the semi-supervised learning method UMAP is applied to identify layers with high interpretability. Hierarchical training is then employed to mitigate the impact of noise and high-dimensional nonlinearity on identification results. Finally, the processed vector curves are input into the attention module layer, where features of each layer are extracted in an encoding-decoding manner. This method has been tested and validated using gas logging data from 12 wells in a gas field offshore China. The overall prediction accuracy can exceed 94%, and the prediction accuracy for a single gas layer is over 88%. When compared to other machine learning approaches, such as K-means, decision tree, and XGBoost, this method exhibits superior identification accuracy and generalization ability, especially for unbalanced samples.

Improved adaptive war strategy optimization algorithm assisted-adaptive multi-head graph attention mechanism network for remaining useful life of complex equipment

The remaining useful life (RUL) of complex equipment is an important criterion to ensure stable operation. In recent years, deep learning-based methods for predicting the RUL of complex equipment have attracted wide attention. However, it is only able to obtain the potential information in the Euclidean space, which hinders their ability to capture the deeply degradation information. Thus, graph neural networks have gradually entered the researchers’ field of vision. Despite the fact that graph neural networks are able to accomplish the task of RUL for complex equipment, there are still limitations that restrict the prediction performance in practical engineering. To address this challenge, an improved adaptive war strategy optimization algorithm assisted-adaptive multi-head graph attention mechanism network (IWSO-LMGAT) is proposed. For one thing, a learnable attention mechanism is proposed to adjust the weights of different heads dynamical and improve the limitation of GAT in obtaining deep degradation information. In addition, since hyperparameters are essential elements affecting the predicted result, inspired by the “no-free lunch” principle, an improved mathematical expression is described to avoid the issue such as precocity, fall into local optimums for WSO so that the optimal hyperparameters of the LMGAT could be obtained. The effectiveness and advancement of IWSO-LMGAT are validated on the CMAPSS dataset, and experimental results show that the proposed method could provide competitive forecasted results compared to traditional methods, that is, R2 = 0.9939, RMSE = 4.3638, and MAPE = 0.0137; this illustrates the IWSO-LMGAT’s potential for the RUL prediction of complex equipment.

SEnSCA: Identifying possible ligand-receptor interactions and its application in cell-cell communication inference.

Multicellular organisms have dense affinity with the coordination of cellular activities, which severely depend on communication across diverse cell types. Cell-cell communication (CCC) is often mediated via ligand-receptor interactions (LRIs). Existing CCC inference methods are limited to known LRIs. To address this problem, we developed a comprehensive CCC analysis tool SEnSCA by integrating single cell RNA sequencing and proteome data. SEnSCA mainly contains potential LRI acquisition and CCC strength evaluation. For acquiring potential LRIs, it first extracts LRI features and reduces the feature dimension, subsequently constructs negative LRI samples through K-means clustering, finally acquires potential LRIs based on Stacking ensemble comprising support vector machine, 1D-convolutional neural networks and multi-head attention mechanism. During CCC strength evaluation, SEnSCA conducts LRI filtering and then infers CCC by combining the three-point estimation approach and single cell RNA sequencing data. SEnSCA computed better precision, recall, accuracy, F1 score, AUC and AUPR under most of conditions when predicting possible LRIs. To better illustrate the inferred CCC network, SEnSCA provided three visualization options: heatmap, bubble diagram and network diagram. Its application on human melanoma tissue demonstrated its reliability in CCC detection. In summary, SEnSCA offers a useful CCC inference tool and is freely available at https://github.com/plhhnu/SEnSCA.

Prediction of Parking Space Availability Using Improved MAT-LSTM Network

The prediction of parking space availability plays a crucial role in information systems providing parking guidance. However, controversy persists regarding the efficiency and accuracy of mainstream time series prediction methods, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). In this study, a comparison was made between a temporal convolutional network (TCN) based on CNNs and a long short-term memory (LSTM) network based on RNNs to determine an appropriate baseline for predicting parking space availability. Subsequently, a multi-head attention (MAT) mechanism was incorporated into an LSTM network, attempting to improve its accuracy. Experiments were conducted on three real and two synthetic datasets. The results indicated that the TCN achieved the fastest convergence, whereas the MAT-LSTM method provided the highest average accuracy, namely 0.0330 and 1.102 × 10−6, on the real and synthetic datasets, respectively. Furthermore, the improved MAT-LSTM model accomplished an increase of up to 48% in accuracy compared with the classic LSTM model. Consequently, we concluded that RNN-based networks are better suited for predicting long-time series. In particular, the MAT-LSTM method proposed in this study holds higher application value for predicting parking space availability with a higher accuracy.

Convolutional Preprocessing Transformer-Based Fault Diagnosis for Rectifier-Filter Circuits in Nuclear Power Plants

Rectifier-filter circuit, as a vital electronic component in nuclear power plants (NPPs), serves the role of converting alternating current into smooth direct current. Affected by intense radiation environmental factors and variable operating profiles, the rectifier-filter circuit is subject to degradation and soft failures, namely, a derated circuit performance but not yet to failure. It is urgent to diagnose the soft failures of the rectifier-filter circuit, so as to ensure the safe operations of NPPs. In this article, an ensemble empirical mode decomposition (EEMD) algorithm is first employed to decompose the three-phase current signals and electrical signals of electronic components. The decomposed signals are fed into a convolutional preprocessing Transformer (CPT) to deeply extract the fault features of the rectifier-filter circuits. Specifically, the multi-directional, multi-scale, and extremely sensitive long-range time-dependent features in the EEMD feature data are extracted by the multi-head attention mechanism of the Transformer. A deep Softmax classifier is, then, devised to reduce the feature dimensionality and identify the soft failure modes of the rectifier-filter circuit. The fault simulation of the rectifier-filter circuit is conducted, and a real case experiment and multiple comparative studies are conducted to validate the effectiveness and diagnosis accuracy of the CPT-based method.

A Real-Time License Plate Detection and Recognition Model in Unconstrained Scenarios.

Accurate and fast recognition of vehicle license plates from natural scene images is a crucial and challenging task. Existing methods can recognize license plates in simple scenarios, but their performance degrades significantly in complex environments. A novel license plate detection and recognition model YOLOv5-PDLPR is proposed, which employs YOLOv5 target detection algorithm in the license plate detection part and uses the PDLPR algorithm proposed in this paper in the license plate recognition part. The PDLPR algorithm is mainly designed as follows: (1) A Multi-Head Attention mechanism is used to accurately recognize individual characters. (2) A global feature extractor network is designed to improve the completeness of the network for feature extraction. (3) The latest parallel decoder architecture is adopted to improve the inference efficiency. The experimental results show that the proposed algorithm has better accuracy and speed than the comparison algorithms, can achieve real-time recognition, and has high efficiency and robustness in complex scenes.

Multi-bolt looseness positioning using multivariate recurrence analytic active sensing method and MHAMCNN model

The piezoelectric active sensing method has been proven to be an effective, nondestructive approach for bolt looseness monitoring. The conventional damage index relies on the energy index of the wavelet packet. However, the method of wavelet packet could not identify the looseness location of the multi-bolt. Therefore, an innovative multi-bolt intelligent method combined the multivariate recurrence analytic active sensing method with an improved convolutional neural network (CNN) model was proposed for multi-bolt looseness positioning. In this method, the multivariate recurrence plots (RPs) are firstly constructed based on the received stress waves and then it is utilized to extract the recursive attributes presented within the stress wave signals. Secondly, a novel model, the CNN improved by the multi-head attention mechanism (MHAMCNN), is constructed to extract nonlinear features hidden in multivariate RPs. Repeated experiments demonstrate that the proposed method, which combines MHAMCNN and un-thresholded multivariate RPs (i.e., mean detection accuracy: 98.62%), outperformed other methods, including multivariate RPs (i.e., mean detection accuracy: 97.06%) and common RPs. Compared with traditional pattern recognition algorithms, MHAMCNN has higher classification accuracy. In summary, the suggested approach presents a promising opportunity for the real-time monitoring of multi-bolt looseness.

WIFI based human activity recognition using multi-head adaptive attention mechanism

Human Activity Recognition (HAR) utilizing Channel State Information (CSI) extracted from WiFi signals has garnered substantial interest across various domains and applications. This field’s potential paths and applications extend beyond CSI-based HAR and include smart homes, assisted living, security, gaming, surveillance, and context-aware computing. The ability of deep learning algorithms to effectively process and interpret CSI data opens up new possibilities for accurate and robust human activity recognition in real-world scenarios. However, traditional Recurrent Neural Networks (RNN) models, such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), rely solely on their internal memory cells to maintain information over time. Important details might be diluted or lost within the memory cells in complex CSI sequences. To address this limitation, we propose a lightweight approach that incorporates a multi-head adaptive attention weight mechanism MHAAM into the HAR framework. The multi-head attention mechanism allows the model to attend to different informative patterns within the CSI data simultaneously, capturing fine-grained temporal dependencies and improving the model’s ability to recognize complex activities. The implemented models effectively filter out noise and irrelevant information by assigning higher weights to informative CSI features, further enhancing activity classification accuracy. Experimental evaluations and comparative analyses of HAR for seven activities demonstrate that attention-based RNN models with multi-head attention consistently outperform traditional RNN models. The multi-head attention mechanism achieves improved generalization and testing for seven common human activities and environments, leading to a higher complex human activity classification accuracy of up to 98.5%.

SwinSOD: Salient object detection using swin-transformer

The Transformer structure has achieved excellent performance in a wide range of applications in computer vision, and Swin-Transformer also shows strong feature representation capabilities. On this basis, we proposed a fusion model SwinSOD for RGB salient object detection. This model used a Swin-Transformer as the encoder to extract hierarchical features, was driven by a multi-head attention mechanism to bridge the gap between hierarchical features, progressively fused adjacent layer feature information under the guidance of global information, and refined the boundaries of saliency objects through the feedback information. Specifically, the Swin-Transformer encoder extracted multi-level features and then recalibrated the channels to optimize intra-layer channel features. The feature fusion module realized feature fusion between each layer under the guidance of global information. In order to clarify the fuzzy boundaries, the second stage feature fusion achieved edge refinement under the guidance of feedback information. The proposed model outperforms state-of-the-art models on five popular SOD datasets, demonstrating the advanced performance of this network. Code released: https://github.com/user-wu/SwinSOD.

Defect Detection of Cell Phone Screen Using a Faster Regional Convolutional Neural Network with Multi-head Attention Mechanism

Since the glass outer screen of a cell phone is the main sensory part of the human eye when using a cell phone, the advantages and disadvantages of the cell phone screen directly affect people's sense of use. Therefore, the defect detection requirements for cell phone screens are high and need to meet the needs of high-volume factory inspection. Most of the traditional defect detection methods use visual methods, the detection results are overly dependent on the subjectivity and experience of workers, the efficiency of this method is low, and the accuracy is poor. Currently, machine learning-based detection methods are applied in numerous industries. In this paper, a faster Regional Convolutional Neural Network (R-CNN) with multi-head attention mechanism for defect detection of cell phone screen is proposed. To enhance the network's capability in extracting feature information, a four-head attention mechanism is added to the last convolutional layer of the ResNet50 network. An improved Region of Interest (ROI) Align is proposed to replace the original ROI Pooling to reduce the localization error of cell phone screen defects. Replace the original Rectified Linear Unit (ReLU) activation function with the Copy Exponential Linear Unit (CELU) activation function to expedite the convergence capability of the network. Finally, by comparing with other classical model training, the evaluation results indicate that the proposed method achieved an average accuracy of 95.71%, which is a 5.34% improvement compared to the original faster R-CNN network.