Attention Mechanism Research Articles

Segmentation of ethnic clothing patterns with fusion of multiple attention mechanisms

AbstractTo address the challenges posed by diverse pattern-background elements, intricate details, and complex textures in the semantic segmentation of ethnic clothing patterns, this research introduces a novel semantic segmentation network model called MST-Unet (Mixed Swin Transformer U-net). The proposed model combines a U-shaped network structure with multiple attention mechanisms. The upper layers of the model employ classical convolutional operations, focusing on local relationships in the initial layers containing high-resolution details. In deeper layers, Swin Transformer modules are utilized, capable of efficient feature extraction with smaller spatial dimensions, maintaining performance while reducing computational burden. An attention gate mechanism is integrated into the decoder, contributing to enhanced performance in ethnic clothing pattern segmentation tasks by allowing the model to better capture crucial image features and achieve precise segmentation results. In visual comparisons of segmentation results, our proposed model demonstrates superior performance. The segmentation results exhibit more complete preservation of edge contours and fewer misclassifications in irrelevant regions within the images. In qualitative and quantitative experiments conducted on the ethnic clothing pattern dataset, our model achieves the highest Dice score for segmentation results in all four subclasses of ethnic clothing patterns. The average Dice score of our model reaches an impressive 89.80%, surpassing other algorithms in the same category. When compared to Deeplab_V3+, ResUnet, SwinUnet, and Unet networks, our model outperforms them by 7.72%, 5.09%, 5.05%, and 0.67%.

Research on Aspect-Level Sentiment Analysis Based on Adversarial Training and Dependency Parsing

Aspect-level sentiment analysis is used to predict the sentiment polarity of a specific aspect in a sentence. However, most current research cannot fully utilize semantic information, and the models lack robustness. Therefore, this article proposes a model for aspect-level sentiment analysis based on a combination of adversarial training and dependency syntax analysis. First, BERT is used to transform word vectors and construct adjacency matrices with dependency syntactic relationships to better extract semantic dependency relationships and features between sentence components. A multi-head attention mechanism is used to fuse the features of the two parts, simultaneously perform adversarial training on the BERT embedding layer to enhance model robustness, and, finally, to predict emotional polarity. The model was tested on the SemEval 2014 Task 4 dataset. The experimental results showed that, compared with the baseline model, the model achieved significant performance improvement after incorporating adversarial training and dependency syntax relationships.

CA-YOLO: Ensembled Model Optimization for Remote Sensing Image Object Detection

Multiple object retrieval methods are solved by the CA-YOLO model to find elements in complex spatial images. This addresses weak multiscale feature learning and the delicate balance between model complexity and performance. CA-YOLO is based on YOLOv5 and uses a small coordinate attention module at the lower layer to recover selected features and reduce data. A fast spatial pyramid with tandem design modules at the bottom uses stochastic pooling to speed up thinking and merge features of different sizes. Anchor box interactions and missing features have been updated to make it easier to find items of different sizes and shapes. CA-YOLO outperforms YOLO in multiple object detection and average inference speed of 125 frames/sec. CA-YOLO is a great option in the same conditions and difficulties. The study also investigated V3 tiny, V4, V5s, V8s, CA Yolos, and V5x6 YOLO models. This suggests that YOLO V5x6 can achieve more than 95% mAP on remote sensing object identification datasets. Index terms - Object detection, attention mechanism, coordinate attention, SPPF, SIoU loss.

A hierarchical occupancy network with multi‐height attention for vision‐centric 3D occupancy prediction

AbstractThe precise geometric representation and ability to handle long‐tail targets have led to the increasing attention towards vision‐centric 3D occupancy prediction, which models the real world as a voxel‐wise model solely through visual inputs. Despite some notable achievements in this field, many prior or concurrent approaches simply adapt existing spatial cross‐attention (SCA) as their 2D–3D transformation module, which may lead to informative coupling or compromise the global receptive field along the height dimension. To overcome these limitations, we propose a hierarchical occupancy (HierOcc) network featuring our innovative height‐aware cross‐attention (HACA) and hierarchical self‐attention (HSA) as its core modules to achieve enhanced precision and completeness in 3D occupancy prediction. The former module enables 2D–3D transformation, while the latter promotes voxels’ intercommunication. The key insight behind both modules is our multi‐height attention mechanism which ensures each attention head corresponds explicitly to a specific height, thereby decoupling height information while maintaining global attention across the height dimension. Extensive experiments show that our method brings significant improvements compared to baseline and surpasses all concurrent methods, demonstrating its superiority.

Sla-former: conformer using shifted linear attention for audio-visual speech recognition

AbstractConformer-based models have proven highly effective in Audio-visual Speech Recognition, integrating auditory and visual inputs to significantly enhance speech recognition accuracy. However, the widely utilized softmax attention mechanism within conformer models encounters scalability issues, with its spatial and temporal complexity escalating quadratically with sequence length. To address these challenges, this paper introduces the Shifted Linear Attention Conformer, an evolved iteration of the conformer architecture. Shifted Linear Attention Conformer adopts shifted linear attention as a scalable alternative to softmax attention. We conducted a thorough analysis of the factors constraining the efficiency of linear attention. To mitigate these issues, we propose the utilization of a straightforward yet potent mapping function and an efficient rank restoration module, enhancing the effectiveness of self-attention while maintaining low computational complexity. Furthermore, we integrate an advanced attention-shifting technique facilitating token manipulation within attentional mechanisms, thereby enhancing information flow across various groups. This three-part approach enhances cognitive computations, particularly beneficial for processing longer sequences. Our model achieves exceptional Word Error Rates of 1.9% and 1.5% on the Lip Reading Sentences 2 and Lip Reading Sentences 3 datasets, respectively, showcasing its state-of-the-art performance in audio-visual speech recognition tasks.

EB-Net: an efficient balanced network for accuracy and speed of remote sensing detection

ABSTRACT Remote sensing detection is a difficult task that requires not only identifying objects with complex background, quality, and angle issues, but also being lightweight enough to be carried by edge devices. In real remote sensing scenarios, achieving accurate, fast, and low-resource-consumption automated detection remains a significant challenge. Therefore, this paper proposes an efficient balanced network (EB-Net) for real remote sensing devices. First, a dynamic sparse attention (DSA) mechanism is proposed and has been proven with high performance via the complexity analysis. In addition, a new dynamic sparse transformer (DSFormer) is constructed using DSA, which enhances feature information and adapts to image resolution by self-attention and multi-headed attention, and achieves more flexible computation by random sampling. Then, three versions of discrete distribution IoU (DDIoU) are defined for adapting various scenarios and tasks in remote sensing, and this loss function makes the model achieve high accuracy and more lightweight. Finally, to make the model more lightweight, a cropped SPPF (CroSPPF) is presented, which significantly improves the computational efficiency by the lightweight of the sequence structure and activation function. Ablation experiments are conducted on the NWPU VHR-10 dataset, and demonstrate the effectiveness of the proposed methods. Numerous comparisons with state-of-the-art detectors are conducted on the NWPU VHR-10, RSOD and DOTA datasets. The experiments show that EB-Net outperforms state-of-the-art remote sensing detection models in comprehensive performance and achieves end-to-end accurate and lightweight detection.

Parallel Spatio-Temporal Attention Transformer for Video Frame Interpolation

Traditional video frame interpolation methods based on deep convolutional neural networks face challenges in handling large motions. Their performance is limited by the fact that convolutional operations cannot directly integrate the rich temporal and spatial information of inter-frame pixels, and these methods rely heavily on additional inputs such as optical flow to model motion. To address this issue, we develop a novel framework for video frame interpolation that uses Transformer to efficiently model the long-range similarity of inter-frame pixels. Furthermore, to effectively aggregate spatio-temporal features, we design a novel attention mechanism divided into temporal attention and spatial attention. Specifically, spatial attention is used to aggregate intra-frame information, integrating both attention and convolution paradigms through the simple mapping approach. Temporal attention is used to model the similarity of pixels on the timeline. This design achieves parallel processing of these two types of information without extra computational cost, aggregating information in the space–time dimension. In addition, we introduce a context extraction network and multi-scale prediction frame synthesis network to further optimize the performance of the Transformer. Our method and state-of-the-art methods are extensively quantitatively and qualitatively experimented on various benchmark datasets. On the Vimeo90K and UCF101 datasets, our model achieves improvements of 0.09 dB and 0.01 dB in the PSNR metrics over UPR-Net-large, respectively. On the Vimeo90K dataset, our model outperforms FLAVR by 0.07 dB, with only 40.56% of its parameters. The qualitative results show that for complex and large-motion scenes, our method generates sharper and more realistic edges and details.

Application of the Nadaraya-Watson estimator based attention mechanism to the field of predictive maintenance

Attention mechanism has recently gained immense importance in the natural language processing (NLP) world. This technique highlights parts of the input text that the NLP task (such as translation) must pay “attention” to. Inspired by this, some researchers have recently applied the NLP domain, deep-learning based, attention mechanism techniques to predictive maintenance. In contrast to the deep-learning based solutions, Industry 4.0 predictive maintenance solutions that often rely on edge-computing, demand lighter predictive models. With this objective, we have investigated the adaptation of a simpler, incredibly fast and compute-resource friendly, “Nadaraya-Watson estimator based” attention method. We develop a method to predict tool-wear of a milling machine using this attention mechanism and demonstrate, with the help of heat-maps, how the attention mechanism highlights regions that assist in predicting onset of tool-wear. We validate the effectiveness of this adaptation on the benchmark IEEEDataPort PHM Society dataset, by comparing against other comparatively “lighter” machine learning techniques – Bayesian Ridge, Gradient Boosting Regressor, SGD Regressor and Support Vector Regressor. Our experiments indicate that the proposed Nadaraya-Watson attention mechanism performed best with an MAE of 0.069, RMSE of 0.099 and R2 of 83.40 %, when compared to the next best technique Gradient Boosting Regressor with figures of 0.100, 0.138, 66.51 % respectively. Additionally, it produced a lighter and faster model as well.•We propose a Nadaraya-Watson estimator based “attention mechanism”, applied to a predictive maintenance problem.•Unlike the deep-learning based attention mechanisms from the NLP domain, our method creates fast, light and high-performance models, suitable for edge computing devices and therefore supports the Industry 4.0 initiative.•Method validated on real tool-wear data of a milling machine.

Vision-based robotic grasping using faster R-CNN–GRCNN dual-layer detection mechanism

Visual grasping technology plays a crucial role in various robotic applications, such as industrial automation, warehousing, and logistics. However, current visual grasping methods face limitations when applied in industrial scenarios. Focusing solely on the workspace where the grasping target is located restricts the camera’s ability to provide additional environmental information. On the other hand, monitoring the entire working area introduces irrelevant data and hinders accurate grasping pose estimation. In this paper, we propose a novel approach that combines a global camera and a depth camera to enable efficient target grasping. Specifically, we introduce a dual-layer detection mechanism based on Faster R-CNN–GRCNN. By enhancing the Faster R-CNN with attention mechanisms, we focus the global camera on the workpiece placement area and detect the target object within that region. When the robot receives the command to grasp the workpiece, the improved Faster R-CNN recognizes the workpiece and guides the robot towards the target location. Subsequently, the depth camera on the robot determines the grasping pose using Generative Residual Convolutional Neural Network and performs the grasping action. We validate the feasibility and effectiveness of our proposed framework through experiments involving collaborative assembly tasks using two robotic arms.

A deep learning based communication traffic prediction approach for smart monitoring of distributed energy resources in virtual power plants

AbstractVirtual power plants (VPPs) have been widely recognized as a key enabler for energy system neutrality. The communication traffic of a VPP fundamentally indicates its activeness in interacting with the power system, thus providing a new dimension in depicting the behaviour characteristics of distributed energy resources in VPPs. Therefore, the prediction of communication traffic is significant in improving the control efficiency of VPPs. However, due to the involvement of numerous interactive agents characterised by both individual randomness and coordinative characteristics, traditional prediction models are no longer capable of fitting VPP communication traffic effectively. Therefore, a novel prediction model is introduced that enhances the prediction accuracy by integrating long short‐term memory (LSTM) and variational mode decomposition (VMD). This model employs VMD as the initial step for extracting the intrinsic modes from the traffic sequence, thereby mitigating the impact of incidental noise. Then, LSTM is applied to fit each intrinsic mode individually. Additionally, considering the outer influencing factors, the attention mechanism is incorporated. Finally, all sub‐prediction algorithms are neatly integrated as a whole prediction model. The proposed model is evaluated through simulation prediction using realistic VPP communication traffic data, and the results demonstrate its effectiveness.

Construction and Implementation of Content-Based National Music Retrieval Model Under Deep Learning

This research mainly studies the construction and implementation of the content-based folk music retrieval model. Firstly, it studies the music automatic annotation method based on deep learning, and then proposes the tag conditional random field music automatic annotation method, and then constructs the music annotation depth neural network model combining a variety of music representation and attention mechanism. Finally, it analyzes the proposed folk music retrieval model the effectiveness of the cable model is verified and its performance is evaluated. The results show that in Glu module, Glu blocks had better performance in music annotation, and the music annotation results of each index in music hierarchical sequence modeling are better, which ensures the effectiveness of music annotation. Compared with other algorithms, the AUC tag score of the proposed method is the highest, which is 0.913; it can better model the mapping relationship between the audio features of music input to the text tag and has higher scores on all evaluation indicators.

Knowledge Graph Reasoning via Dynamic Subgraph Attention with Low Resource Computation

Knowledge graphs (KGs) suffer from inherent incompleteness, which has spurred research into knowledge graph reasoning (KGR), i.e., ways to infer missing facts based on existing triples. The most prevalent approaches employ a static mechanism for entity representation across the entire graph, sharing entity embeddings for different query relations. Nevertheless, these static entity embeddings fail to capture specific semantics for various query scenarios, resulting in inaccurate entity representations and susceptibility to reasoning errors. Furthermore, the whole graph learning style requires substantial computational resources, especially since KGs are often of large-scale. Consequently, these methods are impractical in low-resource environments. To address these issues, we devise a framework called dynamic subgraph attention (DSA), which learns dynamic entity embeddings for different query relations across subgraphs. In our approach, by utilizing multi-hop path history information obtained through path-based learning, we guide a dynamic attention mechanism to generate dynamic entity embeddings for different query relations. To ensure the semantic information of entities, the embedding-based and path-based learning are jointly trained for KGR. Additionally, the proposed dynamic aggregation mechanism operates on subgraphs, resulting in a remarkable 6–7 times resource conservation compared to GPU computations. Empirical experiments further demonstrate that DSA outperforms the current methods significantly on three benchmark datasets.

MCF-YOLOv5: A Small Target Detection Algorithm Based on Multi-Scale Feature Fusion Improved YOLOv5

In recent years, many deep learning-based object detection methods have performed well in various applications, especially in large-scale object detection. However, when detecting small targets, previous object detection algorithms cannot achieve good results due to the characteristics of the small targets themselves. To address the aforementioned issues, we propose the small object algorithm model MCF-YOLOv5, which has undergone three improvements based on YOLOv5. Firstly, a data augmentation strategy combining Mixup and Mosaic is used to increase the number of small targets in the image and reduce the interference of noise and changes in detection. Secondly, in order to accurately locate the position of small targets and reduce the impact of unimportant information on small targets in the image, the attention mechanism coordinate attention is introduced in YOLOv5’s neck network. Finally, we improve the Feature Pyramid Network (FPN) structure and add a small object detection layer to enhance the feature extraction ability of small objects and improve the detection accuracy of small objects. The experimental results show that, with a small increase in computational complexity, the proposed MCF-YOLOv5 achieves better performance than the baseline on both the VisDrone2021 dataset and the Tsinghua Tencent100K dataset. Compared with YOLOv5, MCF-YOLOv5 has improved detection APsmall by 3.3% and 3.6%, respectively.

Incorporating Multi-Source Market Sentiment and Price Data for Stock Price Prediction

The problem of stock price prediction has been a hot research issue. Stock price is influenced by various factors at the same time, and market sentiment is one of the most critical factors. Financial texts such as news and investor comments reflect investor sentiment in the stock market and influence market movements. Previous research models have struggled to accurately mine multiple sources of market sentiment information originating from the Internet and traditional sentiment analysis models are challenging to quantify and combine indicator data from market data and multi-source sentiment data. Therefore, we propose a BERT-LLA stock price prediction model incorporating multi-source market sentiment and technical analysis. In the sentiment analysis module, we propose a semantic similarity and sector heat-based model to screen for related sectors and use fine-tuned BERT models to calculate the text sentiment index, transforming the text data into sentiment index time series data. In the technical indicator calculation module, technical indicator time series are calculated using market data. Finally, in the prediction module, we combine the sentiment index time series and technical indicator time series and employ a two-layer LSTM network prediction model with an integrated attention mechanism to predict stock close price. Our experiment results show that the BERT-LLA model can accurately capture market sentiment and has a strong practicality and forecasting ability in analyzing market sentiment and stock price prediction.

Adap-BDCM: Adaptive Bilinear Dynamic Cascade Model for Classification Tasks on CNV Datasets.

Copy number variation (CNV) is an essential genetic driving factor of cancer formation and progression, making intelligent classification based on CNV feasible. However, there are a few challenges in the current machine learning and deep learning methods, such as the design of base classifier combination schemes in ensemble methods and the selection of layers of neural networks, which often result in low accuracy. Therefore, an adaptive bilinear dynamic cascade model (Adap-BDCM) is developed to further enhance the accuracy and applicability of these methods for intelligent classification on CNV datasets. In this model, a feature selection module is introduced to mitigate the interference of redundant information, and a bilinear model based on the gated attention mechanism is proposed to extract more beneficial deep fusion features. Furthermore, an adaptive base classifier selection scheme is designed to overcome the difficulty of manually designing base classifier combinations and enhance the applicability of the model. Lastly, a novel feature fusion scheme with an attribute recall submoduleis constructed, effectively avoiding getting stuck in local solutions and missing some valuable information. Numerous experiments have demonstrated that our Adap-BDCM model exhibits optimal performance in cancer classification, stage prediction, and recurrence on CNV datasets. This study can assist physicians in making diagnoses faster and better.

Automated assessment of hydrostatic pressure resistance in woven fabrics using the enhanced YOLOv8 network

Hydrostatic pressure resistance serves as a crucial metric in the assessment of water resistance in woven fabrics. The expeditious, precise, and efficient conduct of hydrostatic pressure tests holds paramount importance in advancing the progress and production of high-performance textiles. Addressing the challenges posed by intricate printed patterns on woven fabrics, and the presence of small, widely scattered water droplets, the study leverages the enhanced YOLOv8 model to develop a machine vision-based automated detection technique for assessing water resistance in woven fabric. The proposed method incorporates convolutional block attention module attention mechanisms into the backbone and neck network, replaces the path aggregation network structure of YOLOv8 with the bidirectional feature pyramid network structure, and introduces a dedicated detection head for small targets. These enhancements facilitate accurate identification of water outlet points on the woven fabric and precise recording of frame positions, enabling the precise measurement of hydrostatic pressure. Validation of the proposed model is conducted through a series of comparative experiments utilizing a self-collected dataset. The experimental results underscore the exemplary performance of the proposed model, evidenced by an AP0.5 score of 92.18%, showcasing superior overall efficacy in comparison with alternative models. Notably, the target localization time error is found to be less than 2 s when contrasted with manual detection. This method substantially enhances the accuracy of water droplet detection and localization in hydrostatic pressure resistance testing of woven fabric, characterized by complex surface patterns, thereby contributing to refinement of hydrostatic pressure testing methodologies in woven fabric analysis.

Speaker-aware cognitive network with cross-modal attention for multimodal emotion recognition in conversation

Emotion recognition in conversation (ERC) has gained considerable attention owing to its extensive applications in the field of human-computer interaction. However, previous models have had certain limitations in exploring the potential emotional relationships within the conversation due to their inability to fully leverage speaker information. Additionally, information from various modalities such as text, audio, and video can synergistically enhance and supplement the analysis of emotional context within the conversation. Nonetheless, effectively fusing multimodal features to understand the detailed contextual information in the conversation is challenging. This paper proposes a Speaker-Aware Cognitive network with Cross-Modal Attention (SACCMA) for multimodal ERC to effectively leverage multimodal information and speaker information. Our proposed model primarily consists of the modality encoder and the cognitive module. The modality encoder is employed to fuse multimodal feature information from speech, text, and vision using a cross-modal attention mechanism. Subsequently, the fused features and speaker information are separately fed into the cognitive module to enhance the perception of emotions within the dialogue. Compared to seven common baseline methods, our model increased the Accuracy score by 2.71 % and 1.70 % on the IEMOCAP and MELD datasets, respectively. Additionally, the F1 score improved by 2.92 % and 0.70 % for each dataset. Various experiments also demonstrate the effectiveness of our method.

A Noise-Robust CNN Architecture with Global Attention and Gated Convolutional Kernels for Bearing Fault Detection

Abstract Detecting faults in bearings is essential for the maintenance and operation of rotating machinery. However, achieving high accuracy and noise immunity is challenging due to the involvement of intricate and noisy signals. To address this issue, this paper introduces a multi-scale separable gated convolutional neural network (GCK-MSSC). In the GCK-MSSC model, the gate convolutional kernel replaces the conventional convolutional kernel. It is designed to dynamically adjust the convolution kernel's weights based on the input features. Additionally, the one-dimensional global attention mechanism (1D-GAM) is incorporated, enhancing the model's global awareness within the multi-scale separable neural network (MSSC) framework. The experimental results on two public bearing datasets confirm the performance of the proposed method. It demonstrates improved performance over current leading-edge methods, especially in terms of accuracy, and proves to be significantly robust against various levels of noise. Specifically, achieves accuracies of 99.45 \% and 99.78 \% on the two datasets. Furthermore, even after the addition of noise with a signal-to-noise ratio of 0, it still maintains an accuracy as high as 85.65\% (on the Politecnico di Torino dataset).

Microblog sentiment analysis method using BTCBMA model in Spark big data environment

Abstract Microblogs are currently one of the most well-liked social platforms in China, and sentiment analysis of microblog texts can help further analyze the realization of their media value; however, the current task of sentiment analysis based on microblog information suffers from low accuracy due to the large size and high redundancy of microblog data, a microblog sentiment analysis method using Bidirectional Encoder Representation from Transformers (BERT)–Text Convolutional Neural Network (TextCNN)–Bidirectional Gate Recurrent Unit (BiGRU)–Multihead-Attention model in Spark big data environment is proposed. First, the Chinese pre-trained language model BERT is used to convert the input data into dynamic character-level word vectors; then, TextCNN is used to effectively obtain local features such as keywords and pool the filtered features; then, BiGRU is introduced to quickly capture more comprehensive semantic information; finally, a multi-headed attention mechanism is implemented to emphasize the most significant features in order to accomplish the sentiment classification of microblog information task precisely. By comparing the existing advanced models, the proposed model demonstrates an improvement of at least 4.99% and 0.05 in accuracy and F1-score evaluation indexes, respectively. This enhancement significantly enhances the accuracy of microblog sentiment analysis tasks and aids pertinent authorities in comprehending the inclination of individual’s attitude toward hot topics. Furthermore, it facilitates a prompt prediction of topic trends, enabling them to guide public opinion accordingly.

Development of an artificial intelligence-based multimodal model for assisting in the diagnosis of necrotizing enterocolitis in newborns: a retrospective study

ObjectiveThe purpose of this study is to develop a multimodal model based on artificial intelligence to assist clinical doctors in the early diagnosis of necrotizing enterocolitis in newborns.MethodsThis study is a retrospective study that collected the initial laboratory test results and abdominal x-ray image data of newborns (non-NEC, NEC) admitted to our hospital from January 2022 to January 2024.A multimodal model was developed to differentiate multimodal data, trained on the training dataset, and evaluated on the validation dataset. The interpretability was enhanced by incorporating the Gradient-weighted Class Activation Mapping (GradCAM) analysis to analyze the attention mechanism of the multimodal model, and finally compared and evaluated with clinical doctors on external datasets.ResultsThe dataset constructed in this study included 11,016 laboratory examination data from 408 children and 408 image data. When applied to the validation dataset, the area under the curve was 0.91, and the accuracy was 0.94. The GradCAM analysis shows that the model's attention is focused on the fixed dilatation of the intestinal folds, intestinal wall edema, interintestinal gas, and portal venous gas. External validation demonstrated that the multimodal model had comparable accuracy to pediatric doctors with ten years of clinical experience in identification.ConclusionThe multimodal model we developed can assist doctors in early and accurate diagnosis of NEC, providing a new approach for assisting diagnosis in underdeveloped medical areas.