Multi-layer Attention Research Articles

Intelligent Road Surface State Recognition Method based on Multi-Layer Attention Residual Network

Abstract Data-driven road surface state recognition enhances the efficiency and accuracy of road management, contributing to increased safety and reliability in road traffic. However, traditional machine learning and deep learning-based road surface state recognition typically rely on extensive data for model training, making it challenging to adapt to complex tasks in diverse scenarios. Therefore, this paper proposes a Multi-layer Attention Residual Network (MARN)-based intelligent road surface state recognition method. First, a Residual Convolutional Neural Network (ResNet) is constructed as the backbone model of MARN to mitigate the gradient vanishing problem, allowing the network to extract deeper features. Subsequently, an adaptive multi-layer attention mechanism is introduced in each convolutional layer, enabling adaptive weighting of each feature channel in the dataset to enhance the model's focus on different features for better feature extraction. Furthermore, a cosine annealing learning rate adjuster is designed to improve the accuracy, robustness, and convergence during the model training process. Finally, the proposed MARN is validated using an image dataset containing six different road surface states. Comparative studies are conducted on the recognition accuracy of the proposed MARN, original ResNet, Visual Geometry Group network (VGG16), and Convolutional Neural Network (CNN). The impact of different batch sizes on the convergence speed of road surface state recognition under MARN is also analyzed. Results demonstrate that MARN achieves a training set accuracy of over 95%, surpassing VGG16 and CNN with accuracies below 85%. Compared to ResNet, MARN exhibits a 1.3% higher training set accuracy and a 0.25 lower validation set loss, showcasing superior accuracy and robustness in road surface state recognition.

Spatial layout optimization model integrating layered attention mechanism in the development of smart tourism management

Tourism demand projection is paramount for both corporate operations and destination management, facilitating tourists in crafting bespoke, multifaceted itineraries and enriching their vacation experiences. This study proposes a multi-layer self attention mechanism recommendation algorithm based on dynamic spatial perception, with the aim of refining the analysis of tourists’ emotional inclinations and providing precise estimates of tourism demand. Initially, the model is constructed upon a foundation of multi-layer attention modules, enabling the semantic discovery of proximate entities to the focal scenic locale and employing attention layers to consolidate akin positions, epitomizing them through contiguous vectors. Subsequently, leveraging tourist preferences, the model forecasts the likelihood of analogous attractions as a cornerstone for the recommendation system. Furthermore, an attention mechanism is employed to refine the spatial layout, utilizing the forecasted passenger flow grid to infer tourism demand across multiple scenic locales in forthcoming periods. Ultimately, through scrutiny of data pertaining to renowned tourist destinations in Beijing, the model exhibits an average MAPE of 8.11%, markedly surpassing benchmarks set by alternative deep learning models, thereby underscoring its precision and efficacy. The spatial layout optimization methodology predicated on a multi-layer attention mechanism propounded herein confers substantive benefits to tourism demand prognostication and recommendation systems, promising to elevate the operational standards and customer contentment within the tourism sector.

Aspect based sentiment analysis of consumer reviews using unsupervised attention neural framework

Consumer reviews posted on e-commerce platforms offer an assessment of their opinion and sentiments about various aspects of a product or service. The aspects may belong to a specific category at subsystem and component level. The irrealis often associated with an opinion term, if not handled carefully, may distort the sentiment scores. This paper introduces a clustering and negative-sampling backed feed-forward Unsupervised Attention Deep Neural Framework for Opinion-Aspect-Category-Irrealis-Sentiment (UAN-OACIS) Quintuple extraction. This framework splits the five subtasks into two stages. In the first stage, aspects and opinions are extracted using word embedding-based multi-layer Attention Deep Neural Network (DNN) models; while the second stage projects corresponding categories, irrealis, and sentiment scores by creating a hash table and a couple of algorithms. We also propose a POS-Tag-based algorithm to make an irrealis dictionary to suggest sentiment modification percentage scores; and another algorithm to modify existing sentiment-lexicon based on domain-specific words. We compare UAN-OACIS with four unsupervised and two semi-supervised approaches at subtask level, and two supervised approaches at quadruples level excluding irrealis, and have got comparable results. Since no method is available for overall comparison at quintuple level, we additionally study the impact of irrealis on sentiment scores and observe substantial improvement compared to state-of-the-art lexicons methods.

AI-empowered consumer behavior analysis for trustworthy track recommendation over musical dance electronic products

—In the digital music era, accurate and trustworthy track recommendations for musical dance electronic products are becoming increasingly important to improve user experiences and attract more consumers. Consumer behavior modeling is critical in user interest learning and has been extensively used in recommender systems to improve recommendation accuracy. This paper proposes a novel AI-empowered consumer behavior analysis method for trustworthy track recommendations over musical dance electronic products. Specifically, we first model consumer behavior by integrating collaborative filtering and a hidden Markov model to capture the key interactive patterns between consumers and musical dance electronic products. Then, we develop a trustworthy track recommendation method based on multi-layer attention representation learning, which leverages scattering transform for audio preprocessing and attention-based independent recurrent neural networks for encoding user preferences and product features. Extensive experiments on real-world datasets demonstrate the superiority of our proposed method in terms of recommendation accuracy and trustworthiness.

Fusion of Multi-Layer Attention Mechanisms and CNN-LSTM for Fault Prediction in Marine Diesel Engines

Timely and effective maintenance is imperative to minimize operational disruptions and ensure the reliability of marine vessels. However, given the low early warning rates and poor adaptability under complex conditions of previous data-driven fault prediction methods, this paper presents a hybrid deep learning model based on multi-layer attention mechanisms for predicting faults in a marine diesel engine. Specifically, this hybrid model first introduces a Convolutional Neural Network (CNN) and self-attention to extract local features from multi-feature input sequences. Then, we utilize Long Short-Term Memory (LSTM) and multi-head attention to capture global correlations across time steps. Finally, the hybrid deep learning model is integrated with the Exponential Weighted Moving Average (EWMA) to monitor the operational status and predict potential faults in the marine diesel engine. We conducted extensive evaluations using real datasets under three operating conditions. The experimental results indicate that the proposed method outperforms the current state-of-the-art methods. Moreover, ablation studies and visualizations highlight the importance of fusing multi-layer attention, and the results under various operating conditions and application scenarios demonstrate that this method possesses predictive accuracy and broad applicability. Hence, this approach can provide decision support for condition monitoring and predictive maintenance of marine mechanical systems.

FSN-YOLO: Nearshore Vessel Detection via Fusing Receptive-Field Attention and Lightweight Network

Vessel detection is critical for ensuring maritime transportation and navigational safety, creating a pressing need for detection methodologies that are more efficient, precise, and intelligent in the maritime domain. Nonetheless, accurately detecting vessels across multiple scales remains challenging due to the diversity in vessel types and locations, similarities in ship hull shapes, and disturbances from complex environmental conditions. To address these issues, we introduce an innovative FSN-YOLO framework that utilizes enhanced YOLOv8 with multi-layer attention feature fusion. Specifically, FSN-YOLO employs the backbone structure of FasterNet, enriching feature representations through super-resolution processing with a lightweight Convolutional Neural Network (CNN), thereby achieving a balance between processing speed and model size without compromising accuracy. Furthermore, FSN-YOLO uses the Receptive-Field Attention (RFA) mechanism to adaptively fine-tune the feature responses between channels, significantly boosting the network’s capacity to capture critical information and, in turn, improve the model’s overall performance and enrich the discriminative feature representation of ships. Experimental validation on the Seaship7000 dataset showed that, compared to the baseline YOLOv8l approach, FSN-YOLO considerably increased accuracy, recall rates, and mAP@0.5:0.95 by absolute margins of 0.82%, 1.54%, and 1.56%, respectively, positioning it at the forefront of current state-of-the-art models.

Intelligent recommendation system for College English courses based on graph convolutional networks

With the rapid development of international communication, the number of English courses has shown an explosive growth trend, which has caused a serious problem of information overload, resulting in poor teaching performance of recommended English courses. To solve this problem, this paper proposes a graph convolutional neural network model based on College English course texts, students’ major, English foundation and network structure characteristics. First, by analyzing the relevant data of College English courses and combining with graph neural network, an English course recommendation algorithm model based on the College English learning strategy of proximity comparison is proposed. Then, the College English texts are taken as feature input, and multi-layer graph convolutional neural network is used to process the above graph neural network structure. Attention mechanism is introduced to enhance the representation of graph features in College English skills. Finally, multi-layer attention model is used to process the courses that users have learned, and intelligent course recommendation is made by combining the multi-layer attention modeling of College English skills. The experimental data show that the proposed method achieves the best performance compared with the commonly used College English course recommendation method.

MAGRU: Multi-layer Attention with GRU for Logistics Warehousing Demand Prediction

MAGRU: Multi-layer Attention with GRU for Logistics Warehousing Demand Prediction

Local highlight and shadow adaptively repairing GAN for illumination-robust makeup transfer

Recently, makeup transfer task has been widely explored with the development of deep learning. However, existing methods have shortcomings in more complex lighting situations in the real world because they do not consider the interference of lighting factors on facial features. To solve the above problem, we propose a local highlight and shadow adaptively repairing GAN for illumination-robust makeup transfer. We first map the 2D face images to UV representations and perform makeup transfer in the UV texture space, which explicitly removes the spatial misalignment to achieve pose and expression invariant makeup transfer. Furthermore, we take advantage of the face symmetry in the UV texture space to design an illumination repair module. It can adaptively repair the features affected by asymmetric local highlight and shadow based on a process of flipping and multi-layer attention fusion. In addition, the multi-layer attention maps are obtained by a pre-trained illumination classification network and hence have the ability to indicate local highlight and shadow areas. Comprehensive experiment results demonstrate the consistent effectiveness and clear advantages of our method, which significantly improve the robustness against local light effects and generate natural transfer results.

Frequency and spatial based multi-layer context network (FSCNet) for remote sensing scene classification

Remote Sensing Scene Classification (RSSC) is an important and challenging research topic due to the variety of land cover sizes and spatial combinations, as well as significant interclass similarity and intraclass variability. Currently, convolutional neural network (CNN)-based methods have been widely used in RSSC tasks with significant results. However, CNNs lack the ability to obtain long-term correlations. Transformer addressed this problem, thanks to the global receptive field of multi-head self-attention (MSA). Nevertheless, the vanilla transformer also needs further improvement to accommodate the diverse in type and scale of objects in RS scenes. In addition, the existing RSSC methods either use the last layer features, which is not conducive to process multi-scale remote sensing images, or directly fuse the multi-layer features, which will bring redundant or mutually exclusive information. To address the above issues, a novel RSSC framework, named frequency and spatial based multi-layer attention network (FSCNet) for remote sensing scene classification is proposed in this article. First, to fully extract the pyramid multi resolution features of CNN, a cross resolution injection model (CRIM) is proposed. Second, to generate better understand of the multilevel features, a frequency and spatial MLP (FS-MLP) is designed. Third, in order to aggregate contextual relations among multi-layer features, a multi-layer context align attention (MCAA) is adopted. The final classification is integration of top-layer feature and aggregated multi-layer feature. The experiment results on three well-known RS scene classification datasets (UCM, AID, and NWPU) prove the effectiveness of FSCNet and it outperforms many state-of-the-art methods.

A Hotspot-Aware Personalized News Recommendation Mechanism Based on DistilBERT-TC-MA

Aiming at the problems of existing news recommendation methods, such as inadequate exploration of the semantic information of news, neglecting potential hotspot features of news, and challenging the balance between user preferences and hotspot features, a hotspot-aware personalized news recommendation model (DistilBERT-TC-MA) is suggested, which integrates the distilled version of BERT (DistilBERT), text convolutional neural network (TextCNN), and multilayer attention (MA). First, it takes full advantage of DistilBERT, TextCNN, and self-attention mechanism to achieve news encoding. Following this, representations of trending news are dynamically aggregated using the attention mechanism, while user preferences are mined utilizing user click history. Finally, in order to successfully accomplish the click prediction of candidate news, the hotspot features, user preferences, and candidate news are ultimately combined using a click predictor. The experimental results of the suggested DistilBERT-TC-MA model on MIND dataset are better than several other advanced methods.

Time-Series Forecasting Based on Multi-Layer Attention Architecture

Time-Series Forecasting Based on Multi-Layer Attention Architecture

A Multimodal Sentiment Analysis Method Integrating Multi-Layer Attention Interaction and Multi-Feature Enhancement

To address issues related to the insufficient representation of text semantic information and the lack of deep fusion between internal modal information and intermodal information in current multimodal sentiment analysis (MSA) methods, a new method integrating multi-layer attention interaction and multi-feature enhancement (AM-MF) is proposed. First, multimodal feature extraction (MFE) is performed based on RoBERTa, ResNet, and ViT models for text, audio, and video information, and high-level features of the three modalities are obtained through self-attention mechanisms. Then, a cross modal attention (CMA) interaction module is constructed based on transformer, achieving feature fusion between different modalities. Finally, the use of a soft attention mechanism for the deep fusion of internal and intermodal information effectively achieves multimodal sentiment classification. The experimental results CH-SIMS and CMU-MOSEI datasets show that the classification results of proposed MSA method are significantly superior to other advanced comparative methods.

BVMHA: Text classification model with variable multihead hybrid attention based on BERT

Text classification is an important tasks in natural language processing. Multilayer attention networks have achieved excellent performance in text classification tasks, but they also face challenges such as high temporal and spatial complexity levels and low-rank bottleneck problems. This paper incorporates spatial attention into a neural network architecture that utilizes fewer encoder layers. The proposed model aims to enhance the spatial information of semantic features while addressing the high temporal and spatial demands of traditional multilayer attention networks. This approach utilizes spatial attention to selectively weigh the relevance of the spatial locations in the input feature maps, thereby enabling the model to focus on the most informative regions while ignoring the less important regions. By incorporating spatial attention into a shallower encoder network, the proposed model achieves improved performance on spatially oriented tasks while reducing the computational overhead associated with deeper attention-based models. To alleviate the low-rank bottleneck problem of multihead attention, this paper proposes a variable multihead attention mechanism, which changes the number of attention heads in a layer-by-layer manner with the encoder, achieving a balance between expression power and computational efficiency. We use two Chinese text classification datasets and an English sentiment classification dataset to verify the effectiveness of the proposed model.

Learning Multi-Layer Attention Aggregation Siamese Network for Robust RGBT Tracking

Learning Multi-Layer Attention Aggregation Siamese Network for Robust RGBT Tracking

Research on Innovative Teaching Path of English Literature in Colleges and Universities Based on Multimodal Discourse Analysis

Abstract English literature teaching can improve students’ literary literacy and English proficiency. In this paper, we preprocess the English literature teaching data in colleges and universities from the two modalities of visual information and auditory information, extract the features of auditory modality, text modality, and video modality of the classroom respectively, and represent the features using bidirectional long-time memory neural network. A multilayer attention network mechanism is introduced to complete multimodal fusion in English literature teaching and label multimodal features after the extraction is done. On this basis, a student-centered innovative teaching path for English literature in colleges and universities is constructed, and teaching practice and multimodal discourse analysis are carried out to explore its teaching effect. The results show that the length of time used for facial smiles in inefficient classrooms (780.52/s) is significantly more than that in inefficient classrooms (150.22/s), and the length of time used for eye interactions (832.63/s) is significantly more than that in inefficient classrooms (720.44/s), and the length of time used for animations (450.42/s) is significantly more than that in inefficient classrooms (128.88/s), and efficient classrooms are more emphasized on student interactions. Students’ reading and writing abilities were significantly different from those before the practice (P<0.05). This study suggests that promoting innovation in teaching English literature in colleges and universities can have a positive practical significance.

Rolling bearing fault diagnosis method using time-frequency information integration and multi-scale TransFusion network

Advances in deep learning methods have demonstrated remarkable development in diagnosing faults of rotating machinery. The currently popular deep neural networks suffer from design flaws in their network structure, leading to issues of long-term dependencies in fault diagnosis models built upon conventional deep neural networks. Consequently, such models exhibit insufficient global perceptual capabilities towards fault features. Furthermore, how accurately pre-trained models can diagnose faults is hugely impacted by changes in bearings’ working conditions. To tackle the aforementioned issues, this study puts forth a multi-scale TransFusion (MSTF) model for diagnosing faults in rolling bearings under multiple operating conditions. Firstly, a time-frequency symmetric dot pattern transformation technique is designed to transform the original vibration signals into two-dimensional representations. This method can effectively highlight the distinctions between different fault types. Secondly, a multi-scale feature fusion module is established, which fully extracts low-level features from the time-frequency signals and reduces the complexity of the subsequent attention calculations. Meanwhile, relying on the advantages of the Transformer model in capturing global dependencies, the long-range periodic fault information is deeply mined. Finally, multi-head and multi-layer attention are visualized to enhance the interpretability of the model. After analyzing two case studies with both public and experimental datasets, the examination demonstrated that the developed model outperformed other state-of-the-art models. The diagnostic model developed in this study exhibits the ability to accurately diagnose bearing faults across multiple operating conditions while maintaining high robustness to signals contaminated with noise.

Detection of Changes in Buildings in Remote Sensing Images via Self-Supervised Contrastive Pre-Training and Historical Geographic Information System Vector Maps

The detection of building changes (hereafter ‘building change detection’, BCD) is a critical issue in remote sensing analysis. Accurate BCD faces challenges, such as complex scenes, radiometric differences between bi-temporal images, and a shortage of labelled samples. Traditional supervised deep learning requires abundant labelled data, which is expensive to obtain for BCD. By contrast, there is ample unlabelled remote sensing imagery available. Self-supervised learning (SSL) offers a solution, allowing learning from unlabelled data without explicit labels. Inspired by SSL, we employed the SimSiam algorithm to acquire domain-specific knowledge from remote sensing data. Then, these well-initialised weight parameters were transferred to BCD tasks, achieving optimal accuracy. A novel framework for BCD was developed using self-supervised contrastive pre-training and historical geographic information system (GIS) vector maps (HGVMs). We introduced the improved MS-ResUNet network for the extraction of buildings from new temporal satellite images, incorporating multi-scale pyramid image inputs and multi-layer attention modules. In addition, we pioneered a novel spatial analysis rule for detecting changes in building vectors in bi-temporal images. This rule enabled automatic BCD by harnessing domain knowledge from HGVMs and building upon the spatial analysis of building vectors in bi-temporal images. We applied this method to two extensive datasets in Liuzhou, China, to assess its effectiveness in both urban and suburban areas. The experimental results demonstrated that our proposed approach offers a competitive quantitative and qualitative performance, surpassing existing state-of-the-art methods. Combining HGVMs and high-resolution remote sensing imagery from the corresponding years is useful for building updates.

Point cloud semantic segmentation based on local feature fusion and multilayer attention network

AbstractSemantic segmentation from a three‐dimensional point cloud is vital in autonomous driving, computer vision, and augmented reality. However, current semantic segmentation does not effectively use the point cloud's local geometric features and contextual information, essential for improving segmentation accuracy. A semantic segmentation network that uses local feature fusion and a multilayer attention mechanism is proposed to address these challenges. Specifically, the authors designed a local feature fusion module to encode the geometric and feature information separately, which fully leverages the point cloud's feature perception and geometric structure representation. Furthermore, the authors designed a multilayer attention pooling module consisting of local attention pooling and cascade attention pooling to extract contextual information. Local attention pooling is used to learn local neighbourhood information, and cascade attention pooling captures contextual information from deeper local neighbourhoods. Finally, an enhanced feature representation of important information is obtained by aggregating the features from the two deep attention pooling methods. Extensive experiments on large‐scale point‐cloud datasets Stanford 3D large‐scale indoor spaces and SemanticKITTI indicate that authors network shows excellent advantages over existing representative methods regarding local geometric feature description and global contextual relationships.

Image and Text Aspect Level Multimodal Sentiment Classification Model Using Transformer and Multilayer Attention Interaction

Many existing image and text sentiment analysis methods only consider the interaction between image and text modalities, while ignoring the inconsistency and correlation of image and text data, to address this issue, an image and text aspect level multimodal sentiment analysis model using transformer and multi-layer attention interaction is proposed. Firstly, ResNet50 is used to extract image features, and RoBERTa-BiLSTM is used to extract text and aspect level features. Then, through the aspect direct interaction mechanism and deep attention interaction mechanism, multi-level fusion of aspect information and graphic information is carried out to remove text and images unrelated to the given aspect. The emotional representations of text data, image data, and aspect type sentiments are concatenated, fused, and fully connected. Finally, the designed sentiment classifier is used to achieve sentiment analysis in terms of images and texts. This effectively has improved the performance of sentiment discrimination in terms of graphics and text.