Attention Mechanism Research Articles

Private and utility enhanced intrusion detection based on attack behavior analysis with local differential privacy on IoV

In recent years, with the escalating security demands of the Internet of Vehicles (IoV), concerns over safety have intensified. To prevent security incidents and privacy breaches, IoV must address various threats promptly and effectively. The use of deep learning methods for intrusion detection in IoV has garnered widespread attention. Compared to traditional security defenses, deep learning can learn from heterogeneous data sources, enhancing the accuracy of detecting various security threats. However, current research based on deep learning primarily focuses on constructing intrusion detection models and overlooks the analysis and processing of extensive behavioral data. Moreover, model training requires access to and transmission of sensitive data, which may lead to high communication costs and potential privacy leaks. To ensure the network security of IoV, we propose FDL-IDM, an innovative behavior-analysis-based intrusion detection model leveraging differential privacy within federated learning. It extracts driving behavior spatiotemporally and employs noise perturbation pre-aggregation, reducing communication costs and ensuring privacy without compromising accuracy. Specifically, we process data from both temporal and spatial dimensions. Data are grouped based on sender identity and then sliced according to the time sequence to create state matrices that vary over time, enhancing the performance and robustness of the detection model. Next, we incorporate an attention mechanism to merge outputs from each time step and hidden layer, strengthening the time series model and reducing information loss. Lastly, in federated learning, we add noise perturbation to the uploaded parameters, reducing the risk of privacy breaches. Additionally, we employ a random scheduling strategy during training to select clients and assign an adjusted learning rate that decreases with iterations, enhancing the stability of model training. Therefore, FDL-IDM helps prevent security attacks and protect IoV privacy. Through experiments and privacy analysis, as well as tests on vehicle-level devices, FDL-IDM achieved F1-scores of 0.9751, 0.9851, and 0.9789 on three public datasets, demonstrating not only high accuracy but also robust privacy protection capabilities.

A Novel Bidirectional Long Short-Term Memory Network With Weighted Attention Mechanism for Industrial Soft Sensor Development

A Novel Bidirectional Long Short-Term Memory Network With Weighted Attention Mechanism for Industrial Soft Sensor Development

Unsupervised Monocular Depth Estimation With Channel and Spatial Attention.

Understanding 3-D scene geometry from videos is a fundamental topic in visual perception. In this article, we propose an unsupervised monocular depth and camera motion estimation framework using unlabeled monocular videos to overcome the limitation of acquiring per-pixel ground-truth depth at scale. The photometric loss couples the depth network and pose network together and is essential to the unsupervised method, which is based on warping nearby views to target using the estimated depth and pose. We introduce the channelwise attention mechanism to dig into the relationship between channels and introduce the spatialwise attention mechanism to utilize the inner-spatial relationship of features. Both of them applied in depth networks can better activate the feature information between different convolutional layers and extract more discriminative features. In addition, we apply the Sobel boundary to our edge-aware smoothness for more reasonable accuracy, and clearer boundaries and structures. All of these help to close the gap with fully supervised methods and show high-quality state-of-the-art results on the KITTI benchmark and great generalization performance on the Make3D dataset.

Vietnam Vehicle Number Recognition Based on an Improved CRNN with Attention Mechanism

Vietnam Vehicle Number Recognition Based on an Improved CRNN with Attention Mechanism

Video Summarization using Attention Mechanisms and CNN

Video Summarization using Attention Mechanisms and CNN

Optimizing cardiovascular image segmentation through integrated hierarchical features and attention mechanisms.

Cardiovascular diseases are the top cause of death in China. Manual segmentation of cardiovascular images, prone to errors, demands an automated, rapid, and precise solution for clinical diagnosis. The paper highlights deep learning in automatic cardiovascular image segmentation, efficiently identifying pixel regions of interest for auxiliary diagnosis and research in cardiovascular diseases. In our study, we introduce innovative Region Weighted Fusion (RWF) and Shape Feature Refinement (SFR) modules, utilizing polarized self-attention for significant performance improvement in multiscale feature integration and shape fine-tuning. The RWF module includes reshaping, weight computation, and feature fusion, enhancing high-resolution attention computation and reducing information loss. Model optimization through loss functions offers a more reliable solution for cardiovascular medical image processing. Our method excels in segmentation accuracy, emphasizing the vital role of the RWF module. It demonstrates outstanding performance in cardiovascular image segmentation, potentially raising clinical practice standards. Our method ensures reliable medical image processing, guiding cardiovascular segmentation for future advancements in practical healthcare and contributing scientifically to enhanced disease diagnosis and treatment.

Intelligent deep learning supports biomedical image detection and classification of oral cancer.

Oral cancer is a malignant tumor that usually occurs within the tissues of the mouth. This type of cancer mainly includes tumors in the lining of the mouth, tongue, lips, buccal mucosa and gums. Oral cancer is on the rise globally, especially in some specific risk groups. The early stage of oral cancer is usually asymptomatic, while the late stage may present with ulcers, lumps, bleeding, etc. The objective of this paper is to propose an effective and accurate method for the identification and classification of oral cancer. We applied two deep learning methods, CNN and Transformers. First, we propose a new CANet classification model for oral cancer, which uses attention mechanisms combined with neglected location information to explore the complex combination of attention mechanisms and deep networks, and fully tap the potential of attention mechanisms. Secondly, we design a classification model based on Swim transform. The image is segmented into a series of two-dimensional image blocks, which are then processed by multiple layers of conversion blocks. The proposed classification model was trained and predicted on Kaggle Oral Cancer Images Dataset, and satisfactory results were obtained. The average accuracy, sensitivity, specificity and F1-Socre of Swin transformer architecture are 94.95%, 95.37%, 95.52% and 94.66%, respectively. The average accuracy, sensitivity, specificity and F1-Score of CANet model were 97.00%, 97.82%, 97.82% and 96.61%, respectively. We studied different deep learning algorithms for oral cancer classification, including convolutional neural networks, converters, etc. Our Attention module in CANet leverages the benefits of channel attention to model the relationships between channels while encoding precise location information that captures the long-term dependencies of the network. The model achieves a high classification effect with an accuracy of 97.00%, which can be used in the automatic recognition and classification of oral cancer.

DCNeT: A disease comorbidity network-based temporal deep learning framework to predict cardiovascular risk in patients with mental disorders

Patients with mental disorders (MDs) are at higher subsequent risk of developing cardiovascular diseases (CVDs) than the general population. Early identification of cardiovascular risk in patients with MDs is beneficial for timely intervention and reducing disease burden. Recently, deep learning approaches have been increasingly applied in CVDs risk prediction. However, these methods have three major issues: 1) mostly relying on multiple types of clinical data, 2) not sufficiently mining and utilizing comorbidity patterns hidden in complex correlations among various diseases, and 3) not fully leveraging the time information, including the irregular intervals. To address these issues, we propose a disease comorbidity network-based temporal deep learning framework (DCNeT) to predict the subsequent CVDs risk for patients with MDs based on routinely collected administrative health data. Firstly, to identify the comorbidity patterns of MDs, we construct a disease comorbidity network (DCN) for MDs and apply graph embedding method to generate disease embeddings for each disease in the DCN. Then, a code attention mechanism is proposed to obtain the weight of each disease which is embedded into dense vectors based on the structure of DCN. We present a view attention mechanism to compute the attention weights of different types of features including disease embeddings, basic features, and disease indicators for generating the final representations of patients’ hospitalizations. Furthermore, to fully utilize the information on the irregular time intervals between hospitalizations, a time encoding module is designed, and the time-aware LSTM is adopted to model the irregular time intervals and capture the temporal patterns of patients’ hospitalizations. The experimental results show that DCNeT outperforms the state-of-the-art methods, with the area under the receiver operating characteristic curve of 0.7658, 0.8143, 0.8110, and 0.7839 on four datasets, respectively. The ablation experiments further demonstrate that each module of DCNeT, including the code attention, view attention, and time encoding module, contributes to its superior performance, with average improvements of 1.20 %, 1.65 %, and 1.13 % in accuracy, respectively. Our DCNeT could be utilized as an efficient framework for identifying high-risk groups of CVDs among patients with MDs that may benefit from screening and preventive strategies.

Revolutionizing connectivity: Unleashing the power of 5G wireless networks enhanced by artificial intelligence for a smarter future

In the context of 5G wireless networks, this paper addresses a crucial concern—data loss in Analog-to-Digital Converters (ADCs) embedded within Multiple Input Multiple Output (MIMO) channels, a prevalent challenge in modern wireless communication systems. Despite the integration of ADCs in MIMO configurations to mitigate this issue, their implementation introduces significant complexities. In response to this, the paper introduces an innovative and efficient approach aimed at enhancing channel estimation in 5G MIMO systems. The proposed methodology relies on a heuristic-based optimization technique within a channel prediction framework. The framework uses feedback information collected through a Cascade Fusion Serial Network (CFSN) to analyze the receiver-side error ratio and estimate the channel coefficients of MIMO systems at the transmitter. An attention mechanism is incorporated into the model design through the use of a Convolutional Neural Networks - Gated Recurrent Units (CNN-GRU) and a Multi-Scaled Stacked Auto encoder. Parameters of Attention CNN-GRU, Stacked Auto encoder, and CFSN are refined using the Whale Optimization method. The optimization objective is to minimize key metrics, including Root Mean Squared Error (RMSE), Bit Error Rate (BER), and Mean Squared Error (MSE) associated with the estimated channel. Experimental evaluations underscore the efficacy of the proposed MIMO model in the context of 5G networks. The results demonstrate improved convergence rates, heightened prediction performance, and reduced computational costs compared to existing methods. This research contributes valuable insights into addressing data loss challenges specific to 5G MIMO systems, providing a pioneering solution that integrates heuristic optimization, advanced neural network architectures, and nature-inspired algorithms for parameter tuning.

An Improved YOLOv7-Based Model for Real-Time Meter Reading with PConv and Attention Mechanisms.

With the increasing complexity of the grid meter dial, precise feature extraction is becoming more and more difficult. Many automatic recognition solutions have been proposed for grid meter readings. However, traditional inspection methods cannot guarantee detection accuracy in complex environments. So, deep-learning methods are combined with grid meter recognition. Existing recognition systems that utilize segmentation models exhibit very high computation. It is challenging to ensure high real-time performance in edge computing devices. Therefore, an improved meter recognition model based on YOLOv7 is proposed in this paper. Partial convolution (PConv) is introduced into YOLOv7 to create a lighter network. Different PConv introduction locations on the base module have been used in order to find the optimal approach for reducing the parameters and floating point of operations (FLOPs). Meanwhile, the dynamic head (DyHead) module is utilized to enhance the attention mechanism for the YOLOv7 model. It can improve the detection accuracy of striped objects. As a result, this paper achieves mAP50val of 97.87% and mAP50:90val of 62.4% with only 5.37 M parameters. The improved model's inference speed can reach 108 frames per second (FPS). It enables detection accuracy that can reach ±0.1 degrees in the grid meter.

Plane-wave medical image reconstruction based on dynamic Criss-Cross attention and multi-scale convolution.

Plane-wave imaging is widely employed in medical imaging due to its ultra-fast imaging speed. However, the image quality is compromised. Existing techniques to enhance image quality tend to sacrifice the imaging frame rate. The study aims to reconstruct high-quality plane-wave images while maintaining the imaging frame rate. The proposed method utilizes a U-Net-based generator incorporating a multi-scale convolution module in the encoder to extract information at different levels. Additionally, a Dynamic Criss-Cross Attention (DCCA) mechanism is proposed in the decoder of the U-Net-based generator to extract both local and global features of plane-wave images while avoiding interference caused by irrelevant regions. In the reconstruction of point targets, the experimental images achieved a reduction in Full Width at Half Maximum (FWHM) of 0.0499 mm, compared to the Coherent Plane-Wave Compounding (CPWC) method using 75-beam plane waves. For the reconstruction of cyst targets, the simulated image achieved a 3.78% improvement in Contrast Ratio (CR) compared to CPWC. The proposed model effectively addresses the issue of unclear lesion sites in plane-wave images.

Aspect category sentiment analysis based on multiple attention mechanisms and pre-trained models

Aspect category sentiment analysis based on multiple attention mechanisms and pre-trained models

Graph convolutional network with attention mechanism improve major depressive depression diagnosis based on plasma biomarkers and neuroimaging data

BackgroundThe absence of clinically-validated biomarkers or objective protocols hinders effective major depressive disorder (MDD) diagnosis. Compared to healthy control (HC), MDD exhibits anomalies in plasma protein levels and neuroimaging presentations. Despite extensive machine learning studies in psychiatric diagnosis, a reliable tool integrating multi-modality data is still lacking. MethodsIn this study, blood samples from 100 MDD and 100 HC were analyzed, along with MRI images from 46 MDD and 49 HC. Here, we devised a novel algorithm, integrating graph neural networks and attention modules, for MDD diagnosis based on inflammatory cytokines, neurotrophic factors, and Orexin A levels in the blood samples. Model performance was assessed via accuracy and F1 value in 3-fold cross-validation, comparing with 9 traditional algorithms. We then applied our algorithm to a dataset containing both the aforementioned protein quantifications and neuroimages, evaluating if integrating neuroimages into the model improves performance. ResultsCompared to HC, MDD showed significant alterations in plasma protein levels and gray matter volume revealed by MRI. Our new algorithm exhibited superior performance, achieving an F1 value and accuracy of 0.9436 and 94.08 %, respectively. Integration of neuroimaging data enhanced our novel algorithm's performance, resulting in an improved F1 value and accuracy, reaching 0.9543 and 95.06 %. LimitationsThis single-center study with a small sample size requires future evaluations on a larger test set for improved reliability. ConclusionsIn comparison to traditional machine learning models, our newly developed MDD diagnostic model exhibited superior performance and showed promising potential for inclusion in routine clinical diagnosis for MDD.

Multi-Head Attention in Residual Networks to Improve Coral Reef Structure Classification

Residual Networks (ResNet) mark a crucial advancement in convolutional neural network architecture, effectively tackling challenges like vanishing gradients for improved pattern detection in various image classification tasks. This study introduces a novel adaptation of the ResNet50 architecture that integrates a multi-head attention mechanism (MHA), coined MHA-ResNet50, for discerning coral reef structures within images. Strategic modifications are applied to the input of each stage, leading to the development of an MHA block, which is augmented by separable convolution. The deliberate inclusion of the MHA block at various stages in identity-block Resnet50, in adherence to multiscale gate principles, precedes its traversal through fully connected layers. Furthermore, we implemented the Stratified K-fold concept to ensure that each fold has a comparable proportion of each class. We successfully assessed the efficacy of the MHA-Resnet50 model in several MHA-block placement scenarios and saw improvements in the accuracy of coral reef structure predictions. The most optimal results were achieved by incorporating four attention blocks (MHA-ResNet50-4), yielding an accuracy rate of 85.23% in recognition of coral structure images, comprising a mere 409 images. This model showcases adaptability to small datasets while delivering commendable performance. The ResNet50 architecture undergoes enhancement in our proposed model by integrating multi-head attention, separable convolution, and multiscale gate principles. The MHA-ResNet50 model substantially advances accurately predicting coral reef structures, demonstrating adaptability to limited datasets. Future lines of this research involve digging deeper into the model design and using more significant amounts and classes of data to strengthen a more comprehensive range of generalizations.

MPSA: Multi-Position Supervised Soft Attention-based convolutional neural network for histopathological image classification

In recent years, significant achievements have been made in the field of histopathological image analysis using convolutional neural networks (CNNs). However, existing CNNs fail to fully capture the important local structures and regional information in histopathological images due to the complex tissue structures and variable pathological features present in these images. They often treat all regions equally, which further exacerbates the challenge of accurately analyzing such images. Current network model can’t extract deep layer features efficiently without guiding. To alleviate this problem, we propose a novel network model called Multi-Position Supervised Soft Attention (MPSA). MPSA adds regions of interest (RoI) labels at multiple feature layers for deep supervision, and then uses the supervised layers as soft attention to guide the learning of the classification network, enabling the network to accurately extract features of the lesion target. Additionally, we design a Multi-level Attention Feature Enhancement Module (MAFEM), which combines multiple levels of attention mechanisms to enhance the performance of the convolutional neural network in histopathological image classification. MAFEM includes spatial attention, soft attention of the main branch, and our proposed soft attention for multi-branch feature fusion. The proposed soft attention for multi-branch feature fusion aims to enhance the predictive performance of the classification model by activating relevant neurons in the diagnostic area in a highly activated state, while effectively avoiding noise activation. This innovative approach ensures that the model can focus on the most pertinent information, leading to improved classification accuracy. We conducted classification experiments on the liver cancer histopathological images dataset and the results showed that our method achieved a classification accuracy of 95.79%, indicating that it is very effective in the analysis of liver histopathological images. Our proposed network architecture has also demonstrated good generalization ability in other medical datasets, achieving a classification accuracy of 84.41% on the ultrasound carotid plaque dataset.

Construction of multi-scale feature fusion segmentation model of MRI knee images based on dual attention mechanism weighted aggregation.

Early diagnosis of knee osteoarthritis is an important area of research in the field of clinical medicine. Due to the complexity in the MRI imaging sequences and the diverse structure of cartilage, there are many challenges in the segmentation of knee bone and cartilage. Relevant studies have conducted semantic fusion processing through splicing or summing forms, which results in reduced resolution and the accumulation of redundant information. This study was envisaged to construct an MRI image segmentation model to improve the diagnostic efficiency and accuracy of different grade knee osteoarthritis by adopting the Dual Attention and Multi-scale Feature Fusion Segmentation network (DA-MFFSnet). The feature information of different scales was fused through the Multi-scale Attention Downsample module to extract more accurate feature information, and the Global Attention Upsample module weighted lower-level feature information to reduce the loss of key information. The collected MRI knee images were screened and labeled, and the study results showed that the segmentation effect of DA-MFFSNet model was closer to that of the manually labeled images. The mean intersection over union, the dice similarity coefficient and the volumetric overlap error was 92.74%, 91.08% and 7.44%, respectively, and the accuracy of the differential diagnosis of knee osteoarthritis was 84.42%. The model exhibited better stability and classification effect. Our results indicated that the Dual Attention and Multi-scale Feature Fusion Segmentation model can improve the segmentation effect of MRI knee images in mild and medium knee osteoarthritis, thereby offering an important clinical value and improving the accuracy of the clinical diagnosis.

A multifaceted architecture to Automate Essay Scoring for assessing english article writing: Integrating semantic, thematic, and linguistic representations

We propose a multifaceted architecture for automated essay scoring integrating semantic, thematic, and linguistic representations. Specifically, for the deep semantic content of the essay, we employ Convolutional Neural Network (ConvNet) and attention mechanism module to extract local semantic representation, and use ConvNet, Long Short-Term Memory network (LSTM), and attention mechanism module to capture the global semantic representation, providing the comprehensive deep semantic-facet representations. To answer the presence of prompt in the test essay, we utilize the Doc2Vec model to represent the target essay and the essay prompt as vectors and calculate their textual similarity as thematic-facet representations. In addition, we construct six categories of nine manual features, including grammatical errors, essay length, textual complexity, word complexity, sentence complexity, and the number of clauses, to capture the shallow linguistic representation that deep learning models may struggle to uncover. Finally, we integrate the multifaceted representation through the fully connected layer to automatically score essays. Experimental results on the standard public dataset ASAP and the more complex dataset CLC-FCE, which varies in standards, lengths, and topics, demonstrate that the proposed method effectively captures diverse aspects of textual representations. For the ASAP dataset, our method reduced the training time to 1862 s per essay and the inference time to 0.9778 s, achieving a 1.13 improvement in the Quadratic Weighted Kappa score over the previous best method. On the CLC-FCE dataset, our approach enhanced the Pearson's product-moment correlation coefficient and Spearman's rank correlation coefficient by 0.005 and 0.0337 respectively, surpassing prior benchmarks.

Spatiotemporal polynomial graph neural network for anomaly detection of complex systems

Internet of Things systems are developed rapidly, there is a need for anomaly detection to identify malicious data to guarantee that the systems perform under healthy condition. However, the dynamic system state is not only influenced by sequential patterns in the time dimension but also affected by other sensor channels in the spatial dimension. Although many successful models have been established in past research to handle multivariate time series data, most of these models exhibit shortcomings in capturing spatiotemporal dependencies. The Internet of Things offers a promising platform for advanced data analysis, but its unique characteristics present challenges for anomaly detection. This paper introduces a novel spatiotemporal polynomial graph neural network (STPGNN) is proposed for anomaly detection of complex systems within the Internet of Things (IoT) framework. Specifically, we propose an adaptive spatiotemporal feature modeling module that includes two developed adaptive modules: the Adaptive Temporal Module and the Spatial Module. These modules are designed to capture dynamic spatial dependencies across multiple time steps and temporal dependencies over time, respectively. The results are then output through a decoder. Importantly, the multi-head attention mechanism employed in both sub-modules can effectively explore potential spatiotemporal dependency patterns within different subspaces. The proposed model, utilizing the innovative approach of STPGNN, demonstrates superior performance compared to existing techniques, as evidenced by experimental results on the real-world test run dataset. This marks a significant advancement in the field of anomaly detection, particularly for complex systems.

Image Caption Prediction Using Deep Learning

Abstract: In the era of rapidly growing digital content, the automatic generation of image captions has gain significant attention in the field of deep learning and computer vision. Our proposed model leverages deep learning techniques, including convolutional neural networks (CNNs) for image feature extraction and recurrent neural networks (RNNs) with attention mechanisms for caption generation. To evaluate the performance of our model, we employ a comprehensive dataset containing a diverse range of images and corresponding human-generated captions. In conclusion, our deep learning-based image captioning project offers a promising solution to the challenge of automatically generating meaningful and contextually accurate descriptions for images. The potential applications of the technology are vast, and we anticipate that it will continue to various fields, including artificial intelligence, accessibility, content creation, content management and publishing across various social media platform.

MMANet: A multi-task residual network for Alzheimer's disease classification and brain age prediction

Objective: Alzheimer's disease (AD) is an irreversible neurodegenerative disease, while mild cognitive impairment (MCI) is a clinical precursor of AD, thus differentiation of AD, MCI and normal control (NC) from noninvasive magnetic resonance imaging (MRI) has positive clinical implications. Material and method: We utilize a 3D residual network to classify AD, MCI, and NC, and add a multiscale module to the original network to enhance the feature representation capability of the network, as well as a cross-dimensional attentional mechanism to enhance the network's attention to important brain regions. We experimentally verified that the network is more inclined to overestimate the brain age of patients in AD and MCI subgroups, thus proving that there is a high correlation between the brain age prediction task and the AD classification task. Therefore, we adopted a multi-task learning approach, using brain age prediction as a supplementary task for AD classification to reduce the risk of overfitting of the network during the training process. Results: Our method achieved 96.02% accuracy, 93.40% precision, 91.48% recall, and 92.24% F1 value in AD/MCI/NC classification. Conclusions: Ablation experiments confirmed that our proposed cross-dimensional attention and multiscale modules can improve the diagnostic performance of AD and MCI, and that multi-task learning in conjunction with brain age prediction can further improve the performance.