Fusion Mechanism Research Articles

Deepfake Detection Model Combining Texture Differences and Frequency Domain Information

In recent years, public security incidents caused by deepfake technology have occurred frequently around the world, which makes an efficient and accurate deepfake detection model crucial. The existing advanced methods use the manipulation features in the image to realize the binary classification of real and fake images by training complex neural network models. However, these models rely on a single manipulation feature, and the detection accuracy of these methods will be greatly reduced when the forgery technology or image quality of the training data set and the validation data set are different. Inspired by the existing work, we propose a two-stream collaborative learning framework that combines spatial texture differences and frequency information. The average difference convolution (ADC) is designed to extract the spatial texture difference information of the image, and the gray image frequency-aware decomposition (GFAD) is used to extract the artifact information of the image in the frequency domain. At the same time, the ViT idea is combined with cross attention mechanism for feature fusion to comprehensively mine forged features in forged images. Experimental results show that the proposed model has good detection effects on three benchmark data sets. In terms of cross-dataset evaluation, the AUC on Celeb-DF dataset reaches 82.86%, which is better than the existing advanced methods.

Research on Object Detection in Complex Scenarios Based on ASA‐YOLOv5

The applications of target detection in complex scenarios cover a wide range of fields, such as pedestrian and vehicle detection in self‐driving cars, face recognition and abnormal behavior detection in security monitoring systems, hazardous materials safety detection in public transportation, and so on. These applications demonstrate the importance and the prospect of wide application of target detection techniques in solving practical problems in complex scenarios. However, in these real scenes, there are often problems such as mutual occlusion and scale change. Therefore, how to accurately identify the target in the real complex scenarios has become a big problem to be solved. In order to solve the above problem, the paper proposes a novel algorithm, Adaptive Self‐Attention‐YOLOv5 (ASA‐YOLOv5), which is built upon the YOLOv5s algorithm and demonstrates effectiveness for target identification in complex scenarios. First, the paper implements a fusion mechanism between the trunk and neck networks, enabling the fusion of features across different levels through upsampling and downsampling. This fusion process mitigates detection errors caused by feature loss. Second, the Shuffle Attention mechanism is introduced before upsampling and downsampling to suppress noise and amplify essential semantic information, further enhancing target identification accuracy. Lastly, the Adaptively Spatial Feature Fusion (ASFF) module and Receptive Field Blocks (RFBs) module are added in the head network, and it can improve feature scale invariance and expand the receptive field. The ability of the model to detect the target in the complex scene is improved effectively. Experimental results indicate a notable improvement in the model's mean Average Precision (mAP) by 2.1% on the COCO dataset and 0.7% on the SIXray dataset. The proposed ASA‐YOLOv5 algorithm can enhance the effectiveness for target detection in complex scenarios, and it can be widely used in real‐world settings. © 2024 Institute of Electrical Engineers of Japan and Wiley Periodicals LLC.

Fast-BEV: A Fast and Strong Bird's-Eye View Perception Baseline.

Recently, perception task based on Bird's-Eye View (BEV) representation has drawn more and more attention, and BEV representation is promising as the foundation for next-generation Autonomous Vehicle (AV) perception. However, most existing BEV solutions either require considerable resources to execute on-vehicle inference or suffer from modest performance. This paper proposes a simple yet effective framework, termed Fast-BEV, which is capable of performing faster BEV perception on the on-vehicle chips. Towards this goal, we first empirically find that the BEV representation can be sufficiently powerful without expensive transformer based transformation or depth representation. Our Fast-BEV consists of five parts, we innovatively propose (1) a lightweight deployment-friendly view transformation which fast transfers 2D image features to 3D voxel space, (2) a multi-scale image encoder which leverages multi-scale information for better performance, (3) an efficient BEV encoder which is particularly designed to speed up on-vehicle inference. We further introduce (4) a strong data augmentation strategy for both image and BEV space to avoid over-fitting, (5) a multi-frame feature fusion mechanism to leverage the temporal information. Among them, (1) and (3) enable Fast-BEV to be fast inference and deployment friendly on the on-vehicle chips, (2), (4) and (5) ensure that Fast-BEV has competitive performance. All these make Fast-BEV a solution with high performance, fast inference speed, and deployment-friendly on the on-vehicle chips of autonomous driving. Through experiments, on 2080Ti platform, our R50 model can run 52.6 FPS with 47.3% NDS on the nuScenes validation set, exceeding the 41.3 FPS and 47.5% NDS of the BEVDepth-R50 model (Li et al. 2022) and 30.2 FPS and 45.7% NDS of the BEVDet4D-R50 model (J. Huang and G. Huang, 2022). Our largest model (R101@900×1600) establishes a competitive 53.5% NDS on the nuScenes validation set. We further develop a benchmark with considerable accuracy and efficiency on current popular on-vehicle chips.

A novel multimodal image feature fusion mechanism:Application to rabbit liveweight estimation in commercial farms

In the realm of commercial rabbit farming, assessment of liveweight for each rabbit is crucial for production management. Traditional manual weighing methods are labor-intensive, time-consuming, and can induce stress and disease transmission. To solve this problem, this paper introduced a liveweight estimation method for rabbit utilizing near-infrared and depth image, including a segmentation stage and a weight estimation stage. Specifically, a dual-stream feature fusion mechanism was proposed to effectively integrate information from both near-infrared and depth imaging. The network was trained and validated on a dataset of 1,957 overhead images from 300 rabbits collected from a commercial farm, achieving superior performance with an R-Square (R2) of 0.95, Root Mean Square Error (RMSE) of 194.95 g, Mean Absolute Deviation (MAD) of 155.10 g, Mean Absolute Percentage Error (MAPE) of 4.08%, and Mean Coefficient of Variation (MCV) of 3.03%. Ablation experiments revealed that employing the proposed dual-stream feature fusion mechanism led to reductions of 52.87g, 38.19g, 1.01%, and 0.75% in RMSE, MAD, MAPE, and MCV, respectively, compared to not employing the mechanism. Compared to machine learning method, our proposed approach demonstrates superior performance in RMSE, MAD, MAPE, and MCV, achieving more accurate liveweight estimation for large-scale commercial farms.

BiGCAN: A novel SRS-based bidirectional graph Convolution Attention Network for dynamic user preference and next-item recommendation

Bidirectional graph modeling represents a significant emerging technology that can be effectively integrated into sequential recommender systems. This innovation enables the explicit modeling of both forward and backward interactions in dynamic user preferences and next-item recommendations. However, existing approaches, particularly unidirectional methods, suffer from critical limitations. Unidirectional models are restricted to processing information in a single direction, which limits their ability to fully capture user interaction patterns, leading to suboptimal recommendation performance. Additionally, simplistic fusion mechanisms in current methods fail to effectively learn the intricate relationships between long-term and short-term user preferences. To address these limitations, we propose a novelty of the Bidirectional Graph Convolution Attention Network (BiGCAN) to enhance recommendation quality through three key innovations. First, bidirectional graph convolution network captures long-term preferences by modeling interactions in both forward and backward directions, allowing for a more comprehensive understanding of user behavior over time. Second, time-aware bidirectional graph attention network focuses on short-term preferences by incorporating temporal dynamics, enabling the model to adapt to recent changes in user preferences. Third, Bidirectional Gated Collaborative Filtering (BiGatedCF) mechanism integrates both long-term and short-term preferences through an advanced gating mechanism that efficiently balances these interactions. The BiGatedCF module combines a bidirectional gated recurrent unit to capture complex sequential dependencies, a deep matrix factorization to learn high-quality latent representations, and a multi-layer perceptron to match scores. Empirical experiments on four benchmark datasets demonstrate that BiGCAN surpasses state-of-the-art methods in terms of Top-N recommendations and performance across varying sparsity levels.

Fractal-domain Transformer Based on Learnable Multifractal Spectrum for Chaotic Systems Classification

Conventional deep learning in the spatiotemporal-frequency domain frequently encounter challenges in terms of slow convergence rates and limited generalization, particularly for classification of chaotic systems. To address these limitations, this paper introduces a novel fractal-inspired deep network model, specifically, the Multifractal Spectrum Transformer (MFS-Transformer), grounded in learnable multifractal analysis. Initially, we put forward the conceptual framework of fractal learning, compared with traditional fractal signal processing methodologies and spatial-temporal domain learning paradigms. Subsequently, a Learnable Multifractal Spectrum (LMFS) derived from 3D spatial gridding, coupled with fractal-domain filtering, is proposed to construct the iterative learning process within the fractal domain. Further, we formulate the MFS-Transformer, an innovative architecture that integrates multi-channel embedding, LMFS, fractal-domain filtering, residual fusion mechanisms, a mixer module, and a classifier, tailored for chaotic system classification. Ultimately, we evaluate the efficacy of our model in classifying 3D chaotic systems under stringent conditions of short-term sequences and low Signal-to-Noise Ratio (SNR). Experimental outcomes underscore the substantial performance gains achieved by the MFS-Transformer, with classification accuracy enhancements of 13.34% and 5.00% over existing Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), respectively, under SNR=0dB and 32-sample sequences. These findings validate the superiority of the MFS-Transformer in addressing the complexities of chaotic system classification under complex scenarios. This research not only advances the frontier of fractal deep learning but also presents a novel perspective and methodology for tackling intricate spatiotemporal classification problems.

GUESS: GradUally Enriching SyntheSis for Text-Driven Human Motion Generation.

In this article, we propose a novel cascaded diffusion-based generative framework for text-driven human motion synthesis, which exploits a strategy named GradUally Enriching SyntheSis (GUESS as its abbreviation). The strategy sets up generation objectives by grouping body joints of detailed skeletons in close semantic proximity together and then replacing each of such joint group with a single body-part node. Such an operation recursively abstracts a human pose to coarser and coarser skeletons at multiple granularity levels. With gradually increasing the abstraction level, human motion becomes more and more concise and stable, significantly benefiting the cross-modal motion synthesis task. The whole text-driven human motion synthesis problem is then divided into multiple abstraction levels and solved with a multi-stage generation framework with a cascaded latent diffusion model: an initial generator first generates the coarsest human motion guess from a given text description; then, a series of successive generators gradually enrich the motion details based on the textual description and the previous synthesized results. Notably, we further integrate GUESS with the proposed dynamic multi-condition fusion mechanism to dynamically balance the cooperative effects of the given textual condition and synthesized coarse motion prompt in different generation stages. Extensive experiments on large-scale datasets verify that GUESS outperforms existing state-of-the-art methods by large margins in terms of accuracy, realisticness, and diversity.

Construction error control method of large-span spatial structures based on digital twin

In the process of construction of large-span spatial structures, there are many control elements, which need to carry on the dual control of internal force and shape. The traditional control method cannot measure the construction deviation completely and accurately, and there is a lag in information feedback. In this paper, spatio-temporal information is integrated into digital twin and a construction error control method is proposed for large-span spatial structures. By analyzing the fusion mechanism of spatio-temporal information and digital twin, a closed-loop control system for structural errors before, during and after construction is formed. Driven by the control architecture, the method of structure simulation analysis and real-time monitoring during construction is proposed, and the functional relationship between cable force and cable length is established. Based on point cloud data, construction error optimization detection method is proposed for structure formation state. The nonlinear relation between elevation change rate and cable force deviation is formed and the control measures of formation state error are obtained. The experimental model of cable truss structure is taken as an example to apply and verify the theoretical method. The construction error control twin platform is formed in the experiment. Driven by the platform, the twinning analysis and real-time monitoring of the construction process are carried out to control the error of the structure formation. The results show that this method integrates spatio-temporal information into digital twin to realize data integration processing. The optimized error detection method can save 30% time cost and effectively control the construction error within 3%. The integration of spatio-temporal information and digital twin provides theoretical and algorithm support for the intelligent management and control of large-span spatial structure construction.

A deep learning-enabled visual-inertial fusion method for human pose estimation in occluded human-robot collaborative assembly scenarios

In the context of human-centric smart manufacturing, human-robot collaboration (HRC) systems leverage the strengths of both humans and machines to achieve more flexible and efficient manufacturing. In particular, estimating and monitoring human motion status determines when and how the robots cooperate. However, the presence of occlusion in industrial settings seriously affects the performance of human pose estimation (HPE). Using more sensors can alleviate the occlusion issue, but it may cause additional computational costs and lower workers' comfort. To address this issue, this work proposes a visual-inertial fusion-based method for HPE in HRC, aiming to achieve accurate and robust estimation while minimizing the influence on human motion. A part-specific cross-modal fusion mechanism is designed to integrate spatial information provided by a monocular camera and six Inertial Measurement Units (IMUs). A multi-scale temporal module is developed to model the motion dependence between frames at different granularities. Our approach achieves 34.9 mm Mean Per Joint Positional Error (MPJPE) on the TotalCapture dataset and 53.9 mm on the 3DPW dataset, outperforming state-of-the-art visual-inertial fusion-based methods. Tests on a synthetic-occlusion dataset further validate the occlusion robustness of our network. Quantitative and qualitative experiments on a real assembly case verified the superiority and potential of our approach in HRC. It is expected that this work can be a reference for human motion perception in occluded HRC scenarios.

A new data-driven production scheduling method based on digital twin for smart shop floors

As a mainstream means for solving smart shop floor production scheduling problems, the data-driven scheduling method has gained considerable attention in recent years. However, extant studies have primarily utilized physical shop floor data with limited quantity and quality to train scheduling models, which suffer from the drawbacks of long training time and poor scheduling performance. Therefore, this study proposes a new data-driven scheduling method based on digital twin for smart shop floors, which utilizes the data from physical shop floor and digital shop floor constructed by digital twin to train scheduling models. Specifically, in this method, a model-level data fusion mechanism is designed to achieve the fusion and complementary advantages of these two types of data, thus providing sufficient and high-quality data support for high-precision model training. Additionally, a multi-layer feedforward neural network with a generative adversarial network-based sample expansion mechanism is further integrated to efficiently generate scheduling decisions. Experiments in a semiconductor production shop floor are conducted to confirm the effectiveness of the proposed method.

Unraveling dual fusion mechanisms in BmNPV GP64: critical roles of CARC motifs and signal peptide retention.

Viral membrane fusion is a critical process enabling viruses to invade host cells, driven by viral membrane fusion proteins (MFPs). Cholesterol plays a pivotal role in this process, which is essential for the infectivity of many enveloped viruses. The interaction between MFPs and cholesterol is often facilitated by specific amino acid motifs known as cholesterol recognition/interaction amino acid consensus (CRAC) motifs and reverse CARC motifs. In a previous study, we demonstrated that CRAC1 and CRAC2 in GP64 are required for Bombyx mori nucleopolyhedrovirus (BmNPV) infection. This study further investigates the role of CARC in the GP64 protein of BmNPV, revealing their complex interaction with cholesterol and the influence of signal peptide (SP) retention on viral infectivity. We identified six putative CARC motifs in GP64 and generated mutants to assess their function. Our findings show that CARC1, CARC2, CARC3, and CARC4 are indispensable for viral fusion and infection when the SP is retained, whereas only CARC2 and CARC3 remain essential after SP cleavage. In contrast, CARC1 and CARC4 are necessary for viral infection through a cholesterol-independent mechanism resulting from double mutations in the CRAC1 and CRAC2 motifs of GP64. These insights not only deepen our understanding of BmNPV GP64-mediated fusion but also highlight potential antiviral targets, underscoring the adaptability and resilience of viral fusion mechanisms.IMPORTANCEUnderstanding viral membrane fusion mechanisms is crucial for developing antiviral strategies. This study provides novel insights into the intricate roles of CARC and CRAC motifs in the GP64 protein of BmNPV, particularly their interaction with cholesterol and the influence of signal peptide retention. The discovery that certain CARC motifs are essential for cholesterol-dependent fusion, whereas others function in a cholesterol-independent context advances our understanding of viral fusion processes. These findings emphasize the potential of targeting CARC motifs for therapeutic interventions and underline the importance of cholesterol interactions in viral infections. This research not only deepens our understanding of BmNPV fusion mechanisms but also has broader implications for other enveloped viruses.

Distribution-Aware Dual-LLM Collaborative Framework for Human Preference Prediction

This paper proposes a novel Distribution-aware Dual-LLM Collaborative Framework (D2CF) for human preference prediction in large language model dialogue systems. Through data analysis in the Kaggle LMSYS Chatbot Arena competition, we innovatively selected two complementary base models: Gemma-2-9b and Llama-3.1-8b. The framework's main technical innovations include: (1) A model complementarity quantification method based on Wasserstein distance, optimizing model selection from a data distribution perspective; (2) A parameter-efficient QLoRA improvement strategy that reduced computational overhead by 42.6% through adaptive rank adjustment and quantization optimization; (3) A validation set-driven dynamic weight fusion mechanism that achieves adaptive feature fusion through attention mechanisms. In the competition evaluation, this solution achieved stable performance on both public and private test sets, ultimately winning a silver medal, validating the effectiveness and robustness of distribution-aware strategies in practical applications. The significant performance improvement from public to private test sets demonstrates the superiority of this method in handling different data distributions. This paper details the technical principles and implementation details of the solution, providing reproducible engineering practice references for human preference prediction tasks in large-scale language models.

Sparse Cascade: Multi-stage cascade fusion and shape dictionary guided network for segmentation of microscopic cytology images

Microscopic cytology image instance segmentation is critical for clinical diagnosis, treatment planning, and biomedical research. However, challenges such as cell overlapping, external interferences, and irregular shapes hinder accurate segmentation. In this paper, a microscopic cytology image instance segmentation algorithm based on a multi-stage cascaded fusion and shape-guided strategy is proposed to improve instance segmentation accuracy. To mitigate the issue of external interference and cell overlap, a feedforward fusion mechanism is introduced in the multi-stage cascade architecture to effectively integrate the mask features from multiple stages. To make the shape of the segmentation result more consistent with the shape of the cell, and to alleviate the issues of unclear boundaries and incomplete shape segmentation caused by cellular overlap and irregular shapes, the sparse code guidance branch based on the explicit shape guidance strategy is introduced. Experiments on the private embryonic cell dataset and the ISBI2014 dataset indicate that the network has achieved improvements over the Cascade Mask R-CNN, with enhancements of 2.51% and 3.92% in bounding boxes mean Average Precision (mAP), and 2.82% and 4.70% in segmentation mAP, respectively. The experimental results indicate that the network proposed in this paper can output more accurate bounding boxes and instance masks.

VDMNet: A Deep Learning Framework with Vessel Dynamic Convolution and Multi-Scale Fusion for Retinal Vessel Segmentation

Retinal vessel segmentation is crucial for diagnosing and monitoring ophthalmic and systemic diseases. Optical Coherence Tomography Angiography (OCTA) enables detailed imaging of the retinal microvasculature, but existing methods for OCTA segmentation face significant limitations, such as susceptibility to noise, difficulty in handling class imbalance, and challenges in accurately segmenting complex vascular morphologies. In this study, we propose VDMNet, a novel segmentation network designed to overcome these challenges by integrating several advanced components. Firstly, we introduce the Fast Multi-Head Self-Attention (FastMHSA) module to effectively capture both global and local features, enhancing the network’s robustness against complex backgrounds and pathological interference. Secondly, the Vessel Dynamic Convolution (VDConv) module is designed to dynamically adapt to curved and crossing vessels, thereby improving the segmentation of complex morphologies. Furthermore, we employ the Multi-Scale Fusion (MSF) mechanism to aggregate features across multiple scales, enhancing the detection of fine vessels while maintaining vascular continuity. Finally, we propose Weighted Asymmetric Focal Tversky Loss (WAFT Loss) to address class imbalance issues, focusing on the accurate segmentation of small and difficult-to-detect vessels. The proposed framework was evaluated on the publicly available ROSE-1 and OCTA-3M datasets. Experimental results demonstrated that our model effectively preserved the edge information of tiny vessels and achieved state-of-the-art performance in retinal vessel segmentation across several evaluation metrics. These improvements highlight VDMNet’s superior ability to capture both fine vascular details and overall vessel connectivity, making it a robust solution for retinal vessel segmentation.

Enhanced multi-scale trademark element detection using the improved DETR

The exponential growth in the number of registered trademarks, coupled with the escalating incidents of trademark infringement, has made the automatic detection of such infractions a crucial area of study in the domain of market regulation. In light of the diverse range of elements and the pervasive presence of small targets in trademark images, we present an enhanced version of the DETR-based Multi-Scale Trademark Element Detection Network (MSTED-Net). Our primary innovation lies in incorporating a dual fusion mechanism that integrates the Spatial Attention Module (SAM) and Global Context Network (GCNet) within the backbone network, thereby providing a more robust approach to capture the essential characteristics of the trademark images under investigation. Subsequently, we develop a Multi-scale Feature Augmentation Pyramid (MFA-FPN), which aims to further fortify the model’s ability to extract features and boost the detection efficiency for small targets. The efficacy of our proposed detection network is demonstrated through experimental results, showcasing an outstanding detection accuracy of 91.12% in comparison to other state-of-the-art detection algorithms.

PVT-MA: pyramid vision transformers with multi-attention fusion mechanism for polyp segmentation

PVT-MA: pyramid vision transformers with multi-attention fusion mechanism for polyp segmentation

Adaptive output feedback fault-tolerant control for a class of nonlinear systems based on a sensor fusion mechanism

This paper investigates an output feedback adaptive fault-tolerant tracking control for a class of nonlinear systems with system nonlinearities, sensor failures and external disturbances, in which sensor redundancy is employed to enhance measurement reliability. A sensor fusion mechanism, together with a novel history-based weighted average algorithm is first designed to fuse all sensor outputs. Then, an adaptive controller based on the sensor fusion output, a dynamic gain and a state observer is constructed to handle all the uncertainties caused by system nonlinearities, external disturbances, sensor failures and fusion mechanism. It is shown that by using the proposed scheme, the closed-loop system is stable, the sensor fusion mechanism can eliminate the effects of faulty sensors, and the real tracking error can be driven into a small compact set mainly affected by the fusion error. Experimental results are accomplished to validate the proposed scheme.

Fostering college students’ mental well-being: the impact of social networking site utilization on emotion management and regulation

With the widespread proliferation of the Internet, social networking sites have increasingly become integrated into the daily lives of university students, leading to a growing reliance on these platforms. Several studies have suggested that this emotional dependence on social networking sites stems from unmet psychological needs. Meanwhile, social rejection has been identified as a prevalent phenomenon that exacerbates the deficiency of individual psychological needs. However, existing research on aspect-level sentiment analysis among college students within social networking sites faces challenges such as inadequate feature extraction, ineffective handling of data noise, and the neglect of complex interactions in multimodal data. To address these issues, this paper introduces a novel approach, the Multi-Granular View Dynamic Fusion Model (MVDFM), developed from both coarse-grained and fine-grained perspectives. MVDFM extracts multi-granular view features from textual and visual content, incorporating a dynamic gating self-attention mechanism. Additionally, it proposes a three-view decomposition higher-order pooling mechanism for a two-stage dynamic fusion of these features. Experimental results demonstrate the model’s effectiveness, achieving accuracy and F1 values of 78.78% and 74.48% on the Twitter-2015 dataset, and 73.89% and 72.47% on the Twitter-2017 dataset, respectively. This efficient supervision enables the extraction of deep semantic information from multimodal data generated by college students on social networking sites. The model adeptly mines pertinent information related to target aspect-based words, enhancing the efficacy of aspect-level emotion prediction. Furthermore, it facilitates an effective exploration of the intricate interplay between social rejection, monitoring on social networking sites, the fear of missing out, and dependence on social networking sites, ultimately aiding university students in regulating their emotional management.

Boosting Few-Shot Network Intrusion Detection with Adaptive Feature Fusion Mechanism

In network security, intrusion detection systems (IDSs) are essential for maintaining network integrity. Traditional IDSs primarily use supervised learning, relying on extensive datasets for effective training, which limits their ability to address rapidly evolving cyber threats, especially with limited data samples. To overcome this, prior research has applied meta-learning methods to distinguish between normal and malicious network traffic, showing promising results mainly in binary classification scenarios. However, challenges remain in model information acquisition within few-shot learning (FSL) frameworks. This study introduces a metric-based meta-learning strategy that constructs prototypes for each sample category, improving the model’s ability to manage multi-class scenarios. Additionally, we propose an Adaptive Feature Fusion (AFF) mechanism that dynamically integrates statistical features and binary data streams to extract meaningful insights from limited datasets, thereby enhancing the effectiveness of IDSs in few-shot learning contexts. By introducing a metric-based meta-learning method and the Adaptive Feature Fusion mechanism, this study provides a feasible solution for developing a high-accuracy, multi-class few-shot intrusion detection system. A series of experiments show that this approach significantly improves the effectiveness of the intrusion detection system, achieving an impressive accuracy of 97.78% in multi-class tasks, even when the sample size is reduced to just one.

Automatic Modulation Recognition Based on Multimodal Information Processing: A New Approach and Application

Automatic modulation recognition (AMR) has wide applications in the fields of wireless communications, radar systems, and intelligent sensor networks. The existing deep learning-based modulation recognition models often focus on temporal features while overlooking the interrelations and spatio-temporal relationships among different types of signals. To overcome these limitations, a hybrid neural network based on a multimodal parallel structure, called the multimodal parallel hybrid neural network (MPHNN), is proposed to improve the recognition accuracy. The algorithm first preprocesses the data by parallelly processing the multimodal forms of the modulated signals before inputting them into the network. Subsequently, by combining Convolutional Neural Networks (CNN) and Bidirectional Gated Recurrent Unit (Bi-GRU) models, the CNN is used to extract spatial features of the received signals, while the Bi-GRU transmits previous state information of the time series to the current state to capture temporal features. Finally, the Convolutional Block Attention Module (CBAM) and Multi-Head Self-Attention (MHSA) are introduced as two attention mechanisms to handle the temporal and spatial correlations of the signals through an attention fusion mechanism, achieving the calibration of the signal feature maps. The effectiveness of this method is validated using various datasets, with the experimental results demonstrating that the proposed approach can fully utilize the information of multimodal signals. The experimental results show that the recognition accuracy of MPHNN on multiple datasets reaches 93.1%, and it has lower computational complexity and fewer parameters than other models.