Deep Fusion Research Articles

A Remaining Useful Life Prediction Method for Insulated-Gate Bipolar Transistor Based on Deep Fusion of Nonlinear Features From Multisource Data

A Remaining Useful Life Prediction Method for Insulated-Gate Bipolar Transistor Based on Deep Fusion of Nonlinear Features From Multisource Data

Predicting Protein-Ligand Binding Affinity Using Fusion Model of Spatial-Temporal Graph Neural Network and 3D Structure-Based Complex Graph.

The investigation of molecular interactions between ligands and their target molecules is becoming more significant as protein structure data continues to develop. In this study, we introduce PLA-STGCNnet, a deep fusion spatial-temporal graph neural network designed to study protein-ligand interactions based on the 3D structural data of protein-ligand complexes. Unlike 1D protein sequences or 2D ligand graphs, the 3D graph representation offers a more precise portrayal of the complex interactions between proteins and ligands. Research studies have shown that our fusion model, PLA-STGCNnet, outperforms individual algorithms in accurately predicting binding affinity. The advantage of a fusion model is the ability to fully combine the advantages of multiple different models and improve overall performance by combining their features and outputs. Our fusion model shows satisfactory performance on different data sets, which proves its generalization ability and stability. The fusion-based model showed good performance in protein-ligand affinity prediction, and we successfully applied the model to drug screening. Our research underscores the promise of fusion spatial-temporal graph neural networks in addressing complex challenges in protein-ligand affinity prediction. The Python scripts for implementing various model components are accessible at https://github.com/ligaili01/PLA-STGCN.

SRPNet: stroke risk prediction based on two-level feature selection and deep fusion network

BackgroundStroke is one of the major chronic non-communicable diseases (NCDs) with high morbidity, disability and mortality. The key to preventing stroke lies in controlling risk factors. However, screening risk factors and quantifying stroke risk levels remain challenging.MethodsA novel prediction model for stroke risk based on two-level feature selection and deep fusion network (SRPNet) is proposed to solve the problem mentioned above. First, the two-level feature selection method is used to screen comprehensive features related to stroke risk, enabling accurate identification of significant risk factors while eliminating redundant information. Next, the deep fusion network integrating Transformer and fully connected neural network (FCN) is utilized to establish the risk prediction model SRPNet for stroke patients.ResultsWe evaluate the performance of the SRPNet using screening data from the China Stroke Data Center (CSDC), and further validate its effectiveness with census data on stroke collected in affiliated hospital of Jining Medical University. The experimental results demonstrate that the SRPNet model selects features closely related to stroke and achieves superior risk prediction performance over benchmark methods.ConclusionsSRPNet can rapidly identify high-quality stroke risk factors, improve the accuracy of stroke prediction, and provide a powerful tool for clinical diagnosis.

Enhancing environmental modeling and 3D point cloud construction for unmanned vehicles using laser SLAM and stereo vision with Convolutional Neural Networks

Abstract. This research presents an innovative SLAM algorithm that integrates Convolutional Neural Networks (CNNs) with LIDAR and stereo vision to significantly enhance the accuracy of environmental modeling and the construction of dense 3D point cloud maps in complex and dynamic surroundings. By processing pre-recorded video data and employing advanced image segmentation techniques, this study achieves a deep fusion of visual and geometric data, resulting in highly detailed and precise 3D representations of the environment. The experimental results demonstrate that this approach effectively detects and excludes dynamic objects, thereby significantly improving the overall quality, robustness, and reliability of the constructed maps. This work represents a substantial advancement in SLAM technology, particularly in its capability to model and capture intricate environmental details under varying conditions, making it a powerful tool for precise environmental mapping.

Prompt-guided bidirectional deep fusion network for referring image segmentation

Prompt-guided bidirectional deep fusion network for referring image segmentation

PFEI-Net: A profound feature exploration and interaction network for ceramic substrate surface defect detection

Ceramic substrates serve as the foundational material for numerous electronic devices, and their surface quality directly affects performance and longevity. Therefore, surface defect detection on ceramic substrates is an indispensable task during the manufacturing process. Ceramic substrate surface defects exhibit low contrast, along with intraclass difference and interclass similarity, posing substantial challenges for automated visual inspection. Existing deep learning-based methods utilize carefully crafted backbone networks and feature pyramid structures that perceive multilevel information to deal with the above challenges. However, these methods still suffer from insufficient extraction of discriminative features and low effectiveness in feature fusion, thereby hindering the detection performance. To address these issues, we propose the profound feature exploration and interaction network (PFEI-Net). Initially, we propose the discriminative feature mining backbone (DFM-backbone) to progressively explore the effective discriminative features. In the DFM-backbone, the global information grated perception module is devised to construct long-range feature dependencies and aggregate rich features, whereas the contextual semantic flexible extraction module is designed to meticulously perceive contextual semantic information in a targeted manner, enhancing the comprehensive representation of the discriminative features. Then, we propose the multipath semantic interaction guided-feature pyramid network (MSIG-FPN) to facilitate the fusion and interaction of the valuable information. In the MSIG-FPN, the hierarchical refocus–reaggregation module is constructed to refocus on task-relevant features and provide reliable guidance for the network, whereas the cross-stage semantic deep fusion module is designed to integrate multi-level contextual information deeply across stages and is employed to construct the semantic complementary paths, thereby preventing the loss of crucial features. Finally, built upon the backbone of DFM-backbone and MSIG-FPN, we establish the PFEI-Net, achieving accurate detection of surface defects on ceramic substrates. Experimental results highlight that the PFEI-Net achieves excellent performance with a remarkable 89.3 % mean average precision (mAP) and an inference time of 26.1 ms, which offers a promising and viable solution for automated surface defect detection of ceramic substrate.

Cross-coupled prompt learning for few-shot image recognition

Prompt learning based on large models shows great potential to reduce training time and resource costs, which has been progressively applied to visual tasks such as image recognition. Nevertheless, the existing prompt learning schemes suffer from either inadequate prompt information from a single modality or insufficient prompt interaction between multiple modalities, resulting in low efficiency and performance. To address these limitations, we propose a Cross-Coupled Prompt Learning (CCPL) architecture, which is designed with two novel components (i.e., Cross-Coupled Prompt Generator (CCPG) module and Cross-Modal Fusion (CMF) module) to achieve efficient interaction between visual and textual prompts. Specifically, the CCPG module incorporates a cross-attention mechanism to automatically generate visual and textual prompts, each of which will be adaptively updated using the self-attention mechanism in their respective image and text encoders. Furthermore, the CMF module implements a deep fusion to reinforce the cross-modal feature interaction from the output layer with the Image–Text Matching (ITM) loss function. We conduct extensive experiments on 8 image datasets. The experimental results verify that our proposed CCPL outperforms the SOTA methods on few-shot image recognition tasks. The source code of this project is released at: https://github.com/elegantTechie/CCPL.

The role of the O2O blended teaching model in improving the teaching effectiveness of physical education classes

Abstract The deep fusion of Internet technology and education is constantly pushing forward the reform of university education. Traditional educational ideas, concepts, and models cannot keep pace with the times, and hybrid teaching has become a new way of education in colleges and universities. To improve the teaching effect of physical education classes, the study used a blended teaching model and designed a teaching evaluation and performance prediction model under the blended teaching model based on an improved cluster analysis method and attention mechanism. The lab results indicated that under the blended teaching model, students’ performance increased by 12.89 points, and the level of skill mastery and proficiency increased by 26.52 and 28.55%, respectively, with grades more inclined to high score distribution. “Excellent” grade clustering increased by 77.71%, and “Good” grade clustering increased by 19.01%. The minimum error sum of squares of the improved clustering algorithm was 58.18 and 36.25% lower than the other two algorithms, and the clustering results were more relevant. The two-way attention mechanism algorithm predicted higher accuracy results and performed best on all four evaluation metrics, with a prediction accuracy of 98.23%, an accuracy of 98.42%, and an F1 value of 91.78%. This hybrid teaching model is more in line with the characteristics of the physical education teaching discipline, successfully cultivates students’ independent learning ability, stimulates students’ love for physical education courses, and achieves better teaching results.

Multi-Source Data Fusion for Vehicle Maintenance Project Prediction

Ensuring road safety is heavily reliant on the effective maintenance of vehicles. Accurate predictions of maintenance requirements can substantially reduce ownership costs for vehicle owners. Consequently, this field has attracted increasing attention from researchers in recent years. However, existing studies primarily focus on predicting a limited number of maintenance needs, predominantly based solely on vehicle mileage and driving time. This approach often falls short, as it does not comprehensively monitor the overall health condition of vehicles, thus posing potential safety risks. To address this issue, we propose a deep fusion network model that utilizes multi-source data, including vehicle maintenance record data and vehicle base information data, to provide comprehensive predictions for vehicle maintenance projects. To capture the relationships among various maintenance projects, we create a correlation representation using the maintenance project co-occurrence matrix. Furthermore, building on the correlation representation, we propose a deep fusion network that employs the attention mechanism to efficiently merge vehicle mileage and vehicle base information. Experiments conducted on real data demonstrate the superior performance of our proposed model relative to competitive baseline models in predicting vehicle maintenance projects.

Imaging and reconstruction of asymmetric wrinkles in carbon fibre composites using high-frequency eddy current and full matrix capture-based ultrasound

Ply wrinkles are a common defect found in Carbon Fiber Reinforced Polymer (CFRP) laminates that can lead to failure in composite structures if not correctly evaluated during manufacture stages. The mechanical performance of a structure depends on multiple parameters of a wrinkle, such as amplitude, wavelength, and asymmetry, which ideally need to all be measured to characterise the wrinkle properly. To address this issue, the authors designed and manufactured a set of complex wrinkles with various asymmetry offsets and amplitude and obtained raw data by testing them with a high-frequency eddy-current system. A unique trajectory in the complex Nyquist plot was found on asymmetric wrinkles, demonstrating the dominant influence of wrinkle asymmetry on electrical resistance. The authors also used full matrix capture ultrasound to characterise the wrinkles and measured the profiles by extracting texture phase randomness. The study revealed the superior capability of FMC UT for characterising and retrieving the profile in wrinkle coupons, while HF ECT was more adaptable for identifying the region of interest effectively. This work also implements a state-of-the-art deep fusion strategy to demonstrate a lightweight architecture can be easily used to combine multiple sources of wrinkle profiles and generate a more accurate prediction than traditional methods. Overall, the demonstrated work can be further applied to CFRPs with increasing geometric complexity and defects to facilitate the design efficiency and testing of CFRP components.

Development of an online prediction system for soil organic matter and soil moisture content based on multi-modal fusion

Realizing accurate determination of in situ soil organic matter (SOM) content and soil moisture (SM) content in the field is of great importance for improving agricultural production efficiency. However, the feature information of a single sensor is limited, and there is still a certain gap in modeling accuracy compared to traditional laboratory methods. The variations of SM in the field also interfere with data collection, limiting the application of single sensor. In order to realize the efficient detection of in-situ SOM and SM content, this study developed an online detection system for SOM and SM content based on the fusion of characteristic wavelengths, visible images and thermal imaging image features. Firstly, based on a vehicle-mounted platform, a visible-thermal imaging camera and a characteristic wavelength integration device were integrated in a deep pine plow to realize the simultaneous acquisition of in-situ soil multi-sensor data. Then, a lightweight multimodal network was constructed to obtain thermal visible light image features and characteristic wavelength features through branch networks, achieving deep fusion of different modal data and predicting SOM and SM content. Finally, output the forecasting results of SOM and SM content, and transmit the predictive information to the cloud platform for storage. It was verified that the Multi-modal proposed in this study worked best in the laboratory environment, with a predicted R2 of 0.91 and RMSE of 2.9 g/kg for SOM, and a predicted R2 of 0.92 and RMSE of 0.77 % for SM. Compared with single image or characteristic spectral data, the fusion of visible image and characteristic wavelength effectively improved the prediction accuracy of SOM. The real-time prediction of SM was realized by fusing thermal imaging data, and the elimination of the moisture effect of visible images and characteristic wavelengths was also realized with the help of deep fusion of Multi-modal network. After field validation, the R2 of the Multi-modal system was 0.84 and the RMSE was 5.0 g/kg for SOM, and the R2 of the SM was 0.88 and the RMSE was 1.03 %. Despite the differences in soil types between the validation field and the sampling field, the system still demonstrated strong generalization and achieved high accuracy prediction of in-situ SOM and SM in the field, which effectively improves the efficiency and applicability of field. The efficiency and applicability of soil testing effectively improve the efficiency and provide technical guidance for precision management in the field.

FNNGM: A neural-driven fractional-derivative multivariate fusion model for interpretable real-time CPI forecasts

Integrating models from diverse sources has attracted substantial interest in developing advanced time series forecasting technologies. However, current research lacks a comprehensive and deep fusion model to integrate multiple forecasting methodologies. To this end, this paper proposes a neural-driven fractional-derivative multivariate fusion model (FNNGM (p, n)) to assimilate the fractional-derivative dynamical system, the driving factor in grey multivariate models, and the neural network into a cohesive framework. Consequently, this fusion architecture can benefit from the synergy of the target system's dynamics, extensive exogenous information, and non-linear transformation. Additionally, FNNGM (p, n) fosters extra functionalities through its inherent memory layer and sequence decomposition, bolstering model interpretability with the visible memory mechanism and understandable model workflows. To showcase the utility of FNNGM (p, n), this paper conducts real-time monthly consumer price index (CPI) forecasts that span ten years (from 2013:08 to 2023:07), analyzing the interpretable results from FNNGM (p, n) and contrasting it against many prevailing benchmark models. The comparison results reveal FNNGM (p, n)’s highly concentrated error distributions and the minimum mean absolute percentage forecasting error (APFE), squared forecasting error (SFE), and absolute forecasting error (AFE) values of 0.22 %, 0.59, and 0.56, respectively. Furthermore, the ablation experiments are performed to explore the specific effects and compatibilities of the fusion components, validating the effectiveness of the proposed fusion approach.

IACP-DFSRA: Identification of Anticancer Peptides Based on a Dual-channel Fusion Strategy of ResCNN and Attention

Anticancer peptides (ACPs) have been widely applied in the treatment of cancer owing to good safety, rational side effects, and high selectivity. However, the number of ACPs that have been experimentally validated is limited as identification of ACPs is extremely expensive. Hence, accurate and cost-effective identification methods for ACPs are urgently needed. In this work, we proposed a deep learning-based model, named iACP-DFSRA, for ACPs identification. Specifically, we adopted two kinds of sequence embedding technologies, ProtBert_BFD pre-training language model and handcrafted features to encode protein sequences. Then, the LightGBM was used for feature selection, and the selected features were input into ResCNN and Attention mechanism, respectively, to extract local and global features. Finally, the concatenate features were deeply fused by using the Attention mechanism to allow key features to be paid more attention to by the model and make predictions by fully connected layer. The results of 10-fold cross-validation demonstrated that the iACP-DFSRA model delivered improved results in most metrics with Sp of 94.15%, Sn of 95.32%, Acc of 94.74% and MCC of 89.48% compared to the latest AACFlow model. Indeed, the iACP-DFSRA model is the only model with Acc > 90% and MCC > 80% on this independent test dataset. Furthermore, we have further demonstrated the superiority of our model on additional datasets. In addition, t-SNE and SHAP interpretation analysis demonstrated that it is crucial to use two channels for feature extraction and use the Attention mechanism for deep fusion, which helps the iACP-DFSRA to predict ACPs more effectively.

EDPNet: A Transmission Line Ice-Thickness Recognition End-Side Network Based on Efficient Dynamic Perception

Ice-covered transmission lines seriously affect the stable operation of the power system. Deploying a recognition network for measuring the ice thickness on transmission lines within a deicing robot, and controlling the robot to perform resonant deicing, is an effective solution. In order to solve the problem that the existing recognition network is not suitable for an edge device, an ice-thickness recognition network for transmission lines based on efficient dynamic perception (EDPNet) is proposed. Firstly, a lightweight multidimensional recombination convolution (LMRC) is designed to split the ordinary convolution for lightweight design and extract feature information of different scales for reorganization. Then, a lightweight deep fusion module (LDFM) is designed, which combines the attention mechanism with different features to enhance the information interaction between the encoder and decoder. Then, a new dynamic loss function is adopted in the training process to guide the model to perform refined detection of ice-covered boundaries. Finally, we count the ice pixels and calculate the ice thickness. The model is deployed on an OrangePi5 Plus edge computing board. Compared with the baseline model, the maximum ice-thickness detection error is 4.2%, the model parameters are reduced by 86.1%, and the detection speed is increased by 74.6%. Experimental results show that EDPNet can efficiently complete the task of identifying ice-covered transmission lines and has certain engineering application value.

Power data analysis and mining technology in smart grid

This study proposes a smart grid model named “GridOptiPredict”, which aims to achieve efficient analysis and processing of power system data through deep fusion of deep learning and graph neural network, so as to improve the intelligent level and overall efficiency of power grid operation. The model integrates three core functions of load forecasting, power grid state sensing and resource optimization into one, forming a closely connected and complementary framework. Through carefully designed experimental scheme, the practical value and effectiveness of “Grid OptiPredict” model are fully verified from three aspects: accuracy of load forecasting, sensitivity of power grid state sensing and efficiency of resource allocation strategy. Experimental results show that the model has significant advantages in prediction accuracy, model stability and robustness, resource optimization, security, information security, social and economic benefits and user experience.

ACDF-YOLO: Attentive and Cross-Differential Fusion Network for Multimodal Remote Sensing Object Detection

Object detection in remote sensing images has received significant attention for a wide range of applications. However, traditional unimodal remote sensing images, whether based on visible light or infrared, have limitations that cannot be ignored. Visible light images are susceptible to ambient lighting conditions, and their detection accuracy can be greatly reduced. Infrared images often lack rich texture information, resulting in a high false-detection rate during target identification and classification. To address these challenges, we propose a novel multimodal fusion network detection model, named ACDF-YOLO, basedon the lightweight and efficient YOLOv5 structure, which aims to amalgamate synergistic data from both visible and infrared imagery, thereby enhancing the efficiency of target identification in remote sensing imagery. Firstly, a novel efficient shuffle attention module is designed to assist in extracting the features of various modalities. Secondly, deeper multimodal information fusion is achieved by introducing a new cross-modal difference module to fuse the features that have been acquired. Finally, we combine the two modules mentioned above in an effective manner to achieve ACDF. The ACDF not only enhances the characterization ability for the fused features but also further refines the capture and reinforcement of important channel features. Experimental validation was performed using several publicly available multimodal real-world and remote sensing datasets. Compared with other advanced unimodal and multimodal methods, ACDF-YOLO separately achieved a 95.87% and 78.10% mAP0.5 on the LLVIP and VEDAI datasets, demonstrating that the deep fusion of different modal information can effectively improve the accuracy of object detection.

MAFNet: a two-stage multiple attention fusion network for partial-to-partial point cloud registration

Abstract 3D point cloud registration is a critical technology in the fields of visual measurement and robot automation processing. In actual large-scale industrial production, the accuracy of point cloud registration directly affects the quality of automated welding processes. However, most existing methods are confronted with serious challenges such as the failure of partial-to-partial point cloud model registration when facing robot automatic processing guidance and error analysis work. Therefore, this paper proposes a novel two-stage network architecture for point cloud registration, which aims at robot pose adjustment and visual guidance in the field of automated welding by using 3D point cloud data. Specifically, we propose a neighborhood-based multi-head attention module in the coarse registration stage. The neighborhood information of each point can be aggregated through focusing on different weight coefficients of multi-head inputs. Then the spatial structure features which is used to establish the overlapping constraint of point clouds are obtained based on above neighborhood information. In the fine registration stage, we propose the similarity matching removal module based on multiple attention fusion features to explore deeper features from different aspects. By using deep fusion features to guide the similarity calculation, the interference of non-overlapping points is removed to achieve the finer registration. Eventually, we compare and analyze the proposed method with the SOTA ones through several error metrics and overlap estimation experiments based on the ModelNet40 dataset. The test results indicate that our method, relative to other mainstream techniques, achieves lower error rates and the most superior accuracy of 98.61% and recall of 98.37%. To demonstrate the generalization performance of proposed algorithm, extensive experiments on the Stanford 3D Scanning Repository, 7-Scenes and our own scanning dataset using partially overlapping point clouds individually under clean and noisy conditions show the validity and reliability of our proposed registration network.

Recognition of vehicle license plates in highway scenes with deep fusion network and connectionist temporal classification

AbstractLicense plate recognition is crucial in Intelligent Transportation Systems (ITS) for vehicle management, traffic monitoring, and security inspection. In highway scenarios, this task faces challenges such as diversity, blurriness, occlusion, and illumination variation of license plates. This article explores Recurrent Neural Networks based on Connectionist Temporal Classification (RNN‐CTC) for license plate recognition in challenging highway conditions. Four neural network models: ResNet50, ResNeXt, InceptionV3, and SENet, all combined with RNN‐CTC are comparatively evaluated. Furthermore, a novel architecture named ResNet50 Deep Fusion Network using Connectionist Temporal Classification (ResNet50‐DFN‐CTC) is proposed. Comparative and ablation experiments are conducted using the Highway License Plate Dataset of Southeast University (HLPD‐SU). Results demonstrate the superior performance of ResNet50‐DFN‐CTC in challenging highway conditions, achieving 93.158% accuracy with a processing time of 7.91 ms, outperforming other tested models. This research contributes to advancing license plate recognition technology for real‐world highway applications under adverse conditions.

UAV image object detection based on self-attention guidance and global feature fusion

Unmanned aerial vehicle (UAV) image object detection has garnered considerable attentions in fields such as Intelligent transportation, urban management and agricultural monitoring. However, it suffers from key challenges of the deficiency in multi-scale feature extraction and the inaccuracy when processing complex scenes and small-sized targets in practical applications. To address this challenge, we propose a novel UAV image object detection network based on self-attention guidance and global feature fusion, named SGGF-Net. First, in order to optimizing feature extraction in global perspective and enhancing target localization precision, the global feature extraction module (GFEM) is introduced by exploiting the self-attention mechanism to capture and integrate long-range dependencies within images. Second, a normal distribution-based prior assigner (NDPA) is developed by measuring the resemblance between ground truth and the priors, which improves the precision of target position matching and thus handle the problem of inaccurate localization of small targets. Furthermore, we design an attention-guided ROI pooling module (ARPM) via a deep fusion strategy of multilevel features for optimizing the integration of multi-scale features and improving the quality of feature representation. Finally, experimental results demonstrate the effectiveness of the proposed SGGF-Net approach.

STSNet: A cross-spatial resolution multi-modal remote sensing deep fusion network for high resolution land-cover segmentation

Recently, deep learning models have found extensive application in high-resolution land-cover segmentation research. However, the most current research still suffers from issues such as insufficient utilization of multi-modal information, which limits further improvement in high-resolution land-cover segmentation accuracy. Moreover, differences in the size and spatial resolution of multi-modal datasets collectively pose challenges to multi-modal land-cover segmentation. Therefore, we propose a high-resolution land-cover segmentation network (STSNet) with cross-spatial resolution spatio-temporal-spectral deep fusion. This network effectively utilizes spatio-temporal-spectral features to achieve information complementary among multi-modal data. Specifically, STSNet consists of four components: (1) A high resolution and multi-scale spatial-spectral encoder to jointly extract subtle spatial-spectral features in hyperspectral and high spatial resolution images. (2) A long-term spatio-temporal encoder formulated by spectral convolution and spatio-temporal transformer block to simultaneously delineates the spatial, temporal and spectral information in dense time series Sentinel-2 imagery. (3) A cross-resolution fusion module to alleviate the spatial resolution differences between multi-modal data and effectively leverages complementary spatio-temporal-spectral information. (4) A multi-scale decoder integrates multi-scale information from multi-modal data. We utilized airborne hyperspectral remote sensing imagery from the Shenyang region of China in 2020, with a spatial resolution of 1authors declare that they have no known competm, a spectral number of 249, and a spectral resolution ≤ 5 nm, and its Sentinel dense time-series images acquired in the same period with a spatial resolution of 10 m, a spectral number of 10, and a time-series number of 31. These datasets were combined to generate a multi-modal dataset called WHU-H2SR-MT, which is the first open accessed large-scale high spatio-temporal-spectral satellite remote sensing dataset (i.e., with >2500 image pairs sized 300 m × 300 m for each). Additionally, we employed two open-source datasets to validate the effectiveness of the proposed modules. Extensive experiments show that our multi-scale spatial-spectral encoder, spatio-temporal encoder, and cross-resolution fusion module outperform existing state-of-the-art (SOTA) algorithms in terms of overall performance on high-resolution land-cover segmentation. The new multi-modal dataset will be made available at http://irsip.whu.edu.cn/resources/resources_en_v2.php, along with the corresponding code for accessing and utilizing the dataset at https://github.com/RS-Mage/STSNet.