Convolutional Fusion Research Articles

Hierarchical Fusion of Infrared and Visible Images Based on Channel Attention Mechanism and Generative Adversarial Networks

In order to solve the problem that existing visible and infrared image fusion methods rely only on the original local or global information representation, which has the problem of edge blurring and non-protrusion of salient targets, this paper proposes a layered fusion method based on channel attention mechanism and improved Generative Adversarial Network (HFCA_GAN). Firstly, the infrared image and visible image are decomposed into a base layer and fine layer, respectively, by a guiding filter. Secondly, the visible light base layer is fused with the infrared image base layer by histogram mapping enhancement to improve the contour effect. Thirdly, the improved GAN algorithm is used to fuse the infrared and visible image refinement layer, and the depth transferable module and guided fusion network are added to enrich the detailed information of the fused image. Finally, the multilayer convolutional fusion network with channel attention mechanism is used to correlate the local information of the layered fusion image, and the final fusion image containing contour gradient information and useful details is obtained. TNO and RoadSence datasets are selected for training and testing. The results show that the proposed algorithm retains the global structure features of multilayer images and has obvious advantages in fusion performance, model generalization and computational efficiency.

Remaining Useful Life Prediction Method Based on Dual-Path Interaction Network with Multiscale Feature Fusion and Dynamic Weight Adaptation

In fields such as manufacturing and aerospace, remaining useful life (RUL) prediction estimates the failure time of high-value assets like industrial equipment and aircraft engines by analyzing time series data collected from various sensors, enabling more effective predictive maintenance. However, significant temporal diversity and operational complexity during equipment operation make it difficult for traditional single-scale, single-dimensional feature extraction methods to effectively capture complex temporal dependencies and multi-dimensional feature interactions. To address this issue, we propose a Dual-Path Interaction Network, integrating the Multiscale Temporal-Feature Convolution Fusion Module (MTF-CFM) and the Dynamic Weight Adaptation Module (DWAM). This approach adaptively extracts information across different temporal and feature scales, enabling effective interaction of multi-dimensional information. Using the Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) dataset for comprehensive performance evaluation, our method achieved RMSE values of 0.0969, 0.1316, 0.086, and 0.1148; MAPE values of 9.72%, 14.51%, 8.04%, and 11.27%; and Score results of 59.93, 209.39, 67.56, and 215.35 across four different data categories. Furthermore, the MTF-CFM module demonstrated an average improvement of 7.12%, 10.62%, and 7.21% in RMSE, MAPE, and Score across multiple baseline models. These results validate the effectiveness and potential of the proposed model in improving the accuracy and robustness of RUL prediction.

MLSNet: a deep learning model for predicting transcription factor binding sites.

Accurate prediction of transcription factor binding sites (TFBSs) is essential for understanding gene regulation mechanisms and the etiology of diseases. Despite numerous advances in deep learning for predicting TFBSs, their performance can still be enhanced. In this study, we propose MLSNet, a novel deep learning architecture designed specifically to predict TFBSs. MLSNet innovatively integrates multisize convolutional fusion with long short-term memory (LSTM) networks to effectively capture DNA-sparse higher-order sequence features. Further, MLSNet incorporates super token attention and Bi-LSTM to systematically extract and integrate higher-order DNA shape features. Experimental results on 165 ChIP-seq (chromatin immunoprecipitation followed by sequencing) datasets indicate that MLSNet consistently outperforms several state-of-the-art algorithms in the prediction of TFBSs. Specifically, MLSNet reports average metrics: 0.8306 for ACC, 0.8992 for AUROC, and 0.9035 for AUPRC, surpassing the second-best methods by 1.82%, 1.68%, and 1.54%, respectively. This research delineates the effectiveness of combining multi-size convolutional layers with LSTM and DNA shape-based features in enhancing predictive accuracy. Moreover, this study comprehensively assesses the variability in model performance across different cell lines and transcription factors. The source code of MLSNet is available at https://github.com/minghaidea/MLSNet.

TAFENet: A Two-Stage Attention-Based Feature-Enhancement Network for Strip Steel Surface Defect Detection

Strip steel serves as a crucial raw material in numerous industries, including aircraft and automobile manufacturing. Surface defects in strip steel can degrade the performance, quality, and appearance of industrial steel products. Detecting surface defects in steel strip products is challenging due to the low contrast between defects and background, small defect targets, as well as significant variations in defect sizes. To address these challenges, a two-stage attention-based feature-enhancement network (TAFENet) is proposed, wherein the first-stage feature-enhancement procedure utilizes an attentional convolutional fusion module with convolution to combine all four-level features and then strengthens the features of different levels via a residual spatial-channel attention connection module (RSC). The second-stage feature-enhancement procedure combines three-level features using an attentional self-attention fusion module and then strengthens the features using a RSC attention module. Experiments on the NEU-DET and GC10-DET datasets demonstrated that the proposed method significantly improved detection accuracy, thereby confirming the effectiveness and generalization capability of the proposed method.

Underwater target detection network based on differential routing assistance and bilateral attention synergy

Underwater target detection technology holds significant importance in both military and civilian applications of ocean exploration. However, due to the complex underwater environment, most targets are small and often obscured, leading to low detection accuracy and missed detections in existing target detection algorithms. To address these issues, we propose an underwater target detection algorithm that balances accuracy and speed. Specifically, we first propose the Differentiable Routing Assistance Sampling Network named (DRASN), where differentiable routing participates in training the sampling network but not in the inference process. It replaces the down-sampling network composed of Maxpool and convolution fusion in the backbone network, reducing the feature loss of small and occluded targets. Secondly, we proposed the Bilateral Attention Synergistic Network (BASN), which establishes connections between the backbone and neck with fine-grained information from both channel and spatial perspectives, thereby further enhancing the detection capability of targets in complex backgrounds. Finally, considering the characteristics of real frames, we proposed a scale approximation auxiliary loss function named (Aux-Loss) and modify the allocation strategy of positive and negative samples to enable the network to selectively learn high-quality anchors, thereby improving the convergence capability of the network. Compared with mainstream algorithms, our detection network achieves 82.9% in mAP@0.5 on the URPC2021 dataset, which is 9.5%, 5.7%, and 2.8% higher than YOLOv8s, RT-DETR, and SDBB respectively. The speed reaches 75 FPS and meets the requirements for real-time performance.

Abnormal behavior detection in industrial control systems based on CNN

With the widespread application of Internet of Things technology in industrial control systems, abnormal behavior detection has become a key task to ensure the safety and stable operation of the system. We propose a multi-branch convolutional fusion neural network method to improve the accuracy and efficiency of abnormal behavior detection in Internet of Things industrial control systems. This method achieves efficient detection of abnormal behavior in video data by combining ResNet152 for spatial feature extraction and GRU for temporal feature extraction. Unlike traditional methods, this method adopts a multi-branch structure, which can simultaneously capture multi-scale feature information, significantly enhancing the richness of feature expression and the accuracy of detection. Experimental results show that on the UCF-Crime dataset, the accuracy of this method reaches 85.76%, which is significantly better than that of traditional methods. In addition, on the larger UCF-101 dataset, the accuracy of this method reaches 92.21%, further verifying its excellent generalization performance. Compared with the C3D network, this method improves the accuracy by nearly 6% while maintaining a high processing speed. These results show that the proposed method has great potential in practical applications.

Coupled Spatial-Spectral Constrained Convolutional Fusion Network for Hyperspectral and Panchromatic images

Coupled Spatial-Spectral Constrained Convolutional Fusion Network for Hyperspectral and Panchromatic images

MRCFN: A multi-sensor residual convolutional fusion network for intelligent fault diagnosis of bearings in noisy and small sample scenarios

Bearing fault diagnosis is of great importance to ensure the safe and stable operation of mechanical equipment. The actual collected bearing fault signals are susceptible to strong noise interference and bearing samples for each fault state may be insufficient, which increases the difficulty of capturing effective features. Most of the existing diagnostic methods extract features from a single sensor signal for pattern recognition and fault diagnosis. The fault information provided by a single sensor is limited and incomplete, which is usually very difficult to meet the demand for accurate and reliable fault diagnosis in complex scenarios. To solve these problems, this paper proposes a multi-sensor residual convolutional fusion network (MRCFN) for intelligent fault diagnosis of bearings. Firstly, a convolutional pooling module (CPM) is coupled with the designed double ring residual module (DRRM) to rough feature extraction and deep feature mining, which not only captures the discriminative fault features from multi-sensor signal, but also avoids the performance degradation of network. Secondly, a spatial channel reconstruction module (SCRM) is further introduced to eliminate redundant information in the features and improve the network training efficiency. Finally, the presented global interactive perception fusion module (GIPFM) is connected with a classification block (CB) to globally fuse the features extracted from the acoustic and vibration signals and conduct automatic fault identification, which both can realize the complementarity and calibration of multi-sensor feature information and high precision diagnosis. The experiments and a series of comparisons are implemented on two datasets to efficaciously verify the superiority of the proposed method over five existing representative multi-sensor fusion diagnosis methods (i.e., FAC-CNN, MB-CNN, MsACNN, MRSDF, and IMSFDFL) under strong noise and small samples.

EAAC-Net: An Efficient Adaptive Attention and Convolution Fusion Network for Skin Lesion Segmentation.

Accurate segmentation of skin lesions in dermoscopic images is of key importance for quantitative analysis of melanoma. Although existing medical image segmentation methods significantly improve skin lesion segmentation, they still have limitations in extracting local features with global information, do not handle challenging lesions well, and usually have a large number of parameters and high computational complexity. To address these issues, this paper proposes an efficient adaptive attention and convolutional fusion network for skin lesion segmentation (EAAC-Net). We designed two parallel encoders, where the efficient adaptive attention feature extraction module (EAAM) adaptively establishes global spatial dependence and global channel dependence by constructing the adjacency matrix of the directed graph and can adaptively filter out the least relevant tokens at the coarse-grained region level, thus reducing the computational complexity of the self-attention mechanism. The efficient multiscale attention-based convolution module (EMA⋅C) utilizes multiscale attention for cross-space learning of local features extracted from the convolutional layer to enhance the representation of richly detailed local features. In addition, we designed a reverse attention feature fusion module (RAFM) to enhance the effective boundary information gradually. To validate the performance of our proposed network, we compared it with other methods on ISIC 2016, ISIC 2018, and PH2 public datasets, and the experimental results show that EAAC-Net has superior segmentation performance under commonly used evaluation metrics.

Fault Diagnosis Method of Special Vehicle Bearing Based on Multi-Scale Feature Fusion and Transfer Adversarial Learning.

To address the issues of inadequate feature extraction for rolling bearings, inaccurate fault diagnosis, and overfitting in complex operating conditions, this paper proposes a rolling bearing diagnosis method based on multi-scale feature fusion and transfer adversarial learning. Firstly, a multi-scale convolutional fusion layer is designed to effectively extract fault features from the original vibration signals at multiple time scales. Through a feature encoding fusion module based on the multi-head attention mechanism, feature fusion extraction is performed, which can model long-distance contextual information and significantly improve diagnostic accuracy and anti-noise capability. Secondly, based on the domain adaptation (DA) cross-domain feature adversarial learning strategy of transfer learning methods, the extraction of optimal domain-invariant features is achieved by reducing the gap in data distribution between the target domain and the source domain, addressing the call for research on fault diagnosis across operating conditions, equipment, and virtual-real migrations. Finally, experiments were conducted to verify and optimize the effectiveness of the feature extraction and fusion network. A public bearing dataset was used as the source domain data, and special vehicle bearing data were selected as the target domain data for comparative experiments on the effect of network transfer learning. The experimental results demonstrate that the proposed method exhibits an exceptional performance in cross-domain and variable load environments. In multiple bearing cross-domain transfer learning tasks, the method achieves an average migration fault diagnosis accuracy rate of up to 98.65%. When compared with existing methods, the proposed method significantly enhances the ability of data feature extraction, thereby achieving a more robust diagnostic performance.

Optimization of Remote-Sensing Image-Segmentation Decoder Based on Multi-Dilation and Large-Kernel Convolution

Land-cover segmentation, a fundamental task within the domain of remote sensing, boasts a broad spectrum of application potential. We address the challenges in land-cover segmentation of remote-sensing imagery and complete the following work. Firstly, to tackle the issues of foreground–background imbalance and scale variation, a module based on multi-dilated rate convolution fusion was integrated into a decoder. This module extended the receptive field through multi-dilated convolution, enhancing the model’s capability to capture global features. Secondly, to address the diversity of scenes and background interference, a hybrid attention module based on large-kernel convolution was employed to improve the performance of the decoder. This module, based on a combination of spatial and channel attention mechanisms, enhanced the extraction of contextual information through large-kernel convolution. A convolution kernel selection mechanism was also introduced to dynamically select the convolution kernel of the appropriate receptive field, suppress irrelevant background information, and improve segmentation accuracy. Ablation studies on the Vaihingen and Potsdam datasets demonstrate that our decoder significantly outperforms the baseline in terms of mean intersection over union and mean F1 score, achieving an increase of up to 1.73% and 1.17%, respectively, compared with the baseline. In quantitative comparisons, the accuracy of our improved decoder also surpasses other algorithms in the majority of categories. The results of this paper indicate that our improved decoder achieves significant performance improvement compared with the old decoder in remote-sensing image-segmentation tasks, which verifies its application potential in the field of land-cover segmentation.

Improved Chinese Giant Salamander Parental Care Behavior Detection Based on YOLOv8.

Optimizing the breeding techniques and increasing the hatching rate of Andrias davidianus offspring necessitates a thorough understanding of its parental care behaviors. However, A. davidianus' nocturnal and cave-dwelling tendencies pose significant challenges for direct observation. To address this problem, this study constructed a dataset for the parental care behavior of A. davidianus, applied the target detection method to this behavior for the first time, and proposed a detection model for A. davidianus' parental care behavior based on the YOLOv8s algorithm. Firstly, a multi-scale feature fusion convolution (MSConv) is proposed and combined with a C2f module, which significantly enhances the feature extraction capability of the model. Secondly, the large separable kernel attention is introduced into the spatial pyramid pooling fast (SPPF) layer to effectively reduce the interference factors in the complex environment. Thirdly, to address the problem of low quality of captured images, Wise-IoU (WIoU) is used to replace CIoU in the original YOLOv8 to optimize the loss function and improve the model's robustness. The experimental results show that the model achieves 85.7% in the mAP50-95, surpassing the YOLOv8s model by 2.1%. Compared with other mainstream models, the overall performance of our model is much better and can effectively detect the parental care behavior of A. davidianus. Our research method not only offers a reference for the behavior recognition of A. davidianus and other amphibians but also provides a new strategy for the smart breeding of A. davidianus.

Multi-Source Feature-Fusion Method for the Seismic Data of Cultural Relics Based on Deep Learning.

The museum system is exposed to a high risk of seismic hazards. However, it is difficult to carry out seismic hazard prevention to protect cultural relics in collections due to the lack of real data and diverse types of seismic hazards. To address this problem, we developed a deep-learning-based multi-source feature-fusion method to assess the data on seismic damage caused by collected cultural relics. Firstly, a multi-source data-processing strategy was developed according to the needs of seismic impact analysis of the cultural relics in the collection, and a seismic event-ontology model of cultural relics was constructed. Additionally, a seismic damage data-classification acquisition method and empirical calculation model were designed. Secondly, we proposed a deep learning-based multi-source feature-fusion matching method for cultural relics. By constructing a damage state assessment model of cultural relics using superpixel map convolutional fusion and an automatic data-matching model, the quality and processing efficiency of seismic damage data of the cultural relics in the collection were improved. Finally, we formed a dataset oriented to the seismic damage risk analysis of the cultural relics in the collection. The experimental results show that the accuracy of this method reaches 93.6%, and the accuracy of cultural relics label matching is as high as 82.6% compared with many kinds of earthquake damage state assessment models. This method can provide more accurate and efficient data support, along with a scientific basis for subsequent research on the impact analysis of seismic damage to cultural relics in collections.

STFGCN: Spatial–temporal fusion graph convolutional network for traffic prediction

Accurate traffic prediction plays a crucial role in improving traffic conditions and optimizing road utilization. Effectively capturing the multi-scale temporal dependencies and dynamic spatial dependencies is crucial for accurate traffic prediction. These features can effectively reflect complex dynamic spatial–temporal processes, which have not been comprehensively addressed in most existing research work. Motivated by this issue, the primary contribution of this paper lies in proposing a novel Spatial–Temporal Fusion Graph Neural Network (STFGCN) for accurate traffic prediction, achieved by extracting multi-scale temporal dependencies from multiple semantic environments and constructing a dynamic adaptive graph to model spatial dependencies based on temporal characteristics. Specifically, to capture the multi-scale dynamic temporal dependencies effectively, a Multi-Scale Fusion Convolution (MSFC) module is designed, in which the temporal dependencies are extracted from multiple textual environments by utilizing multi-scale convolution. In order to model dynamic spatial dependencies, a Spatial Adaptive Fusion Convolution (SAFC) module is designed by combining the recent coherence and periodicity to infer dynamic graphs, which are then fused to model dynamic spatial dependencies. Extensive experimental results on five real-world datasets demonstrate that the proposed STFGCN has superior performance. Specifically, compared with the state-of-the-art baselines, STFGCN reduced 1.2% to 16.4% in RMSE measure.

An adaptive dual graph convolution fusion network for aspect-based sentiment analysis

An adaptive dual graph convolution fusion network for aspect-based sentiment analysis

MST-DGCN: A Multi-Scale Spatio-Temporal and Dynamic Graph Convolution Fusion Network for Electroencephalogram Recognition of Motor Imagery

The motor imagery brain-computer interface (MI-BCI) has the ability to use electroencephalogram (EEG) signals to control and communicate with external devices. By leveraging the unique characteristics of task-related brain signals, this system facilitates enhanced communication with these devices. Such capabilities hold significant potential for advancing rehabilitation and the development of assistive technologies. In recent years, deep learning has received considerable attention in the MI-BCI field due to its powerful feature extraction and classification capabilities. However, two factors significantly impact the performance of deep-learning models. The size of the EEG datasets influences how effectively these models can learn. Similarly, the ability of classification models to extract features directly affects their accuracy in recognizing patterns. In this paper, we propose a Multi-Scale Spatio-Temporal and Dynamic Graph Convolution Fusion Network (MST-DGCN) to address these issues. In the data-preprocessing stage, we employ two strategies, data augmentation and transfer learning, to alleviate the problem of an insufficient data volume in deep learning. By using multi-scale convolution, spatial attention mechanisms, and dynamic graph neural networks, our model effectively extracts discriminative features. The MST-DGCN mainly consists of three parts: the multi-scale spatio-temporal module, which extracts multi-scale information and refines spatial attention; the dynamic graph convolution module, which extracts key connectivity information; and the classification module. We conduct experiments on real EEG datasets and achieve an accuracy of 77.89% and a Kappa value of 0.7052, demonstrating the effectiveness of the MST-DGCN in MI-BCI tasks. Our research provides new ideas and methods for the further development of MI-BCI systems.

MFAFNet: An innovative crack intelligent segmentation method based on multi-layer feature association fusion network

The regular and timely detection and segmentation of cracks in asphalt pavements is very important for road evaluation. However, certain problems remain to be solved, such as the difficulty of crack detection under interference from zebra stripes and dark light. A novel intelligent crack segmentation method based on a multi-layer feature association fusion network (MFAFNet) is therefore proposed in this study. Firstly, a fusion convolution with a Transformer called a feature coupling encoder (FC-Encoder) is developed in MFAFNet to enhance the model's global feature awareness and ability to capture local details. Secondly, a two-branch feature association module (TFBA-module) is proposed to obtain strong correlation information for global and local features. Next, a hybrid MLP architecture for lightweight feature decoder (H-MLP Decoder) is constructed to achieve feature recombination of the multi-level fusion information. The accuracy of crack segmentation under interference from zebra stripes and dark light is improved through the use of these three modules. Our model is compared with some well-known existing segmentation networks such as U-Net, DeepLabv3+, PSPNet, HRNet and SegFormer, and the segmentation accuracy, F1-score and IoU of the proposed MFAFNet are found to be 94.39 %, 93.26 % and 89.46 %, respectively. In addition, the segmentation efficiency reaches 21.87FPS. Our experimental results show that the proposed MFAFNet yields more effective performance in the field of asphalt pavement crack segmentation than the alternatives.

A novel spectral super-resolution network with dominant information between spatial and spectral domains

Existing spectral super-resolution (SSR) methods have achieved satisfactory performance by designing complicated deep convolution neural networks (DCNNs) to extract spectral and spatial features. However, these methods ignore the fact that the significance of spatial and spectral information in each hyperspectral image (HSI) is different, and most of them directly fuse two kinds of information with concatenation and convolution operation, which resulting in generating redundant information and have negative effects on reconstruction. To address such inadequacies, this paper proposes a novel adaptive spatial–spectral modulation network (ASSM-Net). Specifically, we propose a new adaptive feature fusion module (AFFM) to replace traditional convolutional fusion schemes. Through explicitly measuring the weights of spatial information and spectral features of HSI, AFFM can select dominant features for each pixel to modulate spatial and spectral information. Additionally, we develop a pixel-weighted aware attention (PAA) mechanism to enhance the feature interdependences for recovering finer structure information. Finally, a large number of quantitative and qualitative experiments reveal that the proposed network achieves competitive reconstruction results on three benchmark datasets (NTIRE 2022, CAVE and Harvard).

Bearing fault diagnosis based on a multiple-constraint modal-invariant graph convolutional fusion network

Multisensor data fusion method can improve the accuracy of bearing fault diagnosis, in order to address the problems of single-sensor data types and the insufficient exploration of redundancy and complementarity between different modal data in most existing multisensor data fusion methods for bearing fault diagnosis, a bearing fault diagnosis method based on a Multiple-Constraint Modal-Invariant Graph Convolutional Fusion Network (MCMI-GCFN) is proposed in this paper. Firstly, a Convolutional Autoencoder (CAE) and Squeeze-and-Excitation Block (SE block) are used to extract features of raw current and vibration signals. Secondly, the model introduces source domain classifiers and domain discriminators to capture modal invariance between different modal data based on domain adversarial training, making use of the redundancy and complementarity between multimodal data. Then, the spatial aggregation property of Graph Convolutional Neural Networks (GCN) is utilized to capture the dependency relationship between current and vibration modes with similar time step features for accurately fusing contextual semantic information. Finally, the validation is conducted on the public bearing damage current and vibration dataset from Paderborn University. The experimental results showed that the delivered fusion method achieved a bearing fault diagnosis accuracy of 99.6 %, which was about 9 %–11.4 % better than that with nonfusion methods.

Residual cosine similar attention and bidirectional convolution in dual-branch network for skin lesion image classification

Skin cancer is one of the most serious threats to human health among skin lesions. Computer-aided diagnosis methods can assist patients in identifying and detecting skin lesion types early, thereby enabling corresponding treatments. In this paper, we propose a dual-branch neural network model Conformer with Residual Cosine Similarity Attention and Bidirectional Convolutional fusion strategy, named RCSABC-Conformer. The core of this network structure comprises three parts: a Convolutional Neural Network (CNN) branch with Residual Cosine Similarity Attention (RCSA), a Transformer branch, and a Feature Couple Unit with Bidirectional Convolutional strategy (BC-FCU). The RCSA module calculates the cosine similarity value between the feature map generated by the convolutional operation and the feature map of the residual edge to assess whether their semantic information is similar. The semantic information of similar parts is weighted by exponential normalization to enhance the network's memory of similar features of the same type of skin lesion. The BC-FCU module interactively fuses local features and global representations of skin lesion images with different resolutions in the two branches. Specifically, when the global representations is integrated into local features, we introduce a new bidirectional convolution strategy to extract the feature map from both forward and backward directions, and then select the element with the smaller feature value from the two directions to fuse into local features. In this way, we can minimize the interference of the artifact features extracted by the Transformer branch on the CNN branch. In addition, taking advantage of the Transformer branch's capacity to construct global representations, our model can learn contextual semantic information of normal skin and lesion areas to enhance model robustness. We conducted experiments on three datasets, consisting of clinical and dermoscopic skin lesion images, as well as a hybrid of both. The experimental results show that RCSABC-Conformer outperforms both advanced and classical classification methods in terms of classification accuracy across all three datasets, without requiring an increase in the number of parameters and computational complexity. Compared with the baseline model, the classification accuracy of our proposed method improves by 2.40%, 5.39%, and 4.44% on the three datasets, respectively. To the best of our knowledge, this is the first study to apply an interactive fusion dual-branch network for multi-disease classification on different modalities of skin lesion databases. Code will be available at https://github.com/AlenLi817/RCSABC-Conformer.