Fusion Network Research Articles

Domain adaptation hyperspectral image fusion based on spatial-spectral domain separation

ABSTRACT Hyperspectral datasets captured by airborne are one of the important data sources in the field of hyperspectral image fusion. However, the limited number of data samples often hinders the optimal performance of deep learning-based methods on these datasets. Domain adaptation is expected to alleviate the issue of data scarcity. However, there are two issues that need to be addressed in applying domain adaptation to hyperspectral image fusion, which are the heterogeneity and spatial degradation diversity of hyperspectral data. In this paper, we propose a domain adaptation hyperspectral image fusion network based on spatial-spectral domain separation. The model, constructed with three-dimensional convolutional layers, effectively addresses the challenge of hyperspectral data heterogeneity. Compared with 2D convolution, 3D convolution is able to consider both spatial and spectral dimensions and extract spatial-spectral features efficiently. Based on this, we design a domain-separation architecture to extract domain-invariant and domain-private features in the spatial-spectral domain. The architecture achieves domain adaptation by extracting domain-invariant features from both datasets to obtain the prior knowledge in the source dataset and apply it to the target dataset. Additionally, the spatial degradation of different datasets varies due to distinct shooting conditions during the acquisition. Therefore, the fusion algorithms should be robust to various degradation scenarios. To tackle this challenge, we design the Observation Module to predict degradation information and the Degradation Information Modulation Module to apply it to the input, thereby enhancing the network’s robustness. Experiments on various datasets demonstrate that our method is qualitatively and quantitatively superior to state-of-the-art methods.

Unsupervised end-to-end multiscale neural network for multi-focus MicroLED image fusion

MicroLED has a broad application prospect in visible light communication, medical detection, and other fields, owing to its small size, high integration, and long service life. However, capturing a full-focus image during microscopic visual inspection of MicroLED is challenging due to the significant thickness of the chip. To address this problem, an end-to-end neural network named MMLFuse is proposed for MicroLED image fusion, which uses unsupervised learning to directly generate fused images from two original images. Firstly, we introduce the Spatial Pyramid Pooling Mixing (SPPM) module for rapid extraction of partially focused image features. The extracted features are then used to obtain a weight map, which is further refined using a moving window smoothing technique. This refined weight map is employed for feature fusion, and the fused image is reconstructed based on the fused features. Specifically, the network uses a two-stage training strategy with different loss functions for each stage to improve the convergence speed of the model and the quality of the fused image. In particular, mask loss is designed in the second stage to ensure that the network pays more attention to the focus area during training to accurately match the corresponding input image. Experimental results demonstrate that MMLFuse achieves superior performance on the MicroLED dataset compared to other classical methods, highlighting its effectiveness and potential in the field.

Parallel proportional fusion of a spiking quantum neural network for optimizing image classification

Parallel proportional fusion of a spiking quantum neural network for optimizing image classification

ETBHD‐HMF: A Hierarchical Multimodal Fusion Architecture for Enhanced Text‐Based Hair Design

AbstractText‐based hair design (TBHD) represents an innovative approach that utilizes text instructions for crafting hairstyle and colour, renowned for its flexibility and scalability. However, enhancing TBHD algorithms to improve generation quality and editing accuracy remains a current research difficulty. One important reason is that existing models fall short in alignment and fusion designs. Therefore, we propose a new layered multimodal fusion network called ETBHD‐HMF, which decouples the input image and hair text information into layered hair colour and hairstyle representations. Within this network, the channel enhancement separation (CES) module is proposed to enhance important signals and suppress noise for text representation obtained from CLIP, thus improving generation quality. Based on this, we develop the weighted mapping fusion (WMF) sub‐networks for hair colour and hairstyle. This sub‐network applies the mapper operations to input image and text representations, acquiring joint information. The WMF then selectively merges image representation and joint information from various style layers using weighted operations, ultimately achieving fine‐grained hairstyle designs. Additionally, to enhance editing accuracy and quality, we design a modality alignment loss to refine and optimize the information transmission and integration of the network. The experimental results of applying the network to the CelebA‐HQ dataset demonstrate that our proposed model exhibits superior overall performance in terms of generation quality, visual realism, and editing accuracy. ETBHD‐HMF (27.8 PSNR, 0.864 IDS) outperformed HairCLIP (26.9 PSNR, 0.828 IDS), with a 3% higher PSNR and a 4% higher IDS.

ECF-Net: Enhanced, Channel-Based, Multi-Scale Feature Fusion Network for COVID-19 Image Segmentation

Accurate segmentation of COVID-19 lesion regions in lung CT images aids physicians in analyzing and diagnosing patients’ conditions. However, the varying morphology and blurred contours of these regions make this task complex and challenging. Existing methods utilizing Transformer architecture lack attention to local features, leading to the loss of detailed information in tiny lesion regions. To address these issues, we propose a multi-scale feature fusion network, ECF-Net, based on channel enhancement. Specifically, we leverage the learning capabilities of both CNN and Transformer architectures to design parallel channel extraction blocks in three different ways, effectively capturing diverse lesion features. Additionally, to minimize irrelevant information in the high-dimensional feature space and focus the network on useful and critical information, we develop adaptive feature generation blocks. Lastly, a bidirectional pyramid-structured feature fusion approach is introduced to integrate features at different levels, enhancing the diversity of feature representations and improving segmentation accuracy for lesions of various scales. The proposed method is tested on four COVID-19 datasets, demonstrating mIoU values of 84.36%, 87.15%, 83.73%, and 75.58%, respectively, outperforming several current state-of-the-art methods and exhibiting excellent segmentation performance. These findings provide robust technical support for medical image segmentation in clinical practice.

Enhancing protein-ligand binding affinity prediction through sequential fusion of graph and convolutional neural networks.

Predicting protein-ligand binding affinity is a crucial and challenging task in structure-based drug discovery. With the accumulation of complex structures and binding affinity data, various machine-learning scoring functions, particularly those based on deep learning, have been developed for this task, exhibiting superiority over their traditional counterparts. A fusion model sequentially connecting a graph neural network (GNN) and a convolutional neural network (CNN) to predict protein-ligand binding affinity is proposed in this work. In this model, the intermediate outputs of the GNN layers, as supplementary descriptors of atomic chemical environments at different levels, are concatenated with the input features of CNN. The model demonstrates a noticeable improvement in performance on CASF-2016 benchmark compared to its constituent CNN models. The generalization ability of the model is evaluated by setting a series of thresholds for ligand extended-connectivity fingerprint similarity or protein sequence similarity between the training and test sets. Masking experiment reveals that model can capture key interaction regions. Furthermore, the fusion model is applied to a virtual screening task for a novel target, PI5P4Kα. The fusion strategy significantly improves the ability of the constituent CNN model to identify active compounds. This work offers a novel approach to enhancing the accuracy of deep learning models in predicting binding affinity through fusion strategies.

Advancing Skarn Iron Ore Detection through Multispectral Image Fusion and 3D Convolutional Neural Networks (3D-CNNs)

This study explores novel techniques to improve the detection accuracy of skarn iron deposits using advanced image-processing methodologies. Leveraging the capabilities of ASTER image, band ratio (BR) images, and principal component analysis (PCA) alongside the power of 3D convolutional neural networks (3D-CNNs), the research aims to enhance the precision and efficiency of ore detection in complex geological environments. The proposed method employs a specific 3D-CNN architecture accepting input as a 7 × 7 × C image patch, where C represents the combined number of selected ASTER image bands, principal component (PC) bands, and computed BR images. To evaluate the accuracy of the proposed method, five distinct image band combinations, including the proposed band combination, were tested and evaluated based on the overall accuracy (OA), average accuracy (AA), and kappa coefficient. The results demonstrated that while the incorporation of BR images alongside ASTER bands initially seemed promising, it introduced significant confusion in certain classifications, leading to unexpected misclassification rates. Surprisingly, utilizing solely ASTER bands as input parameters yielded higher accuracy rates (OA = 93.13%, AA = 91.96%, kappa = 90.91%) compared with scenarios involving the integration with band ratios (OA = 87.02%, AA = 79.15, kappa = 82.60%) or the integration of BR images to PC bands (OA = 87.78%, AA = 82.39%, kappa = 83.81%). However, the amalgamation of ASTER bands with selected PC bands showed slight improvements in accuracy (OA = 94.65%, AA = 92.93%, kappa = 93.45%), although challenges in accurately classifying certain features persisted. Ultimately, the proposed combination of ASTER bands, PC bands, and BR images (proposed band combination) presented the most visually appealing and statistically accurate results (OA = 96.95%, AA = 94.87%, kappa = 95.93%), effectively addressing misclassifications observed in the other combinations. These findings underscore the synergistic contributions of each of the ASTER bands, PC bands, and BR images, with the ASTER bands proving pivotal for optimal skarn classification, the PC bands enhancing intrusions classification accuracy, and the BR images strengthening wall rock classification accuracy. In conclusion, the proposed combination of input image bands emerges as a robust and comprehensive methodology, demonstrating unparalleled accuracy in the remote sensing detection of skarn iron minerals.

Fast Semantic Segmentation of Ultra-High-Resolution Remote Sensing Images via Score Map and Fast Transformer-Based Fusion

For ultra-high-resolution (UHR) image semantic segmentation, striking a balance between computational efficiency and storage space is a crucial research direction. This paper proposes a Feature Fusion Network (EFFNet) to improve UHR image semantic segmentation performance. EFFNet designs a score map that can be embedded into the network for training purposes, enabling the selection of the most valuable features to reduce storage consumption, accelerate speed, and enhance accuracy. In the fusion stage, we improve upon previous redundant multiple feature fusion methods by utilizing a transformer structure for one-time fusion. Additionally, our combination of the transformer structure and multibranch structure allows it to be employed for feature fusion, significantly improving accuracy while ensuring calculations remain within an acceptable range. We evaluated EFFNet on the ISPRS two-dimensional semantic labeling Vaihingen and Potsdam datasets, demonstrating that its architecture offers an exceptionally effective solution with outstanding semantic segmentation precision and optimized inference speed. EFFNet substantially enhances critical performance metrics such as Intersection over Union (IoU), overall accuracy, and F1-score, highlighting its superiority as an architectural innovation in ultra-high-resolution remote sensing image semantic segmentation.

Cotton-YOLO-Seg: An Enhanced YOLOV8 Model for Impurity Rate Detection in Machine-Picked Seed Cotton

The detection of the impurity rate in machine-picked seed cotton is crucial for precision agriculture. This study proposes a novel Cotton-YOLO-Seg cotton-impurity instance segmentation algorithm based on the you only look once version 8 small segmentation model (Yolov8s-Seg). The algorithm achieves precise pixel-level segmentation of cotton and impurities in seed cotton images and establishes a detection model for the impurity rate, enabling accurate detection of the impurity rate in machine-picked cotton. The proposed algorithm removes the Pyramid 4 (P4) feature layer and incorporates Multi-Scale Convolutional Block Attention (MSCBCA) that integrates the Convolutional Block Attention Module (CBAM) and Multi-Scale Convolutional Attention (MSCA) into the Faster Implementation of Cross Stage Partial Bottleneck with 2 Convolutions (C2f) module of the feature extraction network, forming a novel C2f_MSCBCA module. The SlimNeck structure is introduced in the feature fusion network by replacing the P4 feature layer with the small-target detection layer Pyramid 2 (P2). Additionally, transfer learning is employed using the Common Objects in Context (COCO) instance segmentation dataset. The analysis of 100 groups of cotton image samples shows that the Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Mean Absolute Percentage Error (MAPE) for impurity rate detection are 0.29%, 0.33%, and 3.70%, respectively, which are reduced by 52.46%, 48.44%, and 53.75% compared to the Yolov8s-seg model. The Precision (P), Recall (R), and mean Average Precision at an intersection over union of 0.5 (mAP@0.5) are 85.4%, 78.4%, and 80.8%, respectively, which are improved by 4.2%, 6.2%, and 6.4% compared to Yolov8s-seg model, significantly enhancing the segmentation performance of minor impurities. The Cotton-YOLO-Seg model demonstrates practical significance for precisely detecting the impurity rate in machine-picked seed cotton.

Hyperspectral Image Denoising and Compression Using Optimized Bidirectional Gated Recurrent Unit

The availability of a higher resolution fine spectral bandwidth in hyperspectral images (HSI) makes it easier to identify objects of interest in them. The inclusion of noise into the resulting collection of images is a limitation of HSI and has an adverse effect on post-processing and data interpretation. Denoising HSI data is thus necessary for the effective execution of post-processing activities like image categorization and spectral unmixing. Most of the existing models cannot handle many forms of noise simultaneously. When it comes to compression, available compression models face the problems of increased processing time and lower accuracy. To overcome the existing limitations, an image denoising model using an adaptive fusion network is proposed. The denoised output is then processed through a compression model which uses an optimized deep learning technique called "chaotic Chebyshev artificial hummingbird optimization algorithm-based bidirectional gated recurrent unit" (CCAO-BiGRU). All the proposed models were tested in Python and evaluated using the Indian Pines, Washington DC Mall and CAVE datasets. The proposed model underwent qualitative and quantitative analysis and showed a PSNR value of 82 in the case of Indian Pines and 78.4 for the Washington DC Mall dataset at a compression rate of 10. The study proved that the proposed model provides the knowledge about complex nonlinear mapping between noise-free and noisy HSI for obtaining the denoised images and also results in high-quality compressed output.

MFFNet: Multimodal feature fusion network for RGB-D transparent object detection

Transparent objects are ubiquitous in everyday life, but how to detect them is full of challenges. Transparent objects hardly reflect light, and they usually transmit the appearance of their surroundings, making it difficult to distinguish them from their surroundings. Existing methods usually use only RGB (Red Green Blue) images as input, ignoring the role of depth maps in transparent object detection. In this article, we try to improve the detection performance of transparent objects by fusing RGB and depth information. Specifically, we propose a multimodal fusion network that fuses RGB and depth modalities in a complementary way. Moreover, extensive experiments and ablation studies on the RGB-D (RGB-Depth) transparent object dataset demonstrate the excellent performance of our method.

Unsupervised Deep Tensor Network for Hyperspectral-Multispectral Image Fusion.

Fusing low-resolution (LR) hyperspectral images (HSIs) with high-resolution (HR) multispectral images (MSIs) is a significant technology to enhance the resolution of HSIs. Despite the encouraging results from deep learning (DL) in HSI-MSI fusion, there are still some issues. First, the HSI is a multidimensional signal, and the representability of current DL networks for multidimensional features has not been thoroughly investigated. Second, most DL HSI-MSI fusion networks need HR HSI ground truth for training, but it is often unavailable in reality. In this study, we integrate tensor theory with DL and propose an unsupervised deep tensor network (UDTN) for HSI-MSI fusion. We first propose a tensor filtering layer prototype and further build a coupled tensor filtering module. It jointly represents the LR HSI and HR MSI as several features revealing the principal components of spectral and spatial modes and a sharing code tensor describing the interaction among different modes. Specifically, the features on different modes are represented by the learnable filters of tensor filtering layers, the sharing code tensor is learned by a projection module, in which a co-attention is proposed to encode the LR HSI and HR MSI and then project them onto the sharing code tensor. The coupled tensor filtering module and projection module are jointly trained from the LR HSI and HR MSI in an unsupervised and end-to-end way. The latent HR HSI is inferred with the sharing code tensor, the features on spatial modes of HR MSIs, and the spectral mode of LR HSIs. Experiments on simulated and real remote-sensing datasets demonstrate the effectiveness of the proposed method.

Melatonin regulates mitochondrial dynamics and mitophagy: Cardiovascular protection.

Despite extensive progress in the knowledge and understanding of cardiovascular diseases and significant advances in pharmacological treatments and procedural interventions, cardiovascular diseases (CVD) remain the leading cause of death globally. Mitochondrial dynamics refers to the repetitive cycle of fission and fusion of the mitochondrial network. Fission and fusion balance regulate mitochondrial shape and influence physiology, quality and homeostasis. Mitophagy is a process that eliminates aberrant mitochondria. Melatonin (Mel) is a pineal-synthesized hormone with a range of pharmacological properties. Numerous nonclinical trials have demonstrated that Mel provides cardioprotection against ischemia/reperfusion, cardiomyopathies, atherosclerosis and cardiotoxicity. Recently, interest has grown in how mitochondrial dynamics contribute to melatonin cardioprotective effects. This review assesses the literature on the protective effects of Mel against CVD via the regulation of mitochondrial dynamics and mitophagy in both in-vivo and in-vitro studies. The signalling pathways underlying its cardioprotective effects were reviewed. Mel modulated mitochondrial dynamics and mitophagy proteins by upregulation of mitofusin, inhibition of DRP1 and regulation of mitophagy-related proteins. The evidence supports a significant role of Mel in mitochondrial dynamics and mitophagy quality control in CVD.

Real-Time Plant Disease Identification: Fusion of Vision Transformer and Conditional Convolutional Network With C3GAN-Based Data Augmentation

Real-Time Plant Disease Identification: Fusion of Vision Transformer and Conditional Convolutional Network With C3GAN-Based Data Augmentation

Ultrasonic guided Wave Signal Denoising Method Based on one-dimensional convolution and Full connection Model Fusion Denoising autoencoder

Abstract Ultrasonic guided wave (UGW) detection is widely used in pipeline monitoring but faces challenges from weak flaw echo signals within the detection data, making weak UGW signals difficult to recognize. It is essential to denoise the UGW detection signals to identify weak echo signals. This paper proposes an improved denoising autoencoder (DAE) based on the fusion of one-dimensional convolution neural network (1DCNN) and full connection (FC). The model expands the amount of training data by adding noise in batches and uses 1DCNN to enhance the ability of extracting UGW signal features. The model was validated using Several numerical simulation signals. Numerical simulation results show that the signal-to-noise ratio (SNR) of the UGW signals can be improved from -20 dB to 8 dB; it has a strong improved SNR, and the mean square error is greatly reduced while maintaining the original phase almost unchanged. The improved DAE method has significant advantages in denoising effect, and it can effectively reduce the noise of the UGW detection signal and realize the identification of small defects of the simulation pipeline.

Rolling bearings fault diagnosis based on two-stage signal fusion and deep multi-scale multi-sensor network

In order to realize high-precision diagnosis of bearings faults in a multi-sensor detection environment, a fault diagnosis method based on two-stage signal fusion and deep multi-scale multi-sensor networks is proposed. Firstly, the signals are decomposed and fused using weighted empirical wavelet transform to enhance weak features and reduce noise. Secondly, an improved random weighting algorithm is proposed to perform a second weighted fusion of the signals to reduce the total mean square error. The fused signals are input into the deep multi-scale residual network, the feature information of different convolutional layers is extracted through dilated convolution, and the features are fused using pyramid theory. Finally, the bearings states are classified according to the fusion features. Experiment results show the effectiveness and superiority of this method.

3D Multimodal Fusion Network with Disease-induced Joint Learning for Early Alzheimer's Disease Diagnosis.

Multimodal neuroimaging provides complementary information critical for accurate early diagnosis of Alzheimer's disease (AD). However, the inherent variability between multimodal neuroimages hinders the effective fusion of multimodal features. Moreover, achieving reliable and interpretable diagnoses in the field of multimodal fusion remains challenging. To address them, we propose a novel multimodal diagnosis network based on multi-fusion and disease-induced learning (MDL-Net) to enhance early AD diagnosis by efficiently fusing multimodal data. Specifically, MDL-Net proposes a multi-fusion joint learning (MJL) module, which effectively fuses multimodal features and enhances the feature representation from global, local, and latent learning perspectives. MJL consists of three modules, global-aware learning (GAL), local-aware learning (LAL), and outer latent-space learning (LSL) modules. GAL via a self-adaptive Transformer (SAT) learns the global relationships among the modalities. LAL constructs local-aware convolution to learn the local associations. LSL module introduces latent information through outer product operation to further enhance feature representation. MDL-Net integrates the disease-induced region-aware learning (DRL) module via gradient weight to enhance interpretability, which iteratively learns weight matrices to identify AD-related brain regions. We conduct the extensive experiments on public datasets and the results confirm the superiority of our proposed method. Our code will be available at: https://github.com/qzf0320/MDL-Net.

Multiscale Optical Imaging Fusion for Cervical Precancer Diagnosis: Integrating Widefield Colposcopy and High-Resolution Endomicroscopy.

Early detection and treatment of cervical precancers can prevent disease progression. However, in low-resource communities with a high incidence of cervical cancer, high equipment costs and a shortage of specialists hinder preventative strategies. This manuscript presents a low-cost multiscale in vivo optical imaging system coupled with a computer-aided diagnostic system that could enable accurate, real-time diagnosis of high-grade cervical precancers. The system combines portable colposcopy and high-resolution endomicroscopy (HRME) to acquire spatially registered widefield and microscopy videos. A multiscale imaging fusion network (MSFN) was developed to identify cervical intraepithelial neoplasia grade 2 or more severe (CIN 2+). The MSFN automatically identifies and segments the ectocervix and lesions from colposcopy images, extracts nuclear morphology features from HRME videos, and integrates the colposcopy and HRME information. With a threshold value set to achieve sensitivity equal to clinical impression (0.98 [p = 1.0]), the MSFN achieved a significantly higher specificity than clinical impression (0.75 vs. 0.43, p = 0.000006). Our findings show that multiscale optical imaging of the cervix allows the highly sensitive and specific detection of high-grade precancers. The multiscale imaging system and MSFN could facilitate the accurate, real-time diagnosis of cervical precancers in low-resource settings.

Cross-Dimensional Attention Fusion Network for Simulated Single Image Super-Resolution

Cross-Dimensional Attention Fusion Network for Simulated Single Image Super-Resolution

MLFF-Net: a multi-model late feature fusion network for skin disease classification

Early diagnosis is paramount to preventing skin diseases and reducing mortality, given their global prevalence. Visual detection by experts using dermoscopy images has become the gold standard for detecting skin cancer. However, a significant challenge in skin cancer detection and classification lies in the similarity of appearance among skin disease lesions and the complexity of dermoscopic images. In response, we developed multi-model late feature fusion network (MLFF-Net), a multi-model late feature fusion network tailored for skin disease detection. Our approach begins with image pre-processing techniques to enhance image quality. We then employ a two-stream network comprising an enhanced densely linked network (DenseNet-121) and a vision transformer (ViTb16). We leverage shallow and deep feature fusion, late fusion, and an attention module to enhance the model’s feature extraction efficiency. The subsequent feature fusion module constructs multi-receptive fields to capture disease information across various scales and uses generalized mean pooling (GeM) pooling to reduce the spatial dimensions of lesion characteristics. Finally, we implement and test our skin lesion categorization model, demonstrating its effectiveness. Despite the combination, convolutional neural network (CNN) outperforms ViT approaches, with our model enhancing the accuracy of the best model by 6.1%.