Image Fusion Framework Research Articles

Adaptive Transformer-Based Multi-Modal Image Fusion for Real-Time Medical Diagnosis and Object Detection

In recent years, medical diagnosis and object detection have been significantly enhanced by the integration of multi-modal image fusion techniques. This study proposes an Adaptive Transformer-Based Multi-Modal Image Fusion (AT-MMIF) framework designed for real-time medical diagnosis and object detection. The framework employs a Transformer architecture to capture both global and local feature correlations across multiple imaging modalities, including MRI, CT, PET, and X-ray, for more accurate diagnostic results and faster object detection in medical imagery. The fusion process incorporates spatial and frequency-domain information to improve the clarity and detail of the output images, enhancing diagnostic accuracy. The adaptive attention mechanism within the Transformer dynamically adjusts to the relevant features of different image types, optimizing fusion in real time. This leads to an improved sensitivity (98.5%) and specificity (96.7%) in medical diagnosis. Additionally, the model significantly reduces false positives and negatives, with an F1 score of 97.2% in object detection tasks. The AT-MMIF framework is further optimized for real-time processing with an average inference time of 120 ms per image and a model size reduction of 35% compared to existing multi-modal fusion models. By leveraging the strengths of Transformer architectures and adaptive learning, the proposed framework offers a highly efficient and scalable solution for real-time medical diagnosis and object detection in various clinical settings, including radiology, oncology, and pathology.

Bearing fault diagnosis based on enhanced Canberra distance feature in SDP image

Abstract Feature enhancement is important in mechanical equipment fault diagnosis. A limited set of characteristic parameters is insufficient for diagnosing bearing signals with multiple fault types. The presence of noise increases the difficulty of extracting fault features from images. To address the challenge of diagnosing rolling bearing faults in complex environments, this study presents an enhanced weighted image fusion framework aimed at enhancing fault features within the images, which enables accurate diagnosis of bearing faults using a limited number of features. The proposed method encompasses four distinct stages. In the first stage, a symmetrized dot pattern method is employed to transform one-dimensional time-series data into two-dimensional images, visualizing the signal in a 2D format. In the second stage, image binarization and an improved weighted fusion method are utilized to simplify subsequent processing and enhance the image features. The third stage involves extracting the image’s contrast and maximum singular value to improve the Canberra distance calculation. Finally, the enhanced Canberra distance is used for classifying bearing faults. Performance testing of the image feature enhancement is conducted on various datasets containing rolling bearings. Comparative experiments with alternative enhancement methods demonstrate the superiority of the proposed improved weighted image fusion framework. Comparative experiments with the original Canberra distance validate the effectiveness of the enhanced Canberra distance. Additionally, experiments conducted in noisy environments confirm the robustness of the proposed approach. Furthermore, the image feature enhancement method is applied to other bearing datasets, and the experimental results demonstrate its effectiveness in enhancing fault feature representation and achieving accurate diagnosis of rolling bearings.

MixFuse: An iterative mix-attention transformer for multi-modal image fusion

Multi-modal image fusion plays a crucial role in various visual systems. However, existing methods typically involve a multi-stage pipeline, i.e., feature extraction, integration, and reconstruction, which limits the effectiveness and efficiency of feature interaction and aggregation. In this paper, we propose MixFuse, a compact multi-modal image fusion framework based on Transformers. It smoothly unifies the process of feature extraction and integration, As its core, the Mix Attention Transformer Block (MATB) integrates the Cross-Attention Transformer Module (CATM) and the Self-Attention Transformer Module (SATM). The CATM introduces a symmetrical cross-attention mechanism to identify modality-specific and general features, filtering out irrelevant and redundant information. Meanwhile, the SATM is designed to refine the combined features via a self-attention mechanism, enhancing the internal consistency and proper preservation of the features. This successive cross and self-attention modules work together to enhance the generation of more accurate and refined feature maps, which are essential for later reconstruction. Extensive evaluation of MixFuse on five public datasets shows its superior performance and adaptability over state-of-the-art methods. The code and model will be released at https://github.com/Bitlijinfu/MixFuse.

DDBFusion: An unified image decomposition and fusion framework based on dual decomposition and Bézier curves

Existing image fusion algorithms mostly concentrate on the design of network architecture and loss functions, and using unified feature extraction strategies while neglecting the division of redundant and effective information. However, for complementary information, unified feature extractor may not appropriate. Thus, this paper presents a unified image fusion algorithm based on Bézier curves image augmentation and hierarchical decomposition, and a self-supervised learning task is constructed to learn the meaningful information. Where Bézier curves aim to simulate different image features and constructed special self-supervised learning samples, so our method does not require task specific data and can be easily trained on public natural image datasets. Meanwhile, our dual decomposition self-supervised training method can bring redundant information filtering capability to the model. During the decomposition stage, we classify and extract different features of the images and only utilize the extracted effective information in the fusion stage, and the decomposition ability of images provides a foundation for advanced visual tasks, such as image segmentation and object detection. Finally, more detailed and comprehensive fusion images are generated, and the existence of redundant information is effectively reduced. The validity of the proposed method is verified through qualitative and quantitative analysis of multiple image fusion tasks, and our algorithm gets the state-of-the-art results on multiple datasets of different image fusion tasks. The code of our fusion method is available at https://github.com/Yukarizz/DDBFusion.

WaveFusionNet: Infrared and visible image fusion based on multi-scale feature encoder–decoder and discrete wavelet decomposition

To merge complementary information from multimodal images, such as thermal saliency from infrared images and texture details from visible images, traditional multi-scale transform-based methods have been extensively studied, with deep learning-based methods gaining significant popularity in recent years. However, there has been limited research on optimally combining the advantages of these two categories in fusion. In this paper, we propose a novel infrared and visible image fusion (IVIF) framework, WaveFusionNet, which integrates precise frequency feature decomposition from the discrete wavelet transform (DWT) with the comprehensive feature extraction from the multi-scale encoder. Firstly, we train an encoder–decoder network for multi-scale feature extraction and image reconstruction. DWT is used for down-sampling with minimal information loss by decomposing extracted features into low and high-frequency sub-bands. Next, a dual-band feature fusion (DBFF) module is trained to merge these sub-bands by integrating a spatial feature transform-based sub-network for low-frequency fusion and a maximum absolute value selection strategy for fusing high-frequencies. Finally, all fused sub-bands are fed into the pre-trained decoder to reconstruct the final image. Experimental results on three benchmark datasets (TNO, Roadscene, and MSRS) demonstrate that the proposed fusion method outperforms recent IVIF methods in both quantitative assessment and visual perception while maintaining competitive time complexity.

Retraction Note: Optimized multimodal medical image fusion framework using multi-scale geometric and multi-resolution geometric analysis

Retraction Note: Optimized multimodal medical image fusion framework using multi-scale geometric and multi-resolution geometric analysis

Enhanced low-light image fusion through multi-stage processing with Bayesian analysis and quadratic contrast function

This manuscript introduces an innovative multi-stage image fusion framework that adeptly integrates infrared (IR) and visible (VIS) spectrum images to surmount the difficulties posed by low-light settings. The approach commences with an initial preprocessing stage, utilizing an Efficient Guided Image Filter for the infrared (IR) images to amplify edge boundaries and a function for the visible (VIS) images to boost local contrast and brightness. Utilizing a two-scale decomposition technique that incorporates Lipschitz constraints-based smoothing, the images are effectively divided into distinct base and detail layers, thereby guaranteeing the preservation of essential structural information. The process of fusion is carried out in two distinct stages: firstly, a method grounded in Bayesian theory is employed to effectively combine the base layers, so effectively addressing any inherent uncertainty. Secondly, a Surface from Shade (SfS) method is utilized to ensure the preservation of the scene's geometry by enforcing integrability on the detail layers. Ultimately a Choose Max principle is employed to determine the most prominent textural characteristics, resulting in the amalgamation of the base and detail layers to generate an image that exhibits a substantial enhancement in both clarity and detail. The efficacy of our strategy is substantiated by rigorous testing, showcasing notable progressions in edge preservation, detail enhancement, and noise reduction. Consequently, our method presents significant advantages for real-world applications in image analysis.

CFNet: An infrared and visible image compression fusion network

Image fusion aims to acquire a more complete image representation within a limited physical space to more effectively support practical vision applications. Although the currently popular infrared and visible image fusion algorithms take practical applications into consideration. However, they did not fully consider the redundancy and transmission efficiency of image data. To address this limitation, this paper proposes a compression fusion network for infrared and visible images based on joint CNN and Transformer, termed CFNet. First of all, the idea of variational autoencoder image compression is introduced into the image fusion framework, achieving data compression while maintaining image fusion quality and reducing redundancy. Moreover, a joint CNN and Transformer network structure is proposed, which comprehensively considers the local information extracted by CNN and the global long-distance dependencies emphasized by Transformer. Finally, multi-channel loss based on region of interest is used to guide network training. Not only can color visible and infrared images be fused directly but more bits can be allocated to the foreground region of interest, resulting in a superior compression ratio. Extensive qualitative and quantitative analyses affirm that the proposed compression fusion algorithm achieves state-of-the-art performance. In particular, rate–distortion performance experiments demonstrate the great advantages of the proposed algorithm for data storage and transmission. The source code is available at https://github.com/Xiaoxing0503/CFNet.

A dual domain multi-exposure image fusion network based on spatial-frequency integration

Multi-exposure image fusion aims to generate a single high-dynamic image by integrating images with different exposures. Existing deep learning-based multi-exposure image fusion methods primarily focus on spatial domain fusion, neglecting the global modeling ability of the frequency domain. To effectively leverage the global illumination modeling ability of the frequency domain, we propose a novelty perspective on multi-exposure image fusion via the Spatial-Frequency Integration Framework, named MEF-SFI. Initially, we revisit the properties of the Fourier transform on the 2D image, and verify the feasibility of multi-exposure image fusion in the frequency domain where the amplitude and phase component is able to guide the integration of the illumination information. Subsequently, we present the deep Fourier-based multi-exposure image fusion framework, which consists of a spatial path and frequency path for local and global modeling separately. Specifically, we introduce a Spatial-Frequency Fusion Module to facilitate efficient interaction between dual domains and capture complementary information from input images with different exposures. Finally, we combine a dual domain loss function to ensure the retention of complementary information in both the spatial and frequency domains. Extensive experiments on the PQA-MEF dataset demonstrate that our method achieves visual-appealing fusion results against state-of-the-art multi-exposure image fusion approaches. Our code is available at https://github.com/SSyangguang/MEF-freq.

CFIFusion: Dual‐Branch Complementary Feature Injection Network for Medical Image Fusion

ABSTRACTThe goal of fusing medical images is to integrate the diverse information that multimodal medical images hold. However, the challenges lie in the limitations of imaging sensors and the issue of incomplete modal information retention, which make it difficult to produce images encompassing both functional and anatomical information. To overcome these obstacles, several medical image fusion techniques based on CNN or transformer architectures have been presented. Nevertheless, CNN technique struggles to establish extensive dependencies between the fused and source images, and transformer architecture often overlooks shallow complementary features. To augment both the feature extraction capacity and the stability of the model, we introduce a framework, called dual‐branch complementary feature injection fusion (CFIFusion) technique, a for multimodal medical image fusion framework that combines unsupervised models of CNN model and transformer techniques. Specifically, in our framework, the entire source image and segmented source image are input into an adaptive backbone network to learn global and local features, respectively. To further retain the source images' complementary information, we design a multi‐scale complementary feature extraction framework as an auxiliary module, focusing on calculating feature differences at each level to capture the shallow complementary information. Then, we design a shallow information preservation module tailored for sliced image characteristics. Experimental results on the Harvard whole brain atlas dataset demonstrate that CFIFusion shows greater benefits than recent state‐of‐the‐art algorithms in terms of both subjective and objective evaluations.

RDGMEF: a multi-exposure image fusion framework based on Retinex decompostion and guided filter

RDGMEF: a multi-exposure image fusion framework based on Retinex decompostion and guided filter

A general image fusion framework using multi-task semi-supervised learning

Existing image fusion methods primarily focus on solving single-task fusion problems, overlooking the potential information complementarity among multiple fusion tasks. Additionally, there has been no prior research in the field of image fusion that explores the mixed training of labeled and unlabeled data for different fusion tasks. To address these gaps, this paper introduces a novel multi-task semi-supervised learning approach to construct a general image fusion framework. This framework not only facilitates collaborative training for multiple fusion tasks, thereby achieving effective information complementarity among datasets from different fusion tasks, but also promotes the (unsupervised) learning of unlabeled data via the (supervised) learning of labeled data. Regarding the specific network module, we propose a so-called pseudo-siamese Laplacian pyramid transformer (PSLPT), which can effectively distinguish information at different frequencies in source images and discriminatively fuse features from distinct frequencies. More specifically, we take datasets of four typical image fusion tasks into the same PSLPT for weight updates, yielding the final general fusion model. Extensive experiments demonstrate that the obtained general fusion model exhibits promising outcomes for all four image fusion tasks, both visually and quantitatively. Moreover, comprehensive ablation and discussion experiments corroborate the effectiveness of the proposed method. The code is available at https://github.com/wwhappylife/A-general-image-fusion-framework-using-multi-task-semi-supervised-learning.

A Robust Mutual-Reinforcing Framework for 3D Multi-Modal Medical Image Fusion Based on Visual-Semantic Consistency

This work proposes a robust 3D medical image fusion framework to establish a mutual-reinforcing mechanism between visual fusion and lesion segmentation, achieving their double improvement. Specifically, we explore the consistency between vision and semantics by sharing feature fusion modules. Through the coupled optimization of the visual fusion loss and the lesion segmentation loss, visual-related and semantic-related features will be pulled into the same domain, effectively promoting accuracy improvement in a mutual-reinforcing manner. Further, we establish the robustness guarantees by constructing a two-level refinement constraint in the process of feature extraction and reconstruction. Benefiting from full consideration for common degradations in medical images, our framework can not only provide clear visual fusion results for doctor's observation, but also enhance the defense ability of lesion segmentation against these negatives. Extensive evaluations of visual fusion and lesion segmentation scenarios demonstrate the advantages of our method in terms of accuracy and robustness. Moreover, our proposed framework is generic, which can be well-compatible with existing lesion segmentation algorithms and improve their performance. The code is publicly available at https://github.com/HaoZhang1018/RMR-Fusion.

A multi-channel neural network model for multi-focus image fusion

The objective of multi-focus image fusion (MFIF) is to generate a fully focused image through integrating multiple partially focused source images. Most of the existing methods do not fully consider the local gradient variation rate of the source image, which makes it difficult to accurately distinguish the small defocused (focused) region covered by the large focused (defocused) region. In addition, these methods also cause edge blurring because they do not take into account misregistration of the source images. To address these issues, in this paper, we propose a simple and effective multi-focus image fusion framework based on multi-channel Rybak neural network (MCRYNN) model. Specifically, the proposed MCRYNN model is highly sensitive to local gradient changes of images based on input receptive fields, which can process multiple source images in parallel and extract the features of focused regions. Moreover, the decision maps can accurately be generate in proposed method based on the information interaction effect of parallel network structure for multi-focus image fusion task. Finally, we conduct qualitative and quantitative experiments on public datasets, and the results show that the performance of the proposed method outperforms the state-of-the-art methods.

TGFuse: An Infrared and Visible Image Fusion Approach Based on Transformer and Generative Adversarial Network.

The end-to-end image fusion framework has achieved promising performance, with dedicated convolutional networks aggregating the multi-modal local appearance. However, long-range dependencies are directly neglected in existing CNN fusion approaches, impeding balancing the entire image-level perception for complex scenario fusion. In this paper, therefore, we propose an infrared and visible image fusion algorithm based on the transformer module and adversarial learning. Inspired by the global interaction power, we use the transformer technique to learn the effective global fusion relations. In particular, shallow features extracted by CNN are interacted in the proposed transformer fusion module to refine the fusion relationship within the spatial scope and across channels simultaneously. Besides, adversarial learning is designed in the training process to improve the output discrimination via imposing competitive consistency from the inputs, reflecting the specific characteristics in infrared and visible images. The experimental performance demonstrates the effectiveness of the proposed modules, with superior improvement against the state-of-the-art, generalising a novel paradigm via transformer and adversarial learning in the fusion task.

Unified Cloud-Based Framework for Hyperspectral and Multispectral Image Fusion Incorporating Nonlocal Principles and Tensor Decomposition

Unified Cloud-Based Framework for Hyperspectral and Multispectral Image Fusion Incorporating Nonlocal Principles and Tensor Decomposition

A General Spatial-Frequency Learning Framework for Multimodal Image Fusion.

multimodal image fusion involves tasks like pan-sharpening and depth super-resolution. Both tasks aim to generate high-resolution target images by fusing the complementary information from the texture-rich guidance and low-resolution target counterparts. They are inborn with reconstructing high-frequency information. Despite their inherent frequency domain connection, most existing methods only operate solely in the spatial domain and rarely explore the solutions in the frequency domain. This study addresses this limitation by proposing solutions in both the spatial and frequency domains. We devise a Spatial-Frequency Information Integration Network, abbreviated as SFINet for this purpose. The SFINet includes a core module tailored for image fusion. This module consists of three key components: a spatial-domain information branch, a frequency-domain information branch, and a dual-domain interaction. The spatial-domain information branch employs the spatial convolution-equipped invertible neural operators to integrate local information from different modalities in the spatial domain. Meanwhile, the frequency-domain information branch adopts a modality-aware deep Fourier transformation to capture the image-wide receptive field for exploring global contextual information. In addition, the dual-domain interaction facilitates information flow and the learning of complementary representations. We further present an improved version of SFINet, SFINet++, that enhances the representation of spatial information by replacing the basic convolution unit in the original spatial domain branch with the information-lossless invertible neural operator. We conduct extensive experiments to validate the effectiveness of the proposed networks and demonstrate their outstanding performance against state-of-the-art methods in two representative multimodal image fusion tasks: pan-sharpening and depth super-resolution. The source code is publicly available at https://github.com/manman1995/Awaresome-pansharpening.

MsgFusion: Medical Semantic Guided Two-Branch Network for Multimodal Brain Image Fusion

Multimodal image fusion plays an essential role in medical image analysis and application, where computed tomography (CT), magnetic resonance (MR), single-photon emission computed tomography (SPECT), and positron emission tomography (PET) are commonly-used modalities, especially for brain disease diagnoses. Most existing fusion methods do not consider the characteristics of medical images, and they adopt similar strategies and assessment standards to natural image fusion. While distinctive medical semantic information (MS-Info) is hidden in different modalities, the ultimate clinical assessment of the fusion results is ignored. Our MsgFusion first builds a relationship between the key MS-Info of the MR/CT/PET/SPECT images and image features to guide the CNN feature extractions using two branches and the design of the image fusion framework. For MR images, we combine the spatial domain feature and frequency domain feature (SF) to develop one branch. For PET/SPECT/CT images, we integrate the gray color space feature and adapt the HSV color space feature (GV) to develop another branch. A classification-based hierarchical fusion strategy is also proposed to reconstruct the fusion images to persist and enhance the salient MS-Info reflecting anatomical structure and functional metabolism. Fusion experiments are carried out on many pairs of MR-PET/SPECT and MR-CT images. According to seven classical objective quality assessments and one new subjective clinical quality assessment from 30 clinical doctors, the fusion results of the proposed MsgFusion are superior to those of the existing representative methods.

MDDCMA: A Distributed Image Fusion Framework Based on Multiscale Dense Dilated Convolution and Coordinate Mean Attention

MDDCMA: A Distributed Image Fusion Framework Based on Multiscale Dense Dilated Convolution and Coordinate Mean Attention

DTFusion: Infrared and Visible Image Fusion Based on Dense Residual PConv-ConvNeXt and Texture-Contrast Compensation.

Infrared and visible image fusion aims to produce an informative fused image for the same scene by integrating the complementary information from two source images. Most deep-learning-based fusion networks utilize small kernel-size convolution to extract features from a local receptive field or design unlearnable fusion strategies to fuse features, which limits the feature representation capabilities and fusion performance of the network. Therefore, a novel end-to-end infrared and visible image fusion framework called DTFusion is proposed to address these problems. A residual PConv-ConvNeXt module (RPCM) and dense connections are introduced into the encoder network to efficiently extract features with larger receptive fields. In addition, a texture-contrast compensation module (TCCM) with gradient residuals and an attention mechanism is designed to compensate for the texture details and contrast of features. The fused features are reconstructed through four convolutional layers to generate a fused image with rich scene information. Experiments on public datasets show that DTFusion outperforms other state-of-the-art fusion methods in both subjective vision and objective metrics.