Medical Image Fusion Framework Research Articles

Retraction Note: Optimized multimodal medical image fusion framework using multi-scale geometric and multi-resolution geometric analysis

Retraction Note: Optimized multimodal medical image fusion framework using multi-scale geometric and multi-resolution geometric analysis

CFIFusion: Dual‐Branch Complementary Feature Injection Network for Medical Image Fusion

ABSTRACTThe goal of fusing medical images is to integrate the diverse information that multimodal medical images hold. However, the challenges lie in the limitations of imaging sensors and the issue of incomplete modal information retention, which make it difficult to produce images encompassing both functional and anatomical information. To overcome these obstacles, several medical image fusion techniques based on CNN or transformer architectures have been presented. Nevertheless, CNN technique struggles to establish extensive dependencies between the fused and source images, and transformer architecture often overlooks shallow complementary features. To augment both the feature extraction capacity and the stability of the model, we introduce a framework, called dual‐branch complementary feature injection fusion (CFIFusion) technique, a for multimodal medical image fusion framework that combines unsupervised models of CNN model and transformer techniques. Specifically, in our framework, the entire source image and segmented source image are input into an adaptive backbone network to learn global and local features, respectively. To further retain the source images' complementary information, we design a multi‐scale complementary feature extraction framework as an auxiliary module, focusing on calculating feature differences at each level to capture the shallow complementary information. Then, we design a shallow information preservation module tailored for sliced image characteristics. Experimental results on the Harvard whole brain atlas dataset demonstrate that CFIFusion shows greater benefits than recent state‐of‐the‐art algorithms in terms of both subjective and objective evaluations.

A Robust Mutual-Reinforcing Framework for 3D Multi-Modal Medical Image Fusion Based on Visual-Semantic Consistency

This work proposes a robust 3D medical image fusion framework to establish a mutual-reinforcing mechanism between visual fusion and lesion segmentation, achieving their double improvement. Specifically, we explore the consistency between vision and semantics by sharing feature fusion modules. Through the coupled optimization of the visual fusion loss and the lesion segmentation loss, visual-related and semantic-related features will be pulled into the same domain, effectively promoting accuracy improvement in a mutual-reinforcing manner. Further, we establish the robustness guarantees by constructing a two-level refinement constraint in the process of feature extraction and reconstruction. Benefiting from full consideration for common degradations in medical images, our framework can not only provide clear visual fusion results for doctor's observation, but also enhance the defense ability of lesion segmentation against these negatives. Extensive evaluations of visual fusion and lesion segmentation scenarios demonstrate the advantages of our method in terms of accuracy and robustness. Moreover, our proposed framework is generic, which can be well-compatible with existing lesion segmentation algorithms and improve their performance. The code is publicly available at https://github.com/HaoZhang1018/RMR-Fusion.

DFENet: A dual-branch feature enhanced network integrating transformers and convolutional feature learning for multimodal medical image fusion

In recent times, several medical image fusion techniques based on the convolutional neural network (CNN) have been proposed for various medical imaging fusion tasks. However, these methods cannot model the long-range dependencies between the fused image and the source images. To address this limitation, we propose DFENet, a multimodal medical image fusion framework that integrates CNN feature learning and vision transformer feature learning using self-supervised learning. DFENet is based on an encoder-decoder network, which can be trained on large-scale natural image dataset without the need for carefully collated ground truth fusion images. The proposed network consists of an encoder, a feature fuser, and a decoder. The encoder is composed of a CNN module and a transformer module, which is used to extract local and global features of images. In order to avoid the use of simple up-sampling and concatenate processing, a new global semantic information aggregation module is proposed to efficiently aggregate the multi-scale features obtained by the transformer module, which enhances the quality of the reconstructed images. The decoder is composed of six convolution layers with two skip connections, which are used for the reconstruction from fused features. We also propose a fusion strategy combining local energy and gradient information for the feature fusion process of magnetic resonance imaging and functional medical images. Compared to conventional fusion rules, our fusion strategy is more robust to noisy images. And compared with the existing competitive methods, our method retains more texture details of the original images and outputs a more natural and realistic fused image.

Optimized multimodal medical image fusion framework using multi-scale geometric and multi-resolution geometric analysis

Optimized multimodal medical image fusion framework using multi-scale geometric and multi-resolution geometric analysis

Multi-modal medical image fusion framework using co-occurrence filter and local extrema in NSST domain

The fusion of vital imaging information has turned into a primary issue for biomedical applications. In this paper, a new multi-modality medical image fusion method is proposed in the shearlet domain. In the proposed algorithm, input images are decomposed using Non-subsampled shearlet transform (NSST) to get low and high frequencies components. A novel procedure to decompose and combine the base layers and detail layers using the local extrema (LE) approach is used and fused using a Co-occurrence filter (CoF) in low-frequency components. In high-frequency components, an edge-preserving image fusion strategy is performed using sum modified Laplacian (SML) to combine the high-frequency coefficients. The experimental outcomes and comparative analysis are performed over the Multi-modal medical image dataset using proposed and existing methods. It is exhibited through test outcomes and assessments that the proposed technique beats the cutting-edge fusion strategies concerning edge preservation in subjective and objective assessment criteria.

Morphological Component Analysis-Based Perceptual Medical Image Fusion Using Convolutional Sparsity-Motivated PCNN

This paper proposes a perceptual medical image fusion framework based on morphological component analysis combining convolutional sparsity and pulse-coupled neural network, which is called MCA-CS-PCNN for short. Source images are first decomposed into cartoon components and texture components by morphological component analysis, and a convolutional sparse representation of cartoon layers and texture layers is produced by prelearned dictionaries. Then, convolutional sparsity is used as a stimulus to motivate the PCNN for dealing with cartoon layers and texture layers. Finally, the medical fused image is computed via combining fused cartoon layers and texture layers. Experimental results verify that the MCA-CS-PCNN model is superior to the state-of-the-art fusion strategy.

LatLRR-FCNs: Latent Low-Rank Representation With Fully Convolutional Networks for Medical Image Fusion.

Medical image fusion, which aims to derive complementary information from multi-modality medical images, plays an important role in many clinical applications, such as medical diagnostics and treatment. We propose the LatLRR-FCNs, which is a hybrid medical image fusion framework consisting of the latent low-rank representation (LatLRR) and the fully convolutional networks (FCNs). Specifically, the LatLRR module is used to decompose the multi-modality medical images into low-rank and saliency components, which can provide fine-grained details and preserve energies, respectively. The FCN module aims to preserve both global and local information by generating the weighting maps for each modality image. The final weighting map is obtained using the weighted local energy and the weighted sum of the eight-neighborhood-based modified Laplacian method. The fused low-rank component is generated by combining the low-rank components of each modality image according to the guidance provided by the final weighting map within pyramid-based fusion. A simple sum strategy is used for the saliency components. The usefulness and efficiency of the proposed framework are thoroughly evaluated on four medical image fusion tasks, including computed tomography (CT) and magnetic resonance (MR), T1- and T2-weighted MR, positron emission tomography and MR, and single-photon emission CT and MR. The results demonstrate that by leveraging the LatLRR for image detail extraction and the FCNs for global and local information description, we can achieve performance superior to the state-of-the-art methods in terms of both objective assessment and visual quality in some cases. Furthermore, our method has a competitive performance in terms of computational costs compared to other baselines.

Computational Effective Multimodal Medical Image Fusion in NSCT Domain

Multimodal Medical Image Fusion provides comprehensive data in recent medical image diagnosis applications. This paper presents a novel Multimodal Medical Image fusion framework based on the Angular Consistency (AC) and Sub Band Adaptive Filtering (SAF). In the proposed model, the Non-subsampled Contourlet Transform (NSSCT) decomposes the source image pair into high frequency (HF) and low frequency (LF) sub bands. The LF bands are integrated with Angular consistency rule that can fuse the finer details which are very much helpful in medical image diagnosis. Further the HF bands are processed through sub band adaptive filtering such that the redundant information is filtered out and only significant information is preserved. At last the fused sub bands are processed for Inverse Non-subsampled Contourlet Transform to produce a fused image. Experimental evaluation is accomplished through a real time image data set and the performance is measured through Edge Based Similarity index, Local Quality Index and Computational time. The obtained results show that the proposed system achieves a satisfactory results compared to the conventional methods.

A Novel GA-Based Optimized Approach for Regional Multimodal Medical Image Fusion With Superpixel Segmentation

For multimodal medical image fusion problems, most of the existing fusion approaches are based on pixel-level. However, the pixel-based fusion method tends to lose local and spatial information as the relationships between pixels are not considered appropriately, which has much influence on the quality of the fusion results. To address this issue, a region-based multimodal medical image fusion framework is proposed based on superpixel segmentation and a post-processing optimization method in this paper. In this framework, the average image of the source medical images is firstly obtained by a weighted averaging method. To effectively obtain homogeneous regions and preserve the complete information of image details, the fast linear spectral clustering(LSC) superpixel algorithm is carried out to segment the average image and get superpixel labels. For each region of the medical images, log-gabor filter(LGF) and sum modified laplacian(SML) are adopted to extract texture feature and contrast feature for the measurement of region importance. The most important regions are selected and the decision map is generated by comparison. Moreover, to get a more accurate decision map, a new post-processing optimized method based on genetic algorithm(GA) is given. A weighted strategy is applied to the extracted features and the weighting factor can be adaptively adjusted by GA. The effectiveness of the proposed fusion method is validated by conducting experiments on eight pairs of medical images from diverse modalities. In addition, seven other mainstream medical image fusion methods are adopted for comparing the performance of fusion. Experimental results in terms of qualitative and quantitative evaluation demonstrate that the proposed method can achieve state-of-the-art performance for multimodal medical image fusion problems.

Multi-Modal Medical Image Fusion Based on FusionNet in YIQ Color Space.

In order to obtain the physiological information and key features of source images to the maximum extent, improve the visual effect and clarity of the fused image, and reduce the computation, a multi-modal medical image fusion framework based on feature reuse is proposed. The framework consists of intuitive fuzzy processing (IFP), capture image details network (CIDN), fusion, and decoding. First, the membership function of the image is redefined to remove redundant features and obtain the image with complete features. Then, inspired by DenseNet, we proposed a new encoder to capture all the medical information features in the source image. In the fusion layer, we calculate the weight of each feature graph in the required fusion coefficient according to the trajectory of the feature graph. Finally, the filtered medical information is spliced and decoded to reproduce the required fusion image. In the encoding and image reconstruction networks, the mixed loss function of cross entropy and structural similarity is adopted to greatly reduce the information loss in image fusion. To assess performance, we conducted three sets of experiments on medical images of different grayscales and colors. Experimental results show that the proposed algorithm has advantages not only in detail and structure recognition but also in visual features and time complexity compared with other algorithms.

Anatomical‐functional image fusion based on deep convolution neural networks in local Laplacian pyramid domain

AbstractMedical image fusion technology makes clinical diagnosis and treatment more accurate. This technique can solve the existing problem that single mode medical images conveying insufficient information. Therefore, the key to this technology is to retain as much as possible information in the original medical image of multiple modes. However, the existing methods often lose source image detail with a low contrast and cause color distortion. This article proposed a novel multi‐modal medical image fusion framework. Our framework has four key steps. First, extracting the super‐resolution images from anatomical images via a deep neural network is performed. Second, decomposing the source images into detail‐enhanced approximate and residual images, and then processing by local Laplacian filtering. Third, a convolution neural network is used to complete the mapping of the source image to the feature map and the local energy‐based strategy is used to integrate the coefficients. Finally, the inversed local Laplacian pyramid is adopted to reconstruct the fused image. The experimental results prove that the fused image via the proposed method has the characteristics of high contrast, high brightness, and high color saturation and great advantages in retaining the color and structural information.

Brain CT and MRI medical image fusion using convolutional neural networks and a dual-channel spiking cortical model.

The aim of medical image fusion is to improve the clinical diagnosis accuracy, so the fused image is generated by preserving salient features and details of the source images. This paper designs a novel fusion scheme for CT and MRI medical images based on convolutional neural networks (CNNs) and a dual-channel spiking cortical model (DCSCM). Firstly, non-subsampled shearlet transform (NSST) is utilized to decompose the source image into a low-frequency coefficient and a series of high-frequency coefficients. Secondly, the low-frequency coefficient is fused by the CNN framework, where weight map is generated by a series of feature maps and an adaptive selection rule, and then the high-frequency coefficients are fused by DCSCM, where the modified average gradient of the high-frequency coefficients is adopted as the input stimulus of DCSCM. Finally, the fused image is reconstructed by inverse NSST. Experimental results indicate that the proposed scheme performs well in both subjective visual performance and objective evaluation and has superiorities in detail retention and visual effect over other current typical ones. Graphical abstract A schematic diagram of the CT and MRI medical image fusion framework using convolutional neural network and a dual-channel spiking cortical model.

Adaptive Parameter estimation based Multimodal Medical Image Fusion Frame work in SWT domain

Adaptive Parameter estimation based Multimodal Medical Image Fusion Frame work in SWT domain

An Integrated Dictionary-Learning Entropy-Based Medical Image Fusion Framework

Image fusion is widely used in different areas and can integrate complementary and relevant information of source images captured by multiple sensors into a unitary synthetic image. Medical image fusion, as an important image fusion application, can extract the details of multiple images from different imaging modalities and combine them into an image that contains complete and non-redundant information for increasing the accuracy of medical diagnosis and assessment. The quality of the fused image directly affects medical diagnosis and assessment. However, existing solutions have some drawbacks in contrast, sharpness, brightness, blur and details. This paper proposes an integrated dictionary-learning and entropy-based medical image-fusion framework that consists of three steps. First, the input image information is decomposed into low-frequency and high-frequency components by using a Gaussian filter. Second, low-frequency components are fused by weighted average algorithm and high-frequency components are fused by the dictionary-learning based algorithm. In the dictionary-learning process of high-frequency components, an entropy-based algorithm is used for informative blocks selection. Third, the fused low-frequency and high-frequency components are combined to obtain the final fusion results. The results and analyses of comparative experiments demonstrate that the proposed medical image fusion framework has better performance than existing solutions.

A new contrast based multimodal medical image fusion framework

Medical image fusion plays an important role in clinical applications such as image-guided surgery, image-guided radiotherapy, noninvasive diagnosis, and treatment planning. The main motivation is to fuse different multimodal information into a single output. In this instance, we propose a novel framework for spatially registered multimodal medical image fusion, which is primarily based on the non-subsampled contourlet transform (NSCT). The proposed method enables the decomposition of source medical images into low- and high-frequency bands in NSCT domain. Different fusion rules are then applied to the varied frequency bands of the transformed images. Fusion coefficients are achieved by the following fusion rule: low-frequency components are fused using an activity measure based on the normalized Shannon entropy, which essentially selects low-frequency components from the focused regions with high degree of clearness. In contrast, high-frequency components are fused using the directive contrast, which essentially collects all the informative textures from the source. Integrating these fusion rules, more spatial feature and functional information can be preserved and transferred into the fused images. The performance of the proposed framework is illustrated using four groups of human brain and two clinical bone images from different sources as our experimental subjects. The experimental results and comparison with other methods show the superior performance of the framework in both subjective and objective assessment criteria.

Multimodal Medical Image Fusion Framework Based on Simplified PCNN in Nonsubsampled Contourlet Transform Domain

In this paper, we present a new medical image fusion algorithm based on nonsubsampled contourlet transform (NSCT) and spiking cortical model (SCM). The flexible multi-resolution, anisotropy, and directional expansion characteristics of NSCT are associated with global coupling and pulse synchronization features of SCM. Considering the human visual system characteristics, two different fusion rules are used to fuse the low and high frequency sub-bands respectively. Firstly, maximum selection rule (MSR) is used to fuse low frequency coefficients. Secondly, spatial frequency (SF) is applied to motivate SCM network rather than using coefficients value directly, and then the time matrix of SCM is set as criteria to select coefficients of high frequency subband. The effectiveness of the proposed algorithm is achieved by the comparison with existing fusion methods.

Human visual system inspired multi-modal medical image fusion framework

Multi-modal medical image fusion, as a powerful tool for the clinical applications, has developed with the advent of various imaging modalities in medical imaging. The main motivation is to capture most relevant information from sources into a single output, which plays an important role in medical diagnosis. In this paper, a novel framework for medical image fusion based on framelet transform is proposed considering the characteristics of human visual system (HVS). The core idea behind the proposed framework is to decompose all source images by the framelet transform. Two different HVS inspired fusion rules are proposed for combining the low- and high-frequency coefficients respectively. The former is based on the visibility measure, and the latter is based on the texture information. Finally, the fused image is constructed by the inverse framelet transform with all composite coefficients. Experimental results highlight the expediency and suitability of the proposed framework. The efficiency of the proposed method is demonstrated by the different experiments on different multi-modal medical images. Further, the enhanced performance of the proposed framework is understood from the comparison with existing algorithms.