Image Fusion Framework Research Articles

Deep learning-based framework for motion-compensated image fusion in catheterization procedures

Augmenting X-ray (XR) fluoroscopy with 3D anatomic overlays is an essential technique to improve the guidance of the catheterization procedures. Unfortunately, cardiac and respiratory motion compromises the augmented fluoroscopy. Motion compensation methods can be applied to update the overlay of a static model with regard to respiratory and cardiac motion. We investigate the feasibility of motion detection between two fluoroscopic frames by applying a convolutional neural network (CNN). Its integration in the existing open-source software framework 3D-XGuide is demonstrated, such extending its functionality to automatic motion detection and compensation.The CNN is trained on reference data generated from tracking of the rapid pacing catheter tip by applying template matching with normalized cross-correlation (CC). The developed CNN motion compensation model is packaged in a standalone web service, allowing for independent use via a REST API. For testing and demonstration purposes, we have extended the functionality of 3D-XGuide navigation framework by an additional motion compensation module, which uses the displacement predictions of the standalone CNN model service for motion compensation of the static 3D model overlay. We provide the source code on GitHub under BSD license.The performance of the CNN motion compensation model was evaluated on a total of 1690 fluoroscopic image pairs from ten clinical datasets. The CNN model-based motion compensation method clearly overperformed the tracking of the rapid pacing catheter tip with CC with prediction frame rates suitable for live application in the clinical setting.A novel CNN model-based method for automatic motion compensation during fusion of 3D anatomic models with XR fluoroscopy is introduced and its integration with a real software application demonstrated. Automatic motion extraction from 2D XR images using a CNN model appears as a substantial improvement for reliable augmentation during catheter interventions.

Medical image fusion by sparse-based modified fusion framework using block total least-square update dictionary learning algorithm.

Purpose: The technique of fusing or integrating medical images collected from single and various modalities is known as medical image fusion. This is done to improve the quality of the images and combine information from several medical images. The whole procedure aids medical practitioners in gaining correct information from single images. Image fusion is one of the fastest-growing research topics in the medical imaging field. Sparse modeling is a popular signal representation technique used for image fusion with different dictionary learning approaches. We propose a medical image fusion with sparse representation (SR) and block total least-square (BLOTLESS) update dictionary learning. Approach: The domain of dictionary learning is the most significant research domain related to SR. An efficient dictionary increases the effectiveness of sparse modeling. Due to SR being an ongoing interesting research area, the medical image fusion process is done with a modified image fusion framework with recently developed BLOTLESS update dictionary learning. Results: The experimental results are compared for the image fusion process using other state-of-the-art dictionary learning algorithms, such as simultaneous codeword optimization, method of optimal directions, and K-singular value decomposition. The effectiveness of the algorithm is evaluated based on image fusion quantitative parameters. Results show that the BLOTLESS update dictionary algorithm is a promising modification for the sparse-based image fusion with its applicability in the fusion of images related to different diseases. Conclusions: The experiments and results show that the dictionary learning algorithm plays an important role in the sparse-based image fusion general framework. The fusion results also show that the proposed improved image fusion framework for medical images is promising compared with frameworks with other dictionary learning algorithms. As an application, it is also used as a tool for the fusion of different modularities of images related to brain tumor and glioma.

An Infrared and Visible Fusion Framework Based on a Novel Decomposition Method

Image fusion is one of the most rapidly evolving fields in image processing today, and its applications are widely expanded in various fields. In the field of image fusion, the method based on multi-scale decomposition plays an important role. However, it faces many difficult puzzles, such as the risk of over-smoothing during decomposition, blurring of fusion results, and loss of details. Aiming at these problems, this paper proposes a novel decomposition-based image fusion framework, which overcomes the problems of noise, blurring, and loss of details. Both the symmetry and asymmetry between infrared and visible images are important research hotspots in this paper. The experiments confirmed that the fusion framework outperforms other methods in both subjective observation and objective evaluation.

Re2FAD: A differential image registration and robust image fusion method framework for power thermal anomaly detection

Previous image fusion methods may not effectively characterize thermal anomalies in power equipment fault detection. In addition, the source images of power equipment collected on-site are not registered and cannot be directly used for image fusion. To these problems, this paper proposes a framework for differential image registration and robust image fusion to assist thermal anomaly detection (Re2FAD). First, Re2FAD calculates a joint histogram of the source images after low-frequency decomposition and constructs a criterion function to solve the optimal registration parameters. Second, the membership function is constructed to clarify the labels of the segmented regions. Third, the fusion strategy is selected according to the label to reconstruct the fused image. We built the power equipment dataset of the Shanghai Sanlin 500kV substation and proved that the dataset has higher registration accuracy compared with other image registration methods. Meanwhile, the proposed method is compared with state-of-the-art image fusion methods on this dataset. It can be seen from the objective evaluation indicators that the proposed method is 29.03% higher than the comparison method in terms of the accuracy of highlighting thermal anomalies, and the average running time is 2.696 s, which can meet the requirements of auxiliary thermal anomaly detection.

Optimized multimodal medical image fusion framework using multi-scale geometric and multi-resolution geometric analysis

Optimized multimodal medical image fusion framework using multi-scale geometric and multi-resolution geometric analysis

Focus-pixel estimation and optimization for multi-focus image fusion

To integrate the effective information and improve the quality of multi-source images, many spatial or transform domain-based image fusion methods have been proposed in the field of information fusion. The key purpose of multi-focus image fusion is to integrate the focused pixels and remove redundant information of each source image. Theoretically, if the focused pixels and complementary information of different images are detected completely, the fusion image with best quality can be obtained. For this goal, we propose a focus-pixel estimation and optimization based multi-focus image fusion framework in this paper. Because the focused pixels of an image are in the same depth of field (DOF), we propose a multi-scale focus-measure algorithm for the focused pixels matting to integrate the focused region firstly. Then, the boundaries of focused and defocused regions are obtained accurately by the proposed optimizing strategy. And the boundaries are also fused to reduce the influence of insufficient boundary precision. The experimental results demonstrate that the proposed method outperforms some previous typical methods in both objective evaluations and visual perception.

Classification Similarity Network Model for Image Fusion Using Resnet50 and GoogLeNet

The current trend in Image Fusion (IF) algorithms concentrate on the fusion process alone. However, pay less attention to critical issues such as the similarity between the two input images, features that participate in the Image Fusion. This paper addresses these two issues by deliberately attempting a new Image Fusion framework with Convolutional Neural Network (CNN). CNN has features like pre-training and similarity score, but functionalities are limited. A CNN model with classification prediction and similarity estimation are introduced as Classification Similarity Networks (CSN) to address these issues. ResNet50 and GoogLeNet are modified as the classification branches of CSN v1, CSN v2, respectively, to reduce feature dimensions. IF rules depend on the input dataset to fusion the extracted features. The output of the fusion process is fed into CSN v3 to improve the output image quality. The proposed CSN model is pre-trained and Fully Convolutional. At the time of IF, consider the similarities between the input images. This model applies to Multi-Focus, Multi-Modal Medical, Infrared-Visual and Multi-Exposure image datasets, and analyzed outcomes. The suggested model shows a significant improvement than the modern IF algorithms.

Transfuse: A Unified Transformer-Based Image Fusion Framework Using Self-Supervised Learning

Image fusion is a technique to integrate information from multiple source images with complementary information to improve the richness of a single image. Due to insufficient task-specific training data and corresponding ground truth, most existing end-to-end image fusion methods easily fall into overfitting or tedious parameter optimization processes. Two-stage methods avoid the need of large amount of task-specific training data by training encoder-decoder network on large natural image datasets and utilizing the extracted features for fusion, but the domain gap between natural images and different fusion tasks results in limited performance. In this study, we design a novel encoder-decoder based image fusion framework and propose a destruction-reconstruction based self-supervised training scheme to encourage the network to learn task-specific features. Specifically, we propose three destruction-reconstruction self-supervised auxiliary tasks for multi-modal image fusion, multi-exposure image fusion and multi-focus image fusion based on pixel intensity non-linear transformation, brightness transformation and noise transformation, respectively. In order to encourage different fusion tasks to promote each other and increase the generalizability of the trained network, we integrate the three self-supervised auxiliary tasks by randomly choosing one of them to destroy a natural image in model training. In addition, we design a new encoder that combines CNN and Transformer for feature extraction, so that the trained model can exploit both local and global information. Extensive experiments on multi-modal image fusion, multi-exposure image fusion and multi-focus image fusion tasks demonstrate that our proposed method achieves the state-of-the-art performance in both subjective and objective evaluations. The code will be publicly available soon.

Combining Multiple Seismic Attributes using Convolutional Neural Networks

Seismic exploration involves estimating the properties of the Earth's subsurface from reflected seismic waves then visualizing the resulting seismic data and its attributes. These data and derived seismic attributes provide complementary information and reduce the amount of time and effort for the geoscientist. Multiple conventional methods to combine various seismic attributes exist, but the number of attributes is always limited, and the quality of the resulting image varies. This paper proposes a method that can be used to overcome these limitations. In this paper, we propose using Deep Learning-based image fusion models to combine seismic attributes. By using convolutional neural network (CNN) capabilities in feature extraction, the resulting image quality is better than that obtained with conventional methods. This work implemented two models and conducted a number of experiments using them. Several techniques have been used to evaluate the results, such as visual inspection, and using image fusion metrics. The experiments show that the Image-fusion Framework, using the Image Fusion Framework Based on CNN (IFCNN) approach, outperformed all other models in both quantitative and visual analysis. Its QAB/F and MS-SSIM scores are 50% and 10%, respectively, higher than all other models. Also, IFCNN was evaluated against the current state-of-the-art solution, Octree, in a comparative study. IFCNN overcomes the limitation of the Octree method and succeeds in combining nine seismic attributes with a better-combining quality, with QAB/F and NAB/F scores being 40% higher.

Transformer-Based End-to-End Anatomical and Functional Image Fusion

Medical image fusion aims to derive complementary information from medical images with different modalities and is becoming increasingly important in clinical applications. The design of fusion strategy plays a key role in achieving high-quality fusion results. Existing methods usually employ handcrafted fusion rules or convolution-based networks to fuse multi-modal medical images. However, these fusion strategies are insufficiently fine-grained and cannot capture global information of multi-modal images effectively. Moreover, for deep learning-based fusion methods, they always concatenate source images/deep features of different modalities to input into the neural network, easily leading to inadequate information utilization from source images. To address these problems, we propose a transformer-based end-to-end framework for medical image fusion, termed as TransFusion. The proposed TransFusion introduces the transformer as the fusion strategy, which utilizes its self-attention mechanism to incorporate the global contextual information of multi-modal features and fuse them adequately. Besides, unlike traditional parallel multi-branch architectures or shared networks used for multiple inputs, we design branch networks that interact through fusion transformers at multiple scales to utilize the information of different modalities more adequately. A natural advantage of our design is the ability to aggregate global multi-modal features through self-attention. Both qualitative and quantitative experiments demonstrate the superiority of our method over the state-of-the-art fusion methods.

A Deep Framework for Hyperspectral Image Fusion Between Different Satellites.

Recently, fusing a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral image (HR-MSI) of different satellites has become an effective way to improve the resolution of an HSI. However, due to different imaging satellites, different illumination, and adjacent imaging time, the LR-HSI and HR-MSI may not satisfy the observation models established by existing works, and the LR-HSI and HR-MSI are hard to be registered. To solve the above problems, we establish new observation models for LR-HSIs and HR-MSIs from different satellites, then a deep-learning-based framework is proposed to solve the key steps in multi-satellite HSI fusion, including image registration, blur kernel learning, and image fusion. Specifically, we first construct a convolutional neural network (CNN), called RegNet, to produce pixel-wise offsets between LR-HSI and HR-MSI, which are utilized to register the LR-HSI. Next, according to the new observation models, a tiny network, called BKLNet, is built to learn the spectral and spatial blur kernels, where the BKLNet and RegNet can be trained jointly. In the fusion part, we further train a FusNet by downsampling the registered data with the learned spatial blur kernel. Extensive experiments demonstrate the superiority of the proposed framework in HSI registration and fusion accuracy.

MMI-Fuse: Multimodal Brain Image Fusion With Multiattention Module

Medical imaging plays a pivotal role in the clinical diagnosis of brain disease. There are many imaging methods to detect the state of tissues in the brain. While these imaging methods have advantages, they also have shortcomings. For example, magnetic resonance imaging (MRI) contains structural information but no functional characteristics of tissue, while positron emission tomography (PET) possesses functional characteristics but no structural information. The attention mechanism has been widely used in image fusion tasks, such as fusion of infrared and visible images and medical images. However, those attention models lack a balance mechanism for multimodal image features, affecting the final fusion performance. This paper proposes an end-to-end multimodal brain image fusion framework, MMI-fuse. Specifically, we first apply an autoencoder to extract the features of source images. Then, an information preservation weighted channel spatial attention model (ICS) is proposed to fuse the image features. We set an adaptive weight according to the information preservation degree of features. Finally, we use a decoder model to restructure the fused medical image. The proposed method increased the quality of fused images and decreased the fusion time effectively by the help of the improved attention model and encoder-decoder structure. To validate the performance of the proposed method, we collected 1590 pairs of multimodal brain images from the Harvard dataset and performed extensive experiments. Seven methods and five metrics were selected for the comparison experiments. The results demonstrate that the proposed method achieved notable performance on both the visual quality and objective metric score among these seven approaches. Moreover, the proposed method takes the least time among all compared methods.

Similarity Weight Learning: A New Spatial and Temporal Satellite Image Fusion Framework

Spatiotemporal fusion is a topical framework for solving the mutual restricted problem between the spatial and temporal resolution of satellite images. We pioneer an approach to replace similarity measurement steps in spatiotemporal fusion algorithms with convolutional neural networks (CNNs), building a bridge between weight function-based models and the learning-based models. Specifically, we propose a nonlocal form that separates the relational computation part from the value representation part, and construct the CNN-based similarity weight learning block for learning normalized weights. The block can be inserted into spatial and temporal adaptive reflectance fusion model (STARFM) to replace the manually designed weight calculation rules common in weight function-based methods, or into the CNN model StfNet to better utilize neighboring high-resolution images. The trained model outputs a high-resolution prediction from each base date image pair. The final result is a combination of the two predictions. In this regard, we propose the standard deviation-based weights to combine two prediction results. Four experiments are performed on Landsat–Moderate-resolution Imaging Spectroradiometer (MODIS) image pairs to determine the following: 1) the performance of the model at the target training date; 2) the generalization of the model in the target training time period; and 3) the generalization of the model at different dates and different geographical locations, each considering the different cases of giving one and two pairs of known images. Experimental results demonstrate the superiority of the similarity weight learning block and standard deviation-based weights. Among them, STARFM with the similarity weight learning block exhibits strong generalization, which testifies to the practical value of our model.

A Saliency Based Image Fusion Framework for Skin Lesion Segmentation and Classification

Melanoma, due to its higher mortality rate, is considered as one of the most pernicious types of skin cancers, mostly affecting the white populations. It has been reported a number of times and is now widely accepted, that early detection of melanoma increases the chances of the subject’s survival. Computer-aided diagnostic systems help the experts in diagnosing the skin lesion at earlier stages using machine learning techniques. In this work, we propose a framework that accurately segments, and later classifies, the lesion using improved image segmentation and fusion methods. The proposed technique takes an image and passes it through two methods simultaneously; one is the weighted visual saliency-based method, and the second is improved HDCT based saliency estimation. The resultant image maps are later fused using the proposed image fusion technique to generate a localized lesion region. The resultant binary image is later mapped back to the RGB image and fed into the Inception-ResNet-V2 pre-trained model–trained by applying transfer learning. The simulation results show improved performance compared to several existing methods.

Multi-Excitation Infrared Fusion for Impact Evaluation of Aluminium-BFRP/GFRP Hybrid Composites.

Fibre metal laminates are widely implemented in the aerospace industry owing to the merits of fatigue resistance and plastic properties. An effective defect assessment technique needs to be investigated for this type of composite materials. In order to achieve accurate impact-induced damage evaluation, a multi-excitation infrared fusion method is introduced in this study. Optical excitation thermography with high performance on revealing surface and subsurface defects is combined with vibro-thermography to improve the capability of detection on defects. Quantitative analysis is carried out on the temperature curve to assess the impact-induced deformation. A new image fusion framework including feature extraction, feature selection and fusion steps is proposed to fully utilize the information from two excitation modalities. Six fibre metal laminates which contain aluminium-basalt fibre reinforced plastic and aluminium-glass fibre reinforced plastic are investigated. Features from different perspectives are compared and selected via intensity contrast on deformation area for fusion imaging. Both types of defects (i.e., surface and sub-surface) and the internal deformation situation of these six samples are characterized clearly and intuitively.

A unified framework for damaged image fusion and completion based on low-rank and sparse decomposition

Image fusion can integrate the complementary information of multiple images. However, when the images to be fused are damaged, the existing fusion methods cannot recover the lost information. Matrix completion, on the other hand, can be used to recover the missing information of the image. Therefore, the step-by-step operation of image fusion and completion can fuse the damaged images, but it will cause artifact propagation. In view of this, we develop a unified framework for image fusion and completion. Within this framework, we first assume that the image is superimposed by low-rank and sparse components. To obtain the separation of different components to fuse and restore them separately, we propose a low-rank and sparse dictionary learning model. Specifically, we impose low-rank and sparse constraints on low-rank dictionary and sparse component respectively to improve the discrimination of learned dictionaries and introduce the condition constraints of low-rank and sparse components to promote the separation of different components. Furthermore, we integrate the low-rank characteristic of the image into the decomposition model. Based on this design, the lost information can be recovered with the decomposition of the image without using any additional algorithm. Finally, the maximum l1-norm fusion scheme is adopted to merge the coding coefficients of different components. The proposed method can achieve image fusion and completion simultaneously in the unified framework. Experimental results show that this method can well preserve the brightness and details of images, and is superior to the compared methods according to the performance evaluation.

A multi-focus image fusion framework based on multi-scale sparse representation in gradient domain

Existing multi-focus fusion methods cannot identify focus regions of multiple source images at a pixel level well, which limits the fusion performance. To solve this problem, we propose a novel multi-focus image fusion framework that performs focus region detection by a pixel-level focus measure. First, we propose the multi-scale sparse representation in gradient domain (MSSRGD) that combines the advantages of the sparse representation and the multi-scale approach to yield a data-driven and block-level focus measure. Second, the measure values are extended to the pixel level with the adaptive recursive filter (ARF) which can make full use of the original spatial information. Finally, the fused image is generated according to an accurate decision map estimated by a pixel-wise comparison. The experimental results demonstrate that the proposed method outperforms the conventional and state-of-the-art methods in terms of both visual perceptions and objective evaluations.

An effective fuzzy logic and particle swarm optimization based thermal and visible-light image fusion framework using curve-let transform

Thermal and visible-light image fusion is an active and one of the most significant research area in image processing. The result produced through this fusion process can be very much useful in surveillance and military applications especially for improving visibility and situation awareness. This paper proposes an effective fuzzy logic and particle swarm optimization based thermal and visible-light image fusion framework using curve-let transform. A novel fuzzy logic based contrast enhancement method is introduced in this framework for improving the visual appearance of both thermal and visible-light source images. In this framework curve-let transform is used for extracting more multi-scale features from source images. Upon applying curve-let transform on enhanced source images will produce low frequency (approximation) and high frequency (detailed) coefficients. Particle swarm optimization based fusion strategy has been used for producing more featured fused approximation coefficients. Correspondingly Max fusion strategy has been used for generating more textures based fused detailed coefficients. Finally upon applying inverse curve-let transform generates final fused image. The obtained simulation results clearly demonstrate that our framework has shown superior results compared to other fusion methods both in terms of subjective and objective assessments.

A Metaheuristics Framework for Weighted Multi-band Image Fusion

The main objective of hyper/multispectral image fusion is producing a composite color image that allows for an appropriate visualization of the relevant spatial and spectral information. In this paper, we propose a general framework for spectral weighting-based image fusion. The proposed methodology relies on weight updates conducted using nature-inspired algorithms and a goodness-of-fit criterion defined as the average root mean square error. Simulations on four public data sets and a recent Landsat 8 image of Brullus Lake, Egypt, as an area of study prove the efficiency of the proposed framework. The purpose of the study is to present a framework of multi-band image fusion that produces a fused image of high quality to be further used in computer processing and the results show that the image produced by the presented framework has the highest quality compared with some of the state-of-the art algorithms. To prove the increase in the image quality, we used general quality metrics such as Universal Image Quality Index, Mutual Information, the Variance and Information Measure.

Multi-modal medical image fusion framework using co-occurrence filter and local extrema in NSST domain

The fusion of vital imaging information has turned into a primary issue for biomedical applications. In this paper, a new multi-modality medical image fusion method is proposed in the shearlet domain. In the proposed algorithm, input images are decomposed using Non-subsampled shearlet transform (NSST) to get low and high frequencies components. A novel procedure to decompose and combine the base layers and detail layers using the local extrema (LE) approach is used and fused using a Co-occurrence filter (CoF) in low-frequency components. In high-frequency components, an edge-preserving image fusion strategy is performed using sum modified Laplacian (SML) to combine the high-frequency coefficients. The experimental outcomes and comparative analysis are performed over the Multi-modal medical image dataset using proposed and existing methods. It is exhibited through test outcomes and assessments that the proposed technique beats the cutting-edge fusion strategies concerning edge preservation in subjective and objective assessment criteria.