Image Fusion Techniques Research Articles

An ensemble deep learning model for medical image fusion with Siamese neural networks and VGG-19.

Multimodal medical image fusion methods, which combine complementary information from many multi-modality medical images, are among the most important and practical approaches in numerous clinical applications. Various conventional image fusion techniques have been developed for multimodality image fusion. Complex procedures for weight map computing, fixed fusion strategy and lack of contextual understanding remain difficult in conventional and machine learning approaches, usually resulting in artefacts that degrade the image quality. This work proposes an efficient hybrid learning model for medical image fusion using pre-trained and non-pre-trained networks i.e. VGG-19 and SNN with stacking ensemble method. The model leveraging the unique capabilities of each architecture, can effectively preserve the detailed information with high visual quality, for numerous combinations of image modalities in image fusion challenges, notably improved contrast, increased resolution, and lower artefacts. Additionally, this ensemble model can be more robust in the fusion of various combinations of source images that are publicly available from Havard-Medical-Image-Fusion Datasets, GitHub. and Kaggle. Our proposed model performance is superior in terms of visual quality and performance metrics to that of the existing fusion methods in literature like PCA+DTCWT, NSCT, DWT, DTCWT+NSCT, GADCT, CNN and VGG-19.

Erosion of the temporal bone by vestibular schwannoma: morphometrics and predictive modeling.

To perform a comprehensive morphometric analysis of vestibular schwannomas (VS) using multimodal imaging, focusing on the relationship between tumor characteristics and internal acoustic canal (IAC) changes. We analyzed a cohort of patients undergoing radiosurgery for VS, utilizing high-definition MRI and bone CT for detailed anatomical assessment. Image co-registration and fusion techniques were employed to examine VS and IAC dimensions. Advanced statistical methods, including logistic regression, were applied to identify patterns of IAC dilation and establish predictive indicators for these morphological changes. The study included 659 patients (51.1% female, mean age 56.37years) with evenly distributed tumor lateralization. Koos grades were I (22.9%), II (29.9%), III (38.2%), IVa (8.9%), and IVb (0.3%). Most tumors (90.9%) extended both inside and outside the IAC. Ipsilateral IAC (IIAC) dimensions were significantly larger than contralateral, with IIAC volume 49% greater (p < .0001). Higher Koos grades correlated with increased intra-canalicular lesion volume (icLV), which was strongly associated with IIAC size. Logistic regression identified icLV as the strongest predictor of IIAC dilation (AUC = 0.7674, threshold = 137.52 mm3). The icLV appears central to the pathophysiological development of VS and its impact on IAC anatomy. While limited by a selective patient base and static imaging data, these findings enhance the understanding of VS-IAC interactions, offering insights for improved clinical management and further research.

3D micromorphological reconstruction and roughness characterization of wood surface based on sequence images

A novel non-contact optical inspection technique is proposed to precisely reconstruct the three-dimensional surface structure of wood and evaluate its roughness. The feasibility of using image fusion techniques to characterize the 3D roughness of wood surfaces is explored by combining high-resolution serial image acquisition techniques with advanced data processing methods. The work focuses on examining various cut surfaces of three distinct wood species: Larch, Catalpa, and Toona sinensis. The proposed 3D roughness parameters are verified using 3D reconstruction and precise measurements. The experimental results demonstrate that the proposed method exhibits strong adaptability in measuring wood surface roughness and achieves a high degree of consistency with the results obtained from traditional digital microscope systems. This study not only validates the possible use of multilayer sequence image fusion technology in measuring wood roughness, but also establishes a solid basis for the creation of 3D assessment criteria for wood surface roughness and associated inspection systems.

Coupling Image-Fusion Techniques with Machine Learning to Enhance Dynamic Monitoring of Nitrogen Content in Winter Wheat from UAV Multi-Source

Plant nitrogen concentration (PNC) is a key indicator reflecting the growth and development status of plants. The timely and accurate monitoring of plant PNC is of great significance for the refined management of crop nutrition in the field. The rapidly developing sensor technology provides a powerful means for monitoring crop PNC. Although RGB images have rich spatial information, they lack the spectral information of the red edge and near infrared bands, which are more sensitive to vegetation. Conversely, multispectral images offer superior spectral resolution but typically lag in spatial detail compared to RGB images. Therefore, the purpose of this study is to improve the accuracy and efficiency of crop PNC monitoring by combining the advantages of RGB images and multispectral images through image-fusion technology. This study was based on the booting, heading, and early-filling stages of winter wheat, synchronously acquiring UAV RGB and MS data, using Gram–Schmidt (GS) and principal component (PC) image-fusion methods to generate fused images and evaluate them with multiple image-quality indicators. Subsequently, models for predicting wheat PNC were constructed using machine-selection algorithms such as RF, GPR, and XGB. The results show that the RGB_B1 image contains richer image information and more image details compared to other bands. The GS image-fusion method is superior to the PC method, and the performance of fusing high-resolution RGB_B1 band images with MS images using the GS method is optimal. After image fusion, the correlation between vegetation indices (VIs) and wheat PNC has been enhanced to varying degrees in different growth periods, significantly enhancing the response ability of spectral information to wheat PNC. To comprehensively assess the potential of fused images in estimating wheat PNC, this study fully compared the performance of PNC models before and after fusion using machine learning algorithms such as Random Forest (RF), Gaussian Process Regression (GPR), and eXtreme Gradient Boosting (XGB). The results show that the model established by the fusion image has high stability and accuracy in a single growth period, multiple growth periods, different varieties, and different nitrogen treatments, making it significantly better than the MS image. The most significant enhancements were during the booting to early-filling stages, particularly with the RF algorithm, which achieved an 18.8% increase in R2, a 26.5% increase in RPD, and a 19.7% decrease in RMSE. This study provides an effective technical means for the dynamic monitoring of crop nutritional status and provides strong technical support for the precise management of crop nutrition.

Traumatic Brain Injury Structure Detection Using Advanced Wavelet Transformation Fusion Algorithm with Proposed CNN-ViT

Detecting Traumatic Brain Injuries (TBI) through imaging remains challenging due to limited sensitivity in current methods. This study addresses the gap by proposing a novel approach integrating deep-learning algorithms and advanced image-fusion techniques to enhance detection accuracy. The method combines contextual and visual models to effectively assess injury status. Using a dataset of repeat mild TBI (mTBI) cases, we compared various image-fusion algorithms: PCA (89.5%), SWT (89.69%), DCT (89.08%), HIS (83.3%), and averaging (80.99%). Our proposed hybrid model achieved a significantly higher accuracy of 98.78%, demonstrating superior performance. Metrics including Dice coefficient (98%), sensitivity (97%), and specificity (98%) verified that the strategy is efficient in improving image quality and feature extraction. Additional validations with “entropy”, “average pixel intensity”, “standard deviation”, “correlation coefficient”, and “edge similarity measure” confirmed the robustness of the fused images. The hybrid CNN-ViT model, integrating curvelet transform features, was trained and validated on a comprehensive dataset of 24 types of brain injuries. The overall accuracy was 99.8%, with precision, recall, and F1-score of 99.8%. The “average PSNR” was 39.0 dB, “SSIM” was 0.99, and MI was 1.0. Cross-validation across five folds proved the model’s “dependability” and “generalizability”. In conclusion, this study introduces a promising method for TBI detection, leveraging advanced image-fusion and deep-learning techniques, significantly enhancing medical imaging and diagnostic capabilities for brain injuries.

Radar Target Radar Cross-Section Measurement Based on Enhanced Imaging and Scattering Center Extraction.

Accurate measurement of a Radar Cross-Section (RCS) is a critical technical challenge in assessing the stealth performance and scattering characteristics of radar targets. Traditional RCS measurement methods are limited by high costs, sensitivity to environmental conditions, and difficulties in distinguishing local scattering features of targets. To address these challenges, this paper proposes a novel RCS measurement method based on enhanced imaging and scattering center extraction. This method integrates sub-aperture imaging with image fusion techniques to improve imaging quality and enhance the detail of target scattering characteristics. Additionally, an improved sequence CLEAN algorithm is employed to effectively suppress sidelobe effects and ensure the accuracy of scattering center extraction. Experimental results demonstrate that this method achieves higher precision in RCS measurement of complex targets and is particularly effective in environments with strong interference, where it successfully separates the scattering contributions of the target from those of the interference sources. This method offers a new technological approach for precise RCS measurement of radar stealth targets in the future.

Comparative Review Of Image Fusion Techniques For Enhanced Quality

Comparative Review Of Image Fusion Techniques For Enhanced Quality

Comparative study of CTA/CTV and MRTA for preoperative simulation of microvascular decompression in neurovascular compression syndromes.

Neurovascular compression syndrome (NVCS), characterized by cranial nerve compression due to adjacent blood vessels at the root entry zone, frequently presents as trigeminal neuralgia (TN), hemifacial spasm (HFS), or glossopharyngeal neuralgia (GN). Despite its prevalence in NVCS assessment, Magnetic Resonance Tomographic Angiography (MRTA)'s limited sensitivity to small vessels and veins poses challenges. This study aims to refine vessel localization and surgical planning for NVCS patients using a novel 3D multimodal fusion imaging (MFI) technique incorporating computed tomography angiography and venography (CTA/CTV). A retrospective analysis was conducted on 76 patients who underwent MVD surgery and were diagnosed with single-site primary TN, HFS, or GN. Imaging was obtained from MRTA and CTA/CTV sequences, followed by image processing and 3D-MFI using FastSurfer and 3DSlicer. The CTA/CTV-3D-MFI showed higher sensitivity than MRTA-3D-MFI in predicting responsible vessels (98.6% vs. 94.6%) and NVC severity (98.6% vs. 90.8%). Kappa coefficients revealed strong agreement with MRTA-3D-MFI (0.855 for vessels, 0.835 for NVC severity) and excellent agreement with CTA/CTV-3D-MFI (0.951 for vessels, 0.952 for NVC). Resident neurosurgeons significantly preferred CTA/CTV-3D-MFI due to its better correlation with surgical reality, clearer depiction of surgical anatomy, and optimized visualization of approaches (p < 0.001). Implementing CTA/CTV-3D-MFI significantly enhanced diagnostic accuracy and surgical planning for NVCS, outperforming MRTA-3D-MFI in identifying responsible vessels and assessing NVC severity. This innovative imaging modality can potentially improve outcomes by guiding safer and more targeted surgeries, particularly in cases where MRTA may not adequately visualize crucial neurovascular structures.

Enhanced multimodal medical image fusion via modified DWT with arithmetic optimization algorithm

Medical image fusion (MIF) techniques are proficient in combining medical images in distinct morphologies to obtain a reliable medical analysis. A single modality image could not offer adequate data for an accurate analysis. Therefore, a novel multimodal MIF-based artificial intelligence (AI) method has been presented. MIF approaches fuse multimodal medical images for exact and reliable medical recognition. Multimodal MIF improves diagnostic accuracy and clinical decision-making by combining complementary data in different imaging modalities. This article presents a new multimodal medical image fusion model utilizing Modified DWT with an Arithmetic Optimization Algorithm (MMIF-MDWTAOA) approach. The MMIF-MDWTAOA approach aims to generate a fused image with the significant details and features from each modality, leading to an elaborated depiction for precise interpretation by medical experts. The bilateral filtering (BF) approach is primarily employed for noise elimination. Next, the image decomposition process uses a modified discrete wavelet transform (MDWT) approach. However, the approximation coefficient of modality_1 and the detailed coefficient of modality_2 can be fused interchangeably. Furthermore, a fusion rule is derived from combining the multimodality data, and the AOA model is enforced to ensure the optimum selection of the fusion rule parameters. A sequence of simulations is accomplished to validate the enhanced output of the MMIF-MDWTAOA technique. The investigational validation of the MMIF-MDWTAOA technique showed the highest entropy values of 7.568 and 7.741 bits/pixel over other approaches.

A Survey on Deep Learning Techniques for Medical Image Fusion

Medical Image Fusion is a process that involves combining information from multiple medical images, which is essential for healthcare applications like diagnosis, treatment planning, and image-guided interventions. Deep learning techniques have shown significant promise in medical image fusion by integrating information and effectively capturing data from diverse modalities. Additionally, Data augmentation techniques have come to be an important tool for improving model performance and generalization. The objective of this paper is to give a comprehensive overview of deep learning techniques and data augmentation methods utilized in medical image fusion from 2018 to 2023. The paper covers a variety of topics such as image registration, feature extraction, fusion architectures, data augmentation techniques, and evaluation metrics. The survey also discusses the challenges, limitations, and future directions in the field.

Evaluation of the intramammary distribution of breast lesions detected by MRI but not conventional second-look B-mode ultrasound using an MRI/ultrasound fusion technique

The objective of this study was to evaluate the intramammary distribution of MRI-detected mass and focus lesions that were difficult to identify with conventional B-mode ultrasound (US) alone. Consecutive patients with lesions detected with MRI but not second-look conventional B-mode US were enrolled between May 2015 and June 2023. Following an additional supine MRI examination, we performed third-look US using real-time virtual sonography (RVS), an MRI/US image fusion technique. We divided the distribution of MRI-detected mammary gland lesions as follows: center of the mammary gland versus other (superficial fascia, deep fascia, and atrophic mammary gland). We were able to detect 27 (84%) of 32 MRI-detected lesions using third-look US with RVS. Of these 27 lesions, 5 (19%) were in the center of the mammary gland and 22 (81%) were located in other areas. We were able to biopsy all 27 lesions; 8 (30%) were malignant and 19 (70%) were benign. Histopathologically, three malignant lesions were invasive ductal carcinoma (IDC; luminal A), one was IDC (luminal B), and four were ductal carcinoma in situ (low-grade). Malignant lesions were found in all areas. During this study period, 132 MRI-detected lesions were identified and 43 (33%) were located in the center of the mammary gland and 87 (64%) were in other areas. Also, we were able to detect 105 of 137 MRI-detected lesions by second-look conventional-B mode US and 38 (36%) were located in the center of the mammary gland and 67 (64%) were in other areas. In this study, 81% of the lesions identified using third-look US with RVS and 64% lesions detected by second-look conventional-B mode US were located outside the center of the mammary gland. We consider that adequate attention should be paid to the whole mammary gland when we perform third-look US using MRI/US fusion technique.

Fully automated classification of pulmonary nodules in positron emission tomography-computed tomography imaging using a two-stage multimodal learning approach.

Lung cancer is a malignant tumor, for which pulmonary nodules are considered to be significant indicators. Early recognition and timely treatment of pulmonary nodules can contribute to improving the survival rate of patients with cancer. Positron emission tomography-computed tomography (PET/CT) is a noninvasive, fusion imaging technique that can obtain both functional and structural information of lung regions. However, studies of pulmonary nodules based on computer-aided diagnosis have primarily focused on the nodule level due to a reliance on the annotation of nodules, which is superficial and unable to contribute to the actual clinical diagnosis. The aim of this study was thus to develop a fully automated classification framework for a more comprehensive assessment of pulmonary nodules in PET/CT imaging data. We developed a two-stage multimodal learning framework for the diagnosis of pulmonary nodules in PET/CT imaging. In this framework, Stage I focuses on pulmonary parenchyma segmentation using a pretrained U-Net and PET/CT registration. Stage II aims to extract, integrate, and recognize image-level and feature-level features by employing the three-dimensional (3D) Inception-residual net (ResNet) convolutional block attention module architecture and a dense-voting fusion mechanism. In the experiments, the proposed model's performance was comprehensively validated using a set of real clinical data, achieving mean scores of 89.98%, 89.21%, 84.75%, 93.38%, 86.83%, and 0.9227 for accuracy, precision, recall, specificity, F1 score, and area under curve values, respectively. This paper presents a two-stage multimodal learning approach for the automatic diagnosis of pulmonary nodules. The findings reveal that the main reason for limiting model performance is the nonsolitary property of nodules in pulmonary nodule diagnosis, providing direction for future research.

MRI Brain Tumor Classification and Extraction using Deep Learning-Based Decision Level Image Fusion Technique

Objectives: The proposed work emphasizes the tumor region extracted from the multimodal MRI brain scan by deep learning-based decision-level image fusion technique. Methods: Convolutional Neural Network (CNN) architectures such as AlexNet, ResNet50, and VGG16 perform brain tumor classification with multimodal MRI images Flair, T2, and T1c respectively. Flair images are fed to the AlexNet architecture, T2 images are fed to the ResNet50 architecture, and T1c images are fed to the VGG16 architecture to classify brain tumor images. The classification results from these architectures are fused together to perform the decision on the given inputs. If the inputs come under the decision of the tumor affected then the tumor portion will be extracted using the fusion of three images as a post-processing operation. Findings: The experiments are done using BraTS datasets an open-access brain tumor image analysis research repository. The three CNN architectures' performance is measured by accuracy and gives 0.87 for AlexNet, 0.91 for ResNet50, and 0.99 for VGG16. The extracted tumor region from the fused output image is compared with the ground truth image by metrics such as SSIM with 0.93, DC 0.96, and PSNR with 66.57. Better results are received for the proposed work in the evaluation analysis than the existing works. Novelty: Decision level image fusion limitedly experimented with Deep Learning techniques in state-of-art methods. In this proposed method, the decisions made based on the classification result of three CNN architectures. Keywords: MRI, Brain tumor, Deep Learning, Convolutional Neural Network Architectures, Image Fusion, AlexNet, ResNet50, VGG16

A multibranch and multiscale neural network based on semantic perception for multimodal medical image fusion

Medical imaging is indispensable for accurate diagnosis and effective treatment, with modalities like MRI and CT providing diverse yet complementary information. Traditional image fusion methods, while essential in consolidating information from multiple modalities, often suffer from poor image quality and loss of crucial details due to inadequate handling of semantic information and limited feature extraction capabilities. This paper introduces a novel medical image fusion technique leveraging unsupervised image segmentation to enhance the semantic understanding of the fusion process. The proposed method, named DUSMIF, employs a multi-branch, multi-scale deep learning architecture that integrates advanced attention mechanisms to refine the feature extraction and fusion processes. An innovative approach that utilizes unsupervised image segmentation to extract semantic information is introduced, which is then integrated into the fusion process. This not only enhances the semantic relevance of the fused images but also improves the overall fusion quality. The paper proposes a sophisticated network structure that extracts and fuses features at multiple scales and across multiple branches. This structure is designed to capture a comprehensive range of image details and contextual information, significantly improving the fusion outcomes. Multiple attention mechanisms are incorporated to selectively emphasize important features and integrate them effectively across different modalities and scales. This approach ensures that the fused images maintain high quality and detail fidelity. A joint loss function combining content loss, structural similarity loss, and semantic loss is formulated. This function not only guides the network in preserving image brightness and texture but also ensures that the fused image closely resembles the source images in both content and structure. The proposed method demonstrates superior performance over existing fusion techniques in objective assessments and subjective evaluations, confirming its effectiveness in enhancing the diagnostic utility of fused medical images.

An Analytical Study of Image Fusion Techniques in Image Processing for Data Security &amp; Privacy

Security is one of the crucial one in the technologized world, because information grows rapidly day by day. As population grows every year, information shared over internet becomes higher than what we expect and it needs preservation of data because of third party or unauthorized access. There are several techniques and methods are introduced and implemented to protect the data about the users, but security was not upto the level of protecting the data what actually we need. This review article is written about the concepts and methods implemented by various researchers for securing the data using by different users to communicate with known user at destination point without any third-party access with the help of image fusion techniques.

ReLAP-Net: Residual Learning and Attention Based Parallel Network for Hyperspectral and Multispectral Image Fusion

Remote sensing applications require high-resolution images to obtain precise information about the Earth???s surface. Multispectral images have high spatial resolution but low spectral resolution. Hyperspectral images have high spectral resolution but low spatial resolution. This study proposes a residual learning and attention-based parallel network based on residual network and channel attention. The network performs image fusion of a high spatial resolution multispectral image and a low spatial resolution hyperspectral image. The network training and fusion experiments are conducted on four public benchmark data sets to show the effectiveness of the proposed model. The fusion performance is compared with classical signal processing???based image fusion techniques. Four image metrics are used for the quantitative evaluation of the fused images. The proposed network improved fusion ability by reducing the root mean square error and relative dimensionless global error in synthesis and increased the peak signal-to-noise ratio when compared to other state-of-the-art models.

Dual Volume Image Fusion Technique for Anatomical Evaluation of Cavernous Sinus Dural Arteriovenous Fistula: Usefulness for Transvenous Target Segment Embolization

In this preliminary study, we investigated the value of fusion techniques by flat detector computed tomography based dual volume rotational angiography (DVRA) for the evaluation of the anatomical relationship of cavernous sinus dural arteriovenous fistula (CSDAVF) and assessed the possibility of transvenous target segment embolization. Twenty-six patients with CSDAVF supplied by multiple feeders underwent DVRA for each feeding large vessel separately. We assessed the anatomical relationship of feeders, fistula points, and venous drainage with three dual volume image fusion techniques. Transvenous embolization was targeted to the segment of fistulous point for preserving those not involved and reducing coil mass effect. Dual vessel multi-planar reconstruction fusion technique could show which segment of the cavernous sinus supplied by feeding arteries. In the dual vessel volume rendering fusion technique, the association between feeding arteries, fistula points, and draining veins of 2 different vessels could be accurately identified in 3 dimensions. In addition, we could visualize the exact anatomical relationship between the components of CSDAVF and skull anatomy with the single vessel fusion technique. Based on various fusion images, target segment embolization was successfully performed in 8 patients. In this group, we achieved complete or near complete occlusion without complications, including cranial nerve palsy. Detailed anatomical information including accurate fistula point, specific feeding arteries, and draining veins could be obtained with various dual volume image fusion techniques. In addition, the target segment embolization of CSDAVF could be possible with understanding of the precise CSDAVF architectures.

CFIFusion: Dual‐Branch Complementary Feature Injection Network for Medical Image Fusion

ABSTRACTThe goal of fusing medical images is to integrate the diverse information that multimodal medical images hold. However, the challenges lie in the limitations of imaging sensors and the issue of incomplete modal information retention, which make it difficult to produce images encompassing both functional and anatomical information. To overcome these obstacles, several medical image fusion techniques based on CNN or transformer architectures have been presented. Nevertheless, CNN technique struggles to establish extensive dependencies between the fused and source images, and transformer architecture often overlooks shallow complementary features. To augment both the feature extraction capacity and the stability of the model, we introduce a framework, called dual‐branch complementary feature injection fusion (CFIFusion) technique, a for multimodal medical image fusion framework that combines unsupervised models of CNN model and transformer techniques. Specifically, in our framework, the entire source image and segmented source image are input into an adaptive backbone network to learn global and local features, respectively. To further retain the source images' complementary information, we design a multi‐scale complementary feature extraction framework as an auxiliary module, focusing on calculating feature differences at each level to capture the shallow complementary information. Then, we design a shallow information preservation module tailored for sliced image characteristics. Experimental results on the Harvard whole brain atlas dataset demonstrate that CFIFusion shows greater benefits than recent state‐of‐the‐art algorithms in terms of both subjective and objective evaluations.

HPC for SAR-MSS Data Fusion Experiments

Satellite image fusion techniques are compute intensive models mainly due to the availability of voluminous temporal data from multiple sensors. The need and support of High Performance Computing clusters in modelling various satellite data fusion techniques is unavoidable as the 5V’s such as volume, velocity, variety, value and veracity of space based Big Data has to be scientifically integrated over a HPC environment. Any such integration is meaningful, only when a suitable domain specific parallel processing algorithm is developed that can extract the much required critical information in near real time or real time mode. In this regard, this article mainly focuses on defining a HPC system environment and corresponding parallel algorithm which together supports the image fusion experiments of various space borne satellite data sets. Emphasis has been given in fusing different frequency, polarization Synthetic Aperture Radar’s, intensity data with Multi Spectral optical satellite data. The well proved and most commonly followed IHS and PAC image fusion techniques are studied and its parallel version of algorithm that can be enabled over the proposed HPC environment are discussed. In the current world scenario, setting up the proposed HPC system and enablement of proposed parallel models are unavoidable, mainly to address the real time requirements of defense and disaster management operations.

A novel method of ultrasonic tomographic imaging of defects in the coating layer by image fusion and binarization techniques

A novel method of ultrasonic tomographic imaging of defects in the coating layer by image fusion and binarization techniques