CT Image Fusion Research Articles

HBANet: A hybrid boundary-aware attention network for infrared and visible image fusion

Infrared and visible image fusion is an extensively investigated problem in infrared image processing, aiming to extract useful information from source images. However, the automatic fusion of these images presents a significant challenge due to the large domain difference and ambiguous boundaries. In this article, we propose a novel image fusion approach based on hybrid boundary-aware attention, termed HBANet, which models global dependencies across the image and leverages boundary-wise prior knowledge to supplement local details. Specifically, we design a novel mixed boundary-aware attention module that is capable of leveraging spatial information to the fullest extent and integrating long dependencies across different domains. To preserve the integrity of texture and structural information, we introduced a sophisticated loss function that comprises structure, intensity, and variation losses. Our method has been demonstrated to outperform state-of-the-art methods in terms of both visual and quantitative metrics, in our experiments on public datasets. Furthermore, our approach also exhibits great generalization capability, achieving satisfactory results in CT and MRI image fusion tasks.

CT and MRI Image Fusion via Coupled Feature-Learning GAN

The fusion of multimodal medical images, particularly CT and MRI, is driven by the need to enhance the diagnostic process by providing clinicians with a single, comprehensive image that encapsulates all necessary details. Existing fusion methods often exhibit a bias towards features from one of the source images, making it challenging to simultaneously preserve both structural information and textural details. Designing an effective fusion method that can preserve more discriminative information is therefore crucial. In this work, we propose a Coupled Feature-Learning GAN (CFGAN) to fuse the multimodal medical images into a single informative image. The proposed method establishes an adversarial game between the discriminators and a couple of generators. First, the coupled generators are trained to generate two real-like fused images, which are then used to deceive the two coupled discriminators. Subsequently, the two discriminators are devised to minimize the structural distance to ensure the abundant information in the original source images is well-maintained in the fused image. We further empower the generators to be robust under various scales by constructing a discriminative feature extraction (DFE) block with different dilation rates. Moreover, we introduce a cross-dimension interaction attention (CIA) block to refine the feature representations. The qualitative and quantitative experiments on common benchmarks demonstrate the competitive performance of the CFGAN compared to other state-of-the-art methods.

MR–CT image fusion method of intracranial tumors based on Res2Net

BackgroundInformation complementarity can be achieved by fusing MR and CT images, and fusion images have abundant soft tissue and bone information, facilitating accurate auxiliary diagnosis and tumor target delineation.PurposeThe purpose of this study was to construct high-quality fusion images based on the MR and CT images of intracranial tumors by using the Residual-Residual Network (Res2Net) method.MethodsThis paper proposes an MR and CT image fusion method based on Res2Net. The method comprises three components: feature extractor, fusion layer, and reconstructor. The feature extractor utilizes the Res2Net framework to extract multiscale features from source images. The fusion layer incorporates a fusion strategy based on spatial mean attention, adaptively adjusting fusion weights for feature maps at each position to preserve fine details from the source images. Finally, fused features are input into the feature reconstructor to reconstruct a fused image.ResultsQualitative results indicate that the proposed fusion method exhibits clear boundary contours and accurate localization of tumor regions. Quantitative results show that the method achieves average gradient, spatial frequency, entropy, and visual information fidelity for fusion metrics of 4.6771, 13.2055, 1.8663, and 0.5176, respectively. Comprehensive experimental results demonstrate that the proposed method preserves more texture details and structural information in fused images than advanced fusion algorithms, reducing spectral artifacts and information loss and performing better in terms of visual quality and objective metrics.ConclusionThe proposed method effectively combines MR and CT image information, allowing the precise localization of tumor region boundaries, assisting clinicians in clinical diagnosis.

Impact of clinically relevant posthepatectomy liver failure predicted by preoperative evaluation of functional remnant hepatic reserve

Abstract Background: Few studies have investigated the remnant hepatic functional reserve before hepatectomy by calculating the functional remnant liver rate (RLR) using threedimensional computed tomography (3D-CT)/technetium-99m-diethylenetriaminepentaacetic acid galactosyl human serum albumin (99mTc-GSA) single-photon emission CT (SPECT) fusion imaging. We aimed to preoperatively evaluate the predictive value of functional remnant liver rate (RLR) and indocyanine green (ICG) disappearance rate (KICG) in determining the occurrence of posthepatectomy liver failure (PHLF). Summary of Background Data: The conventional method of volumetric rem-KICG calculated from remnant liver volume and the KICG is difficult to accurately reflect heterogenous remnant liver function. Methods: In total, 106 patients who underwent major hepatectomy were retrospectively analysed. Of these, 24 (22.6%) developed clinically relevant PHLF grades B/C. We examined the ICG retention rate at 15 min (ICGR15) and KICG and constructed a 3DCT/ 99mTc-GSA SPECT fusion image to calculate the volumetric RLR, functional RLR, volumetric rem-KICG, and functional rem-KICG. Results: The multivariate analysis showed functional rem-KICG as the strongest independent risk factor for PHLF grade B/C. The functional-to-volumetric RLR ratios in the patients with portal vein obstruction and/or tumor volume of ≥500 mL was significantly higher. The volumetric rem-KICG determined that hepatectomy was unsafe in 7 patients, whereas the functional rem-KICG determined that it was unsafe in 3 patients. Conclusions: Functional rem-KICG was more reliable than volumetric rem-KICG in predicting clinically relevant PHLF grade B/C, as the resected side’s hepatic status highly influenced the function of the remnant liver. This finding could lead to a wider application of this technique.

CT and MRI image fusion via multimodal feature interaction network

CT and MRI image fusion via multimodal feature interaction network

Segmentation-Based Fusion of CT and MR Images.

In this paper, a segmentation-based image fusion method is proposed for the fusion of MR and CT images to obtain a high contrast fused image that contains complementary information from both input images. The proposed method uses the fuzzy C-mean method to extract information about the skull from the CT image. This skull information is used to extract soft tissue information from the MR image. Both the skull information and the soft tissue information are then fused using the fusion rule. The efficiency of the proposed method over other state-of-the-art fusion methods is analyzed and compared using qualitative and quantitative analysis methods. Qualitative analysis shows the improvement in the contrast between the bone and the soft tissue using the proposed method over other state-of-the-art methods without introducing any artifacts or distortions. Classical and gradient-based quantitative analysis also show significant improvement in the fused image obtained using the proposed method over the five state-of-the-art methods. The percentage improvement in the standard deviation, average gradient, entropy, spatial frequency, QABF, and LABF of the proposed method over the best value obtained by the five state-of-the-art methods is 27.11%, 12.06%, 23.64%, 11.30%, 5.59%, and 13.70% respectively.

Brain MRI and CT Image Fusion Using Multiscale Local Extrema and Image Statistics

In medical applications such as radiotherapy and guided-image surgery, data fusion for diagnostic imaging has emerged as a critical issue. Because the objective of medical image fusion is to improve patient diagnosis accuracy, the fused image is created by preserving the source images' prominent details and features. It has been demonstrated that the Multi-Level Local Extrema representation has numerous advantages over traditional image modeling approaches. We propose an innovative MLE-based fusion method for multimodal medical images in this paper. In the MLE schema, the proposed algorithm decomposes source images into coarse and detailed layers, then fuses the source images using weights calculated from these detail images using image statistics. We visually and quantitatively compared the efficacy of the suggested approach to that of existing methods using five different types of medical images from various sources. The experimental results showed that the proposed scheme outperforms other current typical schemes in terms of both qualitative image quality and objective evaluation.

Development and validation of survival prognostic models for head and neck cancer patients using machine learning and dosiomics and CT radiomics features: a multicentric study

BackgroundThis study aimed to investigate the value of clinical, radiomic features extracted from gross tumor volumes (GTVs) delineated on CT images, dose distributions (Dosiomics), and fusion of CT and dose distributions to predict outcomes in head and neck cancer (HNC) patients.MethodsA cohort of 240 HNC patients from five different centers was obtained from The Cancer Imaging Archive. Seven strategies, including four non-fusion (Clinical, CT, Dose, DualCT-Dose), and three fusion algorithms (latent low-rank representation referred (LLRR),Wavelet, weighted least square (WLS)) were applied. The fusion algorithms were used to fuse the pre-treatment CT images and 3-dimensional dose maps. Overall, 215 radiomics and Dosiomics features were extracted from the GTVs, alongside with seven clinical features incorporated. Five feature selection (FS) methods in combination with six machine learning (ML) models were implemented. The performance of the models was quantified using the concordance index (CI) in one-center-leave-out 5-fold cross-validation for overall survival (OS) prediction considering the time-to-event.ResultsThe mean CI and Kaplan-Meier curves were used for further comparisons. The CoxBoost ML model using the Minimal Depth (MD) FS method and the glmnet model using the Variable hunting (VH) FS method showed the best performance with CI = 0.73 ± 0.15 for features extracted from LLRR fused images. In addition, both glmnet-Cindex and Coxph-Cindex classifiers achieved a CI of 0.72 ± 0.14 by employing the dose images (+ incorporated clinical features) only.ConclusionOur results demonstrated that clinical features, Dosiomics and fusion of dose and CT images by specific ML-FS models could predict the overall survival of HNC patients with acceptable accuracy. Besides, the performance of ML methods among the three different strategies was almost comparable.

Anisotropic diffusion filter based fusion of NSST transformed medical images

In this paper, an image fusion method is proposed for the fusion of MR and CT images to obtain high contrast fused image that contains complementary information from both the input images. Proposed method uses NSST based fusion method to fuse the information from the two input images, whereas to enhance the edge information and to increase the contrast in the fused image, anisotropic diffusion based fusion method is used and its result is added to the NSST based fusion method. Proposed method is analyzed and compared with five state-of-the-art fusion methods in terms of both visual perception and objective assessment. Minimum percentage improvement of the proposed method over the five state-of-the-art methods are 5.99%, 129.84%, 7.48%, 7.52%, and 5.25% respectively, whereas maximum percentage improvement of the proposed method over the five state-of-the-art methods are 18.51%, 196.52%, 29.64%, 26.35%, and 16.05% respectively.

Visual transformer and deep CNN prediction of high-risk COVID-19 infected patients using fusion of CT images and clinical data

BackgroundDespite the globally reducing hospitalization rates and the much lower risks of Covid-19 mortality, accurate diagnosis of the infection stage and prediction of outcomes are clinically of interest. Advanced current technology can facilitate automating the process and help identifying those who are at higher risks of developing severe illness. This work explores and represents deep-learning-based schemes for predicting clinical outcomes in Covid-19 infected patients, using Visual Transformer and Convolutional Neural Networks (CNNs), fed with 3D data fusion of CT scan images and patients’ clinical data.MethodsWe report on the efficiency of Video Swin Transformers and several CNN models fed with fusion datasets and CT scans only vs. a set of conventional classifiers fed with patients’ clinical data only. A relatively large clinical dataset from 380 Covid-19 diagnosed patients was used to train/test the models.ResultsResults show that the 3D Video Swin Transformers fed with the fusion datasets of 64 sectional CT scans + 67 clinical labels outperformed all other approaches for predicting outcomes in Covid-19-infected patients amongst all techniques (i.e., TPR = 0.95, FPR = 0.40, F0.5 score = 0.82, AUC = 0.77, Kappa = 0.6).ConclusionsWe demonstrate how the utility of our proposed novel 3D data fusion approach through concatenating CT scan images with patients’ clinical data can remarkably improve the performance of the models in predicting Covid-19 infection outcomes.SignificanceFindings indicate possibilities of predicting the severity of outcome using patients’ CT images and clinical data collected at the time of admission to hospital.

Opt_att-GANC: Fusion of CT and PET images using an optimal attention-based generative adversarial network with classifier for lung cancer classification

Lung cancer incidence and mortality continue to rise rapidly around the world. According to the American Cancer Society, the five-year survivability for individuals in the metastasis phases is significantly lower, highlighting the importance of early lung cancer diagnosis for effective therapy and improved quality of life. To achieve this, it is crucial to combine PET’s sensitivity for recognizing abnormal regions with CT’s anatomical localization for evaluating PET-CT images in computer-assisted detection implementations. Current PET-CT image evaluation methods either run each modality independently or aggregate the data from both, but they often overlook the fact that different visual features encode different types of data from different modalities. For instance, high atypical PET uptake within the lungs is more crucial for identifying tumors compared to physical PET uptake in the heart. To address the challenges of fine-grained issues during feature extraction and fusion, we propose an interpretable deep learning-based solution for lung cancer diagnosis using CT and PET images. This involves building an Optimal Adversarial Network for merging and an Optimal Attention-based Generative Adversarial Network with Classifier (Opt_att-GANC) to augment the classification of the existence and nonexistence of lung cancer based on extracted features. The performance of the Opt_att-GANC is compared with existing methodologies like global-feature encoding U-Net (GEU-Net), 3D Dense-Net, and 3D Convolutional Neural Network Technique (3D-CNN). Results show that the proposed Opt_att-GANC achieves an F1-score of 67.08%, 93.74% accuracy, 92% precision, 92.1% recall, and 93.74% recall. The prospective study aims to enhance the precision degree with reduced duration by incorporating an ensemble neural network paradigm for feature extraction.

CT and MRI image fusion based on variance and pixel significance

In the medical imaging field, images are obtained using various modalities, including the computed tomography procedure and the magnetic resonance imaging procedure. In which every image contains different information from the other image. For better treatment and diagnosis of a patient, a single composite image must be created by fusing all the pertinent data. This process is known as image fusion. We present an innovative and effective image fusion technique utilizing ILWT and DCT for combining brain-related medical images acquired through there are two steps to this strategy. First, the variance is employed as a contrast evaluation in the DCT domain to combine the approximation coefficients generated by the ILWT decomposition. Second, the coefficients representing the fine details are combined by finding the ideal weighted average based on the importance of the pixels in the ILWT domain. Our method is straightforward, making it simple and suitable for deployment in real-time applications. The experimental results demonstrate our method's outstanding performance with regard to both result quality and in contrast to a number of picture fusion techniques.

ADDNS: An asymmetric dual deep network with sharing mechanism for medical image fusion of CT and MR-T2

Medical images with different modalities have different semantic characteristics. Medical image fusion aiming to promotion of the visual quality and practical value has become important in medical diagnostics. However, the previous methods do not fully represent semantic and visual features, and the model generalization ability needs to be improved. Furthermore, the brightness-stacking phenomenon is easy to occur during the fusion process. In this paper, we propose an asymmetric dual deep network with sharing mechanism (ADDNS) for medical image fusion. In our asymmetric model-level dual framework, primal Unet part learns to fuse medical images of different modality into a fusion image, while dual Unet part learns to invert the fusion task for multi-modal image reconstruction. This asymmetry of network settings not only enables the ADDNS to fully extract semantic and visual features, but also reduces the model complexity and accelerates the convergence. Furthermore, the sharing mechanism designed according to task relevance also reduces the model complexity and improves the generalization ability of our model. In the end, we use the intermediate supervision method to minimize the difference between fusion image and source images so as to prevent the brightness-stacking problem. Experimental results show that our algorithm achieves better results on both quantitative and qualitative experiments than several state-of-the-art methods.

GMRE-iUnet: Isomorphic Unet fusion model for PET and CT lung tumor images

Lung tumor PET and CT image fusion is a key technology in clinical diagnosis. However, the existing fusion methods are difficult to obtain fused images with high contrast, prominent morphological features, and accurate spatial localization. In this paper, an isomorphic Unet fusion model (GMRE-iUnet) for lung tumor PET and CT images is proposed to address the above problems. The main idea of this network is as following: Firstly, this paper constructs an isomorphic Unet fusion network, which contains two independent multiscale dual encoders Unet, it can capture the features of the lesion region, spatial localization, and enrich the morphological information. Secondly, a Hybrid CNN-Transformer feature extraction module (HCTrans) is constructed to effectively integrate local lesion features and global contextual information. In addition, the residual axial attention feature compensation module (RAAFC) is embedded into the Unet to capture fine-grained information as compensation features, which makes the model focus on local connections in neighboring pixels. Thirdly, a hybrid attentional feature fusion module (HAFF) is designed for multiscale feature information fusion, it aggregates edge information and detail representations using local entropy and Gaussian filtering. Finally, the experiment results on the multimodal lung tumor medical image dataset show that the model in this paper can achieve excellent fusion performance compared with other eight fusion models. In CT mediastinal window images and PET images comparison experiment, AG, EI, QAB/F, SF, SD, and IE indexes are improved by 16.19%, 26%, 3.81%, 1.65%, 3.91% and 8.01%, respectively. GMRE-iUnet can highlight the information and morphological features of the lesion areas and provide practical help for the aided diagnosis of lung tumors.

Information fusion for fully automated segmentation of head and neck tumors from PET and CT images.

PET/CT images combining anatomic and metabolic data provide complementary information that can improve clinical task performance. PET image segmentation algorithms exploiting the multi-modal information available are still lacking. Our study aimed to assess the performance of PET and CT image fusion for gross tumor volume (GTV) segmentations of head and neck cancers (HNCs) utilizing conventional, deep learning (DL), and output-level voting-based fusions. The current study is based on a total of 328 histologically confirmed HNCs from six different centers. The images were automatically cropped to a 200 × 200 head and neck region box, and CT and PET images were normalized for further processing. Eighteen conventional image-level fusions were implemented. In addition, a modified U2-Net architecture as DL fusion model baseline was used. Three different input, layer, and decision-level information fusions were used. Simultaneous truth and performance level estimation (STAPLE) and majority voting to merge different segmentation outputs (from PET and image-level and network-level fusions), that is, output-level information fusion (voting-based fusions) were employed. Different networks were trained in a 2D manner with a batch size of 64. Twenty percent of the dataset with stratification concerning the centers (20% in each center) were used for final result reporting. Different standard segmentation metrics and conventional PET metrics, such as SUV, were calculated. In single modalities, PET had a reasonable performance with a Dice score of 0.77±0.09, while CT did not perform acceptably and reached a Dice score of only 0.38±0.22. Conventional fusion algorithms obtained a Dice score range of [0.76-0.81] with guided-filter-based context enhancement (GFCE) at the low-end, and anisotropic diffusion and Karhunen-Loeve transform fusion (ADF), multi-resolution singular value decomposition (MSVD), and multi-level image decomposition based on latent low-rank representation (MDLatLRR) at the high-end. All DL fusion models achieved Dice scores of 0.80. Output-level voting-based models outperformed all other models, achieving superior results with a Dice score of 0.84 for Majority_ImgFus, Majority_All, and Majority_Fast. A mean error of almost zero was achieved for all fusions using SUVpeak , SUVmean and SUVmedian . PET/CT information fusion adds significant value to segmentation tasks, considerably outperforming PET-only and CT-only methods. In addition, both conventional image-level and DL fusions achieve competitive results. Meanwhile, output-level voting-based fusion using majority voting of several algorithms results in statistically significant improvements in the segmentation of HNC.

MRI and CT image fusion using cartoon-texture and QWT decomposition and cuckoo search-grey wolf optimization

MRI and CT image fusion using cartoon-texture and QWT decomposition and cuckoo search-grey wolf optimization

Image Fusion of MRI and CT Images Using DTCWT

Image Fusion of MRI and CT Images Using DTCWT

Automatic numeration and localization of I-125 seeds in the post-implant prostate images based on CT and radiography image fusion

Automatic numeration and localization of I-125 seeds in the post-implant prostate images based on CT and radiography image fusion

Medical Image Fusion for Brain Tumor Detection

Medical image fusion is an important task in medical diagnosis that aims to provide a complete representation of medical images by combining multiple imaging modalities. The fused image is then fed into deep learning algorithms for tumor classification. In this paper, we propose an approach for medical image fusion of CT and MRI scan images for brain tumor detection. This work proposes an algorithm for the fusion of several imaging modalities, such as MRI and CT, based on a classifier with a fusion rule. Several qualitative and quantitative evaluation metrics have been used to assess the performance of the proposed method and compare it to cutting-edge image fusion techniques. On the basis of metrics like standard deviation, entropy, mutual information, etc., the experimental findings are assessed. In terms of accuracy and training loss metrics, the experimental results show that the proposed approach outperforms the individual modalities. As a result, the suggested technique can be employed as an effective and accurate instrument for the detection of brain cancers. The method can be used to increase diagnosis precision and decrease the false-negative rate, which will ultimately improve patient outcomes

Effective lung cancer diagnosis using multi-focus fusion of CT and PET images with deep learning strategies

ABSTRACT Lung cancer is among the most prevalent forms of the disease. This study describes multi-modal image fusion for lung cancer diagnosis using a combination of PET and CT images. The proposed approach involves breaking down the images into constituent parts, amalgamating the low and high-frequency bands, and using a Siamese Inception V3 approach to form a weight map that merges the pixel motion data from the images. The fused images are acquired using the inverse DTCWT of the fused coefficient, and deep EfficientNet is used to extract features from the fused image. The embedded whale optimization with mean grey wolf optimization methods are used to choose features. The proposed approach is tested on CT and PET images of normal and lung cancer, and a deep learning technique, CNN/BiLSTM, is used to classify lung cancer. The proposed classifier has an AUC of 92.34 percent, which is superior to other proposed categorizers.