Multimodal Medical Image Research Articles

Organomics: A Concept Reflecting the Importance of PET/CT Healthy Organ Radiomics in Non-Small Cell Lung Cancer Prognosis Prediction Using Machine Learning.

Non-small cell lung cancer is the most common subtype of lung cancer. Patient survival prediction using machine learning (ML) and radiomics analysis proved to provide promising outcomes. However, most studies reported in the literature focused on information extracted from malignant lesions. This study aims to explore the relevance and additional value of information extracted from healthy organs in addition to tumoral tissue using ML algorithms. This study included PET/CT images of 154 patients collected from available online databases. The gross tumor volume and 33 volumes of interest defined on healthy organs were segmented using nnU-Net deep learning-based segmentation. Subsequently, 107 radiomic features were extracted from PET and CT images (Organomics). Clinical information was combined with PET and CT radiomics from organs and gross tumor volumes considering 19 different combinations of inputs. Finally, different feature selection (FS; 5 methods) and ML (6 algorithms) algorithms were tested in a 3-fold data split cross-validation scheme. The performance of the models was quantified in terms of the concordance index (C-index) metric. For an input combination of all radiomics information, most of the selected features belonged to PET Organomics and CT Organomics. The highest C-index (0.68) was achieved using univariate C-index FS method and random survival forest ML model using CT Organomics + PET Organomics as input as well as minimum depth FS method and CoxPH ML model using PET Organomics as input. Considering all 17 combinations with C-index higher than 0.65, Organomics from PET or CT images were used as input in 16 of them. The selected features and C-indices demonstrated that the additional information extracted from healthy organs of both PET and CT imaging modalities improved the ML performance. Organomics could be a step toward exploiting the whole information available from multimodality medical images, contributing to the emerging field of digital twins in health care.

Deep evidential fusion with uncertainty quantification and reliability learning for multimodal medical image segmentation

Single-modality medical images generally do not contain enough information to reach an accurate and reliable diagnosis. For this reason, physicians commonly rely on multimodal medical images for comprehensive diagnostic assessments. This study introduces a deep evidential fusion framework designed for segmenting multimodal medical images, leveraging the Dempster–Shafer theory of evidence in conjunction with deep neural networks. In this framework, features are first extracted from each imaging modality using a deep neural network, and features are mapped to Dempster–Shafer mass functions that describe the evidence of each modality at each voxel. The mass functions are then corrected by the contextual discounting operation, using learned coefficients quantifying the reliability of each source of information relative to each class. The discounted evidence from each modality is then combined using Dempster’s rule of combination. Experiments were carried out on a PET-CT dataset for lymphoma segmentation and a multi-MRI dataset for brain tumor segmentation. The results demonstrate the ability of the proposed fusion scheme to quantify segmentation uncertainty and improve segmentation accuracy. Moreover, the learned reliability coefficients provide some insight into the contribution of each modality to the segmentation process.

Enhanced multimodal medical image fusion via modified DWT with arithmetic optimization algorithm

Medical image fusion (MIF) techniques are proficient in combining medical images in distinct morphologies to obtain a reliable medical analysis. A single modality image could not offer adequate data for an accurate analysis. Therefore, a novel multimodal MIF-based artificial intelligence (AI) method has been presented. MIF approaches fuse multimodal medical images for exact and reliable medical recognition. Multimodal MIF improves diagnostic accuracy and clinical decision-making by combining complementary data in different imaging modalities. This article presents a new multimodal medical image fusion model utilizing Modified DWT with an Arithmetic Optimization Algorithm (MMIF-MDWTAOA) approach. The MMIF-MDWTAOA approach aims to generate a fused image with the significant details and features from each modality, leading to an elaborated depiction for precise interpretation by medical experts. The bilateral filtering (BF) approach is primarily employed for noise elimination. Next, the image decomposition process uses a modified discrete wavelet transform (MDWT) approach. However, the approximation coefficient of modality_1 and the detailed coefficient of modality_2 can be fused interchangeably. Furthermore, a fusion rule is derived from combining the multimodality data, and the AOA model is enforced to ensure the optimum selection of the fusion rule parameters. A sequence of simulations is accomplished to validate the enhanced output of the MMIF-MDWTAOA technique. The investigational validation of the MMIF-MDWTAOA technique showed the highest entropy values of 7.568 and 7.741 bits/pixel over other approaches.

Reliable multi-modal medical image-to-image translation independent of pixel-wise aligned data.

The current mainstream multi-modal medical image-to-image translation methods face a contradiction. Supervised methods with outstanding performance rely on pixel-wise aligned training data to constrain the model optimization. However, obtaining pixel-wise aligned multi-modal medical image datasets is challenging. Unsupervised methods can be trained without paired data, but their reliability cannot be guaranteed. At present, there is no ideal multi-modal medical image-to-image translation method that can generate reliable translation results without the need for pixel-wise aligned data. This work aims to develop a novel medical image-to-image translation model that is independent of pixel-wise aligned data (MITIA), enabling reliable multi-modal medical image-to-image translation under the condition of misaligned trainingdata. The proposed MITIA model utilizes a prior extraction network composed of a multi-modal medical image registration module and a multi-modal misalignment error detection module to extract pixel-level prior information from training data with misalignment errors to the largest extent. The extracted prior information is then used to construct a regularization term to constrain the optimization of the unsupervised cycle-consistent Generative Adversarial Network model, restricting its solution space and thereby improving the performance and reliability of the generator. We trained the MITIA model using six datasets containing different misalignment errors and two well-aligned datasets. Subsequently, we conducted quantitative analysis using peak signal-to-noise ratio and structural similarity as metrics. Moreover, we compared the proposed method with six other state-of-the-art image-to-image translationmethods. The results of both quantitative analysis and qualitative visual inspection indicate that MITIA achieves superior performance compared to the competing state-of-the-art methods, both on misaligned data and aligned data. Furthermore, MITIA shows more stability in the presence of misalignment errors in the training data, regardless of their severity or type. The proposed method achieves outstanding performance in multi-modal medical image-to-image translation tasks without aligned training data. Due to the difficulty in obtaining pixel-wise aligned data for medical image translation tasks, MITIA is expected to generate significant application value in this scenario compared to existingmethods.

Interactive Multi-scale Fusion: Advancing Brain Tumor Detection Through Trans-IMSM Model.

Multi-modal medical image (MI) fusion assists in generating collaboration images collecting complement features through the distinct images of several conditions. The images help physicians to diagnose disease accurately. Hence, this research proposes a novel multi-modal MI fusion modal named guided filter-based interactive multi-scale and multi-modal transformer (Trans-IMSM) fusion approach to develop high-quality computed tomography-magnetic resonance imaging (CT-MRI) fused images for brain tumor detection. This research utilizes the CT and MRI brain scan dataset to gather the input CT and MRI images. At first, the data preprocessing is carried out to preprocess these input images to improve the image quality and generalization ability for further analysis. Then, these preprocessed CT and MRI are decomposed into detail and base components utilizing the guided filter-based MI decomposition approach. This approach involves two phases: such as acquiring the image guidance and decomposing the images utilizing the guided filter. A canny operator is employed to acquire the image guidance comprising robust edge for CT and MRI images, and the guided filter is applied to decompose the guidance and preprocessed images. Then, by applying the Trans-IMSM model, fuse the detail components, while a weighting approach is used for the base components. The fused detail and base components are subsequently processed through a gated fusion and reconstruction network, and the final fused images for brain tumor detection are generated. Extensive tests are carried out to compute the Trans-IMSM method's efficacy. The evaluation results demonstrated the robustness and effectiveness, achieving an accuracy of 98.64% and an SSIM of 0.94.

Retraction Note: Optimized multimodal medical image fusion framework using multi-scale geometric and multi-resolution geometric analysis

Retraction Note: Optimized multimodal medical image fusion framework using multi-scale geometric and multi-resolution geometric analysis

A secure and privacy-preserving technique based on coupled chaotic system and plaintext encryption for multimodal medical images

A secure and privacy-preserving technique based on coupled chaotic system and plaintext encryption for multimodal medical images

Convolutional laplacian gaussian pyramid approach multimodal medical image fusion

Convolutional laplacian gaussian pyramid approach multimodal medical image fusion

OTMorph: Unsupervised Multi-domain Abdominal Medical Image Registration Using Neural Optimal Transport.

Deformable image registration is one of the essential processes in analyzing medical images. In particular, when diagnosing abdominal diseases such as hepatic cancer and lymphoma, multi-domain images scanned from different modalities or different imaging protocols are often used. However, they are not aligned due to scanning times, patient breathing, movement, etc. Although recent learning-based approaches can provide deformations in real-time with high performance, multi-domain abdominal image registration using deep learning is still challenging since the images in different domains have different characteristics such as image contrast and intensity ranges. To address this, this paper proposes a novel unsupervised multi-domain image registration framework using neural optimal transport, dubbed OTMorph. When moving and fixed volumes are given as input, a transport module of our proposed model learns the optimal transport plan to map data distributions from the moving to the fixed volumes and estimates a domain-transported volume. Subsequently, a registration module taking the transported volume can effectively estimate the deformation field, leading to deformation performance improvement. Experimental results on multi-domain image registration using multi-modality and multi-parametric abdominal medical images demonstrate that the proposed method provides superior deformable registration via the domain-transported image that alleviates the domain gap between the input images. Also, we attain the improvement even on out-of-distribution data, which indicates the superior generalizability of our model for the registration of various medical images. Our source code is available at https://github.com/boahK/OTMorph.

Performance analysis of multimodal medical image fusion using AMT-DWT-based pre-processing and customized CNN for denoising

Performance analysis of multimodal medical image fusion using AMT-DWT-based pre-processing and customized CNN for denoising

Adaptive decomposition with guided filtering and Laplacian pyramid-based image fusion method for medical applications

Medical image fusion enhances diagnostic precision and facilitates clinical decision-making by integrating information from multiple medical imaging modalities. However, this field is still challenging as the output integrated image, whether from spatial or transform domain algorithms, may suffer from drawbacks such as low contrast, blurring effect, noise, over smoothness, etc. Also, some existing novel works are restricted to specific image datasets. So, to address such issues, a new multi-modal medical image fusion approach based on the advantageous effects of multiple transforms has been introduced in the present work. For this, we use an adaptive image decomposition tool known as Hilbert vibration decomposition (HVD). HVD decomposes an image into different energy components, and after a proper decomposition of the source images, the desirable features of the decomposed components are then passed through a guided filter (GF) for edge preservation. Then, the Laplacian pyramid integrates these filtered parts using the choose max rule. Since HVD offers better spatial resolution and is independent of fixed cut-off frequencies like other transforms, the subjective outputs from this method for different publicly available medical image datasets are clear and better than the previously 20 state-of-the-art published results. Moreover, the obtained values of different objective evaluation metrics such as information entropy (IE): 7.6943, 5.9737, mean: 110.6453, 54.6346, standard deviation (SD): 85.5376, 61.8129, average gradient (AG): 109.2818, 64.6451, spatial frequency (SF): 0.1475, 0.1100, and edge metric (QHK/S): 0.5400, 0.6511 demonstrate its comparability to others. The algorithm's running period of just 0.161244 s also indicates high computational efficiency.

A multibranch and multiscale neural network based on semantic perception for multimodal medical image fusion

Medical imaging is indispensable for accurate diagnosis and effective treatment, with modalities like MRI and CT providing diverse yet complementary information. Traditional image fusion methods, while essential in consolidating information from multiple modalities, often suffer from poor image quality and loss of crucial details due to inadequate handling of semantic information and limited feature extraction capabilities. This paper introduces a novel medical image fusion technique leveraging unsupervised image segmentation to enhance the semantic understanding of the fusion process. The proposed method, named DUSMIF, employs a multi-branch, multi-scale deep learning architecture that integrates advanced attention mechanisms to refine the feature extraction and fusion processes. An innovative approach that utilizes unsupervised image segmentation to extract semantic information is introduced, which is then integrated into the fusion process. This not only enhances the semantic relevance of the fused images but also improves the overall fusion quality. The paper proposes a sophisticated network structure that extracts and fuses features at multiple scales and across multiple branches. This structure is designed to capture a comprehensive range of image details and contextual information, significantly improving the fusion outcomes. Multiple attention mechanisms are incorporated to selectively emphasize important features and integrate them effectively across different modalities and scales. This approach ensures that the fused images maintain high quality and detail fidelity. A joint loss function combining content loss, structural similarity loss, and semantic loss is formulated. This function not only guides the network in preserving image brightness and texture but also ensures that the fused image closely resembles the source images in both content and structure. The proposed method demonstrates superior performance over existing fusion techniques in objective assessments and subjective evaluations, confirming its effectiveness in enhancing the diagnostic utility of fused medical images.

Retraction Note: MAMIF: multimodal adaptive medical image fusion based on B-spline registration and non-subsampled shearlet transform

Retraction Note: MAMIF: multimodal adaptive medical image fusion based on B-spline registration and non-subsampled shearlet transform

A Novel Fragmented Approach for Securing Medical Health Records in Multimodal Medical Images

A Novel Fragmented Approach for Securing Medical Health Records in Multimodal Medical Images

Latent relation shared learning for endometrial cancer diagnosis with incomplete multi-modality medical images

Magnetic resonance imaging (MRI), ultrasound (US), and contrast-enhanced ultrasound (CEUS) can provide different image data about uterus, which have been used in the preoperative assessment of endometrial cancer. In practice, not all the patients have complete multi-modality medical images due to the high cost or long examination period. Most of the existing methods need to perform data cleansing or discard samples with missing modalities, which will influence the performance of the model. In this work, we propose an incomplete multi-modality images data fusion method based on latent relation shared to overcome this limitation. The shared space contains the common latent feature representation and modality-specific latent feature representation from the complete and incomplete multi-modality data, which jointly exploits both consistent and complementary information among multiple images. The experimental results show that our method outperforms the current representative approaches in terms of classification accuracy, sensitivity, specificity, and area under curve (AUC). Furthermore, our method performs well under varying imaging missing rates.

Enhancing Multi-Modality Medical Imaging: A Novel Approach with Laplacian Filter + Discrete Fourier Transform Pre-Processing and Stationary Wavelet Transform Fusion

Enhancing Multi-Modality Medical Imaging: A Novel Approach with Laplacian Filter + Discrete Fourier Transform Pre-Processing and Stationary Wavelet Transform Fusion

SIMFusion: A semantic information-guided modality-specific fusion network for MR Images

Multi-modal medical image fusion aims to integrate distinct imaging modalities to yield more comprehensive and precise medical images, which can benefit the subsequent image analysis tasks. However, prevailing state-of-the-art image fusion methods, despite substantial advancements, do not explicitly address the efficient handling of complementary information between modalities. Moreover, most current multi-modal medical image fusion methods encounter challenges when integrating with practical tasks, which lack the guidance of semantic information, thus hindering the generation of high-quality images for accurate identification of lesion areas. To deal with these challenges, this paper introduces a novel semantic information-guided modality-specific fusion network for multi-modal magnetic resonance (MR) images, named SIMFusion. Specifically, we propose the decomposition branch to capture common and specific features from MR images of different modalities for fusion to reduce information redundancy through correlated mutual information loss. Subsequently, we obtain their semantic characteristics via the semantic branch utilizing a pre-train segmentation network and, finally, achieve an adaptive balance between the two sets of features through a specialized fusion strategy. Extensive experiments demonstrate the superiority of SIMFusion over existing competing techniques on both BraTS2019 and ISLES2022 datasets, with 8.1% improvement in MI and 38.7% in VIFF in T2-T1ce image pairs, highlighting the potential of the proposed method as a promising solution for MR image fusion in practical applications. Our code will be released at https://github.com/Zhangxw-ustc/SIMFusion.

Deep Attention Fusion Hashing (DAFH) Model for Medical Image Retrieval.

To address this, we propose a novel deep learning-based hashing model, the Deep Attention Fusion Hashing (DAFH) model, which integrates advanced attention mechanisms with medical imaging data. The DAFH model enhances retrieval performance by integrating multi-modality medical imaging data and employing attention mechanisms to optimize the feature extraction process. Utilizing multimodal medical image data from the Cancer Imaging Archive (TCIA), this study constructed and trained a deep hashing network that achieves high-precision classification of various cancer types. At hash code lengths of 16, 32, and 48 bits, the model respectively attained Mean Average Precision (MAP@10) values of 0.711, 0.754, and 0.762, highlighting the potential and advantage of the DAFH model in medical image retrieval. The DAFH model demonstrates significant improvements in the efficiency and accuracy of medical image retrieval, proving to be a valuable tool in clinical settings.

CFIFusion: Dual‐Branch Complementary Feature Injection Network for Medical Image Fusion

ABSTRACTThe goal of fusing medical images is to integrate the diverse information that multimodal medical images hold. However, the challenges lie in the limitations of imaging sensors and the issue of incomplete modal information retention, which make it difficult to produce images encompassing both functional and anatomical information. To overcome these obstacles, several medical image fusion techniques based on CNN or transformer architectures have been presented. Nevertheless, CNN technique struggles to establish extensive dependencies between the fused and source images, and transformer architecture often overlooks shallow complementary features. To augment both the feature extraction capacity and the stability of the model, we introduce a framework, called dual‐branch complementary feature injection fusion (CFIFusion) technique, a for multimodal medical image fusion framework that combines unsupervised models of CNN model and transformer techniques. Specifically, in our framework, the entire source image and segmented source image are input into an adaptive backbone network to learn global and local features, respectively. To further retain the source images' complementary information, we design a multi‐scale complementary feature extraction framework as an auxiliary module, focusing on calculating feature differences at each level to capture the shallow complementary information. Then, we design a shallow information preservation module tailored for sliced image characteristics. Experimental results on the Harvard whole brain atlas dataset demonstrate that CFIFusion shows greater benefits than recent state‐of‐the‐art algorithms in terms of both subjective and objective evaluations.

Research into super-resolution in medical imaging from 2000 to 2023: bibliometric analysis and visualization.

Super-resolution (SR) refers to the use of hardware or software methods to enhance the resolution of low-resolution (LR) images and produce high-resolution (HR) images. SR is applied frequently across a variety of medical imaging contexts, particularly in the enhancement of neuroimaging, with specific techniques including SR microscopy-used for diagnostic biomarkers-and functional magnetic resonance imaging (fMRI)-a neuroimaging method for the measurement and mapping of brain activity. This bibliometric analysis of the literature related to SR in medical imaging was conducted to identify the global trends in this field, and visualization via graphs was completed to offer insights into future research prospects. In order to perform a bibliometric analysis of the SR literature, this study sourced all publications from the Web of Science Core Collection (WoSCC) database published from January 1, 2000, to October 11, 2023. A total of 3,262 articles on SR in medical imaging were evaluated. VOSviewer was used to perform co-occurrence and co-authorship analysis, and network visualization of the literature data, including author, journal, publication year, institution, and keywords, was completed. From 2000 to 2023, the annual publication volume surged from 13 to 366. The top three journals in this field in terms of publication volume were as follows: (I) Scientific Reports (86 publications), (II) IEEE Transactions on Medical Imaging (74 publications), and (III) IEEE Transactions on Ultrasonics Ferroelectrics and Frequency Control (56 publications). The most prolific country, institution, and author were the United States (1,017 publications; 31,301 citations), the Chinese Academy of Sciences (124 publications; 2,758 citations), and Dinggang Shen (20 publications; 671 citations), respectively. A cluster analysis of the top 100 keywords was conducted, which revealed the presence of five co-occurrence clusters: (I) SR and artificial intelligence (AI) for medical image enhancement, (II) SR and inverse problem processing concepts for positron emission tomography (PET) image processing, (III) SR ultrasound through microbubbles, (IV) SR microscopy for Alzheimer and Parkinson diseases, and (V) SR in brain fMRI: rapid acquisition and precise imaging. The most recent high-frequency keywords were deep learning (DL), magnetic resonance imaging (MRI), and convolutional neural networks (CNNs). Over the past two decades, the output of publications by countries, institutions, and authors in the field of SR in medical imaging has steadily increased. Based on bibliometric analysis of international trends, the resurgence of SR in medical imaging has been facilitated by advancements in AI. The increasing need for multi-center and multi-modal medical images has further incentivized global collaboration, leading to the diverse research paths in SR medical imaging among prominent scientists.