Multimodal Medical Image Research Articles

Modal adaptive super-resolution for medical images via continual learning

Details related to tiny abnormal pathologies and textures are crucial for clinical experts and computer-aided diagnosis in medical imaging. Super-resolving medical images provide significant support for disease diagnosis. However, due to differences and diversity in tissue appearance or spatial resolution of images caused by the acquisition principles or parameters in various imaging techniques, it limits their applications in multi-modal medical imaging. To this end, we propose a multi-modal adaptive super-resolution algorithm for reconstructing medical images, named MAda-SR, which improves the traditional Adam optimizer into an adaptive optimizer in terms of parameter updates and optimization strategies. Additionally, we enhance the MSE loss by adjusting its weight space, thereby increasing the transfer potential between multimodal tasks and enabling more extended continual learning. With extensive experimental validation, we demonstrate that a single super-resolution model can handle various modalities of medical datasets without compromising performance. The results indicate that our proposed MAda-SR outperforms comparative methods in terms of controlling forgetting and relearning metrics, achieving both stability and plasticity. The source code is available at https://github.com/wuzheng2022/MAda-SR.

New Approach for Multimodal Medical Image Fusion using PDE-based Technique

Accurate diagnosis of diseases is essential for effective medical treatment, and medical image fusion plays a crucial role in achieving this goal. This article proposes a novel technique for multimodal medical image fusion based on fourth order PDE-based techniques. The proposed technique uses inputted CT and brain MRI images, and applies a PDE-based technique for noise reduction and edge preservation. Subsequently, a computed fusion rule using principal component analysis is applied to generate a fused image of the brain that presents clear and essential components for medical diagnosis and analysis. The article provides a comparative study of the proposed technique with other existing techniques, demonstrating its optimal performance. The proposed technique can significantly enhance medical image diagnosis and analysis for accurate disease detection and treatment.

Pyramid-attentive GAN for multimodal brain image complementation in Alzheimer’s disease classification

Multimodal medical imaging has a larger volume of data compared to unimodal medical imaging, and can reflect different biological information and tissue features of the human body, complementing the structural details and image features missing from unimodal images to achieve a more accurate and comprehensive classification and diagnosis of diseases. In Alzheimer’s disease, the PET scan imaging technique is difficult to operate and expensive to detect, and is not included in routine examinations, resulting in a lack of PET image data in the dataset. In this paper, we propose a Generative Adversarial Network based on pyramidal attention mechanism to generate PET images through pyramidal attention mechanism and standard discriminators, which can effectively solve the problem of lack of PET data, complete the multimodal data sets of MRI and PET, combine the grey matter part of MRI images with the metabolic information in PET images to achieve multimodal medical image information fusion, and achieve classification and diagnosis of fused images through neural network. The experimental results of AD:MCI:NC triple classification show that our method achieves 89.9% accuracy.

Deep learning-based 3D brain multimodal medical image registration.

Medical image registration is a critical preprocessing step in medical image analysis. While traditional medical image registration techniques have matured, their registration speed and accuracy still fall short of clinical requirements. In this paper, we propose an improved VoxelMorph network incorporating ResNet modules and CBAM (RCV-Net), for 3D multimodal unsupervised registration. Unlike popular convolution-based U-shaped registration networks like VoxelMorph, RCV-Net incorporates the convolutional block attention module (CBAM) during the convolution process. This inclusion enhances the feature map information extraction capabilities during training and effectively prevents information loss. Additionally, we introduce a lightweight and residual network module at the network's base, which enhances learning ability without significantly increasing training parameters. To evaluate the superiority of our registration model, we utilize evaluation metrics such as structural similarity (SSIM), peak signal-to-noise ratio (PSNR), and mean square error (MSE). Experimental results demonstrate that our proposed network structure outperforms current state-of-the-art methods, yielding better performance in multimodal registration tasks. Furthermore, generalization testing on databases outside of the training set has confirmed the optimal registration effectiveness of our model.

A lightweight multi-modality medical image semantic segmentation network base on the novel UNeXt and Wave-MLP

Medical images sometimes contain diseased regions that are different sizes and. shapes, which makes it difficult to accurately segment these areas or their edges. However, directly coupling CNN and MLP to construct global and local dependency. models may also cause significant computational complexity issues. In this paper, a. unique, lightweight UNeXt network segmentation model for medical images based on. dynamic aggregation tokens was proposed. Firstly, the Wave Block module in Wave-MLP was introduced to replace the Tok-MLP module in UNeXt. The phase term in Wave Block can dynamically aggregate tokens, improving the segmentation accuracy of the model. Secondly, an AG attention gate module is added at the skip connection to suppress irrelevant feature representations in the sampling path of the encoding. network, thereby reducing computational costs and paying attention to noise and artifacts. Finally, the Focal Tversky Loss was added to handle both binary and multiple classification jobs. Quantitative and qualitative experiments were conducted on two public datasets: COVID-19 CT and BraTS 2018 MRI. The Dice score, Precision score, recall score, and Iou score of the proposed model on the COVID-19 dataset were 0.928, 0.867, 0.916, and 0.940, respectively. On BraTS 2018, the Dice scores of the ET, WT, and TC categories were 0.933, 0.925, and 0.918, respectively, and the HD scores were 1.595, 2.348, and 1.549, respectively. At the same time, the model is lightweight and has a considerably decreased training time with GFLOPs and Params of 0.52 and 0.76, respectively. The proposed lightweight model is superior to other existing methods in terms of segmentation accuracy and computing complexity according to experimental data.

Multi-modal medical image fusion via multi-dictionary and truncated Huber filtering

Multi-modal medical image fusion provides comprehensive and objective descriptions of lesions for clinical medical assistance. However, retaining useful information while achieving noise robustness remains challenging for existing techniques. In this paper, we propose a novel medical image fusion algorithm based on multi-dictionary convolutional sparse representation. Especially, truncated Huber filtering is first introduced to achieve detail-base layer decomposition of source images. Subsequently, multiple-dictionary decisions and nuclear energy-based rules are proposed to fuse the details and base layers, respectively. The fused image is reconstructed by synthesizing the fused detail and base components. The proposed model effectively fuses the source global structure and texture information and exhibits strong robustness against noise. Experiments involving extensive noise-free and noisy anatomical and functional medical image fusion on a public dataset covering five fusion categories demonstrate that the proposed method outperforms other state-of-the-art methods in subjective and objective evaluations. The source code of this study is publicly available at https://github.com/JEI981214/MDHU-fusion.

F-DARTS: Foveated Differentiable Architecture Search Based Multimodal Medical Image Fusion.

Multimodal medical image fusion (MMIF) is highly significant in such fields as disease diagnosis and treatment. The traditional MMIF methods are difficult to provide satisfactory fusion accuracy and robustness due to the influence of such possible human-crafted components as image transform and fusion strategies. Existing deep learning based fusion methods are generally difficult to ensure image fusion effect due to the adoption of a human-designed network structure and a relatively simple loss function and the ignorance of human visual characteristics during weight learning. To address these issues, we have presented the foveated differentiable architecture search (F-DARTS) based unsupervised MMIF method. In this method, the foveation operator is introduced into the weight learning process to fully explore human visual characteristics for the effective image fusion. Meanwhile, a distinctive unsupervised loss function is designed for network training by integrating mutual information, sum of the correlations of differences, structural similarity and edge preservation value. Based on the presented foveation operator and loss function, an end-to-end encoder-decoder network architecture will be searched using the F-DARTS to produce the fused image. Experimental results on three multimodal medical image datasets demonstrate that the F-DARTS performs better than several traditional and deep learning based fusion methods by providing visually superior fused results and better objective evaluation metrics.

Multi-Modal Medical Image Matching Based on Multi-Task Learning and Semantic-Enhanced Cross-Modal Retrieval

With the continuous advancement of medical imaging technology, a vast amount of multi-modal medical image data has been extensively utilized for disease diagnosis, treatment, and research. Effective management and utilization of these data becomes a pivotal challenge, particularly when undertaking image matching and retrieval. Although numerous methods for medical image matching and retrieval exist, they primarily rely on traditional image processing techniques, often limited to manual feature extraction and singular modality handling. To address these limitations, this study introduces an algorithm for medical image matching grounded in multi-task learning, further investigating a semantic-enhanced technique for cross-modal medical image retrieval. By deeply exploring complementary semantic information between different modality medical images, these methods offer novel perspectives and tools for the domain of medical image matching and retrieval.

AQWT: adaptive quaternion wavelet transform and hybrid heuristic improvement for multi-modal medical image fusion model

AQWT: adaptive quaternion wavelet transform and hybrid heuristic improvement for multi-modal medical image fusion model

Multi-modal medical Transformers: A meta-analysis for medical image segmentation in oncology

Multi-modal medical image segmentation is a crucial task in oncology that enables the precise localization and quantification of tumors. The aim of this work is to present a meta-analysis of the use of multi-modal medical Transformers for medical image segmentation in oncology, specifically focusing on multi-parametric MR brain tumor segmentation (BraTS2021), and head and neck tumor segmentation using PET-CT images (HECKTOR2021). The multi-modal medical Transformer architectures presented in this work exploit the idea of modality interaction schemes based on visio-linguistic representations: (i) single-stream, where modalities are jointly processed by one Transformer encoder, and (ii) multiple-stream, where the inputs are encoded separately before being jointly modeled. A total of fourteen multi-modal architectures are evaluated using different ranking strategies based on dice similarity coefficient (DSC) and average symmetric surface distance (ASSD) metrics. In addition, cost indicators such as the number of trainable parameters and the number of multiply-accumulate operations (MACs) are reported. The results demonstrate that multi-path hybrid CNN-Transformer-based models improve segmentation accuracy when compared to traditional methods, but come at the cost of increased computation time and potentially larger model size.

Review on multi-modality medical image fusion using deep learning

Multi-Modality medical image fusion is a method in which multiple images are merged having either single or multiple imaging modalities. This process is carried out to improve the quality of imaging while preserving all the essential and distinct features. Many areas such as Machine Learning, Artificial Intelligence, Image Processing, and Computer Vision are covered by Medical Image Fusion. This method has been adopted on a large scale by physicians to apprehend any damage or injury caused in organ tissues in clinical trials by performing a fusion of images with different modalities. In this review, Deep Learning methods carried out in the medical image fusion field have been discussed along with a comparison between their accuracies. The main objective of this paper is to list some of the most effective techniques in this domain and discuss their performance. At last, the paper concludes with the fact that although the development and growth in this area have increased over the years, many challenges have also come along the way.

One-stop multimodal medical imaging intelligent checkup system

Given the increasing demand for daily health checks and the shortage of related multimodal and multifunctional imaging equipment, a new multifunction and multimodal biomedical tomography system for health examination in which combines five agents is proposed. The main methods are, first, that the architecture framework is based on magneto-acoustic tomography with magnetic induction, integrated four other species of imaging agents, for seamlessly fusing of functional with anatomical information to reconstruct multimodality images. And the image data acquisition and processing is consisted of upper and lower computer which form a virtual instrument: the lower computer is made up by five intelligent agents while all of them were an embedded system severally. Linked with the WIFI networks, the upper computer process the image data and reconstruct the multimodal image. Thirdly, the image fusion is implemented by the system automatically itself and without other post image registrations and software. Most prominent feature of the system is the spatial resolution of images was 0.24 mm, and the imaging speed gets 50 frames per second that meet real time imaging requirements. The phantom system promised a new type noninvasive biomedical imaging multimodality and one-stop health examination system to realize one-stop-shop omni-tomography as well as holographic imaging service. Furthermore, the system and its methods can promote the construction as well as realization of smart medicine and smart hospitals.

Optimized transfer learning based multi-modal medical image retrieval

Optimized transfer learning based multi-modal medical image retrieval

Multimodal medical image protection algorithm based on 3D-CICCM and Fresnel transform

Multimodal medical image protection algorithm based on 3D-CICCM and Fresnel transform

Topology-preserving image registration with novel multi-dimensional Beltrami regularization

Beltrami coefficient provides a proper measure to quantify local distortions for a topology-preserving mapping, a.k.a general quasi-conformal mapping, in 2D. Recent studies managed to extend Beltrami coefficient from 2D to nD, and introduced its corresponding Beltrami-like measures to encode local deformation information in challenging nD topology-preserving image registration tasks. However, all these nD measures are not consistent with the classical 2D Beltrami coefficient. We propose a novel nD measure for evaluating local distortions of an nD diffeomorphic mapping in this work. We show its key mathematical properties, and prove that the new measure coincides with the classical 2D Beltrami coefficient. We further introduce a new nD topology-preserving image registration model by means of regularizing such proposed Beltrami-consistent coefficient, and show the existence of its solution. Meanwhile, a generalized Gauss-Newton scheme is applied to establish a fast multi-level numerical solver. Extensive experiments over different 3D medical image registration tasks, including the challenging multi-modal medical image registration, illustrate that the proposed approach by regularizing the introduced Beltrami-consistent coefficient outperforms the other state-of-the-art topology-preserving methods in both efficiency and accuracy.

Enhanced multimodal medical image fusion based on Pythagorean fuzzy set: an innovative approach

Medical image fusion is the process of combining a multi-modality image into a single output image for superior information and a better visual appearance without any vagueness or uncertainties. It is suitable for better diagnosis. Pythagorean fuzzy set (PFS)-based medical image fusion was proposed in this manuscript. In the first phase, a two-scale gaussian filter was used to decompose the source images into base and detail layers. In the second phase, a spatial frequency (SF)-based fusion rule was employed for detail layers to preserve the more edge-oriented details. However, the base layer images were converted into pythagorean fuzzy images (PFIs) using the optimum value obtained by pythagorean fuzzy entropy (PFE). The blackness and whiteness count fusion rule were performed for image blocks decomposed from two PFIs in the third phase. Finally, the enhanced fused image was obtained by reconstructions of fused PFI blocks, which performed the defuzzification process. The proposed method was evaluated on different datasets for disease diagnosis and achieved better mean (M), standard deviation (SD), average gradient (AG), SF, modified spatial frequency (MSF), mutual information (MI), and fusion symmetry (FS) values than compared to state-of-art methods. This advancement is important in the field of healthcare and medical imaging, including enhanced diagnostics and treatment planning.

Objective Boundary Generation for Gross Target Volume and Organs at Risk Using 3D Multi-Modal Medical Images.

Accurate delineation of Gross Target Volume (GTV) and Organs at Risk (OARs) in medical images is an essential but challenging step in radiotherapy. Deep-learning based automated delineation methods, which learn from manual annotations, are currently prevalent in academic research. However, the limited resolution of medical images and the fuzzy boundaries of lesions and organs present a challenge to the precision of manual annotations. By leveraging the complementary information from multi-modal medical images, this study proposed a novel method to generate objective boundaries of GTV and OARs. We present a novel method of objective boundary generation, inspired by image matting primarily used for 2D RGB natural images, to process 3D grayscale medical images. The proposed method has the following advantages. 1) It allows for flexible input modalities and assigns weights to each modality according to their relative significance when computing information flows in the matting algorithm. 2) It computes 3D spatial information flow among voxels, which has more advantages over its 2D counterpart. 3) It has a closed-form solution that generates deterministic results. To evaluate the characteristics of the generated boundaries, patients with stage I nasopharyngeal carcinoma (NPC) were studied, with CT images and multi-modal MR images (T1, T1C, T2) aligned using deformable registration. Region of Interests (ROIs), i.e., GTV and parotid gland, were used, with a rough trimap marking extremely few foreground voxels, many background voxels, and a large unknown region. The proposed algorithm leverages the connection between each voxel and its nearest neighbors in the feature space, to propagate the opacity information. We evaluated the results by employing both qualitative and quantitative methods. Using qualitative evaluation, experienced clinicians confirmed that the results were in agreement with the input data, especially for areas where borders were visible in most modalities (e.g., between air and tumor). For more challenging regions, where boundaries were unclear in the images, the results displayed fine-grained opacity transitions indicating the confidence of each voxel belonging to the ROI. When compared to the delineations made by clinicians, we found our results are usually more compact. We define a precision metric that evaluates the ratio of the matted foreground inside clinicians' delineations versus the entire matted foreground. Using a threshold of 0.4, our binarized result scored 0.95 for GTV and 0.92 for parotid gland. The proposed method demonstrated satisfactory results on challenging ROIs. The objective boundaries generated by this method have advantages in many aspects, including improvement of delineation protocols, enhancement of manual annotation consistency, and increase of deep-learning based automated delineation accuracy.

Artificial Intelligence in Oncological Hybrid Imaging.

Artificial intelligence (AI) applications have become increasingly relevant across a broad spectrum of settings in medical imaging. Due to the large amount of imaging data that is generated in oncological hybrid imaging, AI applications are desirable for lesion detection and characterization in primary staging, therapy monitoring, and recurrence detection. Given the rapid developments in machine learning (ML) and deep learning (DL) methods, the role of AI will have significant impact on the imaging workflow and will eventually improve clinical decision making and outcomes. The first part of this narrative review discusses current research with an introduction to artificial intelligence in oncological hybrid imaging and key concepts in data science. The second part reviews relevant examples with a focus on applications in oncology as well as discussion of challenges and current limitations. AI applications have the potential to leverage the diagnostic data stream with high efficiency and depth to facilitate automated lesion detection, characterization, and therapy monitoring to ultimately improve quality and efficiency throughout the medical imaging workflow. The goal is to generate reproducible, structured, quantitative diagnostic data for evidence-based therapy guidance in oncology. However, significant challenges remain regarding application development, benchmarking, and clinical implementation. · Hybrid imaging generates a large amount of multimodality medical imaging data with high complexity and depth.. · Advanced tools are required to enable fast and cost-efficient processing along the whole radiology value chain.. · AI applications promise to facilitate the assessment of oncological disease in hybrid imaging with high quality and efficiency for lesion detection, characterization, and response assessment. The goal is to generate reproducible, structured, quantitative diagnostic data for evidence-based oncological therapy guidance.. · Selected applications in three oncological entities (lung, prostate, and neuroendocrine tumors) demonstrate how AI algorithms may impact imaging-based tasks in hybrid imaging and potentially guide clinical decision making..

A Multicenter Study on Deep Learning for Glioblastoma Auto-Segmentation with Prior Knowledge in Multimodal Imaging

A precise radiotherapy plan is required to ensure accurate delineation of gross tumor volumes (GTV) and clinical target volumes (CTV1 and CTV2) of glioblastomas (GBMs). However, traditional manual delineation is labor intensive and highly dependent on oncologists' experience. To construct and evaluate a deep learning-based automatic delineation method using prior knowledge in multimodal medical imaging to automate precise GTV, CTV1 and CTV2 contouring in GBM patients. We retrospectively collected the CT and MRI scans of 55 eligible patients with histologically proven high-grade glioma (HGG) from an institute, these scans were performed with non-enhanced CT (CT), contrast-enhanced T1-weighted (T1C) and T2-FLAIR (T2F) sequences. We proposed a two-stage automatic segmentation framework (PKMI-Net) for GTV, CTV1 and CTV2 based on deep learning using prior knowledge in multimodal medical imaging, and its segmentation performance was evaluated with dice similarity coefficient (DSC), 95% Harsdorff distance (HD95), average surface distance (ASD) and relative volume difference (RVD). To further investigate the generalizability of our method, we designed and conducted two evaluation strategies (Mix and Cross) on four multicenter datasets (including 55 patients, 37 patients, 21 patients and 35 patients). The evaluation results with an 11-patient test set from the single institute were summarized in Table 1, the proposed method demonstrated the best accuracy in segmenting, respectively, GTV, CTV1 and CTV, achieving a DSC of 0.94, 0.95 and 0.92; HD95 of 2.07 mm, 1.18 mm and 3.80 mm; ASD of 0.69 mm, 0.39 mm and 1.13 mm and RVE of 5.50%, 3.97% and 9.68%. In the multicenter evaluation, the segmentation performance of our method implemented with the Cross strategy was comparable to that with the Mix strategy, demonstrating that our method had high and stable generalizability across multicenter datasets in automatically segmenting GTV, CTV1 and CTV2. Our proposed method achieved promising results in automatically segmenting gliomas across various datasets, which could improve the quality and efficiency of glioblastoma radiotherapy.

GMRE-iUnet: Isomorphic Unet fusion model for PET and CT lung tumor images

Lung tumor PET and CT image fusion is a key technology in clinical diagnosis. However, the existing fusion methods are difficult to obtain fused images with high contrast, prominent morphological features, and accurate spatial localization. In this paper, an isomorphic Unet fusion model (GMRE-iUnet) for lung tumor PET and CT images is proposed to address the above problems. The main idea of this network is as following: Firstly, this paper constructs an isomorphic Unet fusion network, which contains two independent multiscale dual encoders Unet, it can capture the features of the lesion region, spatial localization, and enrich the morphological information. Secondly, a Hybrid CNN-Transformer feature extraction module (HCTrans) is constructed to effectively integrate local lesion features and global contextual information. In addition, the residual axial attention feature compensation module (RAAFC) is embedded into the Unet to capture fine-grained information as compensation features, which makes the model focus on local connections in neighboring pixels. Thirdly, a hybrid attentional feature fusion module (HAFF) is designed for multiscale feature information fusion, it aggregates edge information and detail representations using local entropy and Gaussian filtering. Finally, the experiment results on the multimodal lung tumor medical image dataset show that the model in this paper can achieve excellent fusion performance compared with other eight fusion models. In CT mediastinal window images and PET images comparison experiment, AG, EI, QAB/F, SF, SD, and IE indexes are improved by 16.19%, 26%, 3.81%, 1.65%, 3.91% and 8.01%, respectively. GMRE-iUnet can highlight the information and morphological features of the lesion areas and provide practical help for the aided diagnosis of lung tumors.