Multimodal Medical Image Research Articles

Cross-Modal Longitudinal Brain MRI Reconstruction with a Conditional Residual Transformer Network

Multimodal medical image synthesis plays a crucial role in enhancing diagnostic accuracy and understanding disease progression, particularly in Alzheimers disease (AD). However, existing methods often focus on single-modality or single-time synthesis, overlooking the complexities of integrating multiple imaging modalities and longitudinal data. Furthermore, these models tend to ignore patient-specific factors like age, health conditions, and sex, limiting their practical applicability in clinical settings. To address these limitations, we propose CrossSim, a novel residual vision transformer-based framework for multimodal and longitudinal medical image synthesis. Our model integrates cross-attention-based feature fusion to handle personalized data such as age, health state, and sex. This allows for the generation of more clinically relevant synthetic images that better represent the complexities of real medical data. In contrast to existing methods, CrossSim excels in synthesizing images that accurately reflect changes over time and across modalities. We conduct extensive experiments on the ADNI dataset to evaluate the effectiveness of our approach. The results demonstrate significant improvements in key metrics such as PSNR, SSIM, and RMSE, confirming the superior performance of CrossSim in both qualitative and quantitative analyses. This study emphasizes the clinical significance of CrossSim, offering a valuable tool for enhancing diagnostic accuracy and advancing our understanding of Alzheimers disease progression.

OT-CGSF: Optimal Transport and CNN-Guided Score-Based Fusion for Multimodal Medical Image Integration

OT-CGSF: Optimal Transport and CNN-Guided Score-Based Fusion for Multimodal Medical Image Integration

Enhanced Gan-Based Data Augmentation for Multimodal Medical Images Registration

Enhanced Gan-Based Data Augmentation for Multimodal Medical Images Registration

Asymmetric Adaptive Heterogeneous Network for Multi-Modality Medical Image Segmentation

Asymmetric Adaptive Heterogeneous Network for Multi-Modality Medical Image Segmentation

M2OCNN: Many-to-One Collaboration Neural Networks for simultaneously multi-modal medical image synthesis and fusion

M2OCNN: Many-to-One Collaboration Neural Networks for simultaneously multi-modal medical image synthesis and fusion

An adaptive and lightweight YOLOv5 detection model for lung tumor in PET/CT images

Multi-modal medical images are important in tumor lesion detection. However, the existing detection models only use single-modal to detect lesions, a multi-modal semantic correlation is not enough to consider and lacks ability to express the shape, size, and contrast degree features of lesions. A Cross Modal YOLOv5 model (CMYOLOv5) is proposed. Firstly, there are two networks, auxiliary network is consisted by dual-branch structure to extract semantic information from PET and CT, backbone network is consisted by YOLOv5 to extract semantic information from PET/CT. Secondly, Cross-modal Features Fusion (CFF) is designed in auxiliary network to fuse PET functional information and CT anatomical information. Self-Adaptive Attention Fusion (AAF) is designed in backbone network to fuse and enhance three-modal complementary information. Thirdly, Self-Adaptive Transformer (SAT) is designed in feature enhance neck. Using Transformer with deformable attention mechanism to focus on lung tumor region. Using MLP with channel attention mechanism to enhance features representation ability of lung tumor region. Finally, Reparameter Residual Block (RRB) and Reparameter Convolution operation (RC) are designed to fully learn richer PET, CT and PET/CT feature. Comparative experiments are conducted on clinical lung tumor PET/CT multi-modality dataset, the effectiveness of CMYOLOv5 is verified by Precision, Recall, mAP, F1, FPS, and training time, experimental results are 97.16%, 96.41%, 97.18%, 96.78%, 96.37 and 3912 s. CMYOLOv5 has high precision in the detection of irregular lung tumors, which is superior to the existing advanced methods.

Multiple attention channels aggregated network for multimodal medical image fusion.

In clinical practices, doctors usually need to synthesize several single-modality medical images for diagnosis, which is a time-consuming and costly process. With this background, multimodal medical image fusion (MMIF) techniques have emerged to synthesize medical images of different modalities, providing a comprehensive and objective interpretation of the lesion. Although existing MMIF approaches have shown promising results, they often overlook the importance of multiscale feature diversity and attention interaction, which are essential for superior visual outcomes. This oversight can lead to diminished fusion performance. To bridge the gaps, we introduce a novel approach that emphasizes the integration of multiscale features through a structured decomposition and attention interaction. Our method first decomposes the source images into three distinct groups of multiscale features by stacking different numbers of diverse branch blocks. Then, to extract global and local information separately for each group of features, we designed the convolutional and Transformer block attention branch. These two attention branches make full use of channel and spatial attention mechanisms and achieve attention interaction, enabling the corresponding feature channels to fully capture local and global information and achieve effective inter-block feature aggregation. For the MRI-PET fusion type, MACAN achieves average improvements of 24.48%, 27.65%, 19.24%, 27.32%, 18.51%, and 10.33% over the compared methods in terms of Qcb, AG, SSIM, SF, Qabf, and VIF metrics, respectively. Similarly, for the MRI-SPECT fusion type, MACAN outperforms the compared methods with average improvements of 29.13%, 26.43%, 18.20%, 27.71%, 16.79%, and 10.38% in the same metrics. In addition, our method demonstrates promising results in segmentation experiments. Specifically, for the T2-T1ce fusion, it achieves a Dice coefficient of 0.60 and a Hausdorff distance of 15.15. Comparable performance is observed for the Flair-T1ce fusion, with a Dice coefficient of 0.60 and a Hausdorff distance of 13.27. The proposed multiple attention channels aggregated network (MACAN) can effectively retain the complementary information from source images. The evaluation of MACAN through medical image fusion and segmentation experiments on public datasets demonstrated its superiority over the state-of-the-art methods, both in terms of visual quality and objective metrics. Our code is available at https://github.com/JasonWong30/MACAN.

Multi-modal Medical Image Fusion Approach Utilizing Gradient Domain Guided Image Filtering.

Currently, most multimodal medical image fusion techniques focus solely on integrating the edge details of image features, often overlooking color preservation from the source images. Hence, this paper proposes a multi-channel fusion algorithm based on gradient domain-guided image filtering. This study aims to enhance the color preservation of source images in multimodal medical image fusion algorithms. Utilizing gradient field-guided image filters for image smoothing, the process involves constructing different image layers, decomposing using wavelet transforms, and downsampling. Various fusion rules are then applied before inverse wavelet transformation. Regarding MSE, CCI, PSNR, SSIM, DD, SM, and other metrics, the proposed algorithm consistently ranks highest compared to alternative methods. Through both subjective and objective analyses, experimental results substantiate the significant edge-preserving effects of the proposed fusion algorithm while effectively maintaining image fidelity and spectral integrity.

The software investigation of the new multimodal imaging system for one-stop health examination

The present investigation aimed to explore the software for one-stop health examination by multimodal and multifunction imaging. In order to implement the system, the software framework architecture and its views are proposed in the article. Taking magnetic acoustic magnetic induction imaging technology as an example, based on the study of the imaging mechanism such as a deeper understanding of the essence of the crucial charge transfer process in biology of each modality, in order to better understand life phenomena. And a unified data format is used for image fusion, stitching, detection, and recognition of each modality, simultaneously ensuring the reliability and credibility of the software. The multimodal multi-function one-stop medical imaging system is feasible to health examination, and its unique innovation and methodology will certainly promote the commercialization for the target system. Results indicate the system will achieve faster imaging speed, higher resolution of the applications of biomedical imaging.

Overcoming the challenges of multi-modal medical image sharing: A novel data distillation strategy via contrastive learning

Multi-modal medical images serve as a critical basis for disease diagnosis. However, sharing multi-modal medical data between hospitals is a significant challenge due to privacy protection concerns as well as the substantial costs associated with data transfer and storage. These hurdles are further compounded by two primary issues: (1) the complexity involved in sharing data features, which arises from the intricate process of multi-modal image registration; (2) the difficulty and high cost of acquiring accurate data labels. To address these issues, we propose a novel multi-modal medical dataset distillation method based on contrastive learning. We consider each modality as a unique approach to data augmentation for the corresponding imaging region, offering a unique perspective on the same anatomical region, which is then fused to collectively contribute to a comprehensive global representation of the medical image. Validation results on two datasets indicate our method is capable of reducing the original dataset to just 5% of its original size while retaining classification performance comparable to that of the full dataset, and notably, this efficiency is achieved using only 30% of the total labels, which significantly lowers the cost and effort associated with dataset labeling.

Relationship between pancreatic morphological changes and diabetes in autoimmune pancreatitis: Multimodal medical imaging assessment has important potential.

Autoimmune pancreatitis (AIP) is a special type of chronic pancreatitis with clinical symptoms of obstructive jaundice and abdominal discomfort; this condition is caused by autoimmunity and marked by pancreatic fibrosis and dysfunction. Previous studies have revealed a close relationship between early pancreatic atrophy and the incidence rate of diabetes in type 1 AIP patients receiving steroid treatment. Shimada et al performed a long-term follow-up study and reported that the pancreatic volume (PV) of these patients initially exponentially decreased but then slowly decreased, which was considered to be an important factor related to diabetes; moreover, serum IgG4 levels were positively correlated with PV during follow-up. In this letter, regarding the original study presented by Shimada et al, we present our insights and discuss how multimodal medical imaging and artificial intelligence can be used to better assess the relationship between pancreatic morphological changes and diabetes in patients with AIP.

KARAN: Mitigating Feature Heterogeneity and Noise for Efficient and Accurate Multimodal Medical Image Segmentation

KARAN: Mitigating Feature Heterogeneity and Noise for Efficient and Accurate Multimodal Medical Image Segmentation

Enhanced breast cancer diagnosis through integration of computer vision with fusion based joint transfer learning using multi modality medical images

Breast cancer (BC) is a type of cancer which progresses and spreads from breast tissues and gradually exceeds the entire body; this kind of cancer originates in both sexes. Prompt recognition of this disorder is most significant in this phase, and it is measured by providing patients with the essential treatment so their efficient lifetime can be protected. Scientists and researchers in numerous studies have initiated techniques to identify tumours in early phases. Still, misperception in classifying skeptical lesions can be due to poor image excellence and dissimilar breast density. BC is a primary health concern, requiring constant initial detection and improvement in analysis. BC analysis has made major progress recently with combining multi-modal image modalities. These studies deliver an overview of the segmentation, classification, or grading of numerous cancer types, including BC, by employing conventional machine learning (ML) models over hand-engineered features. Therefore, this study uses multi-modality medical imaging to propose a Computer Vision with Fusion Joint Transfer Learning for Breast Cancer Diagnosis (CVFBJTL-BCD) technique. The presented CVFBJTL-BCD technique utilizes feature fusion and DL models to effectively detect and identify BC diagnoses. The CVFBJTL-BCD technique primarily employs the Gabor filtering (GF) technique for noise removal. Next, the CVFBJTL-BCD technique uses a fusion-based joint transfer learning (TL) process comprising three models, namely DenseNet201, InceptionV3, and MobileNetV2. The stacked autoencoders (SAE) model is implemented to classify BC diagnosis. Finally, the horse herd optimization algorithm (HHOA) model is utilized to select parameters involved in the SAE method optimally. To demonstrate the improved results of the CVFBJTL-BCD methodology, a comprehensive series of experimentations are performed on two benchmark datasets. The comparative analysis of the CVFBJTL-BCD technique portrayed a superior accuracy value of 98.18% and 99.15% over existing methods under Histopathological and Ultrasound datasets.

MTCSNet: One-Stage Learning and Two-Point Labeling are Sufficient for Cell Segmentation.

Deep convolution neural networks have been widely used in medical image analysis, such as lesion identification in whole-slide images, cancer detection, and cell segmentation, etc. However, it is often inevitable that researchers try their best to refine annotations so as to enhance the model performance, especially for cell segmentation task. Weakly supervised learning can greatly reduce the workload of annotations, while there is still a huge performance gap between the weakly and fully supervised learning approaches. In this work, we propose a weakly-supervised cell segmentation method, namely Multi-Task Cell Segmentation Network (MTCSNet), for multi-modal medical images, including pathological, brightfield, fluorescent, phase-contrast and differential interference contrast images. MTCSNet is learnt in a single-stage training manner, where only two annotated points for each cell provide supervision information, and the first one is the centroid, the second one is its boundary. Additionally, five auxiliary tasks are elaborately designed to train the network, including two pixel-level classifications, a pixel-level regression, a local temperature scaling and an instance-level distance regression task, which is proposed to regress the distances between the cell centroid and its boundaries in eight orientations. The experimental results indicate that our method outperforms all state-of-the-art weakly-supervised cell segmentation approaches on public multi-modal medical image datasets. The promising performance also shows that a single-stage learning with two-point labeling approach are sufficient for cell segmentation, instead of fine contour delineation. The codes are available at: https://github.com/binging512/MTCSNet.

A Framework for Multimodal Medical Image Interaction.

Medical doctors rely on images of the human anatomy, such as magnetic resonance imaging (MRI), to localize regions of interest in the patient during diagnosis and treatment. Despite advances in medical imaging technology, the information conveyance remains unimodal. This visual representation fails to capture the complexity of the real, multisensory interaction with human tissue. However, perceiving multimodal information about the patient's anatomy and disease in real-time is critical for the success of medical procedures and patient outcome. We introduce a Multimodal Medical Image Interaction (MMII) framework to allow medical experts a dynamic, audiovisual interaction with human tissue in three-dimensional space. In a virtual reality environment, the user receives physically informed audiovisual feedback to improve the spatial perception of anatomical structures. MMII uses a model-based sonification approach to generate sounds derived from the geometry and physical properties of tissue, thereby eliminating the need for hand-crafted sound design. Two user studies involving 34 general and nine clinical experts were conducted to evaluate the proposed interaction framework's learnability, usability, and accuracy. Our results showed excellent learnability of audiovisual correspondence as the rate of correct associations significantly improved (p < 0.001) over the course of the study. MMII resulted in superior brain tumor localization accuracy (p < 0.05) compared to conventional medical image interaction. Our findings substantiate the potential of this novel framework to enhance interaction with medical images, for example, during surgical procedures where immediate and precise feedback is needed.

GeSeNet: A General Semantic-Guided Network With Couple Mask Ensemble for Medical Image Fusion.

At present, multimodal medical image fusion technology has become an essential means for researchers and doctors to predict diseases and study pathology. Nevertheless, how to reserve more unique features from different modal source images on the premise of ensuring time efficiency is a tricky problem. To handle this issue, we propose a flexible semantic-guided architecture with a mask-optimized framework in an end-to-end manner, termed as GeSeNet. Specifically, a region mask module is devised to deepen the learning of important information while pruning redundant computation for reducing the runtime. An edge enhancement module and a global refinement module are presented to modify the extracted features for boosting the edge textures and adjusting overall visual performance. In addition, we introduce a semantic module that is cascaded with the proposed fusion network to deliver semantic information into our generated results. Sufficient qualitative and quantitative comparative experiments (i.e., MRI-CT, MRI-PET, and MRI-SPECT) are deployed between our proposed method and ten state-of-the-art methods, which shows our generated images lead the way. Moreover, we also conduct operational efficiency comparisons and ablation experiments to prove that our proposed method can perform excellently in the field of multimodal medical image fusion. The code is available at https://github.com/lok-18/GeSeNet.

Dual-attention transformer-based hybrid network for multi-modal medical image segmentation

Accurate medical image segmentation plays a vital role in clinical practice. Convolutional Neural Network and Transformer are mainstream architectures for this task. However, convolutional neural network lacks the ability of modeling global dependency while Transformer cannot extract local details. In this paper, we propose DATTNet, DualATTentionNetwork, an encoder-decoder deep learning model for medical image segmentation. DATTNet is exploited in hierarchical fashion with two novel components: (1) Dual Attention module is designed to model global dependency in spatial and channel dimensions. (2) Context Fusion Bridge is presented to remix the feature maps with multiple scales and construct their correlations. The experiments on ACDC, Synapse and Kvasir-SEG datasets are conducted to evaluate the performance of DATTNet. Our proposed model shows superior performance, effectiveness and robustness compared to SOTA methods, with mean Dice Similarity Coefficient scores of 92.2%, 84.5% and 89.1% on cardiac, abdominal organs and gastrointestinal poly segmentation tasks. The quantitative and qualitative results demonstrate that our proposed DATTNet attains favorable capability across different modalities (MRI, CT, and endoscopy) and can be generalized to various tasks. Therefore, it is envisaged as being potential for practicable clinical applications. The code has been released on https://github.com/MhZhang123/DATTNet/tree/main.

Semantic Segmentation of the Prostate Based on Onefold and Joint Multimodal Medical Images Using YOLOv4 and U-Net

Magnetic Resonance Imaging is increasing in importance in prostate cancer diagnosis due to the high accuracy and quality of the examination procedure. However, this process requires a time-consuming analysis of the results. Currently, machine vision is widely used in many areas. It enables automation and support in radiological studies. Successful detection of primary prostate tumors depends on the effective segmentation of the prostate itself. At times, a CT scan may be performed; alternatively, MRI may be the selected option. The data always reach a bottleneck stage. This paper presents the effective training of deep learning models to segment the prostate based on onefold and multimodal medical images. This approach supports the computer-aided diagnosis (CAD) system for radiologists as the first step in cancer exams. A comparison of two approaches designed for prostate segmentation is described. The first combines YOLOv4, the object detection neural network, and U-Net for a semantic segmentation based on onefold modality MRI images. The second presents the same method trained on multimodal images—a CT and MRI mixed dataset. The learning process was carried out in a cloud environment using GPU cards. The experiments are based on data from 120 patients who have undergone MRI and CT examinations. Several metrics evaluated the trained models. In the prostate semantic segmentation process, better results were achieved by mixed MRI with CT datasets. The best model achieved the value of 0.9685 for the Sørensen–Dice coefficient for the threshold value of 0.6.

An ensemble deep learning model for medical image fusion with Siamese neural networks and VGG-19.

Multimodal medical image fusion methods, which combine complementary information from many multi-modality medical images, are among the most important and practical approaches in numerous clinical applications. Various conventional image fusion techniques have been developed for multimodality image fusion. Complex procedures for weight map computing, fixed fusion strategy and lack of contextual understanding remain difficult in conventional and machine learning approaches, usually resulting in artefacts that degrade the image quality. This work proposes an efficient hybrid learning model for medical image fusion using pre-trained and non-pre-trained networks i.e. VGG-19 and SNN with stacking ensemble method. The model leveraging the unique capabilities of each architecture, can effectively preserve the detailed information with high visual quality, for numerous combinations of image modalities in image fusion challenges, notably improved contrast, increased resolution, and lower artefacts. Additionally, this ensemble model can be more robust in the fusion of various combinations of source images that are publicly available from Havard-Medical-Image-Fusion Datasets, GitHub. and Kaggle. Our proposed model performance is superior in terms of visual quality and performance metrics to that of the existing fusion methods in literature like PCA+DTCWT, NSCT, DWT, DTCWT+NSCT, GADCT, CNN and VGG-19.

Brain Tumor Detection through Image Fusion Using Cross Guided Filter and Convolutional Neural Network

This Data fusion has become a significant issue in diagnostic imaging, particularly in medical applications like radiation and guided image surgery. Medical image fusion aims to enhance the precision of tumor diagnosis, by preserving the salient information and characteristics of the original images in the fused image. It has been shown that guided filters are capable of maintaining edges well. In this paper, we propose a novel cross-guided filter-based fusion approach for multimodal medical images utilizing convolutional neural networks. The cross-guided filter is used in the proposed algorithm to extract the detailed features from the source images. Convolutional neural networks are used to generate the feature weights of source images derived from the detail layers. The weighted average rule is used to merge the source images based on these weights. We used thirty distinct types of medical images from diverse sources to compare the effectiveness of the proposed strategy to that of existing methods, both numerically and visually. The experimental findings demonstrated that, in terms of both objective evaluation and qualitative image quality, the suggested system performs better than other standard methods already in use. The quantitative results show that compared to existing methods under consideration for comparison, the proposed algorithm improves mutual information by 25%, image entropy by 9.5%, spatial frequency by 21%, standard deviation by 18.1%, structural similarity index by 30%, and edge strength of the fused image by 39%.