Image Modalities Research Articles

MACTFusion: Lightweight Cross Transformer for Adaptive Multimodal Medical Image Fusion.

Multimodal medical image fusion aims to integrate complementary information from different modalities of medical images. Deep learning methods, especially recent vision Transformers, have effectively improved image fusion performance. However, there are limitations for Transformers in image fusion, such as lacks of local feature extraction and cross-modal feature interaction, resulting in insufficient multimodal feature extraction and integration. In addition, the computational cost of Transformers is higher. To address these challenges, in this work, we develop an adaptive cross-modal fusion strategy for unsupervised multimodal medical image fusion. Specifically, we propose a novel lightweight cross Transformer based on cross multi-axis attention mechanism. It includes cross-window attention and cross-grid attention to mine and integrate both local and global interactions of multimodal features. The cross Transformer is further guided by a spatial adaptation fusion module, which allows the model to focus on the most relevant information. Moreover, we design a special feature extraction module that combines multiple gradient residual dense convolutional and Transformer layers to obtain local features from coarse to fine and capture global features. The proposed strategy significantly boosts the fusion performance while minimizing computational costs. Extensive experiments, including clinical brain tumor image fusion, have shown that our model can achieve clearer texture details and better visual quality than other state-of-the-art fusion methods.

End-to-end Prediction of EGFR Mutation Status with Denseformer.

Accurate genotyping of the epidermal growth factor receptor (EGFR) is critical for the treatment planning of lung adenocarcinoma. Currently, clinical identification of EGFR genotyping highly relies on biopsy and sequence testing which is invasive and complicated. Recent advancements in the integration of computed tomography (CT) imagery with deep learning techniques have yielded a non-invasive and straightforward way for identifying EGFR profiles. However, there are still many limitations for further exploration: 1) most of these methods still require physicians to annotate tumor boundaries, which are time-consuming and prone to subjective errors; 2) most of the existing methods are simply borrowed from computer vision field which does not sufficiently exploit the multi-level features for final prediction. To solve these problems, we propose a Denseformer framework to identify EGFR mutation status in a real end-to-end fashion directly from 3D lung CT images. Specifically, we take the 3D whole-lung CT images as the input of the neural network model without manually labeling the lung nodules. This is inspired by the medical report that the mutational status of EGFR is associated not only with the local tumor nodules but also with the microenvironment surrounded by the whole lung. Besides, we design a novel Denseformer network to fully explore the distinctive information across the different level features. The Denseformer is a novel network architecture that combines the advantages of both convolutional neural network (CNN) and Transformer. Denseformer directly learns from the 3D whole-lung CT images, which preserves the spatial location information in the CT images. To further improve the model performance, we designed a combined Transformer module. This module employs the Transformer Encoder to globally integrate the information of different levels and layers and use them as the basis for the final prediction. The proposed model has been tested on a lung adenocarcinoma dataset collected at the Affiliated Hospital of Zunyi Medical University. Extensive experiments demonstrated the proposed method can effectively extract meaningful features from 3D CT images to make accurate predictions. Compared with other state-of-the-art methods, Denseformer achieves the best performance among current methods using deep learning to predict EGFR mutation status based on a single modality of CT images.

Toward Accurate Cardiac MRI Segmentation With Variational Autoencoder-Based Unsupervised Domain Adaptation.

Accurate myocardial segmentation is crucial in the diagnosis and treatment of myocardial infarction (MI), especially in Late Gadolinium Enhancement (LGE) cardiac magnetic resonance (CMR) images, where the infarcted myocardium exhibits a greater brightness. However, segmentation annotations for LGE images are usually not available. Although knowledge gained from CMR images of other modalities with ample annotations, such as balanced-Steady State Free Precession (bSSFP), can be transferred to the LGE images, the difference in image distribution between the two modalities (i.e., domain shift) usually results in a significant degradation in model performance. To alleviate this, an end-to-end Variational autoencoder based feature Alignment Module Combining Explicit and Implicit features (VAMCEI) is proposed. We first re-derive the Kullback-Leibler (KL) divergence between the posterior distributions of the two domains as a measure of the global distribution distance. Second, we calculate the prototype contrastive loss between the two domains, bringing closer the prototypes of the same category across domains and pushing away the prototypes of different categories within or across domains. Finally, a domain discriminator is added to the output space, which indirectly aligns the feature distribution and forces the extracted features to be more favorable for segmentation. In addition, by combining CycleGAN and VAMCEI, we propose a more refined multi-stage unsupervised domain adaptation (UDA) framework for myocardial structure segmentation. We conduct extensive experiments on the MSCMRSeg 2019, MyoPS 2020 and MM-WHS 2017 datasets. The experimental results demonstrate that our framework achieves superior performances than state-of-the-art methods.

A New 6D Hyperchaos-Based Medical Image Encryption Scheme Using k-Fibonacci Numbers

In the era of information technology, hospitals, medical centers, doctors and health specialists share millions of images. Securing these images is essential to maintaining patients’ privacy. Several cryptographic algorithms are implemented to meet the increasing demands of transmission systems. As a contribution to securing these data, we propose a novel robust and efficient algorithm. We randomly generate numbers from a six-dimensional hyperchaotic system and Arnold’s cat map to confuse the original images. The confused images are then diffused using a k-Fibonacci matrix. The performance of the proposed algorithm was evaluated using several parameters tested on a set of reference images of different sizes and modalities. The obtained results are conclusive and the performances are excellent, sometimes exceeding those found by other recent algorithms proposed in the literature.

Generative Adversarial Network for Trimodal Medical Image Fusion Using Primitive Relationship Reasoning.

Medical image fusion has become a hot biomedical image processing technology in recent years. The technology coalesces useful information from different modal medical images onto an informative single fused image to provide reasonable and effective medical assistance. Currently, research has mainly focused on dual-modal medical image fusion, and little attention has been paid on trimodal medical image fusion, which has greater application requirements and clinical significance. For this, the study proposes an end-to-end generative adversarial network for trimodal medical image fusion. Utilizing a multi-scale squeeze and excitation reasoning attention network, the proposed method generates an energy map for each source image, facilitating efficient trimodal medical image fusion under the guidance of an energy ratio fusion strategy. To obtain the global semantic information, we introduced squeeze and excitation reasoning attention blocks and enhanced the global feature by primitive relationship reasoning. Through extensive fusion experiments, we demonstrate that our method yields superior visual results and objective evaluation metric scores compared to state-of-the-art fusion methods. Furthermore, the proposed method also obtained the best accuracy in the glioma segmentation experiment.

GeSeNet: A General Semantic-Guided Network With Couple Mask Ensemble for Medical Image Fusion.

At present, multimodal medical image fusion technology has become an essential means for researchers and doctors to predict diseases and study pathology. Nevertheless, how to reserve more unique features from different modal source images on the premise of ensuring time efficiency is a tricky problem. To handle this issue, we propose a flexible semantic-guided architecture with a mask-optimized framework in an end-to-end manner, termed as GeSeNet. Specifically, a region mask module is devised to deepen the learning of important information while pruning redundant computation for reducing the runtime. An edge enhancement module and a global refinement module are presented to modify the extracted features for boosting the edge textures and adjusting overall visual performance. In addition, we introduce a semantic module that is cascaded with the proposed fusion network to deliver semantic information into our generated results. Sufficient qualitative and quantitative comparative experiments (i.e., MRI-CT, MRI-PET, and MRI-SPECT) are deployed between our proposed method and ten state-of-the-art methods, which shows our generated images lead the way. Moreover, we also conduct operational efficiency comparisons and ablation experiments to prove that our proposed method can perform excellently in the field of multimodal medical image fusion. The code is available at https://github.com/lok-18/GeSeNet.

GAN Inversion for Data Augmentation to Improve Colonoscopy Lesion Classification.

A major challenge in applying deep learning to medical imaging is the paucity of annotated data. This study explores the use of synthetic images for data augmentation to address the challenge of limited annotated data in colonoscopy lesion classification. We demonstrate that synthetic colonoscopy images generated by Generative Adversarial Network (GAN) inversion can be used as training data to improve polyp classification performance by deep learning models. We invert pairs of images with the same label to a semantically rich and disentangled latent space and manipulate latent representations to produce new synthetic images. These synthetic images maintain the same label as the input pairs. We perform image modality translation (style transfer) between white light and narrow-band imaging (NBI). We also generate realistic synthetic lesion images by interpolating between original training images to increase the variety of lesion shapes in the training dataset. Our experiments show that GAN inversion can produce multiple colonoscopy data augmentations that improve the downstream polyp classification performance by 2.7% in F1-score and 4.9% in sensitivity over other methods, including state-of-the-art data augmentation. Testing on unseen out-of-domain data also showcased an improvement of 2.9% in F1-score and 2.7% in sensitivity. This approach outperforms other colonoscopy data augmentation techniques and does not require re-training multiple generative models. It also effectively uses information from diverse public datasets, even those not specifically designed for the targeted downstream task, resulting in strong domain generalizability. Project code and model: https://github.com/DurrLab/GAN-Inversion.

Tuberculosis Detection From Chest X-Ray Image Modalities Based on Transformer and Convolutional Neural Network

Tuberculosis Detection From Chest X-Ray Image Modalities Based on Transformer and Convolutional Neural Network

Machine Intelligence in Cerebrovascular and Endovascular Neurosurgery.

The advent of different realms of computational neurosurgery-including not only machine intelligence but also visualization techniques such as mixed reality and robotic applications-is beginning to impact both open vascular as well as endovascular neurosurgery. Especially in this relatively common patient population of often very fragile patients, with potential for devastating complications and clinical outcomes and sometimes highly complex pathologies, computer assistance could prove particularly useful. In this chapter, state-of-the-art applications of machine learning toward vascular patients are elucidated: Beginning from simple clinical diagnostic, prognostic, and predictive modeling, to the interpretation of medical imaging (radiomics, segmentation, and diagnostic assistance) and synthetic imaging (image modality conversion, super-resolution, and 2D-to-3D-synthesis), up to intraoperative applications of computer vision (robotic steering, rapid intraoperative histopathology, and anatomical and surgical phase recognition), and natural language processing (enabling model training and big data, documentation, and large language models)-this chapter provides a "tour de force" of machine intelligence in the realm of neurovascular medicine.

Comparative analysis of data augmentation methods for image modality

The object of research is forecasting processes in the case of short sets of tabular data. The subject of research is the data augmentation method for images. Achieving the goal occurs primarily from the study of existing machine learning tools and data augmentation methods for images. Further software development to implement various data augmentation methods and machine learning models for images. Approbation of the work was carried out by analyzing the effectiveness of various methods of data augmentation for images using quality metrics and statistical methods. Due to the results of the research, an analysis of the influence of various methods of data augmentation on the effectiveness of classifiers in images was carried out.

Multimodal Fusion of Different Medical Image Modalities using Optimised Hybrid Network

Multimodal Fusion of Different Medical Image Modalities using Optimised Hybrid Network

Medical Image Segmentation Review: The Success of U-Net.

Automatic medical image segmentation is a crucial topic in the medical domain and successively a critical counterpart in the computer-aided diagnosis paradigm. U-Net is the most widespread image segmentation architecture due to its flexibility, optimized modular design, and success in all medical image modalities. Over the years, the U-Net model has received tremendous attention from academic and industrial researchers who have extended it to address the scale and complexity created by medical tasks. These extensions are commonly related to enhancing the U-Net's backbone, bottleneck, or skip connections, or including representation learning, or combining it with a Transformer architecture, or even addressing probabilistic prediction of the segmentation map. Having a compendium of different previously proposed U-Net variants makes it easier for machine learning researchers to identify relevant research questions and understand the challenges of the biological tasks that challenge the model. In this work, we discuss the practical aspects of the U-Net model and organize each variant model into a taxonomy. Moreover, to measure the performance of these strategies in a clinical application, we propose fair evaluations of some unique and famous designs on well-known datasets. Furthermore, we provide a comprehensive implementation library with trained models. In addition, for ease of future studies, we created an online list of U-Net papers with their possible official implementation. All information is gathered in a GitHub repository https://github.com/NITR098/Awesome-U-Net.

Cascaded regression with dual CNN frame work for time effective detection of gliomas cancers

The determination of brain tumor growth primarily relies on the histopathological examination of biopsy samples. Tumor segmentation in the brain presents a significant challenge in medical image analysis due to its complexity. The ultimate goal is to accurately identify and isolate tumor regions. For the segmentation of brain tumors, a variety of deep-learning techniques have been developed, and they have produced promising results. However, achieving accurate segmentation requires the integration of multiple image modalities with varying contrasts. This makes manual segmentation impractical for larger studies, despite its accuracy. Deep learning's exceptional performance has made it an attractive method for quantitative analysis. The field of medical image analysis presents distinctive challenges that must be overcome to achieve optimal results. The ongoing strategy is obtrusive, tedious and inclined to manual mistakes. These weaknesses show that it is so fundamental to play out a completely computerized technique for the multi-characterization of cerebrum cancers in view of deep learning. Thus, this paper presents an efficient time-optimized and deep-learning model based on cascade regression (DLCR) to segment the tumor grade in the following stages: Data Acquisition in which data were obtained from the well-known brain repository BRATS2017, which included 215 HGG (High-Grade Gliomas) and 80 LGG (Low-Grade Gliomas) glioma cases. Fully Convolutional Neural Network (FCNN) preprocessing was used to remove noise and anomalies from the raw data, and Gaussian Mixture Model feature extraction was used to extract features from the preprocessed image and finally the proposed DLCR model for grade identification. Experimental findings indicate that the suggested system surpasses other pre-existing models in various aspects (accuracy: 0.96, sensitivity:0.97, precision:0.88).

M2Trans: Multi-Modal Regularized Coarse-to-Fine Transformer for Ultrasound Image Super-Resolution.

Ultrasound image super-resolution (SR) aims to transform low-resolution images into high-resolution ones, thereby restoring intricate details crucial for improved diagnostic accuracy. However, prevailing methods relying solely on image modality guidance and pixel-wise loss functions struggle to capture the distinct characteristics of medical images, such as unique texture patterns and specific colors harboring critical diagnostic information. To overcome these challenges, this paper introduces the Multi-Modal Regularized Coarse-to-fine Transformer (M2Trans) for Ultrasound Image SR. By integrating the text modality, we establish joint image-text guidance during training, leveraging the medical CLIP model to incorporate richer priors from text descriptions into the SR optimization process, enhancing detail, structure, and semantic recovery. Furthermore, we propose a novel coarse-to-fine transformer comprising multiple branches infused with self-attention and frequency transforms to efficiently capture signal dependencies across different scales. Extensive experimental results demonstrate significant improvements over state-of-the-art methods on benchmark datasets, including CCA-US, US-CASE, and our newly created dataset MMUS1K, with a minimum improvement of 0.17dB, 0.30dB, and 0.28dB in terms of PSNR. Our code and dataset will be available at: https://github.com/eezkni/M2Trans.

SWI and CTP fusion model based on sparse representation method to predict cerebral infarction trend.

SWI image signal is related to venous reflux disorder and perfusion defect. Computed tomography perfusion (CTP) contains perfusion information in space and time. There is a complementary basis between them to affect the prognosis of cerebral infarction. Sixty-six patients included in the retrospective study were designated as the training set. Effective perfusion indicator features and imaging radiomic features of the peri-infarction area on Susceptibility weighted imaging (SWI) and CTP modality images were extracted from each case. Thirty-three patients from the prospectively included group were designated as the test set of the machine learning model based on a sparse representation method. The predicted results were compared with the DWI results of the patients' 7-10 days review to assess the validity and accuracy of the prediction. The AUC of the SWI + CTP integrated model was 0.952, the ACC was 0.909, the SEN was 0.889, and the SPE was 0.933. The prediction performance is the highest. Compared with the value of AUC: the SWI model is 0.874, inferior to the performance of the SWI + CTP model, and the CTP model is 0.715. The prediction efficiency of the changing trend of infarction volume is further improved by the correlation between the combination of the two image features.

FMCNet + : Feature-Level Modality Compensation for Visible-Infrared Person Re-Identification.

For visible-infrared person re-identification (VI-ReID), current models that compensate modality-specific information strive to generate missing modality images from existing ones to bridge the cross-modality discrepancies. Despite that, those generated images often suffer from low qualities due to the significant modality gap and include interfering information, e.g., inconsistent colors, thus severely degrading the subsequent VI-ReID performance. Alternatively, we propose a feature-level modality compensation network, i.e., FMCNet+, for VI-ReID in this article as an improved version of our previous work (FMCNet). The core of FMCNet + is to compensate for the missing modality-specific information at the feature level, rather than at the image level, enabling our model to generate more person-related and discriminative modality-specific features for VI-ReID. Concretely, FMCNet+ aims to progressively generate missing modality-specific features by fully exploring the relationships among single-modality features, modality-shared features, and modality-specific features, instead of directly generating them through a generative adversarial way as in the previous FMCNet. To this end, three modules, i.e., single-modality feature decomposition (SFD), modality characteristic dictionary learning (MCDL), and missing modality-specific feature compensation (MMFC), are incorporated in FMCNet+. Experimental results demonstrate the superiority of our proposed FMCNet+ over existing ones, especially for those that compensate for modality-specific information at the image level. Our intriguing findings highlight the necessity of feature-level modality compensation in VI-ReID. Our code and pre-trained models will be released on https://github.com/jssyzsfzy/FMCNet_series.

Emotion Recognition through Text, Speech and Image

Abstract: Emotion recognition plays a vital role in various applications, such as human-computer interaction, affective computing, and psychological assessment. This comparative study investigates the effectiveness of emotion recognition through text, image, and speech modalities. Our research aims to analyze and compare state-of-art approaches in each modality and identify their strengths and limitations. Conducting an exhaustive literature review enabled us to understand existing methodologies, datasets, and evaluation metrics. The research methodology and implementation include data collection, preprocessing, feature extraction, and the application of machine learning and deep learning models. The results provide insights into the performance of different modalities, paving the way for advancements in emotion recognition research.

Histopathologic study of sinonasal lesions: A hospital based retrospective study, from January 2016 to August 2020

Background: Sinonasal area is a host to various neoplastic and non- neoplastic lesions. The presenting symptoms of sinonasal lesions are similar and pose difficulty for diagnosis based on clinical features and advanced imaging modalities, which makes histopathology the principal diagnostic tool approach for these lesions. The aim of this study is to determine the various histopathologic types of Sinonasal lesions, their classification and relative distribution with regards to age and sex in our setting. Materials and Methods: A retrospective cross-sectional descriptive study conducted on 306 cases of sinonasal lesions over the period from January 2016 to August 2020. All the sinonasal tissues were received and diagnosed at histopathology section of Department of Pathology in Tikur Anbessa Specialized Hospital, Addis Ababa, Ethiopia. The Pathology reports were collected from the pathology data archives; and the variables of study were extracted by using a data extraction sheet, and data analysis was done using SPSS version 26.0. Results: Most affected age group was 21-30 years 81(26.5%). Male predominance was observed with M: F ratio of 1.4:1.Nasal cavity was the commonest anatomical site involved 237(77.5%).There were 137(44.8%) Non-neoplastic lesions and 169(55.2%) Neoplastic lesions. Inflammatory Sinonasal Polyps 100(73.0%) were the most common among non-neoplastic lesions; inverted Sinonasal Papilloma 34(41%) the most common among the benign neoplastic and squamous cell carcinoma and nasopharyngeal carcinoma accounting for 21(25.6%) cases each, were the commonest malignant lesions. Conclusion: clinical findings and advanced image modalities can reach to a presumptive diagnosis for the sinonasal lesions but histopathology remains the gold standard for categorizing and diagnosis of non-neoplastic and neoplastic lesions of sinonasal tract.

Pattern of Endothelialization in Left Atrial Appendage Occluder by Optic Coherence Tomography: A Pilot Study.

Implantation of the left atrial appendage occluder (LAAO) has been proven to prevent stroke effectively in patients with atrial fibrillation who cannot tolerate anticoagulants. Incomplete endothelization of LAAO may cause device-related thrombus, and currently no good image modality exists to clearly see LAAO endothelialization. We aimed to use coronary optic coherence tomography (OCT) to visualize LAAO endothelialization. We enrolled 14 patients (72.8±9.4 years old) undergoing pulmonary vein isolation with a preexisting LAAO implanted more than 1 year ago (5 Watchman and 9 Amulet). After pulmonary vein isolation, we did OCT via steerable sheath and coronary guiding catheter to adjust OCT probe location and injected contrast medium to visualize the LAAO surface. Invitro testing was also performed to see the bare occluder. Invitro OCT showed the surface of the bare device as an interrupted granule pattern, which included the Watchman surface polytetrafluoroethylene membrane string, Amulet disc metal strut, and inner polytetrafluoroethylene membrane string. In the implanted Watchman, OCT showed endothelialization as a smooth surface layer with noninterrupted coarser granules. In the implanted Amulet, OCT showed endothelialization as thin (early) or thick (late) endothelialization layer covering struts with OCT shadows. Among patients with Watchman, 2 showed no, 2 early, and 1 complete endothelialization. Among patients with Amulet, 2 showed no, 3 early, and 4 late endothelialization. We demonstrated the feasibility of OCT to visualize LAAO endothelization with high resolution. Further studies are needed to determine antithrombotic regimens if incomplete endothelization is detected. A new OCT catheter may be designed specifically for LAAO.

RFTNet: Region–Attention Fusion Network Combined with Dual-Branch Vision Transformer for Multimodal Brain Tumor Image Segmentation

Brain tumor image segmentation plays a significant auxiliary role in clinical diagnosis. Recently, deep learning has been introduced into multimodal segmentation tasks, which construct various Convolutional Neural Network (CNN) structures to achieve excellent performance. However, most CNN-based segmentation methods have poor capability for global feature extraction. Transformer is good at modeling long-distance dependencies, but it can cause local information loss and usually has a high computational complexity. In addition, it is difficult to fully exploit the brain tumor features of different modalities. To address these issues, in this paper, we propose a region–attention fusion (RAF) network that combines a dual-branch vision Transformer (DVT), called RFTNet. In RFTNet, the DVT is exploited to capture the delicate local information and global semantics separately by two branches. Meanwhile, a novel RAF is employed to effectively fuse the images of the different modalities. Finally, we design a new hybrid loss function, called region-mixed loss function (RML) to calculate the importance of each pixel and solve the problem of class imbalance. The experiments on BrasTS2018 and BraTS2020 datasets show that our method obtains a higher segmentation accuracy than other models. Furthermore, ablation experiments prove the effectiveness of each key component in RFTNet.