Medical Image Synthesis Research Articles

A Comprehensive Analysis of GAN-Based Techniques for Medical Image Synthesis

A Comprehensive Analysis of GAN-Based Techniques for Medical Image Synthesis

Artificial Intelligence Techniques in Medical Imaging: A Systematic Review

This scientific review presents a comprehensive overview of medical imaging modalities and their diverse applications in artificial intelligence (AI)-based disease classification and segmentation. The paper begins by explaining the fundamental concepts of AI, machine learning (ML), and deep learning (DL). It provides a summary of their different types to establish a solid foundation for the subsequent analysis. The prmary focus of this study is to conduct a systematic review of research articles that examine disease classification and segmentation in different anatomical regions using AI methodologies. The analysis includes a thorough examination of the results reported in each article, extracting important insights and identifying emerging trends. Moreover, the paper critically discusses the challenges encountered during these studies, including issues related to data availability and quality, model generalization, and interpretability. The aim is to provide guidance for optimizing technique selection. The analysis highlights the prominence of hybrid approaches, which seamlessly integrate ML and DL techniques, in achieving effective and relevant results across various disease types. The promising potential of these hybrid models opens up new opportunities for future research in the field of medical diagnosis. Additionally, addressing the challenges posed by the limited availability of annotated medical images through the incorporation of medical image synthesis and transfer learning techniques is identified as a crucial focus for future research efforts.

Unveiling the future of breast cancer assessment: a critical review on generative adversarial networks in elastography ultrasound.

Elastography Ultrasound provides elasticity information of the tissues, which is crucial for understanding the density and texture, allowing for the diagnosis of different medical conditions such as fibrosis and cancer. In the current medical imaging scenario, elastograms for B-mode Ultrasound are restricted to well-equipped hospitals, making the modality unavailable for pocket ultrasound. To highlight the recent progress in elastogram synthesis, this article performs a critical review of generative adversarial network (GAN) methodology for elastogram generation from B-mode Ultrasound images. Along with a brief overview of cutting-edge medical image synthesis, the article highlights the contribution of the GAN framework in light of its impact and thoroughly analyzes the results to validate whether the existing challenges have been effectively addressed. Specifically, This article highlights that GANs can successfully generate accurate elastograms for deep-seated breast tumors (without having artifacts) and improve diagnostic effectiveness for pocket US. Furthermore, the results of the GAN framework are thoroughly analyzed by considering the quantitative metrics, visual evaluations, and cancer diagnostic accuracy. Finally, essential unaddressed challenges that lie at the intersection of elastography and GANs are presented, and a few future directions are shared for the elastogram synthesis research.

Backdoor attack and defense in federated generative adversarial network-based medical image synthesis.

Deep Learning-based image synthesis techniques have been applied in healthcare research for generating medical images to support open research and augment medical datasets. Training generative adversarial neural networks (GANs) usually require large amounts of training data. Federated learning (FL) provides a way of training a central model using distributed data while keeping raw data locally. However, given that the FL server cannot access the raw data, it is vulnerable to backdoor attacks, an adversarial by poisoning training data. Most backdoor attack strategies focus on classification models and centralized domains. It is still an open question if the existing backdoor attacks can affect GAN training and, if so, how to defend against the attack in the FL setting. In this work, we investigate the overlooked issue of backdoor attacks in federated GANs (FedGANs). The success of this attack is subsequently determined to be the result of some local discriminators overfitting the poisoned data and corrupting the local GAN equilibrium, which then further contaminates other clients when averaging the generator's parameters and yields high generator loss. Therefore, we proposed FedDetect, an efficient and effective way of defending against the backdoor attack in the FL setting, which allows the server to detect the client's adversarial behavior based on their losses and block the malicious clients. Our extensive experiments on two medical datasets with different modalities demonstrate the backdoor attack on FedGANs can result in synthetic images with low fidelity. After detecting and suppressing the detected malicious clients using the proposed defense strategy, we show that FedGANs can synthesize high-quality medical datasets (with labels) for data augmentation to improve classification models' performance.

A generalized dual-domain generative framework with hierarchical consistency for medical image reconstruction and synthesis

Medical image reconstruction and synthesis are critical for imaging quality, disease diagnosis and treatment. Most of the existing generative models ignore the fact that medical imaging usually occurs in the acquisition domain, which is different from, but associated with, the image domain. Such methods exploit either single-domain or dual-domain information and suffer from inefficient information coupling across domains. Moreover, these models are usually designed specifically and not general enough for different tasks. Here we present a generalized dual-domain generative framework to facilitate the connections within and across domains by elaborately-designed hierarchical consistency constraints. A multi-stage learning strategy is proposed to construct hierarchical constraints effectively and stably. We conducted experiments for representative generative tasks including low-dose PET/CT reconstruction, CT metal artifact reduction, fast MRI reconstruction, and PET/CT synthesis. All these tasks share the same framework and achieve better performance, which validates the effectiveness of our framework. This technology is expected to be applied in clinical imaging to increase diagnosis efficiency and accuracy.

2D brain MRI image synthesis based on lightweight denoising diffusion probabilistic model

In recent years, brain health has received increasing attention, but conventional acquisition of brain MRI (magnetic resonance imaging) images still suffer from issues such as missing data, artifacts, and high costs, which hinders research and diagnosis. With the application of deep learning in medical image synthesis, low-cost, efficient, and high-quality medical MRI synthesis techniques have become a prominent research focus and have gradually matured. However, traditional methods for synthesizing magnetic resonance imaging (MRI) mostly rely on generative adversarial networks, which require fine-tuning of parameters and learning rates to achieve stringent Nash equilibrium conditions, leading to problems such as gradient explosions and mode collapse. Building upon the latest research in synthetic models DDPM (denoising diffusion probabilistic model), we propose a novel approach for 2D brain MRI image synthesis based on a lightweight denoising diffusion probabilistic model. This method improves the attention module in the denoising diffusion probabilistic model to make it more lightweight. Additionally, we adopt the smooth L1 loss function as a replacement for the traditional mean absolute error (L1 loss) by comparing the error between the 2D brain MRI images with added noise and the real noise for training the model. Finally, we validate the proposed model on the MRI Brain Tumor Classification dataset, demonstrating that it achieves high-quality synthesis results while significantly reducing the parameter count of the DDPM model.

Machine Learning for Medical Image Translation: A Systematic Review.

CT scans are often the first and only form of brain imaging that is performed to inform treatment plans for neurological patients due to its time- and cost-effective nature. However, MR images give a more detailed picture of tissue structure and characteristics and are more likely to pick up abnormalities and lesions. The purpose of this paper is to review studies which use deep learning methods to generate synthetic medical images of modalities such as MRI and CT. A literature search was performed in March 2023, and relevant articles were selected and analyzed. The year of publication, dataset size, input modality, synthesized modality, deep learning architecture, motivations, and evaluation methods were analyzed. A total of 103 studies were included in this review, all of which were published since 2017. Of these, 74% of studies investigated MRI to CT synthesis, and the remaining studies investigated CT to MRI, Cross MRI, PET to CT, and MRI to PET. Additionally, 58% of studies were motivated by synthesizing CT scans from MRI to perform MRI-only radiation therapy. Other motivations included synthesizing scans to aid diagnosis and completing datasets by synthesizing missing scans. Considerably more research has been carried out on MRI to CT synthesis, despite CT to MRI synthesis yielding specific benefits. A limitation on medical image synthesis is that medical datasets, especially paired datasets of different modalities, are lacking in size and availability; it is therefore recommended that a global consortium be developed to obtain and make available more datasets for use. Finally, it is recommended that work be carried out to establish all uses of the synthesis of medical scans in clinical practice and discover which evaluation methods are suitable for assessing the synthesized images for these needs.

DC-cycleGAN: Bidirectional CT-to-MR synthesis from unpaired data.

Magnetic resonance (MR) and computer tomography (CT) images are two typical types of medical images that provide mutually-complementary information for accurate clinical diagnosis and treatment. However, obtaining both images may be limited due to some considerations such as cost, radiation dose and modality missing. Recently, medical image synthesis has aroused gaining research interest to cope with this limitation. In this paper, we propose a bidirectional learning model, denoted as dual contrast cycleGAN (DC-cycleGAN), to synthesize medical images from unpaired data. Specifically, a dual contrast loss is introduced into the discriminators to indirectly build constraints between real source and synthetic images by taking advantage of samples from the source domain as negative samples and enforce the synthetic images to fall far away from the source domain. In addition, cross-entropy and structural similarity index (SSIM) are integrated into the DC-cycleGAN in order to consider both the luminance and structure of samples when synthesizing images. The experimental results indicate that DC-cycleGAN is able to produce promising results as compared with other cycleGAN-based medical image synthesis methods such as cycleGAN, RegGAN, DualGAN, and NiceGAN. Code is available at https://github.com/JiayuanWang-JW/DC-cycleGAN.

Less Is More: Unsupervised Mask-Guided Annotated CT Image Synthesis With Minimum Manual Segmentations.

As a pragmatic data augmentation tool, data synthesis has generally returned dividends in performance for deep learning based medical image analysis. However, generating corresponding segmentation masks for synthetic medical images is laborious and subjective. To obtain paired synthetic medical images and segmentations, conditional generative models that use segmentation masks as synthesis conditions were proposed. However, these segmentation mask-conditioned generative models still relied on large, varied, and labeled training datasets, and they could only provide limited constraints on human anatomical structures, leading to unrealistic image features. Moreover, the invariant pixel-level conditions could reduce the variety of synthetic lesions and thus reduce the efficacy of data augmentation. To address these issues, in this work, we propose a novel strategy for medical image synthesis, namely Unsupervised Mask (UM)-guided synthesis, to obtain both synthetic images and segmentations using limited manual segmentation labels. We first develop a superpixel based algorithm to generate unsupervised structural guidance and then design a conditional generative model to synthesize images and annotations simultaneously from those unsupervised masks in a semi-supervised multi-task setting. In addition, we devise a multi-scale multi-task Fréchet Inception Distance (MM-FID) and multi-scale multi-task standard deviation (MM-STD) to harness both fidelity and variety evaluations of synthetic CT images. With multiple analyses on different scales, we could produce stable image quality measurements with high reproducibility. Compared with the segmentation mask guided synthesis, our UM-guided synthesis provided high-quality synthetic images with significantly higher fidelity, variety, and utility ( by Wilcoxon Signed Ranked test).

A multimodal comparison of latent denoising diffusion probabilistic models and generative adversarial networks for medical image synthesis

Although generative adversarial networks (GANs) can produce large datasets, their limited diversity and fidelity have been recently addressed by denoising diffusion probabilistic models, which have demonstrated superiority in natural image synthesis. In this study, we introduce Medfusion, a conditional latent DDPM designed for medical image generation, and evaluate its performance against GANs, which currently represent the state-of-the-art. Medfusion was trained and compared with StyleGAN-3 using fundoscopy images from the AIROGS dataset, radiographs from the CheXpert dataset, and histopathology images from the CRCDX dataset. Based on previous studies, Progressively Growing GAN (ProGAN) and Conditional GAN (cGAN) were used as additional baselines on the CheXpert and CRCDX datasets, respectively. Medfusion exceeded GANs in terms of diversity (recall), achieving better scores of 0.40 compared to 0.19 in the AIROGS dataset, 0.41 compared to 0.02 (cGAN) and 0.24 (StyleGAN-3) in the CRMDX dataset, and 0.32 compared to 0.17 (ProGAN) and 0.08 (StyleGAN-3) in the CheXpert dataset. Furthermore, Medfusion exhibited equal or higher fidelity (precision) across all three datasets. Our study shows that Medfusion constitutes a promising alternative to GAN-based models for generating high-quality medical images, leading to improved diversity and less artifacts in the generated images.

A histogram-driven generative adversarial network for brain MRI to CT synthesis

Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) are commonly used tools for medical diagnostic assessment. Considering the ionizing radiation of CT imaging, estimating CT images from radiation-free MRI would be beneficial for medical diagnosis. Although state-of-art generative adversarial networks or end-to-end architecture models can generate realistic natural images, it is challenging to generate medical CT images with high signal-to-noise ratios and are paired with MRI. We propose a histogram-driven generative adversarial network (HisGAN) to address this issue, estimate CT images paired with MR, and develop a histogram-based dynamic scaling factor to facilitate learning different image styles. By employing an adversarial learning strategy to train the end-to-end generator, the generator better simulate the nonlinear mapping from source to target. For the generator, the proposed method applies multiple learnable parameters to adjust the overall weights of the dilated convolution layers to ensure sufficient expansive receptive fields for improved performance. Additionally, the method utilizes deep residual networks to train randomly smoothed generated images and employs adversarial loss to enhance the generation of the discriminator, achieving a balance between the generator and the discriminator. Our approach can synthesize image details at the pixel level in the target domain and has been evaluated using two datasets for MR to CT, T1 to T2, and Flair to T1ce modality synthesis tasks. The proposed method outperforms existing generative adversarial models applied to medical image synthesis.

An attentive-based generative model for medical image synthesis

Magnetic resonance (MR) and computer tomography (CT) imaging are valuable tools for diagnosing diseases and planning treatment. However, limitations such as radiation exposure and cost can restrict access to certain imaging modalities. To address this issue, medical image synthesis can generate one modality from another, but many existing models struggle with high-quality image synthesis when multiple slices are present in the dataset. This study proposes an attention-based dual contrast generative model, called ADC-cycleGAN, which can synthesize medical images from unpaired data with multiple slices. The model integrates a dual contrast loss term with the CycleGAN loss to ensure that the synthesized images are distinguishable from the source domain. Additionally, an attention mechanism is incorporated into the generators to extract informative features from both channel and spatial domains. To improve performance when dealing with multiple slices, the $K$-means algorithm is used to cluster the dataset into $K$ groups, and each group is used to train a separate ADC-cycleGAN. Experimental results demonstrate that the proposed ADC-cycleGAN model produces comparable samples to other state-of-the-art generative models, achieving the highest PSNR and SSIM values of 19.04385 and 0.68551, respectively. We publish the code at https://github.com/JiayuanWang-JW/ADC-cycleGAN.

Variable Augmented Network for Invertible Modality Synthesis and Fusion.

As an effective way to integrate the information contained in multiple medical images under different modalities, medical image synthesis and fusion have emerged in various clinical applications such as disease diagnosis and treatment planning. In this paper, an invertible and variable augmented network (iVAN) is proposed for medical image synthesis and fusion. In iVAN, the channel number of the network input and output is the same through variable augmentation technology, and data relevance is enhanced, which is conducive to the generation of characterization information. Meanwhile, the invertible network is used to achieve the bidirectional inference processes. Empowered by the invertible and variable augmentation schemes, iVAN not only be applied to the mappings of multi-input to one-output and multi-input to multi-output, but also to the case of one-input to multi-output. Experimental results demonstrated superior performance and potential task flexibility of the proposed method, compared with existing synthesis and fusion methods.

Assessing the Ability of Generative Adversarial Networks to Learn Canonical Medical Image Statistics.

In recent years, generative adversarial networks (GANs) have gained tremendous popularity for potential applications in medical imaging, such as medical image synthesis, restoration, reconstruction, translation, as well as objective image quality assessment. Despite the impressive progress in generating high-resolution, perceptually realistic images, it is not clear if modern GANs reliably learn the statistics that are meaningful to a downstream medical imaging application. In this work, the ability of a state-of-the-art GAN to learn the statistics of canonical stochastic image models (SIMs) that are relevant to objective assessment of image quality is investigated. It is shown that although the employed GAN successfully learned several basic first- and second-order statistics of the specific medical SIMs under consideration and generated images with high perceptual quality, it failed to correctly learn several per-image statistics pertinent to the these SIMs, highlighting the urgent need to assess medical image GANs in terms of objective measures of image quality.

2D medical image synthesis using transformer-based denoising diffusion probabilistic model

Objective. Artificial intelligence (AI) methods have gained popularity in medical imaging research. The size and scope of the training image datasets needed for successful AI model deployment does not always have the desired scale. In this paper, we introduce a medical image synthesis framework aimed at addressing the challenge of limited training datasets for AI models. Approach. The proposed 2D image synthesis framework is based on a diffusion model using a Swin-transformer-based network. This model consists of a forward Gaussian noise process and a reverse process using the transformer-based diffusion model for denoising. Training data includes four image datasets: chest x-rays, heart MRI, pelvic CT, and abdomen CT. We evaluated the authenticity, quality, and diversity of the synthetic images using visual Turing assessments conducted by three medical physicists, and four quantitative evaluations: the Inception score (IS), Fréchet Inception Distance score (FID), feature similarity and diversity score (DS, indicating diversity similarity) between the synthetic and true images. To leverage the framework value for training AI models, we conducted COVID-19 classification tasks using real images, synthetic images, and mixtures of both images. Main results. Visual Turing assessments showed an average accuracy of 0.64 (accuracy converging to indicates a better realistic visual appearance of the synthetic images), sensitivity of 0.79, and specificity of 0.50. Average quantitative accuracy obtained from all datasets were IS = 2.28, FID = 37.27, FDS = 0.20, and DS = 0.86. For the COVID-19 classification task, the baseline network obtained an accuracy of 0.88 using a pure real dataset, 0.89 using a pure synthetic dataset, and 0.93 using a dataset mixed of real and synthetic data. Significance. A image synthesis framework was demonstrated for medical image synthesis, which can generate high-quality medical images of different imaging modalities with the purpose of supplementing existing training sets for AI model deployment. This method has potential applications in many data-driven medical imaging research.

QUANTITATIVE BIOMARKERS REPRODUCIBILITY USING GENERATIVE ADVERSARIAL APPROACHES IN REDUCED TO CONVENTIONAL DOSE CT.

In recent years, several techniques for image-to-image translation by means of generative adversarial neural networks (GAN) have been proposed to learn mapping characteristics between a source and a target domain. In particular, in the medical imaging field conditional GAN frameworks with paired samples (cGAN) and unconditional cycle-consistent GANs with unpaired data (CycleGAN) have been demonstrated as a powerful scheme to model non-linear mappings that produce realistic target images from different modality sources. When proposing the usage and adaptation of these frameworks for medical image synthesis, quantitative and qualitative validation are usually performed by assessing the similarity between synthetic and target images in terms of metrics such as mean absolute error (MAE) or structural similarity (SSIM) index. However, an evaluation of clinically relevant markers showing that diagnostic information is not overlooked in the translation process is often missing. In this work, we aim at demonstrating the importance of validating medical image-to-image translation techniques by assessing their effect on the measurement of clinically relevant metrics and biomarkers. We implemented both a conditional and an unconditional approach to synthesize conventional dose chest CT scans from reduced dose CT and show that while both visually and in terms of traditional metrics the network appears to successfully minimize perceptual discrepancies, these methods are not reliable to systematically reproduce quantitative measurements of various chest biomarkers.

A novel approach using the local energy function and its variations for medical image fusion

ABSTRACT Medical image fusion plays a pivotal role in facilitating clinical diagnosis. However, the quality of input medical images may be marred by noise, low contrast, and lack of sharpness, presenting numerous challenges for medical image synthesis algorithms. Additionally, several fusion rules may degrade the brightness and contrast of the fused image. To this end, this paper presents a novel image synthesis approach to tackle the aforementioned issues. First, the input images undergo pre-processing to enhance their quality. Subsequently, we introduce the three-layer image decomposition (TLID) technique, which decomposes an image into three distinct layers: the base layer (), the small-scale structure layer (), and the large-scale structure layer (). Next, we synthesize the base layers utilizing adaptive rules based on the Marine predators algorithm (MPA), ensuring that the output image is not degraded. Finally, we propose an efficient synthesis method for and layers, based on combining the local energy function with its variations. This fusion technique preserves the intricate details present in the original image. We evaluated our approach on 156 medical images using six evaluation metrics and compared it with seven state-of-the-art image synthesis techniques. Our results demonstrate that our method successfully generates high-quality output images and preserves detailed information throughout the image synthesis process.

GANs for Medical Image Synthesis: An Empirical Study.

Generative adversarial networks (GANs) have become increasingly powerful, generating mind-blowing photorealistic images that mimic the content of datasets they have been trained to replicate. One recurrent theme in medical imaging, is whether GANs can also be as effective at generating workable medical data, as they are for generating realistic RGB images. In this paper, we perform a multi-GAN and multi-application study, to gauge the benefits of GANs in medical imaging. We tested various GAN architectures, from basic DCGAN to more sophisticated style-based GANs, on three medical imaging modalities and organs, namely: cardiac cine-MRI, liver CT, and RGB retina images. GANs were trained on well-known and widely utilized datasets, from which their FID scores were computed, to measure the visual acuity of their generated images. We further tested their usefulness by measuring the segmentation accuracy of a U-Net trained on these generated images and the original data. The results reveal that GANs are far from being equal, as some are ill-suited for medical imaging applications, while others performed much better. The top-performing GANs are capable of generating realistic-looking medical images by FID standards, that can fool trained experts in a visual Turing test and comply to some metrics. However, segmentation results suggest that no GAN is capable of reproducing the full richness of medical datasets.

A Framework for Simulating Cardiac MR Images with Varying Anatomy and Contrast.

One of the limiting factors for the development and adoption of novel deep-learning (DL) based medical image analysis methods is the scarcity of labeled medical images. Medical image simulation and synthesis can provide solutions by generating ample training data with corresponding ground truth labels. Despite recent advances, generated images demonstrate limited realism and diversity. In this work, we develop a flexible framework for simulating cardiac magnetic resonance (MR) images with variable anatomical and imaging characteristics for the purpose of creating a diversified virtual population. We advance previous works on both cardiac MR image simulation and anatomical modeling to increase the realism in terms of both image appearance and underlying anatomy. To diversify the generated images, we define parameters: 1) to alter the anatomy, 2) to assign MR tissue properties to various tissue types, and 3) to manipulate the image contrast via acquisition parameters. The proposed framework is optimized to generate a substantial number of cardiac MR images with ground truth labels suitable for downstream supervised tasks. A database of virtual subjects is simulated and its usefulness for aiding a DL segmentation method is evaluated. Our experiments show that training completely with simulated images can perform comparable with a model trained with real images for heart cavity segmentation in mid-ventricular slices. Moreover, such data can be used in addition to classical augmentation for boosting the performance when training data is limited, particularly by increasing the contrast and anatomical variation, leading to better regularization and generalization. The database is publicly available at https://osf.io/ bkzhm/ and the simulation code will be available at https: //github.com/sinaamirrajab/CMRI_Simulation.

EOSA-GAN: Feature enriched latent space optimized adversarial networks for synthesization of histopathology images using Ebola optimization search algorithm

Generative adversarial networks (GAN) represent two deep learning (DL) models positioned in an adversarial manner to generate and evaluate images. This area of research promises to address several issues associated with medical image analysis using deep learning architectures and has been applied to medical image synthesis. The histopathology image samples are the gold standard for detecting and staging cancer since they contain rich latent information. However, this medical image modality is highly problematic: imbalanced class distribution in datasets, the rareness of publicly accessible pathologic findings, the burdensome task of image annotation, the increasing need to anonymize pathology samples, segmentation of regions of interest, and the demand for high-quality with super-resolution histopathology images. In this paper, we present a highly optimized, locally attenuated and two-level optimization strategy to improve the performance of GAN. First, a novel feature space-to-latent space mapping mechanism is designed to enrich the latent space of input to the generator. We applied a DL model to extract discriminant features and used dimensionality reduction to match the number of features to latent space. Secondly, a new metaheuristic algorithm, the Ebola optimization search algorithm (EOSA), optimizes the EOSA-GAN architecture and is experimentally applied to benchmark datasets. Results obtained showed that the quality of generated samples achieved an impressive outcome when evaluated using the Feature Similarity Indexing Method (FSIM), Peak Signal to Noise Ratio (PSNR), Structured Similarity Indexing Method (SSIM) and others. The finding from this study demonstrates the impact of optimization algorithms in stabilizing and speeding up GANs to convergence.