Magnetic Resonance Image Synthesis Research Articles

Multi-scale Transformer Network with Edge-aware Pre-training for Cross-Modality MR Image Synthesis.

Cross-modality magnetic resonance (MR) image synthesis can be used to generate missing modalities from given ones. Existing (supervised learning) methods often require a large number of paired multi-modal data to train an effective synthesis model. However, it is often challenging to obtain sufficient paired data for supervised training. In reality, we often have a small number of paired data while a large number of unpaired data. To take advantage of both paired and unpaired data, in this paper, we propose a Multi-scale Transformer Network (MT-Net) with edge-aware pre-training for cross-modality MR image synthesis. Specifically, an Edge-preserving Masked AutoEncoder (Edge-MAE) is first pre-trained in a self-supervised manner to simultaneously perform 1) image imputation for randomly masked patches in each image and 2) whole edge map estimation, which effectively learns both contextual and structural information. Besides, a novel patch-wise loss is proposed to enhance the performance of Edge-MAE by treating different masked patches differently according to the difficulties of their respective imputations. Based on this proposed pre-training, in the subsequent fine-tuning stage, a Dual-scale Selective Fusion (DSF) module is designed (in our MT-Net) to synthesize missing-modality images by integrating multi-scale features extracted from the encoder of the pre-trained Edge-MAE. Furthermore, this pre-trained encoder is also employed to extract high-level features from the synthesized image and corresponding ground-truth image, which are required to be similar (consistent) in the training. Experimental results show that our MT-Net achieves comparable performance to the competing methods even using 70% of all available paired data. Our code will be released at https://github.com/lyhkevin/MT-Net.

A layer-wise fusion network incorporating self-supervised learning for multimodal MR image synthesis.

Magnetic resonance (MR) imaging plays an important role in medical diagnosis and treatment; different modalities of MR images can provide rich and complementary information to improve the accuracy of diagnosis. However, due to the limitations of scanning time and medical conditions, certain modalities of MR may be unavailable or of low quality in clinical practice. In this study, we propose a new multimodal MR image synthesis network to generate missing MR images. The proposed model comprises three stages: feature extraction, feature fusion, and image generation. During feature extraction, 2D and 3D self-supervised pretext tasks are introduced to pre-train the backbone for better representations of each modality. Then, a channel attention mechanism is used when fusing features so that the network can adaptively weigh different fusion operations to learn common representations of all modalities. Finally, a generative adversarial network is considered as the basic framework to generate images, in which a feature-level edge information loss is combined with the pixel-wise loss to ensure consistency between the synthesized and real images in terms of anatomical characteristics. 2D and 3D self-supervised pre-training can have better performance on feature extraction to retain more details in the synthetic images. Moreover, the proposed multimodal attention feature fusion block (MAFFB) in the well-designed layer-wise fusion strategy can model both common and unique information in all modalities, consistent with the clinical analysis. We also perform an interpretability analysis to confirm the rationality and effectiveness of our method. The experimental results demonstrate that our method can be applied in both single-modal and multimodal synthesis with high robustness and outperforms other state-of-the-art approaches objectively and subjectively.

Deep learning-based 3D MRI contrast-enhanced synthesis from a 2D noncontrast T2Flair sequence.

Gadolinium-based contrast agents (GBCAs) have been successfully applied in magnetic resonance (MR) imaging to facilitate better lesion visualization. However, gadolinium deposition in the human brain raised widespread concerns recently. On the other hand, although high-resolution three-dimensional (3D) MR images are more desired for most existing medical image processing algorithms, their long scan duration and high acquiring costs make 2D MR images still much more common clinically. Therefore, developing alternative solutions for 3D contrast-enhanced MR image synthesis to replace GBCAs injection becomes an urgent requirement. This study proposed a deep learning framework that produces 3D isotropic full-contrast T2Flair images from 2D anisotropic noncontrast T2Flair image stacks. The super-resolution (SR) and contrast-enhanced (CE) synthesis tasks are completed in sequence by using an identical generative adversarial network (GAN) with the same techniques. To solve the problem that intramodality datasets from different scanners have specific combinations of orientations, contrasts, and resolutions, we conducted a region-based data augmentation technique on the fly during training to simulate various imaging protocols in the clinic. We further improved our network by introducing atrous spatial pyramid pooling, enhanced residual blocks, and deep supervision for better quantitative and qualitative results. Our proposed method achieved superior CE-synthesized performance in quantitative metrics and perceptual evaluation. In detail, the PSNR, structural-similarity-index, and AUC are 32.25dB, 0.932, and 0.991 in the whole brain and 24.93dB, 0.851, and 0.929 in tumor regions. The radiologists' evaluations confirmed that our proposed method has high confidence in the diagnosis. Analysis of the generalization ability showed that benefiting from the proposed data augmentation technique, our network can be applied to "unseen" datasets with slight drops in quantitative and qualitative results. Our work demonstrates the clinical potential of synthesizing diagnostic 3D isotropic CE brain MR images from a single 2D anisotropic noncontrast sequence.

Ea-GANs: Edge-Aware Generative Adversarial Networks for Cross-Modality MR Image Synthesis.

Magnetic resonance (MR) imaging is a widely used medical imaging protocol that can be configured to provide different contrasts between the tissues in human body. By setting different scanning parameters, each MR imaging modality reflects the unique visual characteristic of scanned body part, benefiting the subsequent analysis from multiple perspectives. To utilize the complementary information from multiple imaging modalities, cross-modality MR image synthesis has aroused increasing research interest recently. However, most existing methods only focus on minimizing pixel/voxel-wise intensity difference but ignore the textural details of image content structure, which affects the quality of synthesized images. In this paper, we propose edge-aware generative adversarial networks (Ea-GANs) for cross-modality MR image synthesis. Specifically, we integrate edge information, which reflects the textural structure of image content and depicts the boundaries of different objects in images, to reduce this gap. Corresponding to different learning strategies, two frameworks are proposed, i.e., a generator-induced Ea-GAN (gEa-GAN) and a discriminator-induced Ea-GAN (dEa-GAN). The gEa-GAN incorporates the edge information via its generator, while the dEa-GAN further does this from both the generator and the discriminator so that the edge similarity is also adversarially learned. In addition, the proposed Ea-GANs are 3D-based and utilize hierarchical features to capture contextual information. The experimental results demonstrate that the proposed Ea-GANs, especially the dEa-GAN, outperform multiple state-of-the-art methods for cross-modality MR image synthesis in both qualitative and quantitative measures. Moreover, the dEa-GAN also shows excellent generality to generic image synthesis tasks on benchmark datasets about facades, maps, and cityscapes.

Pulse sequence extrapolation with MR image synthesis.

Previous reports have presented validation studies of magnetic resonance (MR) image synthesis in which multiple spin-echo (MSE) source data were used to generate spin-echo images for various echo times and repetition times (TRs). A new method-"pulse sequence extrapolation" -synthesizes images for pulse sequences different from that of the acquisition. MSE data acquired in a time equivalent to a TR of 2,000 msec can be used to generate inversion-recovery (IR) images for arbitrarily chosen TI inversion times. Other combinations of pulse sequences were also studied, and synthetic images were compared visually and quantitatively to directly acquired images with corresponding parameters. Synthetic IR signals of the brain parenchyma consistently matched directly acquired signals to within 6%, with respect to the full magnetization signal. The noise level of synthetic signals was generally no more than twice that of direct acquisition signals, as predicted. This method can achieve selective fat suppression and enhancement in IR imaging.

The precision of TR extrapolation in magnetic resonance image synthesis.

We present a model of noise propagation from acquired magnetic resonance (MR) images to TR-extrapolated synthetic images. This model assumes that images acquired at two repetition times TR1 and TR2 are used to generate synthetic images at arbitrary repetition times TR. The predictions of the model are compared with experimentally acquired phantom data, and show excellent agreement. The model is utilized in an analysis of two applications of MR image synthesis: scan time reduction and multiple-image synthesis. Scan time is reduced by acquiring data at two short repetition times, and synthesizing at a longer repetition time, with TR1 + TR2 less than TR. For T1 = 800 ms, a reduction of 20% in scan time results in a 45% reduction in signal-to-noise ratio SNR, when compared to direct acquisition. Reducing scan time by much more than 20% produces large noise levels in the synthetic image, and is unlikely to be useful. In multiple-image synthesis, images are synthesized at any repetition time in the range 0 to TR1 + TR2, for contrast optimization. If T1 = 800 ms, and TR1 + TR2 = 2000 ms, the optimum combination of TR1, TR2 results in synthetic images whose SNR is at worst 22% less than the SNR of directly acquired images. For many values of TR, the synthetic images have SNR superior to that obtainable by direct acquisition.

Automated MR image synthesis: feasibility studies.

The authors describe an automated technique of magnetic resonance (MR) image synthesis. Given a specific pulse sequence, MR signals are acquired for several pulse delay and/or repetition times and used to compute images of intrinsic parameters T1, T2, and N(H). Both the computed images and operator-specified pulse delay and repetition times are then used to "synthesize" a new image based on equations descriptive of MR signal behavior and comparable to that acquired by using the operator-specified parameters in an actual MR study. Instrumentation enabling rapid operator-interactive generation of synthesized images is described and initial results presented, allowing for dependence of the signal on T2 in spin echo images. Extension to full T1, T2, and N(H) dependence for arbitrary pulse sequences is described. Major advantages of this technique include retrospective optimization of contrast between arbitrary materials, rapid and systematic image analysis, and reduced scanning time; potential limitations include accuracy, noise, motion artifacts, and multicomponent behavior.