Synthetic Images Research Articles

A review of deep learning and Generative Adversarial Networks applications in medical image analysis

Nowadays, computer-aided decision support systems (CADs) for the analysis of images have been a perennial technique in the medical imaging field. In CADs, deep learning algorithms are widely used to perform tasks like classification, identification of patterns, detection, etc. Deep learning models learn feature representations from images rather than handcrafted features. Hence, deep learning models are quickly becoming the state-of-the-art method to achieve good performances in different computer-aided decision-support systems in medical applications. Similarly, deep learning-based generative models called Generative Adversarial Networks (GANs) have recently been developed as a novel method to produce realistic-looking synthetic data. GANs are used in different domains, including medical imaging generation. The common problems, like class imbalance and a small dataset, in healthcare are well addressed by GANs, and it is a leading area of research. Segmentation, reconstruction, detection, denoising, registration, etc. are the important applications of GANs. So in this work, the successes of deep learning methods in segmentation, classification, cell structure and fracture detection, computer-aided identification, and GANs in synthetic medical image generation, segmentation, reconstruction, detection, denoising, and registration in recent times are reviewed. Lately, the review article concludes by raising research directions for DL models and GANs in medical applications.

Potential of synthetic images in landslide segmentation in data-poor scenario: a framework combining GAN and transformer models

Potential of synthetic images in landslide segmentation in data-poor scenario: a framework combining GAN and transformer models

Multi-Step Unsupervised Domain Adaptation in Image and Feature Space for Synthetic Aperture Radar Image Terrain Classification

The significant differences in data domains between SAR images and the expensive and time-consuming process of data labeling pose significant challenges to terrain classification. Current terrain classification methodologies face challenges in addressing domain disparities and detecting uncommon terrain effectively. Based on Style Transformation and Domain Metrics (STDMs), we propose an unsupervised domain adaptive framework named STDM-UDA for terrain classification in this paper, which consists of two steps: image style transfer and domain adaptive segmentation. As a first step, image style transfer is performed within the image space to mitigate the differences in low-level features between SAR image domains. Subsequently, leveraging this process, the segmentation network extracts image features, employing domain metrics and adversarial training to enhance alignment between domain gaps in the semantic feature space. Finally, experiments conducted on several pairs of SAR images, each exhibiting varying degrees of differences in key imaging parameters such as source, resolution, band, and polarization, demonstrate the robustness of the proposed method. It achieves remarkably competitive classification accuracy, particularly for unlabeled, high-resolution broad scenes, effectively overcoming the domain gaps introduced by the diverse imaging parameters under studies.

Syn2Real Detection in the Sky: Generation and Adaptation of Synthetic Aerial Ship Images

Syn2Real Detection in the Sky: Generation and Adaptation of Synthetic Aerial Ship Images

Clinical Turing tests with user certainty analysis to create and validate synthetic electrocardiogram images for artificial intelligence-enhanced algorithm development

Abstract Background Artificial intelligence-enhanced electrocardiogram (AI-ECG) algorithms have primarily been created using digitised signal data, owing to a relative absence of publicly available image-based datasets. ECGs are often scanned or photographed into electronic health records. For maximum clinical utility, AI-ECG algorithms should be applicable to these data. Synthetic data could expedite the creation of extensive, fully anonymised image-based ECG datasets to permit training image-based AI algorithms, but it is essential that such datasets contain the artefacts encountered in clinical practice. We investigated whether iterative clinical Turing tests with user certainty analysis could be used to develop and validate synthetic ECG data. Purpose To create synthetic ECG images containing the artefacts typically encountered in clinical practice, and to validate the images through iterative Turing testing and user certainty analysis. Methods Synthetic ECG images containing artefacts were created using the PTB-XL dataset (a publicly available signal-based dataset comprising 21799 ECGs) as source data. Iterative clinical Turing tests were conducted where healthcare professionals completed an online survey comprising 60 real and synthetic ECGs. Participants were asked to select whether they thought ECGs were real or synthetic. For user certainty analysis, participants were asked to rate their confidence in their answers using a five-point Likert scale (Figure 1). Likert scale responses were converted into a signed ordinal scale representing user certainty in the identification of real or synthetic data. This scale was used to perform Receiver Operating Characteristic (ROC) analysis. Following quantitative survey completion, qualitative feedback was sought and used to iteratively improve the realism of the synthetic images. Results A total of 26 healthcare professionals completed the clinical Turing tests over three rounds. Qualitative feedback was used to improve the fidelity of the synthetic ECG images between rounds (Table 1). During iterative testing, the proportion of synthetic ECGs correctly identified fell from 61.5% to 53.7%, and the proportion of real-world ECGs correctly identified fell from 66.3% to 53.0% (Figure 1). Following the final Turing test, ROC analysis revealed no discriminative ability for identifying synthetic data (C-statistic 0.480, 95% confidence interval 0.432-0.529). Conclusion Iterative Turing testing with user certainty analysis and qualitative user feedback may be used to create synthetic ECG images containing the artefacts typically encountered in clinical practice. Iterative Turing testing improved the images’ realism confirming their potential to augment image-based AI algorithm development. The presented methodology establishes a framework to develop high fidelity, synthetic patient datasets presenting a significant opportunity to enhance the uptake of AI within electrophysiology, cardiology, and medicine.

A deep learning approach for virtual contrast enhancement in Contrast Enhanced Spectral Mammography

Contrast Enhanced Spectral Mammography (CESM) is a dual-energy mammographic imaging technique that first requires intravenously administering an iodinated contrast medium. Then, it collects both a low-energy image, comparable to standard mammography, and a high-energy image. The two scans are combined to get a recombined image showing contrast enhancement. Despite CESM diagnostic advantages for breast cancer diagnosis, the use of contrast medium can cause side effects, and CESM also beams patients with a higher radiation dose compared to standard mammography. To address these limitations, this work proposes using deep generative models for virtual contrast enhancement on CESM, aiming to make CESM contrast-free and reduce the radiation dose. Our deep networks, consisting of an autoencoder and two Generative Adversarial Networks, the Pix2Pix, and the CycleGAN, generate synthetic recombined images solely from low-energy images. We perform an extensive quantitative and qualitative analysis of the model’s performance, also exploiting radiologists’ assessments, on a novel CESM dataset that includes 1138 images. As a further contribution to this work, we make the dataset publicly available. The results show that CycleGAN is the most promising deep network to generate synthetic recombined images, highlighting the potential of artificial intelligence techniques for virtual contrast enhancement in this field.

Personnel Detection in Dark Aquatic Environments Based on Infrared Thermal Imaging Technology and an Improved YOLOv5s Model.

This study presents a novel method for the nighttime detection of waterborne individuals using an enhanced YOLOv5s algorithm tailored for infrared thermal imaging. To address the unique challenges of nighttime water rescue operations, we have constructed a specialized dataset comprising 5736 thermal images collected from diverse aquatic environments. This dataset was further expanded through synthetic image generation using CycleGAN and a newly developed color gamut transformation technique, which significantly improves the data variance and model training effectiveness. Furthermore, we integrated the Convolutional Block Attention Module (CBAM) at the end of the last encoder's feedforward network. This integration maximizes the utilization of channel and spatial information to capture more intricate details in the feature maps. To decrease the computational demands of the network while maintaining model accuracy, Ghost convolution was employed, thereby boosting the inference speed as much as possible. Additionally, we applied hyperparameter evolution to refine the training parameters. The improved algorithm achieved an average detection accuracy of 85.49% on our proprietary dataset, significantly outperforming its predecessor, with a prediction speed of 23.51 FPS. The experimental outcomes demonstrate the proposed solution's high recognition capabilities and robustness, fulfilling the demands of intelligent lifesaving missions.

Ejection and dynamics of aggregates in the coma of comet 67P/Churyumov-Gerasimenko

The process of gas-driven ejection of refractory materials from cometary surfaces continues to pose a challenging question in cometary science. The activity modeling of comet 67P/Churyumov-Gerasimenko, based on data from the Rosetta mission, has significantly enhanced our comprehension of cometary activity. But thermophysical models have difficulties in simultaneously explaining the production rates of various gas species and dust. It has been suggested that different gas species might be responsible for the ejection of refractory material in distinct size ranges. This work focuses on investigating the abundance and the ejection mechanisms of large aggregates (gtrsim 1 cm) from the comet nucleus. We aim to determine their properties and map the distribution of their source regions across the comet surface. This can place constraints on activity models for comets. We examined 189 images acquired at five epochs by the OSIRIS/NAC instrument on board the Rosetta spacecraft. Our goal was to identify bright tracks produced by individual aggregates as they traversed the camera field of view. In parallel, we generated synthetic images based on the output of dynamical simulations involving various types of aggregates. By comparing these synthetic images with the observations, we determined the characteristics of the simulated aggregates that most closely resemble the observations. We have identified over 30000 tracks present in the OSIRIS images, derived constraints on the characteristics of the aggregates, and mapped their origins on the nucleus surface. The aggregates have an average radius of $ cm and a bulk density consistent with that of the comet's nucleus. Due to their size, gas drag exerts only a minor influence on their dynamical behavior, so an initial velocity is needed to bring them into the camera field of view. The source regions of these aggregates are predominantly located near the boundaries of distinct terrains on the surface.

The Lorentz Force at Work: Multiphase Magnetohydrodynamics throughout a Flare Lifespan

The hour-long, gradual phase of solar flares is well observed across the electromagnetic spectrum, demonstrating many multiphase aspects, where cold condensations form within the heated post-flare system, but a complete 3D model is lacking. Using a state-of-the-art 3D magnetohydrodynamic simulation, we identify the key role played by the Lorentz force through the entire flare lifespan, and show that slow variations in the post-flare magnetic field achieve the bulk of the energy release. Synthetic images in multiple passbands closely match flare observations, and we quantify the role of conductive, radiative, and Lorentz force work contributions from flare onset to decay. This highlights how the non-force-free nature of the magnetic topology is crucial to trigger Rayleigh–Taylor dynamics, observed as waving coronal rays in extreme ultraviolet observations. Our C-class solar flare reproduces multiphase aspects such as post-flare coronal rain. In agreement with observations, we find strands of cooler plasma forming spontaneously by catastrophic cooling, leading to cool plasma draining down the post-flare loops. As there is force balance between magnetic pressure and tension and the plasma pressure in gradual-phase flare loops, this has potential for coronal seismology to decipher the magnetic field strength variation from observations.

Characterizing the internal wave wakes and synthetic aperture radar image features of underwater moving objects

In this study, a refined and improved theoretical model was proposed for calculating the synthetic aperture radar (SAR) images of internal wave wakes. This model was based on an equivalent source model of internal wave wakes and the velocity-bunching principle. Wake field, scattering field, and SAR image characteristics were systematically calculated for multiple scenarios. The model described the transfer and solution processes by inputting the background flow field temperature and salinity profiles to map the radar image intensity distribution. The physical mechanisms of the internal wave wake generation and its capture by SAR were comprehensively explained. The effects of various parameters on the internal wave wake SAR images were analyzed comprehensively. The SAR image features containing internal wave wakes were determined based on the gray-level co-occurrence matrix and image power spectrum, which revealed that the maximum surface current induced by the internal wave modes of the same amplitude was considerably larger than that induced by surface-wave mode. When traveling far from the water surface, the rate of change in the total scattering coefficient because of the Bragg effect was approximately 20 times that of the slope effect. In SAR images, the recognizability of wakes depends primarily on the maximum intensity rather than the average disturbance on the sea surface. Therefore, SAR images exhibit considerable differences during downwind radar observations, with the most prominent contrast differences occurring at the maximum spreading angle of wake scattering and 1500 m behind underwater objects. In the image power spectrum, the frequency range of the first modal internal waves was approximately 10–30 Hz, whereas the higher-order modal internal waves had frequencies lower than 10 Hz. Furthermore, the frequency range of the surface-wave mode was larger than that of internal-wave modes, approximately 50–60 Hz.

Synthesizing 4D Magnetic Resonance Angiography From 3D Time-of-Flight Using Deep Learning: A Feasibility Study.

Objective and background This study aimed to develop a deep convolutional neural network (DCNN) model capable of generating synthetic 4D magnetic resonance angiography (MRA) from 3D time-of-flight (TOF) images, allowing estimation of temporal changes in arterial flow. TOF MRA provides static information about arterial structures through maximum intensity projection (MIP) processing, but it does not capture the dynamic information of contrast agent circulation, which is lost during MIP processing. Considering the principles of TOF, it is hypothesized that dynamic information about arterial blood flow is latent within TOF signals. Although arterial spin labeling (ASL) can extract dynamic arterial information, ASL MRA has drawbacks, such as longer imaging times and lower spatial resolution than TOF MRA. This study's primary aim is to extend the utility of TOF MRA by training a machine-learning model on paired TOF and ASL data to extract latent dynamic information from TOF signals. Methods A DCNN combining a modified U-Net and a long-short-term memory (LSTM) network was trained on a dataset of 13 subjects (11 menand two women, aged 42-77 years) using paired 3D TOF MRA and 4D ASL MRA images. Subjects had no history of cerebral vessel occlusion or significant stenosis. The dataset was acquired using a 3T MRI system with a 32-channel head coil. Preprocessing involved resampling and intensity normalization of TOF and ASL images, followed by data augmentation and arterial mask generation. The model learned to extract flow information from TOF images and generate 8-phase 4D MRA images. The precision of flow estimation was evaluated using the coefficient of determination (R²) and Bland-Altman analysis. A board-certified neuroradiologist validated the quality of the images and the absence of significant stenosis in the major cerebral arteries. Results The generated 4D MRA images closely resembled the ground-truth ASL MRA data, with R² values of 0.92, 0.85, and 0.84 for the internal carotid artery (ICA), proximal middle cerebral artery (MCA), and distal MCA, respectively. Bland-Altman analysis revealed a systematic error of -0.06, with 95% agreement limits ranging from -0.18 to 0.12. Additionally, the model successfully identified flow abnormalities in a subject with left MCA stenosis, displaying a delayed peak and subsequent flattening distal to the stenosis, indicative of reduced blood flow. Visualization of the predicted arterial flow overlaid on the original TOF MRA images highlighted the spatial progression and dynamics of the flow. Conclusions The DCNN model effectively generated synthetic 4D MRA images from TOF images, demonstrating its potential to estimate temporal changes in arterial flow accurately. This non-invasive technique offers a promising alternative to conventional methods for visualizing and evaluating healthy and pathological flow dynamics. It has significant potential to improve the diagnosis and treatment of cerebrovascular diseases by providing detailed temporal flow information without the need for contrast agents or invasive procedures. The practical implementation of this model could enable the extraction of dynamic cerebral blood flow information from routine brain MRI examinations, contributing to the early diagnosis and management of cerebrovascular disorders.

Inverse Synthetic Aperture Radar Imaging of Space Targets Using Wideband Pseudo-Noise Signals with Low Peak-to-Average Power Ratio

With the number of new satellites increasing dramatically, comprehensive space surveillance is becoming increasingly important. Therefore, high-resolution inverse synthetic aperture radar (ISAR) imaging of satellites can provide an in-situ assessment of the satellites. This paper demonstrates that pseudo-noise signals can also be used for satellite imaging, in addition to classical linear frequency-modulated chirp signals. Pseudo-noise transmission signals offer the advantage of very low cross-correlation values. This, for instance, enables the possibility of a system with multiple channels transmitting instantaneously. Furthermore, it can significantly reduce signal interference with other systems operating in the same frequency spectrum, which is of particular interest for high-bandwidth, high-power systems such as satellite imaging radars. A new routine has been introduced to generate a wideband pseudo-noise signal with a peak-to-average power ratio (PAPR) similar to that of a chirp signal. This is essential for applications where the transmit signal power budget is sharply limited by the high-power amplifier. The paper presents both theoretical descriptions and analysis of the generated pseudo-noise signal as well as the results of an imaging measurement of a real space target using the introduced pseudo-noise signals.

OffRoadSynth Open Dataset for Semantic Segmentation using Synthetic-Data-Based Weight Initialization for Autonomous UGV in Off-Road Environments

This article concerns the issue of image semantic segmentation for the machine vision system of an autonomous Unmanned Ground Vehicle (UGV) moving in an off-road environment. Determining the meaning (semantics) of the areas visible in the recorded image provides a complete understanding of the scene surrounding the autonomous vehicle. It is crucial for the correct determination of a passable route. Nowadays, semantic segmentation is generally solved using convolutional neural networks (CNN), which can take an image as input and output the segmented image. However, proper training of the neural network requires the use of large amounts of data, which becomes problematic in the situation of low availability of large, dedicated image data sets that consider various off-road situations - driving on various types of roads, surrounded by diverse vegetation and in various weather and light conditions. This study introduces a synthetic image dataset called “OffRoadSynth” to address the training data scarcity for off-road scenarios. It has been shown that pre-training the neural network on this synthetic dataset improves image segmentation accuracy compared to other methods, such as random network weight initialization or using larger, generic datasets. Results suggest that using a smaller but domain-dedicated set of synthetic images to initialize network weights for training on the target real-world dataset may be an effective approach to improving semantic segmentation results of images, including those from off-road environments.

Experimental demonstration of scene-based cophasing in optical synthetic aperture imaging using the SPGD algorithm

Large-aperture telescopes based on optical synthetic aperture imaging are investigated for recent high-resolution spaceborne observations. An enabling technique of aperture synthesis is a cophasing method to suppress a piston-tip-tilt error between sub-apertures. This paper proposes a scene-based cophasing technique using the stochastic parallel gradient descent (SPGD) algorithm, assuming application to high-resolution Earth observation. A significant advantage of the SPGD algorithm is a model-less cophasing capability based on extended scenes, but the simultaneous scene-based piston-tip-tilt correction between multiple apertures has not been demonstrated. In this paper, we developed a tabletop synthetic aperture imaging system with 37 sub-apertures and demonstrated extended-scene-based piston-tip-tilt control by optimizing applied voltages to 111 actuators simultaneously. The demonstration experiments used not only static scenes but also a time-varying dynamic scene for observation targets. In every measurement, the proposed scene-based approach reduced the initially defined piston-tip-tilt errors, and the image sharpness significantly improved, although the correction rate in the dynamic scene observation was slower. Finally, this paper discusses the influence of scene dynamics on image-based cophasing.

Single Image Integrated Deblurring Algorithm in Non-Uniform Environment

Motion blurriness in an image caused by camera shake during exposure is unavoidable. It could lead to information loss and degradation in the image quality. Therefore, many researchers are dedicated to developing image deblurring techniques to recover clear images from blurred images. During the deblurring process, structural edges in images play a vital part in estimating the blur kernels. For images with rich textures, fine-scale edges become more apparent. This will cause vagueness in the image’s structural edges and affect the accuracy of the kernel estimation process. In this study, we propose a single-image motion deblurring by kernel estimation method combined with L0-Regularized Intensity and Gradient Prior, and enhanced Scale Aware Smoothing methods. Two types of non-uniform datasets are used, which are real and synthetic. While synthetic datasets are utilized to assess the consistency in performance across real and synthetic images, real datasets are used to portray the level of detail and variation of actual blurred images. The dataset is divided into five categories (people, nature, manmade, text, and night light). Two image quality metrics were selected: full-reference assessment, including learned perceptual image patch similarity (LPIPS), peak signal-to-noise ratio (PSNR), and structural similarity indexing method (SSIM) for synthetic datasets, and no-reference assessments, including blind/reference-less image spatial quality (BRISQUE), natural image quality evaluator (NIQE) and perception-based image quality evaluator (PIQE) for real datasets. According to the findings, the fusion method performs best in the text category, followed by manmade and in nature, then night light and poor in people. The proposed method not only removes fine-scale edges and preserves the boundary sharpness, but also improved the estimated blur kernel.

Generative Model-Driven Synthetic Training Image Generation: An Approach to Cognition in Railway Defect Detection

Generative Model-Driven Synthetic Training Image Generation: An Approach to Cognition in Railway Defect Detection

Mask-aware transformer with structure invariant loss for CT translation

Multi-phase enhanced computed tomography (MPECT) translation from plain CT can help doctors to detect the liver lesion and prevent patients from the allergy during MPECT examination. Existing CT translation methods directly learn an end-to-end mapping from plain CT to MPECT, ignoring the crucial clinical domain knowledge. As clinicians subtract the plain CT from MPECT images as subtraction image to highlight the contrast-enhanced regions and further to facilitate liver disease diagnosis in the clinical diagnosis, we aim to exploit this domain knowledge for automatic CT translation. To this end, we propose a Mask-Aware Transformer (MAFormer) with structure invariant loss for CT translation, which presents the first effort to exploit this domain knowledge for CT translation. Specifically, the proposed MAFormer introduces a mask estimator to predict the subtraction image from the plain CT image. To integrate the subtraction image into the network, the MAFormer devises a Mask-Aware Transformer based Normalization (MATNorm) as normalization layer to highlight the contrast-enhanced regions and capture the long-range dependencies among these regions. Moreover, aiming to preserve the biological structure of CT slices, a structure invariant loss is designed to extract the structural information and minimize the structural similarity between the plain and synthetic CT images to ensure the structure invariant. Extensive experiments have proven the effectiveness of the proposed method and its superiority to the state-of-the-art CT translation methods. Source code is to be released.

Synthetic spatial aperture holographic third harmonic generation microscopy

Third harmonic generation (THG) provides a valuable, label-free approach to imaging biological systems. To date, THG microscopy has been performed using point-scanning methods that rely on intensity measurements lacking phase information of the complex field. We report the first demonstration, to the best of our knowledge, of THG holographic microscopy and the reconstruction of the complex THG signal field with spatial synthetic aperture imaging. Phase distortions arising from measurement-to-measurement fluctuations and imaging components cause optical aberrations in the reconstructed THG field. We have developed an aberration-correction algorithm that estimates and corrects these phase distortions to reconstruct the spatial synthetic aperture THG field without optical aberrations.

SimCol3D — 3D reconstruction during colonoscopy challenge

Colorectal cancer is one of the most common cancers in the world. While colonoscopy is an effective screening technique, navigating an endoscope through the colon to detect polyps is challenging. A 3D map of the observed surfaces could enhance the identification of unscreened colon tissue and serve as a training platform. However, reconstructing the colon from video footage remains difficult. Learning-based approaches hold promise as robust alternatives, but necessitate extensive datasets. Establishing a benchmark dataset, the 2022 EndoVis sub-challenge SimCol3D aimed to facilitate data-driven depth and pose prediction during colonoscopy. The challenge was hosted as part of MICCAI 2022 in Singapore. Six teams from around the world and representatives from academia and industry participated in the three sub-challenges: synthetic depth prediction, synthetic pose prediction, and real pose prediction. This paper describes the challenge, the submitted methods, and their results. We show that depth prediction from synthetic colonoscopy images is robustly solvable, while pose estimation remains an open research question.

Developing and validating multi-omics prediction models for late patient-reported dysphagia in head and neck radiotherapy

Background and purpose. To investigate models developed using radiomic and dosiomic (multi-omics) features from planning and treatment imaging for late patient-reported dysphagia in head and neck radiotherapy. Materials and methods. Training (n = 64) and testing (n = 23) cohorts of head and neck cancer patients treated with curative intent chemo-radiotherapy with a follow-up time greater than 12 months were retrospectively examined. Patients completed the MD Anderson Dysphagia Inventory and a composite score ≤60 was interpreted as patient-reported dysphagia. A chart review collected baseline dysphagia and clinical factors. Multi-omic features were extracted from planning and last synthetic CT images using the pharyngeal constrictor muscle contours as a region of interest. Late patient-reported dysphagia models were developed using a random forest backbone, with feature selection and up-sampling methods to account for the imbalanced data. Models were developed and validated for multi-omic feature combinations for both timepoints. Results. A clinical and radiomic feature model developed using the planning CT achieved good performance (validation: sensitivity = 80 ± 27% / balanced accuracy = 71 ± 23%, testing: sensitivity = 80 ± 10% / balanced accuracy = 73 ± 11%). The synthetic CT models did not show improvement over the plan CT multi-omics models, with poor reliability of the radiomic features on these images. Dosiomic features extracted from the synthetic CT showed promise in predicting late patient-reported dysphagia. Conclusion. Multi-omics models can predict late patient-reported dysphagia in head and neck radiotherapy patients. Synthetic CT dosiomic features show promise in developing successful models to account for changes in delivered dose distribution. Multi-center or prospective studies are required prior to clinical implementation of these models.