Image Translation Model Research Articles

LuminanceGAN: Controlling the brightness of generated images for various night conditions

There are diverse datasets available for training deep learning models utilized in autonomous driving. However, most of these datasets are composed of images obtained in day conditions, leading to a data imbalance issue when dealing with night condition images. Several day-to-night image translation models have been proposed to resolve the insufficiency of the night condition dataset, but these models often generate artifacts and cannot control the brightness of the generated image. In this study, we propose a LuminanceGAN, for controlling the brightness degree in night conditions to generate realistic night image outputs. The proposed novel Y-control loss converges the brightness degree of the output image to a specific luminance value. Furthermore, the implementation of the self-attention module effectively reduces artifacts in the generated images. Consequently, in qualitative comparisons, our model demonstrates superior performance in day-to-night image translation. Additionally, a quantitative evaluation was conducted using lane detection models, showing that our proposed method improves performance in night lane detection tasks. Moreover, the quality of the generated indoor dark images was assessed using an evaluation metric. It can be proven that our model generates images most similar to real dark images compared to other image translation models.

Deep learning based automated quantification of powders used in additive manufacturing

This study proposes a novel deep learning technique for efficient powder morphology characterization, crucial for successful additive manufacturing. The method segments powder particles in microscopy images using Pix2Pix image translation model, enabling precise quantification of size distribution and extraction of critical morphology parameters like circularity and aspect ratio. The proposed approach achieves high accuracy (Structural Similarity Index of 0.8) and closely matches established methods like laser diffraction in measuring particle size distribution (within a deviation of ∼7 %) and allows determination of additional particle attributes of aspect ratio and circualarity in a reliable, repeated, and automated way. These findings highlight the potential of deep learning for automated powder characterization, offering significant benefits for optimizing additive manufacturing processes.

Towards diverse image-to-image translation via adaptive normalization layer and contrast learning

A nice image-to-image translation framework is able to acquire an explicit and credible mapping relationship between the source domain and target domains while satisfying two requirements. One is simplicity, the other is extensibility over multiple translation tasks. To this end, we design a concise but versatile generative model for image-to-image translation. Our method includes three major ingredients. First, inspired by popular unconditional normalization layers, named Spatially Adaptive Normalization(SPADE). We introduce a novel Semantics-Appearance Spatially Adaptive Normalization (SA-SPADE), taking into account both semantic structure and style appearance. This enables semantic composition and style appearance information to be sufficiently captured and integrated by our normalization layers. Thanks to SA-SPADE, our model extends to multiple image-to-image translation tasks in an unsupervised or supervised way. Second, we carefully designed two symmetrical network branches to provide semantic and appearance information for our normalization layer, namely Semantic Branch (SB) and Appearance Branch(AB) respectively. Third, we propose novel Semantic-aware Contrastive Loss (SCL) and Appearance-aware Contrastive Loss (ACL)based on newly un-/self- supervised contrastive learning. That is, SCL guarantees domain-invariant (e.g., pose, structure) representations between the generated image and the input image, while ACL ensures domain-specific representations (e.g., color, texture) between the generated image and the reference image. As a result, we verify the effectiveness of our method by comparing it with various task-dependent image translation models in both qualitative and quantitative evaluations.

Asymmetric slack contrastive learning for full use of feature information in image translation

Recently, contrastive learning has been proven to be powerful in cross-domain feature learning and has been widely used in image translation tasks. However, these methods often overlook the differences between positive and negative samples regarding model optimization ability and treat them equally. This weakens the feature representation ability of the generative models. In this paper, we propose a novel image translation model based on asymmetric slack contrastive learning. We design a new contrastive loss asymmetrically by introducing a slack adjustment factor. Theoretical analysis shows that it can adaptively optimize and adjust according to different positive and negative samples and significantly improve optimization efficiency. In addition, to better preserve local structural relationships during image translation, we constructed a regional differential structural consistency correction block using differential vectors. Comparative experiments were conducted using seven existing methods on five datasets. The results indicate that our method can maintain structural consistency between cross-domain images at a deeper level. Furthermore, it is more effective in establishing real image-domain mapping relations, resulting in higher-quality images being generated.

Post-operative MRI synthesis from pre-operative MRI and post-operative CT using conditional GAN for the assessment of degree of resection

BackgroundMagnetic resonance imaging (MRI) is an essential part of assessing the extension of resection after craniotomy, allowing for accurate assessment of the brain. However, the application of MRI within 72 h after surgery is limited, owing to its high cost, long time duration, and patients’ limited mobility. Besides, MRI excludes people with contraindications, in which sceneries computerized tomography (CT) is the only available choice. To overcome these limitations, we investigated the use of a deep learning model for synthesizing post-operative MRI images. MethodsWe employed Cross-domain Correspondence Learning for Exemplar-based Image Translation Network (CoCosNet), an exemplar-based image translation model to synthesize T1 contrast-enhanced (T1ce) MRI images by combining post-operative T1ce and post-operative CT images. 233 cases were retrospectively collected at Sun Yat-sen University Cancer Center and underwent complete MRI and CT scans. ResultsAfter 200 epochs and a batch size of 10, CoCosNet achieved comparable results compared to other existing models, with objective indicators structural similarity index = 0.75, peak signal to noise ratio = 21.68, and mean absolute error = 0.007. Furthermore, assessments of the extension of resection between the true and synthesized T1ce images achieve great consistency. ConclusionsOur study demonstrates the effectiveness of the CoCosNet model in synthesizing post-operative MRI images from pre-operative MRI and post-operative CT scans. Quantitative analysis indicates that the synthesized images have comparable quality to real MRI images and can accurately discriminate the extent of resection. This approach has a promising future by providing a reliable alternative to traditional post-operative MRI.

A novel transformer‐based graph generation model for vectorized road design

AbstractRoad network design, as an important part of landscape modeling, shows a great significance in automatic driving, video game development, and disaster simulation. To date, this task remains labor‐intensive, tedious and time‐consuming. Many improved techniques have been proposed during the last two decades. Nevertheless, most of the state‐of‐the‐art methods still encounter problems of intuitiveness, usefulness and/or interactivity. As a rapid deviation from the conventional road design, this paper advocates an improved road modeling framework for automatic and interactive road production driven by geographical maps (including elevation, water, vegetation maps). Our method integrates the capability of flexible image generation models with powerful transformer architecture to afford a vectorized road network. We firstly construct a dataset that includes road graphs, density map and their corresponding geographical maps. Secondly, we develop a density map generation network based on image translation model with an attention mechanism to predict a road density map. The usage of density map facilitates faster convergence and better performance, which also serves as the input for road graph generation. Thirdly, we employ the transformer architecture to evolve density maps to road graphs. Our comprehensive experimental results have verified the efficiency, robustness and applicability of our newly‐proposed framework for road design.

Evaluating Biases and Quality Issues in Intermodality Image Translation Studies for Neuroradiology: A Systematic Review.

Intermodality image-to-image translation is an artificial intelligence technique for generating one technique from another. This review was designed to systematically identify and quantify biases and quality issues preventing validation and clinical application of artificial intelligence models for intermodality image-to-image translation of brain imaging. PubMed, Scopus, and IEEE Xplore were searched through August 2, 2023, for artificial intelligence-based image translation models of radiologic brain images. This review collected 102 works published between April 2017 and August 2023. Eligible studies were evaluated for quality using the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) and for bias using the Prediction model Risk Of Bias ASsessment Tool (PROBAST). Medically-focused article adherence was compared with that of engineering-focused articles overall with the Mann-Whitney U test and for each criterion using the Fisher exact test. Median adherence to the relevant CLAIM criteria was 69% and 38% for PROBAST questions. CLAIM adherence was lower for engineering-focused articles compared with medically-focused articles (65% versus 73%, P < .001). Engineering-focused studies had higher adherence for model description criteria, and medically-focused studies had higher adherence for data set and evaluation descriptions. Our review is limited by the study design and model heterogeneity. Nearly all studies revealed critical issues preventing clinical application, with engineering-focused studies showing higher adherence for the technical model description but significantly lower overall adherence than medically-focused studies. The pursuit of clinical application requires collaboration from both fields to improve reporting.

Data generation using generative adversarial networks to increase data volume

The article is an in-depth analysis of two leading approaches in the field of generative modeling: generative adversarial networks (GANs) and the pixel-to-pixel (Pix2Pix) image translation model. Given the growing interest in automation and improved image processing, the authors focus on the key operating principles of each model, analyzing their unique characteristics and features. The article also explores in detail the various applications of these approaches, highlighting their impact on modern research in computer vision and artificial intelligence. The purpose of the study is to provide readers with a scientific understanding of the effectiveness and potential of each of the models, and to highlight the opportunities and limitations of their application. The authors strive not only to cover the technical aspects of the models, but also to provide a broad overview of their impact on various industries, including medicine, the arts, and solving real-world problems in image processing. In addition, we have identified prospects for the use of these technologies in various fields, such as medicine, design, art, entertainment, and in unmanned aerial vehicle systems. The ability of GANs and Pix2Pix to adapt to a variety of tasks and produce high-quality results opens up broad prospects for industry and research.

SeRL: Style-embedding representation learning for unsupervised CT images synthesis from unpaired MR images

Hepatocellular carcinoma (HCC) is the third leading cause of cancer-related deaths worldwide, and the fastest-growing cause of cancer deaths in the United States. Computed tomography (CT) and magnetic resonance (MR) imaging are the key image-based examination methods for HCC patients. Patients may alternatively undergo CT and MR exams given a radiation dose of the former and the cost of the latter. Due to organ representation differences between CT and MR, it’s a great challenge to detect the tumors from a time series of data consisting of both CT and MR by a single model. The annotations for these time series data will consume much time and labor for the physicians. Thus, we propose our style-embedding representation learning (SeRL) for unsupervised and unpaired abdomen CT and MR translation. Different from current medical image translation models, the style-representation information from real CT and real MR images has been embedded in the translation process to bypass some local minima during the convergence process and improve the synthesis results. Our patch-corrosion augmentation method enhances the style-embedding representation learning by bringing more diversity to the training data. Combined with the self-attention module, SeRL eliminates the noise caused by low grayscale pixel values during translation. Results on the unpaired HCC patient's CT and MR images show that our proposed SeRL is able to generate high quality CT images from MR ones. Evaluations such as Frechet Inception Distance (FID), Sliced Wasserstein Distance (SWD), and liver segmentation dice score are utilized to demonstrate our advantages over other state-of-the-art unsupervised methods.

Style transfer for converting images into Chinese landscape painting based on CycleGAN

Chinese landscape painting is a prevalent and unique mode of artistic expression within traditional Chinese art. It boasts intricate techniques and demands a relatively high level of artistic skill. Recent advancements in artificial intelligence have ushered in the era of image style transfer techniques, making it feasible to convert landscape photographs into stunning Chinese landscape paintings. In this study, the author has developed an image translation model that enables mutual transformation between images of landscapes and Chinese landscape paintings. This technology significantly reduces the difficulty of creating Chinese landscape paintings, allowing more ordinary people in China to experience and appreciate the joy brought by traditional Chinese art. Experimental results indicate that the style transfer model based on CycleGAN has achieved significant success in this scenario. The generated artworks successfully integrate the style of Chinese landscape painting into the original images without altering the original composition and details. As a result, the original photos gain a certain level of artistic value. Additionally, this study innovatively explores the goal of reverse restoration of Chinese landscape paintings into images, highlighting both the similarities and differences with the current research, thus laying the foundation for future studies.

Dynamic Color Design Model Construction Considering Ambient Light Changes

Abstract The study takes dynamic color design as the main research object, firstly, based on the atmospheric scattering model and the improved dark primary color a priori algorithm, designing the image defogging algorithm based on the ambient light variation, which is used to carry out the optimization processing of images. Aiming at the lack of the user’s visual cognitive semantics of color matching in the process of color design, it is proposed to construct a dynamic color design model based on the theoretical foundation of the image translation model, combined with the visual aesthetics data flow. The model obtains the user’s visual aesthetic cognition of the optimized image samples through eye tracking technology and, at the same time, applies the image translation model to the color design process to generate a color design scheme that is more in line with the human visual characteristics by learning the color matching law of the samples and their visual cognitive semantics. The experimental analysis shows that the image-defogging algorithm based on ambient light variation has a better image-processing effect and operation efficiency, exhibiting a higher average gradient (11.83), information entropy (7.86) and visual fidelity (2.99). Meanwhile, the color design scheme generated by the method of this paper is highly recognized by the subjects, with evaluation scores of more than 4 points in each dimension. The dynamic color design model constructed in this paper, by considering ambient light changes, can generate color schemes that are both rational and comfortable.

Quantitative characterization of retinal features in translated OCTA.

This study explores the feasibility of quantitative Optical Coherence Tomography Angiography (OCTA) features translated from OCT using generative machine learning (ML) for characterizing vascular changes in retina. A generative adversarial network framework was employed alongside a 2D vascular segmentation and a 2D OCTA image translation model, trained on the OCT-500 public dataset and validated with data from the University of Illinois at Chicago (UIC) retina clinic. Datasets are categorized by scanning range (Field of view) and disease status. Validation involved quality and quantitative metrics, comparing translated OCTA (TR-OCTA) with ground truth OCTAs (GT-OCTA) to assess the feasibility for objective disease diagnosis. In our study, TR-OCTAs showed high image quality in both 3 and 6mm datasets (high-resolution and contrast quality, moderate structural similarity compared to GT-OCTAs). Vascular features like tortuosity and vessel perimeter index exhibits more consistent trends compared to density features which are affected by local vascular distortions. For the validation dataset (UIC), the metrics show similar trend with a slightly decreased performance since the model training was blind on UIC data, to evaluate inference performance. Overall, this study presents a promising solution to the limitations of OCTA adoption in clinical practice by using vascular features from TR-OCTA for disease detection. By making detailed vascular imaging more widely accessible and reducing reliance on expensive OCTA equipment, this research has the potential to significantly enhance the diagnostic process for retinal diseases.

Face Recognition System Based Car Security System

Modern car security systems have been developed as a result of the growing demand in the automotive sector for increased convenience and security. In order to provide safe access management and authentication for cars, this paper suggests a face recognition system-based auto security system that makes use of facial recognition techniques. When a match is detected, the system uses computer vision algorithms to identify and detect the faces of approved drivers, enabling the car to be unlocked and started without any issues. In order to convert near infrared face images into their corresponding visible images, we develop a nir-vis image translation model. Using two distinct nir databases of face images, the pre-trained face embedding model is applied to present the evaluation performances. The proposed system offers improved security by eliminating the need for traditional key-based systems and providing a robust biometric-based authentication mechanism. Anti-spoofing procedures are also integrated to guarantee the system's dependability and defense against fraudulent attempts. One of the disadvantages of the current systems is that not everyone finds it comfortable to have to memorize their password for certain systems. An RFID card is another device that has been developed to verify the identification of the user. Hackers have the ability to modify RFID data and substitute it with their own. Certain systems utilize fingerprints to verify an individual's identity. The primary reasons for the low adoption of biometric fingerprint locks are their expensive cost and the potential for falsified or damaged fingerprints. Certain systems identify an obstruction ahead of the car, sound an alert, tell the driver to pull over, but do nothing to actually avoid the obstruction. Through tests and assessments, the usefulness and practicability of the suggested system are shown, highlighting its potential as a workable option for safe and practical access management in automotive settings

Image Translation for Estimating Two-Dimensional Axial Amyloid-Beta PET From Structural MRI.

Amyloid-beta and brain atrophy are hallmarks for Alzheimer's Disease that can be targeted with positron emission tomography (PET) and MRI, respectively. MRI is cheaper, less-invasive, and more available than PET. There is a known relationship between amyloid-beta and brain atrophy, meaning PET images could be inferred from MRI. To build an image translation model using a Conditional Generative Adversarial Network able to synthesize Amyloid-beta PET images from structural MRI. Retrospective. Eight hundred eighty-two adults (348 males/534 females) with different stages of cognitive decline (control, mild cognitive impairment, moderate cognitive impairment, and severe cognitive impairment). Five hundred fifty-two subjects for model training and 331 for testing (80%:20%). 3 T, T1-weighted structural (T1w). The testing cohort was used to evaluate the performance of the model using the Structural Similarity Index Measure (SSIM) and Peak Signal-to-Noise Ratio (PSNR), comparing the likeness of the overall synthetic PET images created from structural MRI with the overall true PET images. SSIM was computed in the overall image to include the luminance, contrast, and structural similarity components. Experienced observers reviewed the images for quality, performance and tried to determine if they could tell the difference between real and synthetic images. Pixel wise Pearson correlation was significant, and had an R2 greater than 0.96 in example images. From blinded readings, a Pearson Chi-squared test showed that there was no significant difference between the real and synthetic images by the observers (P = 0.68). A high degree of likeness across the evaluation set, which had a mean SSIM = 0.905 and PSNR = 2.685. The two observers were not able to determine the difference between the real and synthetic images, with accuracies of 54% and 46%, respectively. Amyloid-beta PET images can be synthesized from structural MRI with a high degree of similarity to the real PET images. 3 TECHNICAL EFFICACY: Stage 1.

Development of Pericardial Fat Count Images Using a Combination of Three Different Deep-Learning Models: Image Translation Model From Chest Radiograph Image to Projection Image of Three-Dimensional Computed Tomography

Pericardial fat (PF)-the thoracic visceral fat surrounding the heart-promotes the development of coronary artery disease by inducing inflammation of the coronary arteries. To evaluate PF, we generated pericardial fat count images (PFCIs) from chest radiographs (CXRs) using a dedicated deep-learning model. We reviewed data of 269 consecutive patients who underwent coronary computed tomography (CT). We excluded patients with metal implants, pleural effusion, history of thoracic surgery, or malignancy. Thus, the data of 191 patients were used. We generated PFCIs from the projection of three-dimensional CT images, wherein fat accumulation was represented by a high pixel value. Three different deep-learning models, including CycleGAN were combined in the proposed method to generate PFCIs from CXRs. A single CycleGAN-based model was used to generate PFCIs from CXRs for comparison with the proposed method. To evaluate the image quality of the generated PFCIs, structural similarity index measure (SSIM), mean squared error (MSE), and mean absolute error (MAE) of (i) the PFCI generated using the proposed method and (ii) the PFCI generated using the single model were compared. The mean SSIM, MSE, and MAE were 8.56×10-1, 1.28×10-2, and 3.57×10-2, respectively, for the proposed model, and 7.62×10-1, 1.98×10-2, and 5.04×10-2, respectively, for the single CycleGAN-based model. PFCIs generated from CXRs with the proposed model showed better performance than those generated with the single model. The evaluation of PF without CT may be possible using the proposed method.

Underwater Image Translation via Multi-Scale Generative Adversarial Network

The role that underwater image translation plays assists in generating rare images for marine applications. However, such translation tasks are still challenging due to data lacking, insufficient feature extraction ability, and the loss of content details. To address these issues, we propose a novel multi-scale image translation model based on style-independent discriminators and attention modules (SID-AM-MSITM), which learns the mapping relationship between two unpaired images for translation. We introduce Convolution Block Attention Modules (CBAM) to the generators and discriminators of SID-AM-MSITM to improve its feature extraction ability. Moreover, we construct style-independent discriminators that enable the discriminant results of SID-AM-MSITM to be not affected by the style of images and retain content details. Through ablation experiments and comparative experiments, we demonstrate that attention modules and style-independent discriminators are introduced reasonably and SID-AM-MSITM performs better than multiple baseline methods.

Toward cell nuclei precision between OCT and H&E images translation using signal-to-noise ratio cycle-consistency

Medical image-to-image translation is often difficult and of limited effectiveness due to the differences in image acquisition mechanisms and the diverse structure of biological tissues. This work presents an unpaired image translation model between in-vivo optical coherence tomography (OCT) and ex-vivo Hematoxylin and eosin (H&E) stained images without the need for image stacking, registration, post-processing, and annotation. The model can generate high-quality and highly accurate virtual medical images, and is robust and bidirectional. Our framework introduces random noise to (1) blur redundant features, (2) defend against self-adversarial attacks, (3) stabilize inverse conversion, and (4) mitigate the impact of OCT speckles. We also demonstrate that our model can be pre-trained and then fine-tuned using images from different OCT systems in just a few epochs. Qualitative and quantitative comparisons with traditional image-to-image translation models show the robustness of our proposed signal-to-noise ratio (SNR) cycle-consistency method.

P13.01.A POST-OPERATIVE MRI SYNTHESIS FROM PRE-OPERATIVE MRI AND POST-OPERATIVE CT USING CONDITIONAL GAN FOR THE ASSESSMENT OF DEGREE OF RESECTION

Abstract BACKGROUND Magnetic resonance imaging (MRI) is an essential part of assessing the extension of resection after craniotomy, allowing for accurate assessment of the brain. However, the application of MRI within 72 hours after surgery is limited, owing to its high cost, long time duration, and patients’ limited mobility. Besides, MRI excludes people with contraindications, including implantable electronic devices or artificial limbs, in which sceneries computerized tomography (CT) is the only available choice. To overcome these limitations, we investigated the use of a deep learning model for synthesizing post-operative MRI images. MATERIAL AND METHODS We aim to employ Cross-domain Correspondence Learning for Exemplar-based Image Translation Network (CoCosNet), an exemplar-based image translation model to synthesize T1 contrast-enhanced (T1ce) MRI images by combining post-operative T1ce and post-operative CT images. 233 cases were retrospectively collected at Sun Yat-sen University Cancer Center, who underwent complete MRI and CT scans. RESULTS After 200 epochs and a batch size of 10, CoCosNet achieved comparable results compared to other existing models, with objective indicators structural similarity index = 0.75, peak signal-to-noise ratio = 21.68, and mean absolute error = 0.007. Furthermore, assessments of the extent of resection between the true and synthesized T1ce images achieve great consistency. CONCLUSION Our study demonstrates the effectiveness of the CoCosNet deep learning model in synthesizing post-operative MRI images from pre-operative MRI and post-operative CT scans. Our quantitative analysis indicates that the synthesized images have comparable quality to real MRI images and are able to accurately assess the extent of resection. This approach has a promising future by providing a reliable alternative to traditional post-operative MRI.

GAN-based Image Translation Model with Self-Attention for Nighttime Dashcam Data Augmentation

GAN-based Image Translation Model with Self-Attention for Nighttime Dashcam Data Augmentation

Synthesis of diffusion-weighted MRI scalar maps from FLAIR volumes using generative adversarial networks.

Acquisition and pre-processing pipelines for diffusion-weighted imaging (DWI) volumes are resource- and time-consuming. Generating synthetic DWI scalar maps from commonly acquired brain MRI sequences such as fluid-attenuated inversion recovery (FLAIR) could be useful for supplementing datasets. In this work we design and compare GAN-based image translation models for generating DWI scalar maps from FLAIR MRI for the first time. We evaluate a pix2pix model, two modified CycleGANs using paired and unpaired data, and a convolutional autoencoder in synthesizing DWI fractional anisotropy (FA) and mean diffusivity (MD) from whole FLAIR volumes. In total, 420 FLAIR and DWI volumes (11,957 images) from multi-center dementia and vascular disease cohorts were used for training/testing. Generated images were evaluated using two groups of metrics: (1) human perception metrics including peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), (2) structural metrics including a newly proposed histogram similarity (Hist-KL) metric and mean squared error (MSE). Pix2pix demonstrated the best performance both quantitatively and qualitatively with mean PSNR, SSIM, and MSE metrics of 23.41 dB, 0.8, 0.004, respectively for MD generation, and 24.05 dB, 0.78, 0.004, respectively for FA generation. The new histogram similarity metric demonstrated sensitivity to differences in fine details between generated and real images with mean pix2pix MD and FA Hist-KL metrics of 11.73 and 3.74, respectively. Detailed analysis of clinically relevant regions of white matter (WM) and gray matter (GM) in the pix2pix images also showed strong significant (p < 0.001) correlations between real and synthetic FA values in both tissue types (R = 0.714 for GM, R = 0.877 for WM). Our results show that pix2pix's FA and MD models had significantly better structural similarity of tissue structures and fine details than other models, including WM tracts and CSF spaces, between real and generated images. Regional analysis of synthetic volumes showed that synthetic DWI images can not only be used to supplement clinical datasets, but demonstrates potential utility in bypassing or correcting registration in data pre-processing.