Image Translation Research Articles

Multi-Energy Guided Image Translation with Stochastic Differential Equations for Near-Infrared Facial Expression Recognition

Illumination variation has been a long-term challenge in real-world facial expression recognition (FER). Under uncontrolled or non-visible light conditions, near-infrared (NIR) can provide a simple and alternative solution to obtain high-quality images and supplement the geometric and texture details that are missing in the visible (VIS) domain. Due to the lack of large-scale NIR facial expression datasets, directly extending VIS FER methods to the NIR spectrum may be ineffective. Additionally, previous heterogeneous image synthesis methods are restricted by low controllability without prior task knowledge. To tackle these issues, we present the first approach, called for NIR-FER Stochastic Differential Equations (NFER-SDE), that transforms face expression appearance between heterogeneous modalities to the overfitting problem on small-scale NIR data. NFER-SDE can take the whole VIS source image as input and, together with domain-specific knowledge, guide the preservation of modality-invariant information in the high-frequency content of the image. Extensive experiments and ablation studies show that NFER-SDE significantly improves the performance of NIR FER and achieves state-of-the-art results on the only two available NIR FER datasets, Oulu-CASIA and Large-HFE.

Learning to Manipulate Artistic Images

Recent advancement in computer vision has significantly lowered the barriers to artistic creation. Exemplar-based image translation methods have attracted much attention due to flexibility and controllability. However, these methods hold assumptions regarding semantics or require semantic information as the input, while accurate semantics is not easy to obtain in artistic images. Besides, these methods suffer from cross-domain artifacts due to training data prior and generate imprecise structure due to feature compression in the spatial domain. In this paper, we propose an arbitrary Style Image Manipulation Network (SIM-Net), which leverages semantic-free information as guidance and a region transportation strategy in a self-supervised manner for image generation. Our method balances computational efficiency and high resolution to a certain extent. Moreover, our method facilitates zero-shot style image manipulation. Both qualitative and quantitative experiments demonstrate the superiority of our method over state-of-the-art methods.Code is available at https://github.com/SnailForce/SIM-Net.

Frequency-Controlled Diffusion Model for Versatile Text-Guided Image-to-Image Translation

Recently, text-to-image diffusion models have emerged as a powerful tool for image-to-image translation (I2I), allowing flexible image translation via user-provided text prompts. This paper proposes frequency-controlled diffusion model (FCDiffusion), an end-to-end diffusion-based framework contributing a novel solution to text-guided I2I from a frequency-domain perspective. At the heart of our framework is a feature-space frequency-domain filtering module based on Discrete Cosine Transform, which extracts image features carrying different DCT spectral bands to control the text-to-image generation process of the Latent Diffusion Model, realizing versatile I2I applications including style-guided content creation, image semantic manipulation, image scene translation, and image style translation. Different from related methods, FCDiffusion establishes a unified text-driven I2I framework suiting diverse I2I application scenarios simply by switching among different frequency control branches. The effectiveness and superiority of our method for text-guided I2I are demonstrated with extensive experiments both qualitatively and quantitatively. Our project is publicly available at: https://xianggao1102.github.io/FCDiffusion/.

Patch-Wise Graph Contrastive Learning for Image Translation

Recently, patch-wise contrastive learning is drawing attention for the image translation by exploring the semantic correspondence between the input image and the output image. To further explore the patch-wise topology for high-level semantic understanding, here we exploit the graph neural network to capture the topology-aware features. Specifically, we construct the graph based on the patch-wise similarity from a pretrained encoder, whose adjacency matrix is shared to enhance the consistency of patch-wise relation between the input and the output. Then, we obtain the node feature from the graph neural network, and enhance the correspondence between the nodes by increasing mutual information using the contrastive loss. In order to capture the hierarchical semantic structure, we further propose the graph pooling. Experimental results demonstrate the state-of-art results for the image translation thanks to the semantic encoding by the constructed graphs.

SEIT: Structural Enhancement for Unsupervised Image Translation in Frequency Domain

For the task of unsupervised image translation, transforming the image style while preserving its original structure remains challenging. In this paper, we propose an unsupervised image translation method with structural enhancement in frequency domain named SEIT. Specifically, a frequency dynamic adaptive (FDA) module is designed for image style transformation that can well transfer the image style while maintaining its overall structure by decoupling the image content and style in frequency domain. Moreover, a wavelet-based structure enhancement (WSE) module is proposed to improve the intermediate translation results by matching the high-frequency information, thus enriching the structural details. Furthermore, a multi-scale network architecture is designed to extract the domain-specific information using image-independent encoders for both the source and target domains. The extensive experimental results well demonstrate the effectiveness of the proposed method.

Contrast-enhanced to non-contrast-enhanced image translation to exploit a clinical data warehouse of T1-weighted brain MRI

BackgroundClinical data warehouses provide access to massive amounts of medical images, but these images are often heterogeneous. They can for instance include images acquired both with or without the injection of a gadolinium-based contrast agent. Harmonizing such data sets is thus fundamental to guarantee unbiased results, for example when performing differential diagnosis. Furthermore, classical neuroimaging software tools for feature extraction are typically applied only to images without gadolinium. The objective of this work is to evaluate how image translation can be useful to exploit a highly heterogeneous data set containing both contrast-enhanced and non-contrast-enhanced images from a clinical data warehouse.MethodsWe propose and compare different 3D U-Net and conditional GAN models to convert contrast-enhanced T1-weighted (T1ce) into non-contrast-enhanced (T1nce) brain MRI. These models were trained using 230 image pairs and tested on 77 image pairs from the clinical data warehouse of the Greater Paris area.ResultsValidation using standard image similarity measures demonstrated that the similarity between real and synthetic T1nce images was higher than between real T1nce and T1ce images for all the models compared. The best performing models were further validated on a segmentation task. We showed that tissue volumes extracted from synthetic T1nce images were closer to those of real T1nce images than volumes extracted from T1ce images.ConclusionWe showed that deep learning models initially developed with research quality data could synthesize T1nce from T1ce images of clinical quality and that reliable features could be extracted from the synthetic images, thus demonstrating the ability of such methods to help exploit a data set coming from a clinical data warehouse.

U-SeqNet: learning spatiotemporal mapping relationships for multimodal multitemporal cloud removal

ABSTRACT Optical remotely sensed time series data have various key applications in Earth surface dynamics. However, cloud cover significantly hampers data analysis and interpretation. Despite synthetic aperture radar (SAR)-to-optical image translation techniques emerging as a promising solution, their effectiveness is diminished by their inability to adequately account for the intertwined nature of temporal and spatial dimensions. This study introduces U-SeqNet, an innovative model that integrates U-Net and Sequence-to-Sequence (Seq2Seq) architectures. Leveraging a pioneering spatiotemporal teacher forcing strategy, U-SeqNet excels in adapting and reconstructing data, capitalizing on available cloud-free observations to improve accuracy. Rigorous assessments through No Reference and Full Reference Image Quality Assessments (NR – IQA and FR – IQA) affirm U-SeqNet’s exceptional performance, marked by a Natural Image Quality Evaluator (NIQE) score of 5.85 and Mean Absolute Error (MAE) of 0.039. These results underline U-SeqNet’s exceptional capabilities in image reconstruction and its potential to improve remote sensing analysis by enabling more accurate and efficient multimodal and multitemporal cloud removal techniques.

Image-to-Image Translation with CNN Based Perceptual Similarity Metrics

Image-to-image translation is the process of transforming images from different domains. Generative Adversarial Networks (GANs), and Convolutional Neural Networks (CNNs) are widely used in image translation. This study aims to find the most effective loss function for GAN architectures and synthesize better images. For this, experimental results were obtained by changing the loss functions on the Pix2Pix method, one of the basic GAN architectures. The exist loss function used in the Pix2Pix method is the Mean Absolute Error (MAE). It is called the L_1metric. In this study, the effect of convolutional-based perceptual similarity CONTENT, LPIPS, and DISTS metrics on image-to-image translation was applied on the loss function in Pix2Pix architecture. In addition, the effects on image-to-image translation were analyzed using perceptual similarity metrics ( L_1_CONTENT, L_1_LPIPS, and L_1_DISTS) with the original L_1 loss at a rate of 50%. Performance analyzes of the methods were performed with the Cityscapes, Denim2Mustache, Maps, and Papsmear datasets. Visual results were analyzed with conventional (FSIM, HaarPSI, MS-SSIM, PSNR, SSIM, VIFp and VSI) and up-to-date (FID and KID) image comparison metrics. As a result, it has been observed that better results are obtained when convolutional-based methods are used instead of conventional methods for the loss function of GAN architectures. It has been observed that LPIPS and DISTS methods can be used in the loss function of GAN architectures in the future

Experimental Environment in Radiology: Sustainable Research

Research requires large sums of money that are indirectly provided by taxpayers. It is therefore important that research is sustainable and does not just serve the career development of individuals. The aim of this article is to discuss what sustainable research in radiology is, how it can be organised and, above all, to show that it is possible.There are various approaches to achieving sustainability, ranging from purely gaining knowledge to translated devices and contrast agents, and to new clinical applications. The first step is to clarify exactly what is intended to be achieved with the research and critically weigh up the novelty value and the expected impact. This should be followed by careful, long-term planning of the project over a period of 5-15 years with the definition of clear sub-steps. Securing funding is just as important here as regular communication of the results. It often makes sense to involve the regulatory authorities and commercialisation partners in the project at an early stage.Academic radiology should not limit itself to serving as a test platform for imaging devices and contrast agents from industry, but should try to realise its own ideas and developments. Many academic centres around the world have shown that this is possible. Examples from my own research, particularly in relation to the development and translation of super-resolution ultrasound imaging and the development of diagnostics and nanopharmaceuticals, are explained in this article and challenges at various stages of development are discussed. Young radiologists are encouraged to set bigger and more long-term goals in order to influence and develop our field in a sustainable way. · Sustainable research requires creativity and careful planning. · Sustainable research can start at several stages of the technical readiness level. · Long-term planning of the overall concept (5-15 years) with clear intermediate steps is essential. · Cooperation with industry is often useful. · Acquisition of third-party funding must be ensured at the same time.

Zero-Shot Medical Image Translation via Frequency-Guided Diffusion Models.

Recently, the diffusion model has emerged as a superior generative model that can produce high quality and realistic images. However, for medical image translation, the existing diffusion models are deficient in accurately retaining structural information since the structure details of source domain images are lost during the forward diffusion process and cannot be fully recovered through learned reverse diffusion, while the integrity of anatomical structures is extremely important in medical images. For instance, errors in image translation may distort, shift, or even remove structures and tumors, leading to incorrect diagnosis and inadequate treatments. Training and conditioning diffusion models using paired source and target images with matching anatomy can help. However, such paired data are very difficult and costly to obtain, and may also reduce the robustness of the developed model to out-of-distribution testing data. We propose a frequency-guided diffusion model (FGDM) that employs frequency-domain filters to guide the diffusion model for structure-preserving image translation. Based on its design, FGDM allows zero-shot learning, as it can be trained solely on the data from the target domain, and used directly for source-to-target domain translation without any exposure to the source-domain data during training. We evaluated it on three cone-beam CT (CBCT)-to-CT translation tasks for different anatomical sites, and a cross-institutional MR imaging translation task. FGDM outperformed the state-of-the-art methods (GAN-based, VAE-based, and diffusion-based) in metrics of Fréchet Inception Distance (FID), Peak Signal-to-Noise Ratio (PSNR), and Structural Similarity Index Measure (SSIM), showing its significant advantages in zero-shot medical image translation.

Photo-realistic photo synthesis using improved conditional generative adversarial networks

<span lang="EN-US">There are a wide range of potential uses for both the forward (generating face drawings from actual images) and backward (generating photos from synthetic face sketches). However, photo/sketch synthesis is still a difficult problem to solve because of the distinct differences between photos and sketches. Existing frameworks often struggle to acquire a strong mapping among the geometry of drawing and its corresponding photo-realistic pictures because of the little amount of paired sketch-photo training data available. In this study, we adopt the perspective that this is an image-to-image translation issue and investigate the usage of the well-known enhanced pix2pix generative adversarial networks (GANs) to generate high-quality photo-realistic pictures from drawings; we make use of three distinct datasets. While recent GAN-based approaches have shown promise in image translation, they still struggle to produce high-resolution, photorealistic pictures. This technique uses supervised learning to train the generator's hidden layers to produce low-resolution pictures initially, then uses the network's implicit refinement to produce high-resolution images. Extensive tests on three sketch-photo datasets (two publicly accessible and one we produced) are used to evaluate. Our solution outperforms existing image translation techniques by producing more photorealistic visuals with a peak signal-to-noise ratio of 59.85% and pixel accuracy of 82.7%.</span>

The Value of Computed Tomography Scan in Three-dimensional Planning and Intraoperative Navigation in Primary Total Hip Arthroplasty.

Total hip arthroplasty (THA) is a frequently performed procedure; the objective is restoration of native hip biomechanics and achieving functional range of motion (ROM) through precise positioning of the prosthetic components. Advanced three-dimensional (3D) imaging and computed tomography (CT)-based navigation are valuable tools in both the preoperative planning and intraoperative execution. The aim of this study is to provide a thorough overview on the applications of CT scans in both the preoperative and intraoperative settings of primary THA. Preoperative planning using CT-based 3D imaging enables greater accuracy in prediction of implant sizes, leading to enhancement of surgical workflow with optimization of implant inventory. Surgeons can perform a more thorough assessment of posterior and anterior acetabular wall coverage, acetabular osteophytes, anatomical landmarks, and thus achieve more functional implant positioning. Intraoperative CT-based navigation can facilitate precise execution of the preoperative plan, to attain optimal positioning of the prosthetic components to avoid impingement. Medial reaming can be minimized preserving native bone stock, which can enable restoration of femoral, acetabular, and combined offsets. In addition, it is associated with greater accuracy in leg length adjustment, a critical factor in patients' postoperative satisfaction. Despite the higher costs and radiation exposure, which currently limits its widespread adoption, it offers many benefits, and the increasing interest in robotic surgery has facilitated its integration into routine practice. Conducting additional research on ultra-low-dose CT scans and examining the potential for translation of 3D imaging into improved clinical outcomes will be necessary to warrant its expanded application.

Three-dimensional coherence beamforming using row-column arrays and matrix arrays

Much advancement in 3D ultrasound imaging has been achieved in recent years, allowing for its implementation in many research capacities. Translation of 3D imaging to a clinical setting has been hindered by small fields of view, expensive equipment and poor image quality. For ultrafast imaging, these challenges are accentuated by the need for a small number of transmission events. The row-column array is an emerging technology which can produce large field-of-view 3D ultrasound images, however, they can produce major artefacts when operated at high frame rates. Here we present some techniques which have been developed to improve the image quality and reduce the artifact level of row-column arrays by exploiting the incoherence of the transmission schemes. Presented will be the Row-Column specific Frame Multiply and Sum beamforming along with a technique for implementing Acoustic Sub-Aperture Processing which improves signal-to-noise ratio whilst also increasing the effective frame rate. Additional coherence-based algorithms, which have previously been developed for 2D imaging, are implemented and presented here for 3D imaging using a matrix array probe. All of the above algorithms have been optimised for super-resolution but should improve all types of 3D ultrasound imaging.

Exploring semantic consistency in unpaired image translation to generate data for surgical applications.

In surgical computer vision applications, data privacy and expert annotation challenges impede the acquisition of labeled training data. Unpaired image-to-image translation techniques have been explored to automatically generate annotated datasets by translating synthetic images into a realistic domain. The preservation of structure and semantic consistency, i.e., per-class distribution during translation, poses a significant challenge, particularly in cases of semantic distributional mismatch. This study empirically investigates various translation methods for generating data in surgical applications, explicitly focusing on semantic consistency. Through our analysis, we introduce a novel and simple combination of effective approaches, which we call ConStructS. The defined losses within this approach operate on multiple image patches and spatial resolutions during translation. Various state-of-the-art models were extensively evaluated on two challenging surgical datasets. With two different evaluation schemes, the semantic consistency and the usefulness of the translated images on downstream semantic segmentation tasks were evaluated. The results demonstrate the effectiveness of the ConStructS method in minimizing semantic distortion, with images generated by this model showing superior utility for downstream training. In this study, we tackle semantic inconsistency in unpaired image translation for surgical applications with minimal labeled data. The simple model (ConStructS) enhances consistency during translation and serves as a practical way of generating fully labeled and semantically consistent datasets at minimal cost. Our code is available at https://gitlab.com/nct_tso_public/constructs .

Extracting interpretable features for pathologists using weakly supervised learning to predict p16 expression in oropharyngeal cancer

One drawback of existing artificial intelligence (AI)-based histopathological prediction models is the lack of interpretability. The objective of this study is to extract p16-positive oropharyngeal squamous cell carcinoma (OPSCC) features in a form that can be interpreted by pathologists using AI model. We constructed a model for predicting p16 expression using a dataset of whole-slide images from 114 OPSCC biopsy cases. We used the clustering-constrained attention-based multiple-instance learning (CLAM) model, a weakly supervised learning approach. To improve performance, we incorporated tumor annotation into the model (Annot-CLAM) and achieved the mean area under the receiver operating characteristic curve of 0.905. Utilizing the image patches on which the model focused, we examined the features of model interest via histopathologic morphological analysis and cycle-consistent adversarial network (CycleGAN) image translation. The histopathologic morphological analysis evaluated the histopathological characteristics of image patches, revealing significant differences in the numbers of nuclei, the perimeters of the nuclei, and the intercellular bridges between p16-negative and p16-positive image patches. By using the CycleGAN-converted images, we confirmed that the sizes and densities of nuclei are significantly converted. This novel approach improves interpretability in histopathological morphology-based AI models and contributes to the advancement of clinically valuable histopathological morphological features.

Fluorinated Hafnium and Zirconium Coenable the Tunable Biodegradability of Core-Multishell Heterogeneous Nanocrystals for Bioimaging.

Upconversion (UC)/downconversion (DC)-luminescent lanthanide-doped nanocrystals (LDNCs) with near-infrared (NIR, 650-1700 nm) excitation have been gaining increasing popularity in bioimaging. However, conventional NIR-excited LDNCs cannot be degraded and eliminated eventually in vivo owing to intrinsic "rigid" lattices, thus constraining clinical applications. A biodegradability-tunable heterogeneous core-shell-shell luminescent LDNC of Na3HfF7:Yb,Er@Na3ZrF7:Yb,Er@CaF2:Yb,Zr (abbreviated as HZC) was developed and modified with oxidized sodium alginate (OSA) for multimode bioimaging. The dynamic "soft" lattice-Na3Hf(Zr)F7 host and the varying Zr4+ doping content in the outmoster CaF2 shell endowed HZC with tunable degradability. Through elaborated core-shell-shell coating, Yb3+/Er3+-coupled UC red and green and DC second near-infrared (NIR-II) emissions were, respectively, enhanced by 31.23-, 150.60-, and 19.42-fold when compared with core nanocrystals. HZC generated computed tomography (CT) imaging contrast effects, thus enabling NIR-II/CT/UC trimodal imaging. OSA modification not only ensured the exemplary biocompatibility of HZC but also enabled tumor-specific diagnosis. The findings would benefit the clinical imaging translation of LDNCs.

Controllable image expansion of rock castings based on deep learning

Abstract Digital rock physics (DRP) offers an effective method of deriving elastic parameters from digital rock images, but its practical application is always limited to limited datasets. Recently, deep learning techniques have presented a promising avenue for generating more extensive and cost-effective samples. However, generating controllable samples according to user definition remains very difficult due to high dependence on sufficient datasets. To resolve this problem, a new network was proposed based on the UNet framework through image translation (UNet-IT) to expand rock castings by given porosity in relatively fewer datasets. Practical tests on carbonate rock images demonstrate that the proposed method can generate samples tailored to specific porosity requirements, which achieved a minimum porosity relative error of less than 1%. Compared with the unextended samples, the generated ones have completely different pore structures in terms of two-point probability, two-point cluster, and lineal path functions. Furthermore, the elastic parameters of the generated images obtained through the finite element method (FEM) and practical logging data matched well, with an average relative error of ∼9%. This indicates that the generated samples can be used as effective data to estimate fine rock physics templates and then improve inversion accuracy.

Semantic and Geometric-Aware Day-to-Night Image Translation Network.

Autonomous driving systems heavily depend on perception tasks for optimal performance. However, the prevailing datasets are primarily focused on scenarios with clear visibility (i.e., sunny and daytime). This concentration poses challenges in training deep-learning-based perception models for environments with adverse conditions (e.g., rainy and nighttime). In this paper, we propose an unsupervised network designed for the translation of images from day-to-night to solve the ill-posed problem of learning the mapping between domains with unpaired data. The proposed method involves extracting both semantic and geometric information from input images in the form of attention maps. We assume that the multi-task network can extract semantic and geometric information during the estimation of semantic segmentation and depth maps, respectively. The image-to-image translation network integrates the two distinct types of extracted information, employing them as spatial attention maps. We compare our method with related works both qualitatively and quantitatively. The proposed method shows both qualitative and qualitative improvements in visual presentation over related work.

Nighttime Thermal Infrared Image Translation Integrating Visible Images

Nighttime Thermal InfraRed (NTIR) image colorization, also known as the translation of NTIR images into Daytime Color Visible (DCV) images, can facilitate human and intelligent system perception of nighttime scenes under weak lighting conditions. End-to-end neural networks have been used to learn the mapping relationship between temperature and color domains, and translate NTIR images with one channel into DCV images with three channels. However, this mapping relationship is an ill-posed problem with multiple solutions without constraints, resulting in blurred edges, color disorder, and semantic errors. To solve this problem, an NTIR2DCV method that includes two steps is proposed: firstly, fuse Nighttime Color Visible (NCV) images with NTIR images based on an Illumination-Aware, Multilevel Decomposition Latent Low-Rank Representation (IA-MDLatLRR) method, which considers the differences in illumination conditions during image fusion and adjusts the fusion strategy of MDLatLRR accordingly to suppress the adverse effects of nighttime lights; secondly, translate the Nighttime Fused (NF) image to DCV image based on HyperDimensional Computing Generative Adversarial Network (HDC-GAN), which ensures feature-level semantic consistency between the source image (NF image) and the translated image (DCV image) without creating semantic label maps. Extensive comparative experiments and the evaluation metrics values show that the proposed algorithms perform better than other State-Of-The-Art (SOTA) image fusion and translation methods, such as FID and KID, which decreased by 14.1 and 18.9, respectively.

Hubble Meets Webb: Image-to-Image Translation in Astronomy.

This work explores the generation of James Webb Space Telescope (JWSP) imagery via image-to-image translation from the available Hubble Space Telescope (HST) data. Comparative analysis encompasses the Pix2Pix, CycleGAN, TURBO, and DDPM-based Palette methodologies, assessing the criticality of image registration in astronomy. While the focus of this study is not on the scientific evaluation of model fairness, we note that the techniques employed may bear some limitations and the translated images could include elements that are not present in actual astronomical phenomena. To mitigate this, uncertainty estimation is integrated into our methodology, enhancing the translation's integrity and assisting astronomers in distinguishing between reliable predictions and those of questionable certainty. The evaluation was performed using metrics including MSE, SSIM, PSNR, LPIPS, and FID. The paper introduces a novel approach to quantifying uncertainty within image translation, leveraging the stochastic nature of DDPMs. This innovation not only bolsters our confidence in the translated images but also provides a valuable tool for future astronomical experiment planning. By offering predictive insights when JWST data are unavailable, our approach allows for informed preparatory strategies for making observations with the upcoming JWST, potentially optimizing its precious observational resources. To the best of our knowledge, this work is the first attempt to apply image-to-image translation for astronomical sensor-to-sensor translation.