Latent Code Research Articles

Conditional reiterative High-Fidelity GAN inversion for image editing

Our work introduces a conditional reiteration mechanism for High-Fidelity GAN (Generative Adversarial Networks) inversion (HFGI), preserving image-specific details (like background, appearance, etc.) for both normal and out-of-domain images (e.g. heavy makeup faces). The HFGI encoder’s single-stage conditional latent maps result in blurry regions in restored images and loss of detailed information during editing. To address this, we proposed a reiterative conditional latents method that restores image-specific details sharply. The process involves two stages of iterations, reconstructing the image in the first stage, and refining image-specific details using conditional latent codes in the second stage. Our model successfully inverts out-of-domain images while preserving all details and supports InterfaceGAN, GANspace, and StyleClip for editing. We compare our approach with state-of-the-art GAN inversion methods on FFHQ (Flickr-Faces-HQ) Dataset, demonstrating significant improvements in inversion and editing quality.

HQ‐I2IT: Redesign the optimization scheme to improve image quality in CycleGAN‐based image translation systems

AbstractThe image‐to‐image translation (I2IT) task aims to transform images from the source domain into the specified target domain. State‐of‐the‐art CycleGAN‐based translation algorithms typically use cycle consistency loss and latent regression loss to constrain translation. In this work, it is demonstrated that the model parameters constrained by the cycle consistency loss and the latent regression loss are equivalent to optimizing the medians of the data distribution and the generative distribution. In addition, there is a style bias in the translation. This bias interacts between the generator and the style encoder and visually exhibits translation errors, e.g. the style of the generated image is not equal to the style of the reference image. To address these issues, a new I2IT model termed high‐quality‐I2IT (HQ‐I2IT) is proposed. The optimization scheme is redesigned to prevent the model from optimizing the median of the data distribution. In addition, by separating the optimization of the generator and the latent code estimator, the redesigned model avoids error interactions and gradually corrects errors during training, thereby avoiding learning the median of the generated distribution. The experimental results demonstrate that the visual quality of the images produced by HQ‐I2IT is significantly improved without changing the generator structure, especially when guided by the reference images. Specifically, the Fréchet inception distance on the AFHQ and CelebA‐HQ datasets are reduced from 19.8 to 10.2 and from 23.8 to 17.0, respectively.

Optimal transport-based unsupervised semantic disentanglement: A novel approach for efficient image editing in GANs

The latent space of pre-trained generative adversarial networks (GANs) is rich in semantic information, which often becomes highly entangled. It is crucial to identify semantic directions within this latent space, as these directions correlate with image attributes and are vital for image editing tasks. Existing methods for semantic discovery usually involve labor-intensive procedures such as manual labeling and training attribute classifiers, which limits their practicality. In response to this issue, the paper proposes the Optimal Transport-based Unsupervised Semantic Disentanglement (OTUSD) algorithm. This novel method efficiently uncovers semantic directions in the latent space of GANs by utilizing the concepts of manifold learning and optimal transport (OT) theory. OTUSD applies singular value decomposition (SVD) to the OT matrix that links latent codes to generated images. This process yields singular vectors that correspond to semantically meaningful directions. Unlike traditional methods, OTUSD bypasses the need for time-consuming labeling and training processes, thus enhancing efficiency and revealing a wider array of semantically meaningful directions. Experimental results demonstrate the effectiveness of OTUSD in discovering semantic directions from several state-of-the-art GAN models, including StyleGAN, StyleGAN2, and BigGAN. This performance emphasizes the potential applicability of OTUSD to image editing and other related tasks, and illuminates its value in harnessing the manifold learning and OT mapping capabilities inherent in GANs for semantic disentanglement. The implementation code is available at https://github.com/LuckAlex/OTUSD.

Structural MRI Harmonization via Disentangled Latent Energy-Based Style Translation.

Multi-site brain magnetic resonance imaging (MRI) has been widely used in clinical and research domains, but usually is sensitive to non-biological variations caused by site effects (e.g., field strengths and scanning protocols). Several retrospective data harmonization methods have shown promising results in removing these non-biological variations at feature or whole-image level. Most existing image-level harmonization methods are implemented through generative adversarial networks, which are generally computationally expensive and generalize poorly on independent data. To this end, this paper proposes a disentangled latent energy-based style translation (DLEST) framework for image-level structural MRI harmonization. Specifically, DLEST disentangles site-invariant image generation and site-specific style translation via a latent autoencoder and an energy-based model. The autoencoder learns to encode images into low-dimensional latent space, and generates faithful images from latent codes. The energy-based model is placed in between the encoding and generation steps, facilitating style translation from a source domain to a target domain implicitly. This allows highly generalizable image generation and efficient style translation through the latent space. We train our model on 4,092 T1-weighted MRIs in 3 tasks: histogram comparison, acquisition site classification, and brain tissue segmentation. Qualitative and quantitative results demonstrate the superiority of our approach, which generally outperforms several state-of-the-art methods.

3D Point Cloud Completion Method Based on Building Contour Constraint Diffusion Probability Model

Building point cloud completion is the process of reconstructing missing parts of a building’s point cloud, which have been affected by external factors during data collection, to restore the original geometric shape of the building. However, the uncertainty in filling point positions in the areas where building features are missing makes it challenging to recover the original distribution of the building’s point cloud shape. To address this issue, we propose a point cloud generation diffusion probability model based on building outline constraints. This method constructs building-outline-constrained regions using information related to the walls on the building’s surface and adjacent roofs. These constraints are encoded by an encoder and fused into latent codes representing the incomplete building point cloud shape. This ensures that the completed point cloud adheres closely to the real geometric shape of the building by constraining the generated points within the missing areas. The quantitative and qualitative results of the experiment clearly show that our method performs better than other methods in building point cloud completion.

GAN inversion-based semi-supervised learning for medical image segmentation

The crux of medical image segmentation stems from learning pixel-wise semantic consistency from a mount of labeled samples through exhaustive annotation. Most existing methods rely heavily on labeled image pairs and are optimized in a fully supervised manner, often resulting in unsatisfactory results due to limited paired training samples. In this paper, we propose a GAN inversion-based semi-supervised learning framework InvSSL for medical image segmentation, which generates corresponding variants from labeled images by GAN inversion. Specifically, it first trains a StyleGAN as an inversion generator which is able to generate samples with the same distribution as training samples, and then obtains variant samples from a training sample by performing GAN inversion to strengthen the segmentation performance in a semi-supervised learning manner. In particular, inversion generator inverts a training sample into the latent space and adds interference to the latent codes to evolve variant samples. To model high-level consistency between pixels and decouple semantic components in the latent space, our InvSSL designs a multi-level dense contrastive learning mechanism. Extensive experiments on lung segmentation and skin lesion segmentation demonstrate that our InvSSL outperforms state-of-the-art methods for medical image segmentation. Our code will be released at https://github.com/funkdub/InvSSL.

Generative modeling of living cells with SO(3)-equivariant implicit neural representations

Data-driven cell tracking and segmentation methods in biomedical imaging require diverse and information-rich training data. In cases where the number of training samples is limited, synthetic computer-generated data sets can be used to improve these methods. This requires the synthesis of cell shapes as well as corresponding microscopy images using generative models. To synthesize realistic living cell shapes, the shape representation used by the generative model should be able to accurately represent fine details and changes in topology, which are common in cells. These requirements are not met by 3D voxel masks, which are restricted in resolution, and polygon meshes, which do not easily model processes like cell growth and mitosis. In this work, we propose to represent living cell shapes as level sets of signed distance functions (SDFs) which are estimated by neural networks. We optimize a fully-connected neural network to provide an implicit representation of the SDF value at any point in a 3D+time domain, conditioned on a learned latent code that is disentangled from the rotation of the cell shape. We demonstrate the effectiveness of this approach on cells that exhibit rapid deformations (Platynereis dumerilii), cells that grow and divide (C. elegans), and cells that have growing and branching filopodial protrusions (A549 human lung carcinoma cells). A quantitative evaluation using shape features and Dice similarity coefficients of real and synthetic cell shapes shows that our model can generate topologically plausible complex cell shapes in 3D+time with high similarity to real living cell shapes. Finally, we show how microscopy images of living cells that correspond to our generated cell shapes can be synthesized using an image-to-image model.

Unobtrusive biometric data de-identification of fundus images using latent space disentanglement.

With the incremental popularity of ophthalmic imaging techniques, anonymization of the clinical image datasets is becoming a critical issue, especially the fundus images, which would have unique patient-specific biometric content. Towards achieving a framework to anonymize ophthalmic images, we propose an image-specific de-identification method on the vascular structure of retinal fundus images while preserving important clinical features such as hard exudates. Our method calculates the contribution of latent code in latent space to the vascular structure by computing the gradient map of the generated image with respect to latent space and then by computing the overlap between the vascular mask and the gradient map. The proposed method is designed to specifically target and effectively manipulate the latent code with the highest contribution score in vascular structures. Extensive experimental results show that our proposed method is competitive with other state-of-the-art approaches in terms of identity similarity and lesion similarity, respectively. Additionally, our approach allows for a better balance between identity similarity and lesion similarity, thus ensuring optimal performance in a trade-off manner.

A uniform representation model for OCT-based fingerprint presentation attack detection and reconstruction

In current Optical Coherence Tomography (OCT)-based fingerprint recognition systems, Presentation Attack Detection (PAD) and subsurface fingerprint reconstruction are treated as two independent branches, resulting in high computation and system building complexity. Therefore, this paper proposes a uniform representation model for simultaneous PAD and subsurface fingerprint reconstruction. A novel semantic segmentation network using attention mechanisms was designed to extract and segment multiple subsurface structures from real finger slices (i.e., B-scans). The latent codes derived from the network are directly used to effectively detect PA because they contain abundant subsurface biological information, which is independent of PA materials and has strong robustness for unknown PAs. Segmented subsurface structures were adopted to reconstruct multiple subsurface 2D fingerprints. Extensive experiments were carried out on an in-house database, which is the largest public OCT-based fingerprint database with 2449 volumes. PAD performance was evaluated by comparing the results of the existing methods with those of other segmentation networks. The proposed uniform representation model can obtain an accuracy (Acc) of 96.63%, which achieves a state-of-the-art performance. The effectiveness of subsurface reconstruction was evaluated from the segmentation and recognition results. In the segmentation experiments, the proposed method achieved the best results with an Intersection of Union (mIOU) of 0.834 and a Pixel Accuracy of 0.937. By comparing the recognition performance on surface 2D fingerprints (e.g., commercial and high-resolution), the lowest results with an Equal Error Rate (EER) of 2.25% by minutiae matching and an EER of 5.42% by pore matching are achieved, which indicates the excellent reconstruction capability of the proposed uniform representation model.

Maximum Gaussianality training for deep speaker vector normalization

Automatic Speaker Verification (ASV) is a critical task in pattern recognition and has been applied to various security-sensitive scenarios. The current state-of-the-art technique for ASV is based on deep embedding. However, a significant challenge with this approach is that the resulting deep speaker vectors tend to be irregularly distributed. To address this issue, this paper proposes a novel training method called Maximum Gaussianality (MG), which regulates the distribution of the speaker vectors. Compared to the conventional normalization approach based on maximum likelihood (ML), the new approach directly maximizes the Gaussianality of the latent codes, and therefore can both normalize the between-class and within-class distributions in a controlled and reliable way and eliminate the unbound likelihood problem associated with the conventional ML approach. Our experiments on several datasets demonstrate that our MG-based normalization can deliver much better performance than the baseline systems without normalization and outperform discriminative normalization flow (DNF), an ML-based normalization method, particularly when the training data is limited. In theory, the MG criterion can be applied to any task in any research domain where Gaussian distributions are needed, making the MG training a versatile tool.

Hairstyle transfer via manipulating decoupled latent codes of StyleGAN2

Hairstyle transfer aims to combine the hairstyle attributes of the reference images with the face and background of the identity image into one realistic image. This is challenging due to the differences in semantic structure between reference images and identity image, as well as the tangled hairstyle attributes. Existing GAN-based methods tend to simply combine image patches of hair, face, and background, which leads to discordant semantic structures and obvious artifacts in the results. And the GAN inversion-based methods require elaborate orthogonalization of loss gradients with additional calculations to alleviate the entanglement between the complex hairstyle attributes. To address these limitations, we propose a novel framework for hairstyle transfer based on StyleGAN inversion. Specifically, instead of simply combining the image patches, we use StyleGAN’s generative power to get more reasonable target semantic masks. During the alignment with the target masks, background inpainting is performed by StyleGAN rather than auxiliary models, which makes the results more realistic. Furthermore, our method decomposes hairstyle into three attributes: structure, texture, and color to better control the hairstyle transfer process. Besides, we observe that the spatial scales of these attributes match those of the StyleGAN’s latent codes. Then, we decouple the latent code in the hierarchical dimension and use different layers of the latent code to complete the transfer of different hair attributes, which reduces the entanglement between attributes without extra calculations. Correspondingly, our method divides hairstyle transfer into three steps, making the transfer process more controllable. By comparing with state-of-the-art methods, we verify the effectiveness of our method.

Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features.

Decoding human visual neural representations is a challenging task with great scientific significance in revealing vision-processing mechanisms and developing brain-like intelligent machines. Most existing methods are difficult to generalize to novel categories that have no corresponding neural data for training. The two main reasons are 1) the under-exploitation of the multimodal semantic knowledge underlying the neural data and 2) the small number of paired (stimuli-responses) training data. To overcome these limitations, this paper presents a generic neural decoding method called BraVL that uses multimodal learning of brain-visual-linguistic features. We focus on modeling the relationships between brain, visual and linguistic features via multimodal deep generative models. Specifically, we leverage the mixture-of-product-of-experts formulation to infer a latent code that enables a coherent joint generation of all three modalities. To learn a more consistent joint representation and improve the data efficiency in the case of limited brain activity data, we exploit both intra- and inter-modality mutual information maximization regularization terms. In particular, our BraVL model can be trained under various semi-supervised scenarios to incorporate the visual and textual features obtained from the extra categories. Finally, we construct three trimodal matching datasets, and the extensive experiments lead to some interesting conclusions and cognitive insights: 1) decoding novel visual categories from human brain activity is practically possible with good accuracy; 2) decoding models using the combination of visual and linguistic features perform much better than those using either of them alone; 3) visual perception may be accompanied by linguistic influences to represent the semantics of visual stimuli.

MCDIP‐ADMM: Overcoming overfitting in DIP‐based CT reconstruction

AbstractThis paper investigates the application of unsupervised learning methods for computed tomography reconstruction. To motivate our work, we review several existing priors, namely the truncated Gaussian prior, the prior, the total variation prior, and the deep image prior (DIP). We find that DIP outperforms the other three priors in terms of representational capability and visual performance. However, the performance of DIP deteriorates when the number of iterations exceeds a certain threshold due to overfitting. To address this issue, we propose a novel method (MCDIP‐ADMM) based on multi‐code deep image prior (MCDIP) and plug‐and‐play alternative direction method of multipliers (ADMM). Specifically, MCDIP utilizes multiple latent codes to generate a series of feature maps at an intermediate layer within a generator model. These maps are then composed with trainable weights, representing the complete image prior. Experimental results demonstrate the superior performance of the proposed MCDIP‐ADMM compared to three existing competitors. In the case of parallel beam projection with Gaussian noise, MCDIP‐ADMM achieves an average improvement of 4.3 dB over DIP, 1.7 dB over ADMM DIP‐weighted total variation (WTV) and 1.2 dB over PnP‐DIP in terms of peak‐signal‐to‐noise ratio (PSNR). Similarly, for fan‐beam projection with Poisson noise, MCDIP‐ADMM achieves an average improvement of 3.09 dB over DIP, 1.86 dB over ADMM DIP‐WTV and 0.84 dB over PnP‐DIP in terms of PSNR.

CLIP-guided StyleGAN Inversion for Text-driven Real Image Editing

Researchers have recently begun exploring the use of StyleGAN-based models for real image editing. One particularly interesting application is using natural language descriptions to guide the editing process. Existing approaches for editing images using language either resort to instance-level latent code optimization or map predefined text prompts to some editing directions in the latent space. However, these approaches have inherent limitations. The former is not very efficient, while the latter often struggles to effectively handle multi-attribute changes. To address these weaknesses, we present CLIPInverter, a new text-driven image editing approach that is able to efficiently and reliably perform multi-attribute changes. The core of our method is the use of novel, lightweight text-conditioned adapter layers integrated into pretrained GAN-inversion networks. We demonstrate that by conditioning the initial inversion step on the Contrastive Language-Image Pre-training (CLIP) embedding of the target description, we are able to obtain more successful edit directions. Additionally, we use a CLIP-guided refinement step to make corrections in the resulting residual latent codes, which further improves the alignment with the text prompt. Our method outperforms competing approaches in terms of manipulation accuracy and photo-realism on various domains including human faces, cats, and birds, as shown by our qualitative and quantitative results.

Non-intrusive model order reduction for parametric radiation transport simulations

We present a parametric, non-intrusive reduced-order model (ROM) for multigroup radiation transport, addressing the high computational cost and resource requirements associated with transport simulations in scenarios involving multiple queries. To achieve this, we employ Proper Orthogonal Decomposition (POD) to learn a groupwise reduced basis representation of the observed data (obtained from full-order simulations). We utilize Gaussian Process Regression and Multivariate Adaptive Regression Splines to learn the parametric, low-dimensional latent code (the reduced coordinates in the subspace spanned by the observed data). The approach is non-invasive, making it applicable to any software without the need for accessing the source code. We provide numerical results for various parametric problems, including a 3D multigroup transport case. Comparing the ROM-based surrogate to the full-order radiation transport model, solved using discrete ordinates in angle and discontinuous finite elements in space, we achieve a scalar flux solution that is 20,000 times faster with less than 1% relative error in a 3D transport scenario. Moreover, by utilizing the ROM, we demonstrate how a simple uncertainty quantification study can be completed within minutes, which would otherwise require significantly larger computational resources.

Flow time history representation and reconstruction based on machine learning

Based on deep learning technology, a new spatiotemporal flow data representation and reconstruction scheme is proposed by using flow time history (FTH) data instead of flow snapshots. First, the high-dimensional nonlinear flow system is reduced to a low-dimensional representation latent code using the FTH autoencoder model. Second, the mapping from physical space to latent code space is built using mathematical and machine-learning schemes. Finally, FTH at unavailable positions in physical space is generated by the FTH generator. The proposed scheme is validated by three case studies: (i) representing and recovering the FTH data of periodic laminar flow around a circular cylinder at Re = 200 and generating high-resolution laminar flow data; (ii) reconstructing complex FTH of flow past cylinder at Re = 3900 which including laminar and turbulent flow region and generating three-dimensional high-resolution turbulent flow data, respectively; (iii) representing and generating multi-variable turbulent flow data simultaneously using the multi-channel model. The results show that the proposed scheme is an effective low-dimensional representation for complex flow time variant features, which is suitable for both laminar and turbulent FTH data to generate spatiotemporal high-resolution FTH data in three-dimensional space.

Implicit Neural Representations With Structured Latent Codes for Human Body Modeling.

This paper addresses the challenge of novel view synthesis for a human performer from a very sparse set of camera views. Some recent works have shown that learning implicit neural representations of 3D scenes achieves remarkable view synthesis quality given dense input views. However, the representation learning will be ill-posed if the views are highly sparse. To solve this ill-posed problem, our key idea is to integrate observations over video frames. To this end, we propose Neural Body, a new human body representation which assumes that the learned neural representations at different frames share the same set of latent codes anchored to a deformable mesh, so that the observations across frames can be naturally integrated. The deformable mesh also provides geometric guidance for the network to learn 3D representations more efficiently. Furthermore, we combine Neural Body with implicit surface models to improve the learned geometry. To evaluate our approach, we perform experiments on both synthetic and real-world data, which show that our approach outperforms prior works by a large margin on novel view synthesis and 3D reconstruction. We also demonstrate the capability of our approach to reconstruct a moving person from a monocular video on the People-Snapshot dataset.

3DCascade-GAN: Shape completion from single-view depth images

Depth images can be easily acquired using depth cameras. However, these images only contain partial information about the shape due to unavoidable self-occlusion. Thanks to the availability of large datasets of shapes, it is possible to use a learning-based approach to produce complete shapes from single depth images. State-of-the-art generative adversarial network (GAN) architectures can produce reasonable results. However, the use of relatively local convolutions restricts GAN architectures from producing globally plausible shapes. In this study, we develop a novel dynamic latent code selection mechanism in which the model learns to select only important codes from the latent space. Furthermore, a novel 3D self-attention (3DSA) layer is introduced that is able to capture non-local relationships across the 3D space. We further design a GAN architecture that uses a multistage encoder–decoder to recover the shape, where our 3DSA layer is introduced to the discriminator to help attend to global features, which stabilizes the model learning and encourages shape refinement, making our reconstruction more structurally plausible. Through extensive experiments, we demonstrate that our method outperforms other state-of-the-art methods for single depth image 3D reconstruction.

Unsupervised Category-Specific Partial Point Set Registration via Joint Shape Completion and Registration.

We propose a self-supervised method for partial point set registration. Although recently proposed learning-based methods demonstrate impressive registration performance on full shape observations, these methods often suffer from performance degradation when dealing with partial shapes. To bridge the performance gap between partial and full point set registration, we propose to incorporate a shape completion network to benefit the registration process. To achieve this, we introduce a learnable latent code for each pair of shapes, which can be regarded as the geometric encoding of the target shape. By doing so, our model does not require an explicit feature embedding network to learn the feature encodings. More importantly, both our shape completion and point set registration networks take the shared latent codes as input, which are optimized simultaneously with the parameters of two decoder networks in the training process. Therefore, the point set registration process can benefit from the joint optimization process of latent codes, which are enforced to represent the information of full shapes instead of partial ones. In the inference stage, we fix the network parameters and optimize the latent codes to obtain the optimal shape completion and registration results. Our proposed method is purely unsupervised and does not require ground truth supervision. Experiments on the ModelNet40 dataset demonstrate the effectiveness of our model for partial point set registration.

EC-GAN: Emotion-Controllable GAN for Face Image Completion

Image completion methods based on deep learning, such as Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs), have succeeded in producing semantically plausible results. However, existing facial image completion methods can either produce only one result or, although they can provide multiple results, cannot attribute particular emotions to the results. We propose EC-GAN, a novel facial Emotion-Controllable GAN-based image completion model that can infer and customize generative facial emotions. We propose an emotion inference module that infers the emotions of faces based on the unmasked regions of the faces. The emotion inference module is trained in a supervised manner and enforces the encoder to disentangle the emotion semantics from the native latent space. We also developed an emotion control module to modify the latent codes of emotions, moving the latent codes of the initial emotion toward the desired one while maintaining the remaining facial features. Extensive experiments were conducted on two facial datasets, CelebA-HQ and CFEED. Quantitative and qualitative results indicate that EC-GAN produces images with diverse desired expressions even when the main features of the faces are masked. On the other hand, EC-GAN promotes semantic inference capability with irregularly masked holes, resulting in more natural facial expressions.