Latent Code Research Articles

Private Face Image Generation Method Based on Deidentification in Low Light.

The existing face image recognition algorithm can accurately identify underexposed facial images, but the abuse of face image recognition technology can associate face features with personally identifiable information, resulting in privacy disclosure of the users. The paper puts forward a method for private face image generation based on deidentification under low light. First of all, the light enhancement and attenuation networks are pretrained using the training set, and low-light face images in the test set are input into the light enhancement network for photo enhancement. Then the facial area is captured by the face interception network, and corresponding latent code will be created through the latent code generation network and feature disentanglement will be done. Tiny noise will be added to the latent code by the face generation network to create deidentified face images which will be input in a light attenuation network to generate private facial images in a low-lighting style. At last, experiments show that, compared with other state-of-the-art algorithms, this method is more successful in generating low-light private face images with the most similar structure to original photos. It protects users' privacy effectively by reducing the accuracy of the face recognition network, while also ensuring the practicability of the images.

A Bottleneck Auto-Encoder for F0 Transformations on Speech and Singing Voice

In this publication, we present a deep learning-based method to transform the f0 in speech and singing voice recordings. f0 transformation is performed by training an auto-encoder on the voice signal’s mel-spectrogram and conditioning the auto-encoder on the f0. Inspired by AutoVC/F0, we apply an information bottleneck to it to disentangle the f0 from its latent code. The resulting model successfully applies the desired f0 to the input mel-spectrograms and adapts the speaker identity when necessary, e.g., if the requested f0 falls out of the range of the source speaker/singer. Using the mean f0 error in the transformed mel-spectrograms, we define a disentanglement measure and perform a study over the required bottleneck size. The study reveals that to remove the f0 from the auto-encoder’s latent code, the bottleneck size should be smaller than four for singing and smaller than nine for speech. Through a perceptive test, we compare the audio quality of the proposed auto-encoder to f0 transformations obtained with a classical vocoder. The perceptive test confirms that the audio quality is better for the auto-encoder than for the classical vocoder. Finally, a visual analysis of the latent code for the two-dimensional case is carried out. We observe that the auto-encoder encodes phonemes as repeated discontinuous temporal gestures within the latent code.

SofGAN: A Portrait Image Generator with Dynamic Styling

Recently, Generative Adversarial Networks (GANs) have been widely used for portrait image generation. However, in the latent space learned by GANs, different attributes, such as pose, shape, and texture style, are generally entangled, making the explicit control of specific attributes difficult. To address this issue, we propose a SofGAN image generator to decouple the latent space of portraits into two subspaces: a geometry space and a texture space. The latent codes sampled from the two subspaces are fed to two network branches separately, one to generate the 3D geometry of portraits with canonical pose, and the other to generate textures. The aligned 3D geometries also come with semantic part segmentation, encoded as a semantic occupancy field (SOF). The SOF allows the rendering of consistent 2D semantic segmentation maps at arbitrary views, which are then fused with the generated texturemaps and stylized to a portrait photo using our semantic instance-wise module. Through extensive experiments, we show that our system can generate high-quality portrait images with independently controllable geometry and texture attributes. The method also generalizes well in various applications, such as appearance-consistent facial animation and dynamic styling.

Super-encoder with cooperative autoencoder networks

Dimensionality reduction plays a crucial role in classification, object detection, and pattern recognition tasks. Its main objective is to decrease the dimension of the original data while retaining the most distinctive information. With the emergence of deep learning, an autoencoder has become a state-of-the-art non-linear dimensionality-reduction method. Nonetheless, as the existing autoencoder models are devised to follow the data distribution and employ similarity techniques, preserving distinctive information can be problematic. To tackle this issue, we propose super-encoder (SE) networks trained in a supervised and cooperative manner. The SE consists of an encoder, separator, and decoder networks. The encoder combined with separator networks are dedicated to generating separable latent representation based on the label, and the decoder network should be able to reconstruct it to the original data simultaneously. Herein, we introduce a novel cooperative learning mechanism with a new loss function; therefore, the encoder, separator, and decoder networks can cooperate to achieve these objectives. Extensive experiments using benchmark datasets were conducted. The results indicated that the SE is more effective in extracting separable latent code than the existing supervised and unsupervised dimensionality-reduction models. Furthermore, as a generator, it can obtain highly competitive realistic images.

Exploring the Hysteresis Properties of Nanocrystalline Permanent Magnets Using Deep Learning

We demonstrate the use of model order reduction and neural networks for estimating the hysteresis properties of nanocrystalline permanent magnets from microstructure. With a data-driven approach, we learn the demagnetization curve from data-sets created by grain growth and micromagnetic simulations. We show that the granular structure of a magnet can be encoded within a low-dimensional latent space. Latent codes are constructed using a variational autoencoder. The mapping of structure code to hysteresis properties is a multi-target regression problem. We apply deep neural network and use parameter sharing, in order to predict anchor points along the demagnetization curves from the magnet's structure code. The method is applied to study the magnetic properties of nanocrystalline permanent magnets. We show how new grain structures can be generated by interpolation between two points in the latent space and how the magnetic properties of the resulting magnets can be predicted.

Feature-Based Inversion Using Variational Autoencoder for Electrical Impedance Tomography

A feature-based inversion method is presented to incorporate structural prior information of the human thorax for absolute imaging in electrical impedance tomography (EIT). A set of EIT images are generated from open-source CT scans with embedded structural priors of the human thorax. A variational autoencoder (VAE) is applied to learn high-level features of these EIT images and construct a mapping between EIT images and latent codes in a low-dimensional feature space. Then, the parameters of the latent code are served as unknowns to be inverted under a deterministic framework by the Gauss-Newton (GN) method. In this way, the number of unknowns is greatly reduced. Both synthetic and experimental data validate the proposed method. The reconstructed image quality is significantly improved, and the proposed method is relatively robust to measurement noise and modeling errors. This method provides a flexible and effective way to incorporate structural information from CT scans into EIT inversion, which is a potential algorithm for human thorax imaging.

Adversarial Detection by Latent Style Transformations

Machine learning models have demonstrated vulnerability to adversarial attacks, more specifically misclassification of adversarial examples. In this paper, we investigate an attack-agnostic defense against adversarial attacks on high-resolution images by detecting suspicious inputs. The intuition behind our approach is that the essential characteristics of a normal image are generally consistent with non-essential style transformations, e.g., slightly changing the facial expression of human portraits. In contrast, adversarial examples are generally sensitive to such transformations. In our approach to detect adversarial instances, we propose an in\underline{V}ertible \underline{A}utoencoder based on the \underline{S}tyleGAN2 generator via \underline{A}dversarial training (VASA) to inverse images to disentangled latent codes that reveal hierarchical styles. We then build a set of edited copies with non-essential style transformations by performing latent shifting and reconstruction, based on the correspondences between latent codes and style transformations. The classification-based consistency of these edited copies is used to distinguish adversarial instances.

Diversifying Tire-Defect Image Generation Based on Generative Adversarial Network

With the development of data-driven models, deep learning has been increasingly applied in the field of defect detection. However, the performance of deep learning models is greatly restricted by costly labeling and sample scarcity. One of the best approaches to solve the data imbalance problem is increasing the quantity and diversity of defect samples. Meanwhile, the current models based on generative adversarial network (GAN) cannot readily control the category and shape of generated defect samples, which results in inefficient data augmentation. Thus, to simultaneously achieve the category and shape adjustability of defect samples, a novel model named diversifying tire-defect GAN (DTD-GAN) is proposed in this article by integrating the advantages of latent space decomposition and feature decoupling algorithm. In this model, an auxiliary classifier is designed to control the defect category by discrete latent codes, and the mutual information maximization approach is adopted to diversify the tire-defect shape via continuous latent codes that greatly improve the diversity of the generated tire-defect samples. To illustrate the effectiveness of the proposed DTD-GAN model, extensive experiments on image quality and diversity assessment are carried out on the images generated by this model and other state-of-the-art approaches, and the achieved improvement in deep learning model performance in a tire-defect detection task is also discussed.

Towards Disentangling Latent Space for Unsupervised Semantic Face Editing.

Facial attributes in StyleGAN generated images are entangled in the latent space which makes it very difficult to independently control a specific attribute without affecting the others. Supervised attribute editing requires annotated training data which is difficult to obtain and limits the editable attributes to those with labels. Therefore, unsupervised attribute editing in an disentangled latent space is key to performing neat and versatile semantic face editing. In this paper, we present a new technique termed Structure-Texture Independent Architecture with Weight Decomposition and Orthogonal Regularization (STIA-WO) to disentangle the latent space for unsupervised semantic face editing. By applying STIA-WO to GAN, we have developed a StyleGAN termed STGAN-WO which performs weight decomposition through utilizing the style vector to construct a fully controllable weight matrix to regulate image synthesis, and employs orthogonal regularization to ensure each entry of the style vector only controls one independent feature matrix. To further disentangle the facial attributes, STGAN-WO introduces a structure-texture independent architecture which utilizes two independently and identically distributed (i.i.d.) latent vectors to control the synthesis of the texture and structure components in a disentangled way. Unsupervised semantic editing is achieved by moving the latent code in the coarse layers along its orthogonal directions to change texture related attributes or changing the latent code in the fine layers to manipulate structure related ones. We present experimental results which show that our new STGAN-WO can achieve better attribute editing than state of the art methods.

E2Style: Improve the Efficiency and Effectiveness of StyleGAN Inversion.

This paper studies the problem of StyleGAN inversion, which plays an essential role in enabling the pretrained StyleGAN to be used for real image editing tasks. The goal of StyleGAN inversion is to find the exact latent code of the given image in the latent space of StyleGAN. This problem has a high demand for quality and efficiency. Existing optimization-based methods can produce high-quality results, but the optimization often takes a long time. On the contrary, forward-based methods are usually faster but the quality of their results is inferior. In this paper, we present a new feed-forward network "E2Style" for StyleGAN inversion, with significant improvement in terms of efficiency and effectiveness. In our inversion network, we introduce: 1) a shallower backbone with multiple efficient heads across scales; 2) multi-layer identity loss and multi-layer face parsing loss to the loss function; and 3) multi-stage refinement. Combining these designs together forms an effective and efficient method that exploits all benefits of optimization-based and forward-based methods. Quantitative and qualitative results show that our E2Style performs better than existing forward-based methods and comparably to state-of-the-art optimization-based methods while maintaining the high efficiency as well as forward-based methods. Moreover, a number of real image editing applications demonstrate the efficacy of our E2Style. Our code is available at https://github.com/wty-ustc/e2style.

SAR Image Compression Using Discretized Gaussian Adaptive Model and Generalized Subtractive Normalization

Synthetic aperture radar (SAR) image compression plays an important role in the manipulation of images. However, existing optical compression methods cannot properly handle SAR compression due to the absence of feature learning and representation for SAR images. In this letter, we propose an end-to-end trainable model to effectively fit the feature distribution and reduce information dependencies toward SAR image compression. To better parameterize the distribution of latent codes, a discretized Gaussian adaptive model is designed to achieve a flexible entropy process. To further remove the remaining redundancies, generalized subtractive normalization is introduced to reduce the statistical dependencies in SAR images. Extensive experiments show that the proposed compression method outperforms the traditional compression methods and learning-based algorithms on both the ICEYE and Sandia datasets.

DVAEGMM: Dual Variational Autoencoder With Gaussian Mixture Model for Anomaly Detection on Attributed Networks

A significant aspect of today’s digital information is attributed networks, which combine multiple node attributes with the basic network topology in order to extract knowledge. Anomaly Detection on attributed networks has recently drawn significant attention from researchers and is widely used in a number of high-impact areas. The majority of current approaches focus on shallow learning methods such as community analysis, ego network or selection of subspace method. These approaches have network sparsity and data nonlinearity problems, and they do not even capture the intricate relationships between various information sources. Deep learning approaches like graph autoencoders are utilized to perform anomaly detection through obtaining node embeddings while dealing with the network nonlinearity and sparsity issues. However, they suffer from the problem of ignoring the latent codes’ embedding distribution, which results in poor representation in many instances. In this paper, we propose a new framework called DVAEGMM to detect anomalies on attributed networks. First, our framework utilizes a dual variational autoencoder for capturing the complex cross-modality relationships between node attributes and network structure, similar to vanilla autoencoders, but it also considers the potential data distribution and makes use of a generative adversarial network (GAN) for an adversarial regularization approach. An adversarial mechanism makes the encoder make more accurate estimates of how potential features might be distributed. As a result, decoders can make graphs that are more like the original graph. Each input data point is represented by a low-dimensional representation and a probability of reconstruction by the algorithm. Lastly, the Gaussian Mixture Model, a distinct estimation network, is used to approximate the latent vector density, resulting in the detection of anomalies from measuring sample energy. They are trained jointly as an end-to-end framework. DVAEGMM helps in the simultaneous optimization of the mixture model, generative adversarial network, and variational autoencoder parameters. The joint optimization balances the reconstruction probability, the latent representation density approximation, and regularization. Extensive experiments on attributed networks prove that DVAEGMM significantly beats the existing methods, proving the efficiency of the presented approach. The AUC scores of our proposed framework for the BlogCatalog, Flickr, Enron, and Amazon datasets are 0.89380, 0.87130, 0.72480, and 0.75102, respectively.

Self-Supervised Matting-Specific Portrait Enhancement and Generation.

We resolve the ill-posed alpha matting problem from a completely different perspective. Given an input portrait image, instead of estimating the corresponding alpha matte, we focus on the other end, to subtly enhance this input so that the alpha matte can be easily estimated by any existing matting models. This is accomplished by exploring the latent space of GAN models. It is demonstrated that interpretable directions can be found in the latent space and they correspond to semantic image transformations. We further explore this property in alpha matting. Particularly, we invert an input portrait into the latent code of StyleGAN, and our aim is to discover whether there is an enhanced version in the latent space which is more compatible with a reference matting model. We optimize multi-scale latent vectors in the latent spaces under four tailored losses, ensuring matting-specificity and subtle modifications on the portrait. We demonstrate that the proposed method can refine real portrait images for arbitrary matting models, boosting the performance of automatic alpha matting by a large margin. In addition, we leverage the generative property of StyleGAN, and propose to generate enhanced portrait data which can be treated as the pseudo GT. It addresses the problem of expensive alpha matte annotation, further augmenting the matting performance of existing models.

Pro-PULSE: Learning Progressive Encoders of Latent Semantics in GANs for Photo Upsampling.

The state-of-the-art photo upsampling method, PULSE, demonstrates that a sharp, high-resolution (HR) version of a given low-resolution (LR) input can be obtained by exploring the latent space of generative models. However, mapping an extreme LR input (162) directly to an HR image (10242) is too ambiguous to preserve faithful local facial semantics. In this paper, we propose an enhanced upsampling approach, Pro-PULSE, that addresses the issues of semantic inconsistency and optimization complexity. Our idea is to learn an encoder that progressively constructs the HR latent codes in the extended W+ latent space of StyleGAN. This design divides the complex 64× upsampling problem into several steps, and therefore small-scale facial semantics can be inherited from one end to the other. In particular, we train two encoders, the base encoder maps latent vectors in W space and serves as a foundation of the HR latent vector, while the second scale-specific encoder performed in W+ space gradually replaces the previous vector produced by the base encoder at each scale. This process produces intermediate side-outputs, which injects deep supervision into the training of encoder. Extensive experiments demonstrate superiorities over the latest latent space exploration methods, in terms of efficiency, quantitative quality metrics, and qualitative visual results.

DiverGAN: An Efficient and Effective Single-Stage Framework for Diverse Text-to-Image Generation

In this paper, we concentrate on the text-to-image synthesis task that aims at automatically producing perceptually realistic pictures from text descriptions. Recently, several single-stage methods have been proposed to deal with the problems of a more complicated multi-stage modular architecture. However, they often suffer from the lack-of-diversity issue, yielding similar outputs given a single textual sequence. To this end, we present an efficient and effective single-stage framework (DiverGAN) to generate diverse, plausible and semantically consistent images according to a natural-language description. DiverGAN adopts two novel word-level attention modules, i.e., a channel-attention module (CAM) and a pixel-attention module (PAM), which model the importance of each word in the given sentence while allowing the network to assign larger weights to the significant channels and pixels semantically aligning with the salient words. After that, Conditional Adaptive Instance-Layer Normalization (CAdaILN) is introduced to enable the linguistic cues from the sentence embedding to flexibly manipulate the amount of change in shape and texture, further improving visual-semantic representation and helping stabilize the training. Also, a dual-residual structure is developed to preserve more original visual features while allowing for deeper networks, resulting in faster convergence speed and more vivid details. Furthermore, we propose to plug a fully-connected layer into the pipeline to address the lack-of-diversity problem, since we observe that a dense layer will remarkably enhance the generative capability of the network, balancing the trade-off between a low-dimensional random latent code contributing to variants and modulation modules that use high-dimensional and textual contexts to strength feature maps. Inserting a linear layer after the second residual block achieves the best variety and quality. Both qualitative and quantitative results on benchmark data sets demonstrate the superiority of our DiverGAN for realizing diversity, without harming quality and semantic consistency.

Generative modelling of BRDF textures from flash images

We learn a latent space for easy capture, consistent interpolation, and efficient reproduction of visual material appearance. When users provide a photo of a stationary natural material captured under flashlight illumination, first it is converted into a latent material code. Then, in the second step, conditioned on the material code, our method produces an infinite and diverse spatial field of BRDF model parameters (diffuse albedo, normals, roughness, specular albedo) that subsequently allows rendering in complex scenes and illuminations, matching the appearance of the input photograph. Technically, we jointly embed all flash images into a latent space using a convolutional encoder, and -conditioned on these latent codes- convert random spatial fields into fields of BRDF parameters using a convolutional neural network (CNN). We condition these BRDF parameters to match the visual characteristics (statistics and spectra of visual features) of the input under matching light. A user study compares our approach favorably to previous work, even those with access to BRDF supervision. Project webpage: https://henzler.github.io/publication/neuralmaterial/.

Decomposing normal and abnormal features of medical images for content-based image retrieval of glioma imaging.

In medical imaging, the characteristics purely derived from a disease should reflect the extent to which abnormal findings deviate from the normal features. Indeed, physicians often need corresponding images without abnormal findings of interest or, conversely, images that contain similar abnormal findings regardless of normal anatomical context. This is called comparative diagnostic reading of medical images, which is essential for a correct diagnosis. To support comparative diagnostic reading, content-based image retrieval (CBIR) that can selectively utilize normal and abnormal features in medical images as two separable semantic components will be useful. In this study, we propose a neural network architecture to decompose the semantic components of medical images into two latent codes: normal anatomy code and abnormal anatomy code. The normal anatomy code represents counterfactual normal anatomies that should have existed if the sample is healthy, whereas the abnormal anatomy code attributes to abnormal changes that reflect deviation from the normal baseline. By calculating the similarity based on either normal or abnormal anatomy codes or the combination of the two codes, our algorithm can retrieve images according to the selected semantic component from a dataset consisting of brain magnetic resonance images of gliomas. Moreover, it can utilize a synthetic query vector combining normal and abnormal anatomy codes from two different query images. To evaluate whether the retrieved images are acquired according to the targeted semantic component, the overlap of the ground-truth labels is calculated as metrics of the semantic consistency. Our algorithm provides a flexible CBIR framework by handling the decomposed features with qualitatively and quantitatively remarkable results.

Modeling clothing as a separate layer for an animatable human avatar

We have recently seen great progress in building photorealistic animatable full-body codec avatars, but generating high-fidelity animation of clothing is still difficult. To address these difficulties, we propose a method to build an animatable clothed body avatar with an explicit representation of the clothing on the upper body from multi-view captured videos. We use a two-layer mesh representation to register each 3D scan separately with the body and clothing templates. In order to improve the photometric correspondence across different frames, texture alignment is then performed through inverse rendering of the clothing geometry and texture predicted by a variational autoencoder. We then train a new two-layer codec avatar with separate modeling of the upper clothing and the inner body layer. To learn the interaction between the body dynamics and clothing states, we use a temporal convolution network to predict the clothing latent code based on a sequence of input skeletal poses. We show photorealistic animation output for three different actors, and demonstrate the advantage of our clothed-body avatars over the single-layer avatars used in previous work. We also show the benefit of an explicit clothing model that allows the clothing texture to be edited in the animation output.

Interpolation and extrapolation between the magnetic chiral states using autoencoder

Autoencoder, an artificial neural network, was adopted to generate spin structures that interpolate and extrapolate between two distinct magnetic chiral states, the labyrinth structure and the skyrmion structure. We trained the autoencoder using two distinct magnetic chiral structures. Each input data is encoded through a deep learning process into a latent code, a new representation of the information in a reduced dimensional space. We investigated the latent space to acquire information on the structure of the latent code distribution. With the acquired information, we successfully produced various magnetic structures that exhibit plausible properties under various external fields not provided in the training data. The latent codes were modified by two algorithms. The first algorithm utilizes inversion and translation operation in the latent space and the second algorithm uses recursive flow with a modification bias. The first produced structures preserving the chiral structure of original data and the second produced statistically plausible states.

HSGAN: Reducing mode collapse in GANs by the latent code distance of homogeneous samples

In this paper, we propose HSGAN, a novel generative adversarial network (GAN) variant that plays an adversarial game on the distance between two homogeneous samples (HS) in the latent space. HSGAN alleviates the notorious problem of mode collapse by maintaining a certain distance between the latent code of the generated data. Moreover, HSGAN is directly trained on the encoder and the generator, thereby gaining the ability to conduct inference without introducing any other model complexity. We prove theoretically that the objective function is designed to minimize the f-divergence between the distributions of the generated data and the real data. Extensive experiments on a series of synthetic and real image benchmark datasets demonstrate that HSGAN generates diverse images while keeping high quality, and it generally outperforms other GANs that target at the mode collapse problem.