Latent Code Research Articles

VMP: Versatile Motion Priors for Robustly Tracking Motion on Physical Characters

AbstractRecent progress in physics‐based character control has made it possible to learn policies from unstructured motion data. However, it remains challenging to train a single control policy that works with diverse and unseen motions, and can be deployed to real‐world physical robots. In this paper, we propose a two‐stage technique that enables the control of a character with a full‐body kinematic motion reference, with a focus on imitation accuracy. In a first stage, we extract a latent space encoding by training a variational autoencoder, taking short windows of motion from unstructured data as input. We then use the embedding from the time‐varying latent code to train a conditional policy in a second stage, providing a mapping from kinematic input to dynamics‐aware output. By keeping the two stages separate, we benefit from self‐supervised methods to get better latent codes and explicit imitation rewards to avoid mode collapse. We demonstrate the efficiency and robustness of our method in simulation, with unseen user‐specified motions, and on a bipedal robot, where we bring dynamic motions to the real world.

BundleMoCap++: Efficient, robust and smooth motion capture from sparse multiview videos

Producing smooth and accurate motions from sparse videos without requiring specialized equipment and markers is a long-standing problem in the research community. Most approaches typically involve complex processes such as temporal constraints, multiple stages combining data-driven regression and optimization techniques, and bundle solving over temporal windows. These increase the computational burden and introduce the challenge of hyperparameter tuning for the different objective terms. In contrast, BundleMoCap++ offers a simple yet effective approach to this problem. It solves the motion in a single stage, eliminating the need for temporal smoothness objectives while still delivering smooth motions without compromising accuracy. BundleMoCap++ outperforms the state-of-the-art without increasing complexity. Our approach is based on manifold interpolation between latent keyframes. By relying on a local manifold smoothness assumption and appropriate interpolation schemes, we efficiently solve a bundle of frames using two or more latent codes. Additionally, the method is implemented as a sliding window optimization and requires only the first frame to be properly initialized, reducing the overall computational burden. BundleMoCap++’s strength lies in achieving high-quality motion capture results with fewer computational resources. To do this efficiently, we propose a novel human pose prior that focuses on the geometric aspect of the latent space, modeling it as a hypersphere, allowing for the introduction of sophisticated interpolation techniques. We also propose an algorithm for optimizing the latent variables directly on the learned manifold, improving convergence and performance. Finally, we introduce high-order interpolation techniques adapted for the hypersphere, allowing us to increase the solving temporal window, enhancing performance and efficiency.

Self-Organizing a Latent Hierarchy of Sketch Patterns for Controllable Sketch Synthesis.

Encoding sketches as Gaussian mixture model (GMM)-distributed latent codes is an effective way to control sketch synthesis. Each Gaussian component represents a specific sketch pattern, and a code randomly sampled from the Gaussian can be decoded to synthesize a sketch with the target pattern. However, existing methods treat the Gaussians as individual clusters, which neglects the relationships between them. For example, the giraffe and horse sketches heading left are related to each other by their face orientation. The relationships between sketch patterns are important messages to reveal cognitive knowledge in sketch data. Thus, it is promising to learn accurate sketch representations by modeling the pattern relationships into a latent structure. In this article, we construct a tree-structured taxonomic hierarchy over the clusters of sketch codes. The clusters with the more specific descriptions of sketch patterns are placed at the lower levels, while the ones with the more general patterns are ranked at the higher levels. The clusters at the same rank relate to each other through the inheritance of features from common ancestors. We propose a hierarchical expectation-maximization (EM)-like algorithm to explicitly learn the hierarchy, jointly with the training of encoder-decoder network. Moreover, the learned latent hierarchy is utilized to regularize sketch codes with structural constraints. Experimental results show that our method significantly improves controllable synthesis performance and obtains effective sketch analogy results.

Retinotopic coding organizes the opponent dynamic between internally and externally oriented brain networks.

How the human brain integrates internally- (i.e., mnemonic) and externally-oriented (i.e., perceptual) information is a long-standing puzzle in neuroscience. In particular, the internally-oriented networks like the default network (DN) and externally-oriented dorsal attention networks (dATNs) are thought to be globally competitive, which implies DN disengagement during cognitive states that drive the dATNs and vice versa. If these networks are globally opposed, how is internal and external information integrated across these networks? Here, using precision neuroimaging methods, we show that these internal/external networks are not as dissociated as traditionally thought. Using densely sampled high-resolution fMRI data, we defined individualized whole-brain networks from participants at rest, and the retinotopic preferences of individual voxels within these networks during an independent visual mapping task. We show that while the overall network activity between the DN and dATN is opponent at rest, a latent retinotopic code structures this global opponency. Specifically, the anti-correlation (i.e., global opponency) between the DN and dATN at rest is structured at the voxel-level by each voxel's retinotopic preferences, such that the spontaneous activity of voxels preferring similar visual field locations are more anti-correlated than those that prefer different visual field locations. Further, this retinotopic scaffold integrates with the domain-specific preferences of subregions within these networks, enabling efficient, parallel processing of retinotopic and domain-specific information. Thus, DN and dATN dynamics are opponent, but not competitive: voxel-scale anti-correlation between these networks preserves and encodes information in the negative BOLD responses, even in the absence of visual input or task demands. These findings suggest that retinotopic coding may serve as a fundamental organizing principle for brain-wide communication, providing a new framework for understanding how the brain balances and integrates internal cognition with external perception.

Text-Driven Video Prediction

Current video generation models usually convert signals indicating appearance and motion received from inputs (e.g., image and text) or latent spaces (e.g., noise vectors) into consecutive frames, fulfilling a stochastic generation process for the uncertainty introduced by latent code sampling. However, this generation pattern lacks deterministic constraints for both appearance and motion, leading to uncontrollable and undesirable outcomes. To this end, we propose a new task called Text-driven Video Prediction (TVP). Taking the first frame and text caption as inputs, this task aims to synthesize the following frames. Specifically, appearance and motion components are provided by the image and caption separately. The key to addressing the TVP task depends on fully exploring the underlying motion information in text descriptions, thus facilitating plausible video generation. In fact, this task is intrinsically a cause-and-effect problem, as the text content directly influences the motion changes of frames. To investigate the capability of text in causal inference for progressive motion information, our TVP framework contains a Text Inference Module (TIM), producing step-wise embeddings to regulate motion inference for subsequent frames. In particular, a refinement mechanism incorporating global motion semantics guarantees coherent generation. Extensive experiments are conducted on Something-Something V2 and Single Moving MNIST datasets. Experimental results demonstrate that our model achieves better results over other baselines, verifying the effectiveness of the proposed framework.

VoiceStyle: Voice-Based Face Generation via Cross-Modal Prototype Contrastive Learning

Can we predict a person’s appearance solely based on their voice? This article explores this question by focusing on generating a face from an unheard voice segment. Our proposed method, VoiceStyle, combines cross-modal representation learning with generation modeling, enabling us to incorporate voice semantic cues into the generated face. In the first stage, we introduce cross-modal prototype contrastive (CMPC) learning to establish the association between voice and face. Recognizing the presence of false negative and deviate positive instances in real-world unlabeled data, we not only use voice–face pairs in the same video but also construct additional semantic positive pairs through unsupervised clustering, enhancing the learning process. Moreover, we recalibrate instances based on their similarity to cluster centers in the other modality. In the second stage, we harness the powerful generative capabilities of StyleGAN to produce faces. We optimize the latent code in StyleGAN’s latent space, guided by the learned voice–face alignment. To address the importance of selecting an appropriate starting point for optimization, we aim to automatically find an optimal starting point by utilizing the face prototype derived from the voice input. The entire pipeline can be implemented in a self-supervised manner, eliminating the need for manually labeled annotations. Through extensive experiments, we demonstrate the effectiveness and performance of our VoiceStyle method in both cross-modal representation learning and voice-based face generation.

LoopNet for fine-grained fashion attributes editing

Generative Adversarial Networks (GANs) have revolutionized the field of image synthesis by transforming randomly sampled latent codes into high-fidelity synthesized images. However, current methods fall short in manipulating a wide range of fashion attributes due to semantic ambiguity and lack of disentanglement. This work focuses on fashion attribute editing by proposing an encoder-based GAN inversion method, namely LoopNet. To enable high-fidelity image inversion and fine-grained attribute editing, it refines edited images through two encoder–decoder stages, utilizing predefined directions from principal component analysis on latent codes and canny loss for detail enhancement. Experiments show LoopNet’s effectiveness in attribute disentanglement and manipulation, outperforming seven state-of-the-art image inversion methods.

Quantum deep generative prior with programmable quantum circuits

Exploiting the utility of near-term quantum devices is a long-standing challenge whereas hybrid quantum machine learning emerges as a promising candidate. Here we propose a quantum-enhanced deep generative algorithm based on programmable quantum circuit-induced quantum latent codes. To validate the effectiveness of the algorithm, we conduct optical ghost imaging experiments, collecting dataset under varying physical sampling rates. Leveraging a physically enhanced loss function and a pretrained neural network, the quantum algorithm exhibits superior reconstruction performance compared to conventional algorithms. We observe the learnable quantum latent space enhances the out-of-distribution generalization capability of the pretrained model. In the context of computer vision problems, numerical results indicate that quantum latent space can increase the generation diversity of the pretrained model. Furthermore, the algorithm outperforms classical counterparts, particularly in image inpainting and colorization, by a significant margin. Our study demonstrates the utility of hybrid quantum-classical algorithms in enhancing generalization capability, highlighting the potential of near-term quantum devices in large-scale generative artificial intelligence.

Audio2Gestures: Generating Diverse Gestures From Audio.

People may perform diverse gestures affected by various mental and physical factors when speaking the same sentences. This inherent one-to-many relationship makes co-speech gesture generation from audio particularly challenging. Conventional CNNs/RNNs assume one-to-one mapping, and thus tend to predict the average of all possible target motions, easily resulting in plain/boring motions during inference. So we propose to explicitly model the one-to-many audio-to-motion mapping by splitting the cross-modal latent code into shared code and motion-specific code. The shared code is expected to be responsible for the motion component that is more correlated to the audio while the motion-specific code is expected to capture diverse motion information that is more independent of the audio. However, splitting the latent code into two parts poses extra training difficulties. Several crucial training losses/strategies, including relaxed motion loss, bicycle constraint, and diversity loss, are designed to better train the VAE. Experiments on both 3D and 2D motion datasets verify that our method generates more realistic and diverse motions than previous state-of-the-art methods, quantitatively and qualitatively. Besides, our formulation is compatible with discrete cosine transformation (DCT) modeling and other popular backbones (i.e., RNN, Transformer). As for motion losses and quantitative motion evaluation, we find structured losses/metrics (e.g. STFT) that consider temporal and/or spatial context complement the most commonly used point-wise losses (e.g. PCK), resulting in better motion dynamics and more nuanced motion details. Finally, we demonstrate that our method can be readily used to generate motion sequences with user-specified motion clips on the timeline.

ANISE: Assembly-Based Neural Implicit Surface Reconstruction.

We present ANISE, a method that reconstructs a 3D shape from partial observations (images or sparse point clouds) using a part-aware neural implicit shape representation. The shape is formulated as an assembly of neural implicit functions, each representing a different part instance. In contrast to previous approaches, the prediction of this representation proceeds in a coarse-to-fine manner. Our model first reconstructs a structural arrangement of the shape in the form of geometric transformations of its part instances. Conditioned on them, the model predicts part latent codes encoding their surface geometry. Reconstructions can be obtained in two ways: (i) by directly decoding the part latent codes to part implicit functions, then combining them into the final shape; or (ii) by using part latents to retrieve similar part instances in a part database and assembling them in a single shape. We demonstrate that, when performing reconstruction by decoding part representations into implicit functions, our method achieves state-of-the-art part-aware reconstruction results from both images and sparse point clouds. When reconstructing shapes by assembling parts retrieved from a dataset, our approach significantly outperforms traditional shape retrieval methods even when significantly restricting the database size. We present our results in well-known sparse point cloud reconstruction and single-view reconstruction benchmarks.

LC-NeRF: Local Controllable Face Generation in Neural Radiance Field.

3D face generation has achieved high visual quality and 3D consistency thanks to the development of neural radiance fields (NeRF). However, these methods model the whole face as a neural radiance field, which limits the controllability of the local regions. In other words, previous methods struggle to independently control local regions, such as the mouth, nose, and hair. To improve local controllability in NeRF-based face generation, we propose LC-NeRF, which is composed of a Local Region Generators Module (LRGM) and a Spatial-Aware Fusion Module (SAFM), allowing for geometry and texture control of local facial regions. The LRGM models different facial regions as independent neural radiance fields and the SAFM is responsible for merging multiple independent neural radiance fields into a complete representation. Finally, LC-NeRF enables the modification of the latent code associated with each individual generator, thereby allowing precise control over the corresponding local region. Qualitative and quantitative evaluations show that our method provides better local controllability than state-of-the-art 3D-aware face generation methods. A perception study reveals that our method outperforms existing state-of-the-art methods in terms of image quality, face consistency, and editing effects. Furthermore, our method exhibits favorable performance in downstream tasks, including real image editing and text-driven facial image editing.

BrepGen: A B-rep Generative Diffusion Model with Structured Latent Geometry

This paper presents BrepGen , a diffusion-based generative approach that directly outputs a Boundary representation (B-rep) Computer-Aided Design (CAD) model. BrepGen represents a B-rep model as a novel structured latent geometry in a hierarchical tree. With the root node representing a whole CAD solid, each element of a B-rep model (i.e., a face, an edge, or a vertex) progressively turns into a child-node from top to bottom. B-rep geometry information goes into the nodes as the global bounding box of each primitive along with a latent code describing the local geometric shape. The B-rep topology information is implicitly represented by node duplication. When two faces share an edge, the edge curve will appear twice in the tree, and a T-junction vertex with three incident edges appears six times in the tree with identical node features. Starting from the root and progressing to the leaf, BrepGen employs Transformer-based diffusion models to sequentially denoise node features while duplicated nodes are detected and merged, recovering the B-Rep topology information. Extensive experiments show that BrepGen advances the task of CAD B-rep generation, surpassing existing methods on various benchmarks. Results on our newly collected furniture dataset further showcase its exceptional capability in generating complicated geometry. While previous methods were limited to generating simple prismatic shapes, BrepGen incorporates free-form and doubly-curved surfaces for the first time. Additional applications of BrepGen include CAD autocomplete and design interpolation. The code, pretrained models, and dataset are available at https://github.com/samxuxiang/BrepGen.

Face swapping with adaptive exploration-fusion mechanism and dual en-decoding tactic

Face swapping represents a formidable challenge—it involves a technique that can seamlessly transplant the identity traits from a source figure into the target figure while meticulously preserving the remaining attributes of the target’s figure. In the paper, we present HQFace, our framework to achieve the high-quality face swapping. Our exploration revealed that within the latent codes of facial images, identity information are dispersed across a spectrum of feature dimensions to varying extents. Based on this discovery, we design an innovative adaptive ‘exploration-fusion’ mechanism that adaptive searches identity information across these dimensions, and deftly fuses together the threads of identity-specific and the remaining attributes of the target’s figure. To enhance the verisimilitude of the facial details even further, we integrated a dual encoding and decoding tactic into the aforementioned fusion process. This novel strategy mitigates the potential hue distortions in facial complexions, which may arise from encoding drifts compared to latent spatial anchors. HQFace is put to the test through a comprehensive series of experiments, which stands as a testament to the framework’s ability to deliver high-quality face swapping with strikingly true and devoid of artifacts.

DeCoGAN: MVCT image denoising via coupled generative adversarial network

Objective. In helical tomotherapy, image-guided radiotherapy employs megavoltage computed tomography (MVCT) for precise targeting. However, the high voltage of megavoltage radiation introduces substantial noise, significantly compromising MVCT image clarity. This study aims to enhance MVCT image quality using a deep learning-based denoising method. Approach. We propose an unpaired MVCT denoising network using a coupled generative adversarial network framework (DeCoGAN). Our approach assumes that a universal latent code within a shared latent space can reconstruct any given pair of images. By employing an encoder, we enforce this shared-latent space constraint, facilitating the conversion of low-quality (noisy) MVCT images into high-quality (denoised) counterparts. The network learns the joint distribution of images from both domains by leveraging samples from their respective marginal distributions, enhanced by adversarial training for effective denoising. Main Results. Compared to an analytical algorithm (BM3D) and three deep learning-based methods (RED-CNN, WGAN-VGG and CycleGAN), the proposed method excels in preserving image details and enhancing human visual perception by removing most noise and retaining structural features. Quantitative analysis demonstrates that our method achieves the highest peak signal-to-noise ratio and Structural Similarity Index Measurement values, indicating superior denoising performance. Significance. The proposed DeCoGAN method shows remarkable MVCT denoising performance, making it a promising tool in the field of radiation therapy.

SAC-GAN: Structure-Aware Image Composition.

We introduce an end-to-end learning framework for image-to-image composition, aiming to plausibly compose an object represented as a cropped patch from an object image into a background scene image. As our approach emphasizes more on semantic and structural coherence of the composed images, rather than their pixel-level RGB accuracies, we tailor the input and output of our network with structure-aware features and design our network losses accordingly, with ground truth established in a self-supervised setting through the object cropping. Specifically, our network takes the semantic layout features from the input scene image, features encoded from the edges and silhouette in the input object patch, as well as a latent code as inputs, and generates a 2D spatial affine transform defining the translation and scaling of the object patch. The learned parameters are further fed into a differentiable spatial transformer network to transform the object patch into the target image, where our model is trained adversarially using an affine transform discriminator and a layout discriminator. We evaluate our network, coined SAC-GAN, for various image composition scenarios in terms of quality, composability, and generalizability of the composite images. Comparisons are made to state-of-the-art alternatives, including Instance Insertion, ST-GAN, CompGAN and PlaceNet, confirming superiority of our method.

HairStyle Editing via Parametric Controllable Strokes.

In this work, we propose a stroke-based hairstyle editing network, dubbed HairstyleNet, allowing users to conveniently change the hairstyles of an image in an interactive fashion. Different from previous works, we simplify the hairstyle editing process where users can manipulate local or entire hairstyles by adjusting the parameterized hair regions. Our HairstyleNet consists of two stages: a stroke parameterization stage and a stroke-to-hair generation stage. In the stroke parameterization stage, we first introduce parametric strokes to approximate the hair wisps, where the stroke shape is controlled by a quadratic Bézier curve and a thickness parameter. Since rendering strokes with thickness to an image is not differentiable, we opt to leverage a neural renderer to construct the mapping from stroke parameters to a stroke image. Thus, the stroke parameters can be directly estimated from hair regions in a differentiable way, enabling us to flexibly edit the hairstyles of input images. In the stroke-to-hair generation stage, we design a hairstyle refinement network that first encodes coarsely composed images of hair strokes, face, and background into latent representations and then generates high-fidelity face images with desirable new hairstyles from the latent codes. Extensive experiments demonstrate that our HairstyleNet achieves state-of-the-art performance and allows flexible hairstyle manipulation.

Diffusion-Based Causal Representation Learning.

Causal reasoning can be considered a cornerstone of intelligent systems. Having access to an underlying causal graph comes with the promise of cause-effect estimation and the identification of efficient and safe interventions. However, learning causal representations remains a major challenge, due to the complexity of many real-world systems. Previous works on causal representation learning have mostly focused on Variational Auto-Encoders (VAEs). These methods only provide representations from a point estimate, and they are less effective at handling high dimensions. To overcome these problems, we propose a Diffusion-based Causal Representation Learning (DCRL) framework which uses diffusion-based representations for causal discovery in the latent space. DCRL provides access to both single-dimensional and infinite-dimensional latent codes, which encode different levels of information. In a first proof of principle, we investigate the use of DCRL for causal representation learning in a weakly supervised setting. We further demonstrate experimentally that this approach performs comparably well in identifying the latent causal structure and causal variables.

InfoSTGCAN: An Information-Maximizing Spatial-Temporal Graph Convolutional Attention Network for Heterogeneous Human Trajectory Prediction

Predicting the future trajectories of multiple interacting pedestrians within a scene has increasingly gained importance in various fields, e.g., autonomous driving, human–robot interaction, and so on. The complexity of this problem is heightened due to the social dynamics among different pedestrians and their heterogeneous implicit preferences. In this paper, we present Information Maximizing Spatial-Temporal Graph Convolutional Attention Network (InfoSTGCAN), which takes into account both pedestrian interactions and heterogeneous behavior choice modeling. To effectively capture the complex interactions among pedestrians, we integrate spatial-temporal graph convolution and spatial-temporal graph attention. For grasping the heterogeneity in pedestrians’ behavior choices, our model goes a step further by learning to predict an individual-level latent code for each pedestrian. Each latent code represents a distinct pattern of movement choice. Finally, based on the observed historical trajectory and the learned latent code, the proposed method is trained to cover the ground-truth future trajectory of this pedestrian with a bi-variate Gaussian distribution. We evaluate the proposed method through a comprehensive list of experiments and demonstrate that our method outperforms all baseline methods on the commonly used metrics, Average Displacement Error and Final Displacement Error. Notably, visualizations of the generated trajectories reveal our method’s capacity to handle different scenarios.

Radio Galaxy Zoo: Leveraging latent space representations from variational autoencoder

We propose to learn latent space representations of radio galaxies, and train a very deep variational autoencoder (VDVAE) on RGZ DR1, an unlabeled dataset, to this end. We show that the encoded features can be leveraged for downstream tasks such as classifying galaxies in labeled datasets, and similarity search. Results show that the model is able to reconstruct its given inputs, capturing the salient features of the latter. We use the latent codes of galaxy images, from MiraBest Confident and FR-DEEP NVSS datasets, to train various non-neural network classifiers. It is found that the latter can differentiate FRI from FRII galaxies achieving accuracy ≥ 76%, roc-auc ≥ 0.86, specificity ≥ 0.73 and recall ≥ 0.78 on MiraBest Confident dataset, comparable to results obtained in previous studies. The performance of simple classifiers trained on FR-DEEP NVSS data representations is on par with that of a deep learning classifier (CNN based) trained on images in previous work, highlighting how powerful the compressed information is. We successfully exploit the learned representations to search for galaxies in a dataset that are semantically similar to a query image belonging to a different dataset. Although generating new galaxy images (e.g. for data augmentation) is not our primary objective, we find that the VDVAE model is a relatively good emulator. Finally, as a step toward detecting anomaly/novelty, a density estimator — Masked Autoregressive Flow (MAF) — is trained on the latent codes, such that the log-likelihood of data can be estimated. The downstream tasks conducted in this work demonstrate the meaningfulness of the latent codes.

Application of C-InGAN Model in Interpretable Feature of Bearing Fault Diagnosis.

Although traditional fault diagnosis methods are proficient in extracting signal features, their diagnostic interpretability remains challenging. Consequently, this article proposes a conditionally interpretable generative adversarial network (C-InGAN) model for the interpretable feature fault diagnosis of bearings. Initially, the vibration signal is denoised and transformed into a frequency domain signal. The model consists of the two primary networks, each employing a convolutional layer and an attention module, generator (G) and discriminator (D), respectively. Latent code was incorporated into G to constrain the generated samples, and a discriminant layer was added to D to identify the interpretable features. During training, the two networks were alternately trained, and the feature mapping relationship of the pre-normalized encoder was learned by maximizing the information from the latent code and the discriminative result. The encoding that represents specific features in the vibration signal was extracted from the random noise. Ultimately, after completing adversarial learning, G is capable of generating a simulated signal of the specified feature, and D can assess the interpretable features in the vibration signal. The effectiveness of the model is validated through three typical experimental cases. This method effectively separates the discrete and continuous feature coding in the signal.