Canonical Space Research Articles

PCLC‐Net: Point Cloud Completion in Arbitrary Poses with Learnable Canonical Space

AbstractRecovering the complete structure from partial point clouds in arbitrary poses is challenging. Recently, many efforts have been made to address this problem by developing SO(3)‐equivariant completion networks or aligning the partial point clouds with a predefined canonical space before completion. However, these approaches are limited to random rotations only or demand costly pose annotation for model training. In this paper, we present a novel Network for Point cloud Completion with Learnable Canonical space (PCLC‐Net) to reduce the need for pose annotations and extract SE(3)‐invariant geometry features to improve the completion quality in arbitrary poses. Without pose annotations, our PCLC‐Net utilizes self‐supervised pose estimation to align the input partial point clouds to a canonical space that is learnable for an object category and subsequently performs shape completion in the learned canonical space. Our PCLC‐Net can complete partial point clouds with arbitrary SE(3) poses without requiring pose annotations for supervision. Our PCLC‐Net achieves state‐of‐the‐art results on shape completion with arbitrary SE(3) poses on both synthetic and real scanned data. To the best of our knowledge, our method is the first to achieve shape completion in arbitrary poses without pose annotations during network training.

Archimedean ℓ-groups with strong unit: cozero-sets and coincidence of types of ideals

Abstract W* is the category of the title. For G ∈ W *, we have the canonical compact space YG, and Yosida representation G ≤ C(YG), thus, for g ∈ G, one has the cozero-set coz(g) in YG. The ideals at issue in G include the principal ideals and polars, G(g) and g ⊥⊥, respectively, and the W *-kernels of W *-morphisms from G. The “coincidences of types” include these properties of G: (M) Each G(g) = g ⊥⊥; (Y) Each G(g) is a W *-kernel; (CR) Each g ⊥⊥ is a W *-kernel (iff each coz(g) is regular open). For each of these, we give numerous “rephrasings” and examples, and note that (M) = (Y) ∩ (CR). This paper is a companion to a paper in preparation by the present authors, which includes the present thrust in contexts less restrictive and more algebraic. Here, the focus on W * brings topology to bear, and sharpens the view.

TriHuman: A Real-time and Controllable Tri-plane Representation for Detailed Human Geometry and Appearance Synthesis

Creating controllable, photorealistic, and geometrically detailed digital doubles of real humans solely from video data is a key challenge in Computer Graphics and Vision, especially when real-time performance is required. Recent methods attach a neural radiance field (NeRF) to an articulated structure, e.g., a body model or a skeleton, to map points into a pose canonical space while conditioning the NeRF on the skeletal pose. These approaches typically parameterize the neural field with a multi-layer perceptron (MLP) leading to a slow runtime. To address this drawback, we propose TriHuman a novel human-tailored, deformable, and efficient tri-plane representation, which achieves real-time performance, state-of-the-art pose-controllable geometry synthesis as well as photorealistic rendering quality. At the core, we non-rigidly warp global ray samples into our undeformed tri-plane texture space, which effectively addresses the problem of global points being mapped to the same tri-plane locations. We then show how such a tri-plane feature representation can be conditioned on the skeletal motion to account for dynamic appearance and geometry changes. Our results demonstrate a clear step toward higher quality in terms of geometry and appearance modeling of humans as well as runtime performance.

A Modular Neural Motion Retargeting System Decoupling Skeleton and Shape Perception.

Motion mapping between characters with different structures but corresponding to homeomorphic graphs, meanwhile preserving motion semantics and perceiving shape geometries, poses significant challenges in skinned motion retargeting. We propose M-R 2 ET, a modular neural motion retargeting system to comprehensively address these challenges. The key insight driving M-R 2 ET is its capacity to learn residual motion modifications within a canonical skeleton space. Specifically, a cross-structure alignment module is designed to learn joint correspondences among diverse skeletons, enabling motion copy and forming a reliable initial motion for semantics and geometry perception. Besides, two residual modification modules, i.e., the skeleton-aware module and shape-aware module, preserving source motion semantics and perceiving target character geometries, effectively reduce interpenetration and contact-missing. Driven by our distance-based losses that explicitly model the semantics and geometry, these two modules learn residual motion modifications to the initial motion in a single inference without post-processing. To balance these two motion modifications, we further present a balancing gate to conduct linear interpolation between them. Extensive experiments on the public dataset Mixamo demonstrate that our M-R 2 ET achieves the state-of-the-art performance, enabling cross-structure motion retargeting, and providing a good balance among the preservation of motion semantics, as well as the attenuation of interpenetration and contact-missing.

Learning facial structural dependency in 3D aligned space for face alignment

Facial structure's statistical characteristics offer pivotal prior information in facial landmark prediction, forming inter-dependencies among different landmarks. Such inter-dependencies ensure that predictions adhere to the shape distribution typical of natural faces. In challenging scenarios like occlusions or extreme facial poses, this structure becomes indispensable, which can help to predict elusive landmarks based on more discernible ones. While current deep learning methods do capture these landmark dependencies, it's often an implicit process heavily reliant on vast training datasets. We contest that such implicit modeling approaches fail to manage more challenging situations. In this paper, we propose a new method that harnesses the facial structure and explicitly explores inter-dependencies among facial landmarks in an end-to-end fashion. We propose a Structural Dependency Learning Module (SDLM). It uses 3D face information to map facial features into a canonical UV space, in which the facial structure is explicitly 3D semantically aligned. Besides, to explore the global relationships between facial landmarks, we take advantage of the self-attention mechanism in the image and UV spaces. We name the proposed method Facial Structure-based Face Alignment (FSFA). FSFA reinforces the landmark structure, especially under challenging conditions. Extensive experiments demonstrate that FSFA achieves state-of-the-art performance on the WFLW, 300W, AFLW, and COFW68 datasets.

Neural Novel Actor: Learning a Generalized Animatable Neural Representation for Human Actors.

We propose a new method for learning a generalized animatable neural human representation from a sparse set of multi-view imagery of multiple persons. The learned representation can be used to synthesize novel view images of an arbitrary person and further animate them with the user's pose control. While most existing methods can either generalize to new persons or synthesize animations with user control, none of them can achieve both at the same time. We attribute this accomplishment to the employment of a 3D proxy for a shared multi-person human model, and further the warping of the spaces of different poses to a shared canonical pose space, in which we learn a neural field and predict the person- and pose-dependent deformations, as well as appearance with the features extracted from input images. To cope with the complexity of the large variations in body shapes, poses, and clothing deformations, we design our neural human model with disentangled geometry and appearance. Furthermore, we utilize the image features both at the spatial point and on the surface points of the 3D proxy for predicting person- and pose-dependent properties. Experiments show that our method significantly outperforms the state-of-the-arts on both tasks.

ReN Human: Learning Relightable Neural Implicit Surfaces for Animatable Human Rendering

Recently, implicit neural representation has been widely used to learn the appearance of human bodies in the canonical space, which can be further animated using a parametric human model. However, how to decompose the material properties from the implicit representation for relighting has not yet been investigated thoroughly. We propose to address this problem with a novel framework, ReN Human, that takes sparse or even monocular input videos collected in unconstrained lighting to produce a 3D human representation that can be rendered with novel views, poses, and lighting. Our method represents humans as deformable implicit neural representation and decomposes the geometry, material of humans as well as environment illumination for capturing a relightable and animatable human model. Moreover, we introduce a volumetric lighting grid consisting of spherical Gaussian mixtures to learn the spatially varying illumination and animatable visibility probes to model the dynamic self-occlusion caused by human motion. Specifically, we learn the material property fields and illumination using a physically-based rendering layer that uses Monte Carlo importance sampling to facilitate differentiation of the complex rendering integral. We demonstrate that our approach outperforms recent novel views and poses synthesis methods in a challenging benchmark with sparse videos, enabling high-fidelity human relighting.

Generalized Yang Poisson Models on Canonical Phase Space

We discuss the generalized Yang Poisson models. We construct generalizations of the Yang Poisson algebra related to $\mathfrak{o}(1,5)$ algebra discussed by Meljanac and Mignemi (2023). The exact realizations of this generalized algebra on canonical phase space are presented and the corresponding differential equations are solved in simple cases. Furthermore, we discuss the Poisson algebras related to $\mathfrak{o}(3,3)$ and $\mathfrak{o}(2,4)$ algebras.

A collision operator for describing dissipation in noncanonical phase space

The phase space of a noncanonical Hamiltonian system is partially inaccessible due to dynamical constraints (Casimir invariants) arising from the kernel of the Poisson tensor. When an ensemble of noncanonical Hamiltonian systems is allowed to interact, dissipative processes eventually break the phase space constraints, resulting in a thermodynamic equilibrium described by a Maxwell–Boltzmann distribution. However, the time scale required to reach Maxwell–Boltzmann statistics is often much longer than the time scale over which a given system achieves a state of thermal equilibrium. Examples include diffusion in rigid mechanical systems, as well as collisionless relaxation in magnetized plasmas and stellar systems, where the interval between binary Coulomb or gravitational collisions can be longer than the time scale over which stable structures are self-organized. Here, we focus on self-organizing phenomena over spacetime scales such that particle interactions respect the noncanonical Hamiltonian structure, but yet act to create a state of thermodynamic equilibrium. We derive a collision operator for general noncanonical Hamiltonian systems, applicable to fast, localized interactions. This collision operator depends on the interaction exchanged by colliding particles and on the Poisson tensor encoding the noncanonical phase space structure, is consistent with entropy growth and conservation of particle number and energy, preserves the interior Casimir invariants, reduces to the Landau collision operator in the limit of grazing binary Coulomb collisions in canonical phase space, and exhibits a metriplectic structure. We further show how thermodynamic equilibria depart from Maxwell–Boltzmann statistics due to the noncanonical phase space structure, and how self-organization and collisionless relaxation in magnetized plasmas and stellar systems can be described through the derived collision operator.

ShellNeRF: Learning a Controllable High‐resolution Model of the Eye and Periocular Region

AbstractEye gaze and expressions are crucial non‐verbal signals in face‐to‐face communication. Visual effects and telepresence demand significant improvements in personalized tracking, animation, and synthesis of the eye region to achieve true immersion. Morphable face models, in combination with coordinate‐based neural volumetric representations, show promise in solving the difficult problem of reconstructing intricate geometry (eyelashes) and synthesizing photorealistic appearance variations (wrinkles and specularities) of eye performances. We propose a novel hybrid representation ‐ ShellNeRF ‐ that builds a discretized volume around a 3DMM face mesh using concentric surfaces to model the deformable ‘periocular’ region. We define a canonical space using the UV layout of the shells that constrains the space of dense correspondence search. Combined with an explicit eyeball mesh for modeling corneal light‐transport, our model allows for animatable photorealistic 3D synthesis of the whole eye region. Using multi‐view video input, we demonstrate significant improvements over state‐of‐the‐art in expression re‐enactment and transfer for high‐resolution close‐up views of the eye region.

HQ3DAvatar: High-quality Implicit 3D Head Avatar

Multi-view volumetric rendering techniques have recently shown great potential in modeling and synthesizing high-quality head avatars. A common approach to capture full head dynamic performances is to track the underlying geometry using a mesh-based template or 3D cube-based graphics primitives. While these model-based approaches achieve promising results, they often fail to learn complex geometric details such as the mouth interior, hair, and topological changes over time. This article presents a novel approach to building highly photorealistic digital head avatars. Our method learns a canonical space via an implicit function parameterized by a neural network. It leverages multiresolution hash encoding in the learned feature space, allowing for high quality, faster training, and high-resolution rendering. At test time, our method is driven by a monocular RGB video. Here, an image encoder extracts face-specific features that also condition the learnable canonical space. This encourages deformation-dependent texture variations during training. We also propose a novel optical flow-based loss that ensures correspondences in the learned canonical space, thus encouraging artifact-free and temporally consistent renderings. We show results on challenging facial expressions and show free-viewpoint renderings at interactive real-time rates for a resolution of 480 x 270. Our method outperforms related approaches both visually and numerically. We will release our multiple-identity dataset to encourage further research.

Inflationary α-attractor from type-IIB/F theory

We derive an potential of slow-roll inflation in the warped D brane set-up featuring three intersecting D7 branes under type of CY 3-compactification within type-IIB/F theory with some near-conifold regions. The underlying quadratic structure of the kinetic poles is found to arise from a correction in the Kähler potential when an extra contribution of open string moduli is turned on. While the closed string sector of the moduli spectrum is completely stabilized via quantum corrections of perturbative and non-perturbative origin, the open string sector plays the lead role in driving the inflationary expansion in the radial direction. A generic asymptotic behaviour of the inflaton field near the pole boundaries manifests itself as the slow-roll plateau in canonical field space, which becomes responsible for giving universal predictions of the cosmological parameters. We find that the presence of the open strings near conifold regions brings the realization of pole inflation in the present set-up. Finally we compare our results with similar models and discuss the importance of exploring precise values of α in the light of ongoing and forthcoming cosmological surveys.

Quantization of non-Abelian Yang-Mills theories

A non-Abelian theory of fermions interacting with gauge bosons, the constrained system, is studied. The equations of motion for a singular system are obtained as total differential equations in many variables. The integrability conditions are investigated, and the set of equations of motion is integrable. The Senjanovic and the canonical methods are used to quantize the system, and the integration is taken over the canonical phase space coordinates.

DLCA-Recon: Dynamic Loose Clothing Avatar Reconstruction from Monocular Videos

Reconstructing a dynamic human with loose clothing is an important but difficult task. To address this challenge, we propose a method named DLCA-Recon to create human avatars from monocular videos. The distance from loose clothing to the underlying body rapidly changes in every frame when the human freely moves and acts. Previous methods lack effective geometric initialization and constraints for guiding the optimization of deformation to explain this dramatic change, resulting in the discontinuous and incomplete reconstruction surface.To model the deformation more accurately, we propose to initialize an estimated 3D clothed human in the canonical space, as it is easier for deformation fields to learn from the clothed human than from SMPL.With both representations of explicit mesh and implicit SDF, we utilize the physical connection information between consecutive frames and propose a dynamic deformation field (DDF) to optimize deformation fields. DDF accounts for contributive forces on loose clothing to enhance the interpretability of deformations and effectively capture the free movement of loose clothing. Moreover, we propagate SMPL skinning weights to each individual and refine pose and skinning weights during the optimization to improve skinning transformation. Based on more reasonable initialization and DDF, we can simulate real-world physics more accurately. Extensive experiments on public and our own datasets validate that our method can produce superior results for humans with loose clothing compared to the SOTA methods.

Investigating the ability of deep learning-based structure prediction to extrapolate and/or enrich the set of antibody CDR canonical forms.

Deep learning models have been shown to accurately predict protein structure from sequence, allowing researchers to explore protein space from the structural viewpoint. In this paper we explore whether "novel" features, such as distinct loop conformations can arise from these predictions despite not being present in the training data. Here we have used ABodyBuilder2, a deep learning antibody structure predictor, to predict the structures of ~1.5M paired antibody sequences. We examined the predicted structures of the canonical CDR loops and found that most of these predictions fall into the already described CDR canonical form structural space. We also found a small number of "new" canonical clusters composed of heterogeneous sequences united by a common sequence motif and loop conformation. Analysis of these novel clusters showed their origins to be either shapes seen in the training data at very low frequency or shapes seen at high frequency but at a shorter sequence length. To evaluate explicitly the ability of ABodyBuilder2 to extrapolate, we retrained several models whilst withholding all antibody structures of a specific CDR loop length or canonical form. These "starved" models showed evidence of generalisation across CDRs of different lengths, but they did not extrapolate to loop conformations which were highly distinct from those present in the training data. However, the models were able to accurately predict a canonical form even if only a very small number of examples of that shape were in the training data. Our results suggest that deep learning protein structure prediction methods are unable to make completely out-of-domain predictions for CDR loops. However, in our analysis we also found that even minimal amounts of data of a structural shape allow the method to recover its original predictive abilities. We have made the ~1.5 M predicted structures used in this study available to download at https://doi.org/10.5281/zenodo.10280181.

The existence of universal qualitative belief spaces

This paper constructs a canonical representation of players' interactive beliefs, irrespective of natures of beliefs: whether beliefs are qualitative, truthful (i.e., knowledge), or probabilistic (e.g., countably-additive, finitely-additive, or non-additive). The canonical model is the “largest” interactive belief model to which any particular model can be mapped in a unique belief-preserving way. The key insight for the construction is the need to specify players' possible depth of reasoning up to which they can interactively reason about their beliefs (e.g., their beliefs, their beliefs about their beliefs, their beliefs about their beliefs about their beliefs, and so on). The possible depth of reasoning may be a transfinite level (beyond any finite level) when beliefs are qualitative. The specification of possible depth of reasoning also has game-theoretic implications for characterizations of some solution concepts using the canonical space. For instance, for any strategic game with ordinal payoffs, there exists a canonical interactive belief model which characterizes iterated elimination of strictly dominated actions as an implication of common belief in rationality.

Animatable Implicit Neural Representations for Creating Realistic Avatars From Videos.

This paper addresses the challenge of reconstructing an animatable human model from a multi-view video. Some recent works have proposed to decompose a non-rigidly deforming scene into a canonical neural radiance field and a set of deformation fields that map observation-space points to the canonical space, thereby enabling them to learn the dynamic scene from images. However, they represent the deformation field as translational vector field or SE(3) field, which makes the optimization highly under-constrained. Moreover, these representations cannot be explicitly controlled by input motions. Instead, we introduce blend weight fields to produce the deformation fields. Based on the skeleton-driven deformation, blend weight fields are used with 3D human skeletons to generate observation-to-canonical and canonical-to-observation correspondences. Since 3D human skeletons are more observable, they can regularize the learning of deformation fields. Moreover, the blend weight fields can be combined with input skeletal motions to generate new deformation fields to animate the human model. To improve the quality of human modeling, we further represent the human geometry as a signed distance field in the canonical space. Additionally, a neural point displacement field is introduced to enhance the capability of the blend weight field on modeling detailed human motions. Experiments show that our approach significantly outperforms recent human modeling methods.

MPS-NeRF: Generalizable 3D Human Rendering From Multiview Images.

There has been rapid progress recently on 3D human rendering, including novel view synthesis and pose animation, based on the advances of neural radiance fields (NeRF). However, most existing methods focus on person-specific training and their training typically requires multi-view videos. This paper deals with a new challenging task - rendering novel views and novel poses for a person unseen in training, using only multiview still images as input without videos. For this task, we propose a simple yet surprisingly effective method to train a generalizable NeRF with multiview images as conditional input. The key ingredient is a dedicated representation combining a canonical NeRF and a volume deformation scheme. Using a canonical space enables our method to learn shared properties of human and easily generalize to different people. Volume deformation is used to connect the canonical space with input and target images and query image features for radiance and density prediction. We leverage the parametric 3D human model fitted on the input images to derive the deformation, which works quite well in practice when combined with our canonical NeRF. The experiments on both real and synthetic data with the novel view synthesis and pose animation tasks collectively demonstrate the efficacy of our method.

CPPF++: Uncertainty-Aware Sim2Real Object Pose Estimation by Vote Aggregation.

Object pose estimation constitutes a critical area within the domain of 3D vision. While contemporary state-of-the-art methods that leverage real-world pose annotations have demonstrated commendable performance, the procurement of such real training data incurs substantial costs. This paper focuses on a specific setting wherein only 3D CAD models are utilized as a priori knowledge, devoid of any background or clutter information. We introduce a novel method, CPPF++, designed for sim-to-real category-level pose estimation. This method builds upon the foundational point-pair voting scheme of CPPF, reformulating it through a probabilistic view. To address the challenge posed by vote collision, we propose a novel approach that involves modeling the voting uncertainty by estimating the probabilistic distribution of each point pair within the canonical space. Furthermore, we augment the contextual information provided by each voting unit through the introduction of N-point tuples. To enhance the robustness and accuracy of the model, we incorporate several innovative modules, including noisy pair filtering, online alignment optimization, and a tuple feature ensemble. Alongside these methodological advancements, we introduce a new category-level pose estimation dataset, named DiversePose 300. Empirical evidence demonstrates that our method significantly surpasses previous sim-to-real approaches and achieves comparable or superior performance on novel datasets. Our code is available on https://github.com/qq456cvb/CPPF2.

Modular geodesics and wedge domains in non-compactly causal symmetric spaces

We continue our investigation of the interplay between causal structures on symmetric spaces and geometric aspects of Algebraic Quantum Field Theory. We adopt the perspective that the geometric implementation of the modular group is given by the flow generated by an Euler element of the Lie algebra (an element defining a 3-grading). Since any Euler element of a semisimple Lie algebra specifies a canonical non-compactly causal symmetric space M=G/H\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$M = G/H$$\\end{document}, we turn in this paper to the geometry of this flow. Our main results concern the positivity region W of the flow (the corresponding wedge region): If G has trivial center, then W is connected, it coincides with the so-called observer domain, specified by a trajectory of the modular flow which at the same time is a causal geodesic. It can also be characterized in terms of a geometric KMS condition, and it has a natural structure of an equivariant fiber bundle over a Riemannian symmetric space that exhibits it as a real form of the crown domain of G/K. Among the tools that we need for these results are two observations of independent interest: a polar decomposition of the positivity domain and a convexity theorem for G-translates of open H-orbits in the minimal flag manifold specified by the 3-grading.