Canonical Space Research Articles

Realizations of the Yang–Poisson model on canonical phase space

In this paper, we discuss exact realizations of the Yang–Poisson model on canonical phase space. The Yang model is an example of noncommutative geometry on a background space–time of constant curvature and is notable for its duality between position and momentum manifolds. We call Yang–Poisson model its classical limit, with commutators replaced by Poisson brackets. The structure is simpler in the classical case, and exact realizations can be found.

AvatarStudio: Text-Driven Editing of 3D Dynamic Human Head Avatars

Capturing and editing full-head performances enables the creation of virtual characters with various applications such as extended reality and media production. The past few years witnessed a steep rise in the photorealism of human head avatars. Such avatars can be controlled through different input data modalities, including RGB, audio, depth, IMUs, and others. While these data modalities provide effective means of control, they mostly focus on editing the head movements such as the facial expressions, head pose, and/or camera viewpoint. In this paper, we propose AvatarStudio, a text-based method for editing the appearance of a dynamic full head avatar. Our approach builds on existing work to capture dynamic performances of human heads using Neural Radiance Field (NeRF) and edits this representation with a text-to-image diffusion model. Specifically, we introduce an optimization strategy for incorporating multiple keyframes representing different camera viewpoints and time stamps of a video performance into a single diffusion model. Using this personalized diffusion model, we edit the dynamic NeRF by introducing view-and-time-aware Score Distillation Sampling (VT-SDS) following a model-based guidance approach. Our method edits the full head in a canonical space and then propagates these edits to the remaining time steps via a pre-trained deformation network. We evaluate our method visually and numerically via a user study, and results show that our method outperforms existing approaches. Our experiments validate the design choices of our method and highlight that our edits are genuine, personalized, as well as 3D- and time-consistent.

Revisiting Maxwell-Smoluchowski theory: Low surface roughness in straight channels

The Maxwell-Smoluchowski (MS) theory of gas diffusion is revisited here in the context of gas transport in straight channels in the Knudsen regime of large mean free path. This classical theory is based on a phenomenological model of gas-surface interaction that posits that a fraction ϑ of molecular collisions with the channel surface consists of diffuse collisions, i.e., the direction of post-collision velocities is distributed according to the Knudsen Cosine Law, and a fraction 1−ϑ undergoes specular reflection. From this assumption one obtains the value D=2−ϑϑDK for the self-diffusivity constant, where DK is a reference value corresponding to ϑ=1. In this paper we show that ϑ can be expressed in terms of micro- and macro-geometric parameters for a model consisting of hard spheres colliding elastically against a rigid surface with prescribed microgeometry.Our refinement of the MS theory is based on the observation that the classical surface scattering operator associated to the microgeometry has a canonical velocity space diffusion approximation by a generalized Legendre differential operator whose spectral theory is known explicitly. More specifically, starting from an explicit description of the effective channel surface microgeometry—a concept which incorporates both the actual surface microgeometry and the molecular radius—and using this operator approximation, we show that ϑ can be resolved into easily obtained geometric parameters, ϑ=λh/C, having the following interpretation: C is a macroscopic parameter determined by the shape of the channel cross-section; h is a parameter that precisely captures the degree of roughness of the effective microgeometry, and λ is a parameter that characterizes the overall curvature of the surface microgeometry independent of h. Thus ϑ is resolved as the quotient of microscopic (λh) over macroscopic (C) signature parameters of the channel geometry. The identity ϑ=λh/C holds up to higher order terms in the roughness parameter h, so our main result better applies to well polished, or low roughness, surfaces.

Deform2NeRF: Non-Rigid Deformation and 2D–3D Feature Fusion with Cross-Attention for Dynamic Human Reconstruction

Reconstructing dynamic human body models from multi-view videos poses a substantial challenge in the field of 3D computer vision. Currently, the Animatable NeRF method addresses this challenge by mapping observed points from the viewing space to a canonical space. However, this mapping introduces positional shifts in predicted points, resulting in artifacts, particularly in intricate areas. In this paper, we propose an innovative approach called Deform2NeRF that incorporates non-rigid deformation correction and image feature fusion modules into the Animatable NeRF framework to enhance the reconstruction of animatable human models. Firstly, we introduce a non-rigid deformation field network to address the issue of point position shift effectively. This network adeptly corrects positional discrepancies caused by non-rigid deformations. Secondly, we introduce a 2D–3D feature fusion learning module with cross-attention and integrate it with the NeRF network to mitigate artifacts in specific detailed regions. Our experimental results demonstrate that our method significantly improves the PSNR index by approximately 5% compared to representative methods in the field. This remarkable advancement underscores the profound importance of our approach in the domains of new view synthesis and digital human reconstruction.

Virtual lighting environment and real human fusion based on multiview videos

The realistic fusion of human bodies and virtual lighting environments has important application value in digital tourism, virtual conferences, and other fields. To achieve this goal, it is necessary not only to reconstruct the geometry of the human body but also to estimate the reflectance for the human relighting in a virtual environment. Existing methods only focus on either reconstructing the human body geometry or relighting the human body under specific poses and cannot accommodate new poses in fusion-relighting images. In this paper, we propose a novel approach that enables the relighting of new poses from multiview human videos under unknown illumination, whereby a relationship is established between the observation space and canonical space through a pose-mapping process that connects the geometry and reflectance at different temporal and spatial locations. The geometry and reflectance results obtained from the neural fields are input into the physically based renderer along with the optimized environment map. Human relighting in new poses can be generated by inputting skeletal poses, and the entire pipeline can be trained in a self-supervised manner. The proposed technique was verified on multiple datasets. The experimental results show that the proposed method is capable of fulfilling the goal of a virtual lighting environment and real human fusion under new human poses, outperforming recent methods both subjectively and quantitatively.

Fast-SNARF: A Fast Deformer for Articulated Neural Fields.

Neural fields have revolutionized the area of 3D reconstruction and novel view synthesis of rigid scenes. A key challenge in making such methods applicable to articulated objects, such as the human body, is to model the deformation of 3D locations between the rest pose (a canonical space) and the deformed space. We propose a new articulation module for neural fields, Fast-SNARF, which finds accurate correspondences between canonical space and posed space via iterative root finding. Fast-SNARF is a drop-in replacement in functionality to our previous work, SNARF, while significantly improving its computational efficiency. We contribute several algorithmic and implementation improvements over SNARF, yielding a speed-up of 150×. These improvements include voxel-based correspondence search, pre-computing the linear blend skinning function, and an efficient software implementation with CUDA kernels. Fast-SNARF enables efficient and simultaneous optimization of shape and skinning weights given deformed observations without correspondences (e.g. 3D meshes). Because learning of deformation maps is a crucial component in many 3D human avatar methods and since Fast-SNARF provides a computationally efficient solution, we believe that this work represents a significant step towards the practical creation of 3D virtual humans.

MVP-Human Dataset for 3-D Clothed Human Avatar Reconstruction From Multiple Frames

In this paper, we consider a novel problem of reconstructing a 3D clothed human avatar from multiple frames, independent of assumptions on camera calibration, capture space, and constrained actions. We contribute a large-scale dataset, Multi-View and multi-Pose 3D human (MVP-Human in short) to help address this problem. The dataset contains 400 subjects, each of which has 15 scans in different poses and 8-view images for each pose, providing 6,000 3D scans and 48,000 images in total. In addition, a baseline method that takes multiple images as inputs, and generates a shape-with-skinning avatar in the canonical space, finished in one feed-forward pass is proposed. It first reconstructs the implicit skinning fields in a multi-level manner, and then the image features from multiple images are aligned and integrated to estimate a pixel-aligned implicit function that represents the clothed shape. With the newly collected dataset and the baseline method, it shows promising performance on 3D clothed avatar reconstruction. We release the MVP-Human dataset and the baseline method in, hoping to promote research and development in this field.

Temporally consistent reconstruction of 3D clothed human surface with warp field

Implicit functions are widely used in 3D human surface reconstruction due to their advantage to represent details. However, human reconstruction based on implicit functions struggles to maintain the integrity (unbroken body structure) and accuracy (no non-human parts) of human models. To address these issues, we propose a method, called TCR, for temporally consistent reconstruction of 3D clothed human surface with warp field. The fact that the general shape of a person does not change largely over time inspires us to exploit the temporally consistent shape information from previous frames to refine the human model of current frame. Therefore, we construct a canonical space and then store the shape information by updating the canonical model. To align the observed space with the canonical space, a warp field is firstly estimated for the forward and inverse warping of the human model. A probabilistic fusion strategy is then used to update the canonical model. In addition, the reconstructed result is further refined through the orthogonality constraints between the surface and its normal, which fully exploits the detailed information of estimated normal maps. Experiments on the Adobe and MonoPerfCap datasets show that TCR achieves the state-of-the-art performance. Furthermore, TCR is more robust and can maintain the integrity and accuracy of the reconstructed human body even with extreme poses and partial occlusions.

Progressive Multi-View Human Mesh Recovery with Self-Supervision

To date, little attention has been given to multi-view 3D human mesh estimation, despite real-life applicability (e.g., motion capture, sport analysis) and robustness to single-view ambiguities. Existing solutions typically suffer from poor generalization performance to new settings, largely due to the limited diversity of image/3D-mesh pairs in multi-view training data. To address this shortcoming, people have explored the use of synthetic images. But besides the usual impact of visual gap between rendered and target data, synthetic-data-driven multi-view estimators also suffer from overfitting to the camera viewpoint distribution sampled during training which usually differs from real-world distributions. Tackling both challenges, we propose a novel simulation-based training pipeline for multi-view human mesh recovery, which (a) relies on intermediate 2D representations which are more robust to synthetic-to-real domain gap; (b) leverages learnable calibration and triangulation to adapt to more diversified camera setups; and (c) progressively aggregates multi-view information in a canonical 3D space to remove ambiguities in 2D representations. Through extensive benchmarking, we demonstrate the superiority of the proposed solution especially for unseen in-the-wild scenarios.

External gauge field-coupled quantum dynamics: Gauge choices, Heisenberg algebra representations and gauge invariance in general, and the Landau problem in particular

Even though its classical equations of motion are then left invariant, when an action is redefined by an additive total derivative or divergence term (in time, in the case of a mechanical system) such a transformation induces non-trivial consequences for the system’s canonical phase space formulation. This is even more true and then in more subtle ways for the canonically quantized dynamics, with, in particular, an induced transformation in the unitary configuration space representation of the Heisenberg algebra being used for the quantum system. When coupled to a background gauge field, such considerations become crucial for a proper understanding of the consequences for the system’s quantum dynamics of gauge transformations of that classical external background gauge field, while under such transformations, the system’s degrees of freedom, abstract quantum states and quantum dynamics are certainly strictly invariant. After a detailed analysis of these different points in a general context, these are then illustrated specifically in the case of the quantum Landau problem with its classical external background magnetic vector potential for which the most general possible parametrized gauge choice is implemented herein. The latter discussion aims as well to clarify some perplexing statements in the literature regarding the status of gauge choices to be made for the magnetic vector potential for that quantum system. The role of the global spacetime symmetries of the Landau problem and their gauge invariant Noether charges is then also emphasized.

Holography of information in massive gravity using Dirac brackets

The principle of holography of information states that in massless gravity, it is possible to extract bulk information using asymptotic boundary operators. In our work, we study this principle in a linearized setting about empty flat space and formulate it using Dirac brackets between boundary Hamiltonian and bulk operators. We then address whether the storage of bulk information in flat space linearized massive gravity resembles that of massless gravity. For linearized massless gravity, using Dirac brackets, we recover the necessary criteria for the holography of information. In contrast, we show that the Dirac bracket of the relevant boundary observable with bulk operators vanishes for massive gravity. We use this important distinction to outline the canonical Hilbert space. This leads to split states, and consequently, one cannot use asymptotic boundary observables to extract bulk information in massive gravity. We also argue the split property directly without an explicit reference to the Hilbert space. The result reflects that we can construct local bulk operators in massive gravity about the vacuum, which are obscured from boundary observables due to the lack of diffeomorphism invariance. Our analysis sheds some light on evaporating black holes in the context of the islands proposal.

Cauchy slice holography: a new AdS/CFT dictionary

We investigate a new approach to holography in asymptotically AdS spacetimes, in which time rather than space is the emergent dimension. By making a sufficiently large T2-deformation of a Euclidean CFT, we define a holographic theory that lives on Cauchy slices of the Lorentzian bulk. (More generally, for an arbitrary Hamiltonian constraint equation that closes, we show how to obtain it by an irrelevant deformation from a CFT with suitable anomalies.) The partition function of this theory defines a natural map between the bulk canonical quantum gravity theory Hilbert space, and the Hilbert space of the usual (undeformed) boundary CFT. We argue for the equivalence of the ADM and CFT Hamiltonians. We also explain how bulk unitarity emerges naturally, even though the boundary theory is not reflection-positive. This allows us to reformulate the holographic principle in the language of Wheeler-DeWitt canonical quantum gravity.Along the way, we outline a procedure for obtaining a bulk Hilbert space from the gravitational path integral with Dirichlet boundary conditions. Following previous conjectures, we postulate that this finite-cutoff gravitational path integral agrees with the T2-deformed theory living on an arbitrary boundary manifold — at least near the semiclassical regime. However, the T2-deformed theory may be easier to UV complete, in which case it would be natural to take it as the definition of nonperturbative quantum gravity.

Truncated generalized coherent states

A generalization of the canonical coherent states of a quantum harmonic oscillator has been performed by requiring the conditions of normalizability, continuity in the label, and resolution of the identity operator with a positive weight function. Relying on this approach, in the present scenario, coherent states are generalized over the canonical or finite dimensional Fock space of the harmonic oscillator. A class of generalized coherent states is determined such that the corresponding distributions of the number of excitations depart from the Poisson statistics according to combinations of stretched exponential decays, power laws, and logarithmic forms. The analysis of the Mandel parameter shows that the generalized coherent states exhibit (non-classical) sub-Poissonian or super-Poissonian statistics of the number of excitations, based on the realization of determined constraints. Mittag-Leffler and Wright generalized coherent states are analyzed as particular cases.

Characterizing Uncertainty and Enhancing Utility in Remotely Sensed Land Cover Using Error Matrices Localized in Canonical Correspondence Analysis Ordination Space

In response to uncertainty in remotely sensed land cover products, there is continuing research on accuracy assessment and analysis. Given reference sample data, accuracy indicators are commonly estimated based on error matrices, from which areal extents of different cover types are also estimated. There are merits to explore the ways utilities of land cover products may be further enhanced beyond map face values and conventional area estimation. This paper presents an integrative method (CCAErrMat) for uncertainty characterization and utility enhancement. This works through reference-map cover type co-occurrence analyses based on error matrices localized in canonical correspondence analysis (CCA) ordination space rather than in geographic space to overcome the sparsity of reference sample data. The aforementioned co-occurrence analyses facilitate quantification of accuracy indicators, identification of correctly classified and perfectly misclassified pixels, and prediction of reference class probabilities, all at individual pixels. Moreover, these predicted reference class probabilities are used as auxiliary variables to formulate model-assisted area estimation, further enhancing map utilities. Extensions to CCAErrMat are also investigated as a way to bypass the pre-computing of map class occurrence pattern indices as candidate explanatory variables for CCAErrMat, leading to two variant methods: CCACCAErrMat and CNNCCAErrMat. A case study based in Wuhan municipality, central China was undertaken to compare the proposed method against alternative methods, including CCA-separate and CNN-separate. The advantages of CCAErrMat and CCACCAErrMat were confirmed. The proposed method is recommendable for characterizing uncertainty and enhancing utilities in land cover maps by analyzing locally constrained error matrices. The method is also cost-effective in terms of reference sample data, as requirements for them are similar to those for conventional accuracy assessments.

Polyharmonic Green functions and nonlocal Bondi-Metzner-Sachs transformations of a free scalar field

We express the nonlocal Bondi-Metzner-Sachs (BMS) charges of a free massless Klein-Gordon scalar field in $2+1$ in terms of the Green functions of the polyharmonic operators. Using the properties of these Green functions, we are able to discuss the asymptotic behavior of the fields that ensures the existence of the charges and prove that one obtains a realization of the $2+1$ BMS algebra in canonical phase space. We also discuss the transformations in configuration space and show that in this case the algebra closes only up to skew-symmetric combinations of the equations of motion. The formulation of the charges in terms of Green functions opens the way to the generalization of the formalism to other dimensions and systems.

Toward 3D Face Reconstruction in Perspective Projection: Estimating 6DoF Face Pose From Monocular Image.

In 3D face reconstruction, orthogonal projection has been widely employed to substitute perspective projection to simplify the fitting process. This approximation performs well when the distance between camera and face is far enough. However, in some scenarios that the face is very close to camera or moving along the camera axis, the methods suffer from the inaccurate reconstruction and unstable temporal fitting due to the distortion under the perspective projection. In this paper, we aim to address the problem of single-image 3D face reconstruction under perspective projection. Specifically, a deep neural network, Perspective Network (PerspNet), is proposed to simultaneously reconstruct 3D face shape in canonical space and learn the correspondence between 2D pixels and 3D points, by which the 6DoF (6 Degrees of Freedom) face pose can be estimated to represent perspective projection. Besides, we contribute a large ARKitFace dataset to enable the training and evaluation of 3D face reconstruction solutions under the scenarios of perspective projection, which has 902,724 2D facial images with ground-truth 3D face mesh and annotated 6DoF pose parameters. Experimental results show that our approach outperforms current state-of-the-art methods by a significant margin. The code and data are available at https://github.com/cbsropenproject/6dof_face.

ISP Distillation

Nowadays, many of the images captured are ‘observed’ by machines only and not by humans, e.g., in autonomous systems. High-level machine vision models, such as object recognition or semantic segmentation, assume images are transformed into some canonical image space by the camera Image Signal Processor (ISP). However, the camera ISP is optimized for producing visually pleasing images for human observers and not for machines. Therefore, one may spare the ISP compute time and apply vision models directly to RAW images. Yet, it has been shown that training such models directly on RAW images results in a performance drop. To mitigate this drop, we use a RAW and RGB image pairs dataset, which can be easily acquired with no human labeling. We then train a model that is applied directly to the RAW data by using knowledge distillation such that the model predictions for RAW images will be aligned with the predictions of an off-the-shelf pre-trained model for processed RGB images. Our experiments show that our performance on RAW images for object classification and semantic segmentation is significantly better than models trained on labeled RAW images. It also reasonably matches the predictions of a pre-trained model on processed RGB images, while saving the ISP compute overhead.

Moduli space with a boundary

We find that for various solitonic processes the corresponding canonical moduli space can have a boundary which is accessible in a finite time evolution. We show that such a boundary is not a failure of the moduli space approach but has a physical meaning. In our example, it corresponds to the complete annihilation of a colliding kink and antikink after a finite time. We further find that, close to the boundary, the solutions have an approximate self-similar form.

Concentrate on Hardware Imperfection via Aligning Reconstructed States

Specific emitter identification (SEI) is to authenticate communication devices via analyzing unintentional modulation (UIM) from hardware imperfection. A great challenge existing in current methods is the interference of intentional modulation (IM) and channel noise. To address this issue, this letter firstly analyzes how IM, UIM, and additive noise (AN) affect reconstructed states, which describes the dynamic characteristics of communication devices. The findings indicate that IM and UIM bring a scaling transform, while AN corresponds to a translation transform. Based on theoretical analysis, a novel SEI method concentrating on UIM is proposed. An advanced PointNet architecture is designed with stacked spatial transformer networks (STN), namely PointNet-Alignment Point by Point (PointNet-APBP). PointNet-APBP aligns reconstructed states into a canonical space and extracts spatial interaction as fingerprint features. Simulated and real-world experiments validate that the proposed method can mitigate the effects of IM as well as AN, concentrate feature extractors on UIM, and in turn achieve higher accuracy and stronger robustness.

Human Performance Modeling and Rendering via Neural Animated Mesh

We have recently seen tremendous progress in the neural advances for photo-real human modeling and rendering. However, it's still challenging to integrate them into an existing mesh-based pipeline for downstream applications. In this paper, we present a comprehensive neural approach for high-quality reconstruction, compression, and rendering of human performances from dense multi-view videos. Our core intuition is to bridge the traditional animated mesh workflow with a new class of highly efficient neural techniques. We first introduce a neural surface reconstructor for high-quality surface generation in minutes. It marries the implicit volumetric rendering of the truncated signed distance field (TSDF) with multi-resolution hash encoding. We further propose a hybrid neural tracker to generate animated meshes, which combines explicit non-rigid tracking with implicit dynamic deformation in a self-supervised framework. The former provides the coarse warping back into the canonical space, while the latter implicit one further predicts the displacements using the 4D hash encoding as in our reconstructor. Then, we discuss the rendering schemes using the obtained animated meshes, ranging from dynamic texturing to lumigraph rendering under various bandwidth settings. To strike an intricate balance between quality and bandwidth, we propose a hierarchical solution by first rendering 6 virtual views covering the performer and then conducting occlusion-aware neural texture blending. We demonstrate the efficacy of our approach in a variety of mesh-based applications and photo-realistic free-view experiences on various platforms, i.e., inserting virtual human performances into real environments through mobile AR or immersively watching talent shows with VR headsets.