Motion Clips Research Articles

A Novel Method for Extracting DBH and Crown Base Height in Forests Using Small Motion Clips

The diameter at breast height (DBH) and crown base height (CBH) are important indicators in forest surveys. To enhance the accuracy and convenience of DBH and CBH extraction for standing trees, a method based on understory small motion clips (a series of images captured with slight viewpoint changes) has been proposed. Histogram equalization and quadtree uniformization algorithms are employed to extract image features, improving the consistency of feature extraction. Additionally, the accuracy of depth map construction and point cloud reconstruction is improved by minimizing the variance cost function. Six 20 m × 20 m square sample plots were selected to verify the effectiveness of the method. Depth maps and point clouds of the sample plots were reconstructed from small motion clips, and the DBH and CBH of standing trees were extracted using a pinhole imaging model. The results indicated that the root mean square error (RMSE) for DBH extraction ranged from 0.60 cm to 1.18 cm, with relative errors ranging from 1.81% to 5.42%. Similarly, the RMSE for CBH extraction ranged from 0.08 m to 0.21 m, with relative errors ranging from 1.97% to 5.58%. These results meet the accuracy standards required for forest surveys. The proposed method enhances the efficiency of extracting tree structural parameters in close-range photogrammetry (CRP) for forestry. A rapid and accurate method for DBH and CBH extraction is provided by this method, laying the foundation for subsequent forest resource management and monitoring.

Audio2Gestures: Generating Diverse Gestures From Audio.

People may perform diverse gestures affected by various mental and physical factors when speaking the same sentences. This inherent one-to-many relationship makes co-speech gesture generation from audio particularly challenging. Conventional CNNs/RNNs assume one-to-one mapping, and thus tend to predict the average of all possible target motions, easily resulting in plain/boring motions during inference. So we propose to explicitly model the one-to-many audio-to-motion mapping by splitting the cross-modal latent code into shared code and motion-specific code. The shared code is expected to be responsible for the motion component that is more correlated to the audio while the motion-specific code is expected to capture diverse motion information that is more independent of the audio. However, splitting the latent code into two parts poses extra training difficulties. Several crucial training losses/strategies, including relaxed motion loss, bicycle constraint, and diversity loss, are designed to better train the VAE. Experiments on both 3D and 2D motion datasets verify that our method generates more realistic and diverse motions than previous state-of-the-art methods, quantitatively and qualitatively. Besides, our formulation is compatible with discrete cosine transformation (DCT) modeling and other popular backbones (i.e., RNN, Transformer). As for motion losses and quantitative motion evaluation, we find structured losses/metrics (e.g. STFT) that consider temporal and/or spatial context complement the most commonly used point-wise losses (e.g. PCK), resulting in better motion dynamics and more nuanced motion details. Finally, we demonstrate that our method can be readily used to generate motion sequences with user-specified motion clips on the timeline.

Keyframe control for customizable choreography with style maintenance

Intelligent choreography has been a popular research topic in recent years. However, the whole music-based dance generation process is uncontrollable and non-intervenable to users. To bridge this gap, we propose CtrlDanceFr, a controllable dance generation framework that allows precise interactive adjustments of generated dance sequence through keyframe modification with style maintenance. CtrlDanceFr utilizes an innovative MoDanceRev module to modify the short-term clips according to user-input keyframes. Unlike traditional motion completion tasks, which only consider motion, our MoDanceRev also thinks about music and style, aiming to make the generated motion clip rhythmic and bring in the original sequence’s dance style. This approach significantly enhances the customization of long-term dance sequences from existing music-based dance generation methods. Furthermore, to verify the style maintenance of our framework, we design a dance classifier to evaluate the style of the generated sequences. Our rigorous experimental results, both qualitatively and quantitatively, substantiate the effectiveness and robustness of CtrlDanceFr in generating authentic and smooth dance sequences.

Neural Categorical Priors for Physics-Based Character Control

Recent advances in learning reusable motion priors have demonstrated their effectiveness in generating naturalistic behaviors. In this paper, we propose a new learning framework in this paradigm for controlling physics-based characters with improved motion quality and diversity over existing methods. The proposed method uses reinforcement learning (RL) to initially track and imitate life-like movements from unstructured motion clips using the discrete information bottleneck, as adopted in the Vector Quantized Variational AutoEncoder (VQ-VAE). This structure compresses the most relevant information from the motion clips into a compact yet informative latent space, i.e., a discrete space over vector quantized codes. By sampling codes in the space from a trained categorical prior distribution, high-quality life-like behaviors can be generated, similar to the usage of VQ-VAE in computer vision. Although this prior distribution can be trained with the supervision of the encoder's output, it follows the original motion clip distribution in the dataset and could lead to imbalanced behaviors in our setting. To address the issue, we further propose a technique named prior shifting to adjust the prior distribution using curiosity-driven RL. The outcome distribution is demonstrated to offer sufficient behavioral diversity and significantly facilitates upper-level policy learning for downstream tasks. We conduct comprehensive experiments using humanoid characters on two challenging downstream tasks, sword-shield striking and two-player boxing game. Our results demonstrate that the proposed framework is capable of controlling the character to perform considerably high-quality movements in terms of behavioral strategies, diversity, and realism. Videos, codes, and data are available at https://tencent-roboticsx.github.io/NCP/.

Multi-timescale boosting for efficient and improved event camera face pose alignment

The success of event camera (EC) vision in certain types of applications has been steadily shown thanks to energy-efficient sparse sensing, high dynamic range, and extremely high temporal resolution. However, the utilization of ECs for facial processing tasks has remained rather limited. To enable high energy efficiency for large face pose alignment, which is a crucial facial pre-processing stage, we aim at leveraging EC by effective adaptation of the processing rate proportional to facial movement intensity. For this purpose, we propose a novel alternative to the commonly employed constant time frame and event count frame strategies which combines their advantages and provides the benefits of supervised learning. This is realized by a multi-timescale boosting framework that can generate highly sparse pose-events at a variable rate via detection-based online timescale selection. Although detectors of multiple scales with boosted sensitivities operate as a cascade, our method provides minimal delay essential for real-time applications. Comprehensive evaluations show that the proposed multi-timescale processing substantially improves the performance–efficiency trade-off over single-timescale frames and markedly over event count frames. Mega-floating-point-operations-per-second ranges from 2.5 at the moderate motion clips to 6.5 at the intense motion clips, with negligible computation in the absence of activity. Also, alignment errors are considerably reduced by online selection of small timescales at fast head motion and of bigger timescales at slower motion or local activity of lips and eyes. Being orthogonal and complementary to spatial domain techniques, the proposed approach can also be conveniently integrated with future advances for further performance/efficiency improvements or for alignment extensions.

Hierarchical Planning and Control for Box Loco-Manipulation

Humans perform everyday tasks using a combination of locomotion and manipulation skills. Building a system that can handle both skills is essential to creating virtual humans. We present a physically-simulated human capable of solving box rearrangement tasks, which requires a combination of both skills. We propose a hierarchical control architecture, where each level solves the task at a different level of abstraction, and the result is a physics-based simulated virtual human capable of rearranging boxes in a cluttered environment. The control architecture integrates a planner, diffusion models, and physics-based motion imitation of sparse motion clips using deep reinforcement learning. Boxes can vary in size, weight, shape, and placement height. Code and trained control policies are provided.

GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents

The automatic generation of stylized co-speech gestures has recently received increasing attention. Previous systems typically allow style control via predefined text labels or example motion clips, which are often not flexible enough to convey user intent accurately. In this work, we present GestureDiffuCLIP, a neural network framework for synthesizing realistic, stylized co-speech gestures with flexible style control. We leverage the power of the large-scale Contrastive-Language-Image-Pre-training (CLIP) model and present a novel CLIP-guided mechanism that extracts efficient style representations from multiple input modalities, such as a piece of text, an example motion clip, or a video. Our system learns a latent diffusion model to generate high-quality gestures and infuses the CLIP representations of style into the generator via an adaptive instance normalization (AdaIN) layer. We further devise a gesture-transcript alignment mechanism that ensures a semantically correct gesture generation based on contrastive learning. Our system can also be extended to allow fine-grained style control of individual body parts. We demonstrate an extensive set of examples showing the flexibility and generalizability of our model to a variety of style descriptions. In a user study, we show that our system outperforms the state-of-the-art approaches regarding human likeness, appropriateness, and style correctness.

Multimodal dance style transfer

This paper first presents CycleDance, a novel dance style transfer system that transforms an existing motion clip in one dance style into a motion clip in another dance style while attempting to preserve the motion context of the dance. CycleDance extends existing CycleGAN architectures with multimodal transformer encoders to account for the music context. We adopt a sequence length-based curriculum learning strategy to stabilize training. Our approach captures rich and long-term intra-relations between motion frames, which is a common challenge in motion transfer and synthesis work. Building upon CycleDance, we further propose StarDance, which enables many-to-many mappings across different styles using a single generator network. Additionally, we introduce new metrics for gauging transfer strength and content preservation in the context of dance movements. To evaluate the performance of our approach, we perform an extensive ablation study and a human study with 30 participants, each with 5 or more years of dance experience. Our experimental results show that our approach can generate realistic movements with the target style, outperforming the baseline CycleGAN and its variants on naturalness, transfer strength, and content preservation. Our proposed approach has potential applications in choreography, gaming, animation, and tool development for artistic and scientific innovations in the field of dance.

ZeroEGGS: Zero‐shot Example‐based Gesture Generation from Speech

AbstractWe present ZeroEGGS, a neural network framework for speech‐driven gesture generation with zero‐shot style control by example. This means style can be controlled via only a short example motion clip, even for motion styles unseen during training. Our model uses a Variational framework to learn a style embedding, making it easy to modify style through latent space manipulation or blending and scaling of style embeddings. The probabilistic nature of our framework further enables the generation of a variety of outputs given the input, addressing the stochastic nature of gesture motion. In a series of experiments, we first demonstrate the flexibility and generalizability of our model to new speakers and styles. In a user study, we then show that our model outperforms previous state‐of‐the‐art techniques in naturalness of motion, appropriateness for speech, and style portrayal. Finally, we release a high‐quality dataset of full‐body gesture motion including fingers, with speech, spanning across 19 different styles. Our code and data are publicly available at https://github.com/ubisoft/ubisoft‐laforge‐ZeroEGGS.

Synthesizing Get‐Up Motions for Physics‐based Characters

AbstractWe propose a method for synthesizing get‐up motions for physics‐based humanoid characters. Beginning from a supine or prone state, our objective is not to imitate individual motion clips, but to produce motions that match input curves describing the style of get‐up motion. Our framework uses deep reinforcement learning to learn control policies for the physics‐based character. A latent embedding of natural human poses is computed from a motion capture database, and the embedding is furthermore conditioned on the input features. We demonstrate that our approach can synthesize motions that follow the style of user authored curves, as well as curves extracted from reference motions. In the latter case, motions of the physics‐based character resemble the original motion clips. New motions can be synthesized easily by changing only a small number of controllable parameters. We also demonstrate the success of our controllers on rough and inclined terrain.

ASE

The incredible feats of athleticism demonstrated by humans are made possible in part by a vast repertoire of general-purpose motor skills, acquired through years of practice and experience. These skills not only enable humans to perform complex tasks, but also provide powerful priors for guiding their behaviors when learning new tasks. This is in stark contrast to what is common practice in physics-based character animation, where control policies are most typically trained from scratch for each task. In this work, we present a large-scale data-driven framework for learning versatile and reusable skill embeddings for physically simulated characters. Our approach combines techniques from adversarial imitation learning and unsupervised reinforcement learning to develop skill embeddings that produce life-like behaviors, while also providing an easy to control representation for use on new downstream tasks. Our models can be trained using large datasets of unstructured motion clips, without requiring any task-specific annotation or segmentation of the motion data. By leveraging a massively parallel GPU-based simulator, we are able to train skill embeddings using over a decade of simulated experiences, enabling our model to learn a rich and versatile repertoire of skills. We show that a single pre-trained model can be effectively applied to perform a diverse set of new tasks. Our system also allows users to specify tasks through simple reward functions, and the skill embedding then enables the character to automatically synthesize complex and naturalistic strategies in order to achieve the task objectives.

Dynamic Motion Matching: Design and Implementation of a Context-Aware Animation System for Games

Despite modern game systems adopting motion matching to retrieve an appropriate short motion clip from a database in real-time, existing methods struggle to support complex gaming scenes due to their inability to adapt live the motion retrieval based on the context. This paper presents the design and implementation of a context-aware character animation system, synthesizing realistic animations suitable for complex game scenes from a large-scale motion database. This system, called dynamic motion matching (DyMM), enables geometry and objects aware motion synthesis by introducing a two-phase context computation: an offline subspace decomposition of motion clips for creating a set of retrieval sub-spaces tailored to specific contexts and a subspace ensemble matching to compare relevant sub-features to determine the most appropriate motion clip. We also show the system architecture and implementation details applicable to a production-grade game engine. We verified the effectiveness of our method with industry-level motion data captured by professional game artists for multiple configurations and character controllers. The results of this study show that, by finding motion clips that comply well with the scene context, one can leverage large motion capture datasets to create practical systems that generate believable and controllable animations for games.

Real-Time Style Modelling of Human Locomotion via Feature-Wise Transformations and Local Motion Phases

Controlling the manner in which a character moves in a real-time animation system is a challenging task with useful applications. Existing style transfer systems require access to a reference content motion clip, however, in real-time systems the future motion content is unknown and liable to change with user input. In this work we present a style modelling system that uses an animation synthesis network to model motion content based on local motion phases. An additional style modulation network uses feature-wise transformations to modulate style in real-time. To evaluate our method, we create and release a new style modelling dataset, 100STYLE, containing over 4 million frames of stylised locomotion data in 100 different styles that present a number of challenges for existing systems. To model these styles, we extend the local phase calculation with a contact-free formulation. In comparison to other methods for real-time style modelling, we show our system is more robust and efficient in its style representation while improving motion quality.

The modelling and implementation of the virtual 3D animation scene based on the geometric centre-of-mass algorithm

Abstract In the process of modernisation, the essence of virtual 3D animation scene modelling is to enhance the rendering effect of animation, so designers need to understand and skilfully use virtual reality technology. During the scene modelling, the system perception module will obtain the perception information on the basis of understanding the behaviour state of the role, and thus provide the data basis for the emotion module and the behaviour module. Among them, the emotion module is mainly used to adjust the emotional changes of the role, and the behaviour module is mainly used to build the motion database. So this article after understanding the current situation on the basis of the virtual 3D animation scene modelling, use the geometric centroid algorithm for virtual 3D animation scene modelling technology to realise in-depth understanding, and the final results prove that the build system can not only clear intuitive present animation image, but can be in effective connection also, on the basis of multiple motion clips, to maintain the original movement in detail.

Write‐An‐Animation: High‐level Text‐based Animation Editing with Character‐Scene Interaction

Abstract3D animation production for storytelling requires essential manual processes of virtual scene composition, character creation, and motion editing, etc. Although professional artists can favorably create 3D animations using software, it remains a complex and challenging task for novice users to handle and learn such tools for content creation. In this paper, we present Write‐An‐Animation, a 3D animation system that allows novice users to create, edit, preview, and render animations, all through text editing. Based on the input texts describing virtual scenes and human motions in natural languages, our system first parses the texts as semantic scene graphs, then retrieves 3D object models for virtual scene composition and motion clips for character animation. Character motion is synthesized with the combination of generative locomotions using neural state machine as well as template action motions retrieved from the dataset. Moreover, to make the virtual scene layout compatible with character motion, we propose an iterative scene layout and character motion optimization algorithm that jointly considers character‐object collision and interaction. We demonstrate the effectiveness of our system with customized texts and public film scripts. Experimental results indicate that our system can generate satisfactory animations from texts.

Diverse Motion Stylization for Multiple Style Domains via Spatial-Temporal Graph-Based Generative Model

This paper presents a novel deep learning-based framework for translating a motion into various styles within multiple domains. Our framework is a single set of generative adversarial networks that learns stylistic features from a collection of unpaired motion clips with style labels to support mapping between multiple style domains. We construct a spatio-temporal graph to model a motion sequence and employ the spatial-temporal graph convolution networks (ST-GCN) to extract stylistic properties along spatial and temporal dimensions. Through spatial-temporal modeling, our framework shows improved style translation results between significantly different actions and on a long motion sequence containing multiple actions. In addition, we first develop a mapping network for motion stylization that maps a random noise to style, which allows for generating diverse stylization results without using reference motions. Through various experiments, we demonstrate the ability of our method to generate improved results in terms of visual quality, stylistic diversity, and content preservation.

Learning a family of motor skills from a single motion clip

Learning a family of motor skills from a single motion clip

AMP

Synthesizing graceful and life-like behaviors for physically simulated characters has been a fundamental challenge in computer animation. Data-driven methods that leverage motion tracking are a prominent class of techniques for producing high fidelity motions for a wide range of behaviors. However, the effectiveness of these tracking-based methods often hinges on carefully designed objective functions, and when applied to large and diverse motion datasets, these methods require significant additional machinery to select the appropriate motion for the character to track in a given scenario. In this work, we propose to obviate the need to manually design imitation objectives and mechanisms for motion selection by utilizing a fully automated approach based on adversarial imitation learning. High-level task objectives that the character should perform can be specified by relatively simple reward functions, while the low-level style of the character's behaviors can be specified by a dataset of unstructured motion clips, without any explicit clip selection or sequencing. For example, a character traversing an obstacle course might utilize a task-reward that only considers forward progress, while the dataset contains clips of relevant behaviors such as running, jumping, and rolling. These motion clips are used to train an adversarial motion prior, which specifies style-rewards for training the character through reinforcement learning (RL). The adversarial RL procedure automatically selects which motion to perform, dynamically interpolating and generalizing from the dataset. Our system produces high-quality motions that are comparable to those achieved by state-of-the-art tracking-based techniques, while also being able to easily accommodate large datasets of unstructured motion clips. Composition of disparate skills emerges automatically from the motion prior, without requiring a high-level motion planner or other task-specific annotations of the motion clips. We demonstrate the effectiveness of our framework on a diverse cast of complex simulated characters and a challenging suite of motor control tasks.

Learning a family of motor skills from a single motion clip

We present a new algorithm that learns a parameterized family of motor skills from a single motion clip. The motor skills are represented by a deep policy network, which produces a stream of motions in physics simulation in response to user input and environment interaction by navigating continuous action space. Three novel technical components play an important role in the success of our algorithm. First, it explicitly constructs motion parameterization that maps action parameters to their corresponding motions. Simultaneous learning of motion parameterization and motor skills significantly improves the performance and visual quality of learned motor skills. Second, continuous-time reinforcement learning is adopted to explore temporal variations as well as spatial variations in motion parameterization. Lastly, we present a new automatic curriculum generation method that explores continuous action space more efficiently. We demonstrate the flexibility and versatility of our algorithm with highly dynamic motor skills that can be parameterized by task goals, body proportions, physical measurements, and environmental conditions.

Robust 3D reconstruction from uncalibrated small motion clips

Small motion can be induced from burst video clips captured by a handheld camera when the shutter button is pressed. Although uncalibrated burst video clip conveys valuable parallax information, it generally has small baseline between frames, making it difficult to reconstruct 3D scenes. Existing methods usually employ a simplified camera parameterization process with keypoint-based structure from small motion (SFSM), followed by a tailored dense reconstruction. However, such SFSM methods are sensitive to insufficient or unreliable keypoint features, and the subsequent dense reconstruction may fail to recover the detailed surface. In this paper, we propose a robust 3D reconstruction pipeline by leveraging both keypoint and line segment features from video clips to alleviate the uncertainty induced by small baseline. A joint feature-based structure from small motion method is first presented to improve the robustness of the self-calibration with line segment constraints, and then, a noise-aware PatchMatch stereo module is proposed to improve the accuracy of the dense reconstruction. Finally, a confidence weighted fusion process is utilized to further suppress depth noise and mitigate erroneous depth. The proposed method can reduce the failure cases of self-calibration when the keypoints are insufficient, while recovering the detailed 3D surfaces. In comparison with state of the arts, our method achieves more robust and accurate 3D reconstruction results for a variety of challenging scenes.