Motion Synthesis Research Articles

Robust Diffusion‐based Motion In‐betweening

AbstractThe emergence of learning‐based motion in‐betweening techniques offers animators a more efficient way to animate characters. However, existing non‐generative methods either struggle to support long transition generation or produce results that lack diversity. Meanwhile, diffusion models have shown promising results in synthesizing diverse and high‐quality motions driven by text and keyframes. However, in these methods, keyframes often serve as a guide rather than a strict constraint and can sometimes be ignored when keyframes are sparse. To address these issues, we propose a lightweight yet effective diffusion‐based motion in‐betweening framework that generates animations conforming to keyframe constraints. We incorporate keyframe constraints into the training phase to enhance robustness in handling various constraint densities. Moreover, we employ relative positional encoding to improve the model's generalization on long range in‐betweening tasks. This approach enables the model to learn from short animations while generating realistic in‐betweening motions spanning thousands of frames. We conduct extensive experiments to validate our framework using the newly proposed metrics K‐FID, K‐Diversity, and K‐Error, designed to evaluate generative in‐betweening methods. Results demonstrate that our method outperforms existing diffusion‐based methods across various lengths and keyframe densities. We also show that our method can be applied to text‐driven motion synthesis, offering fine‐grained control over the generated results.

Nonlinear seismic response analysis of long-span railway cable-stayed bridges crossing strike-slip faults

Active faults along railways in the mountainous regions of western China pose significant challenges to bridge safety. To ensure the safe operation of long-span railway bridges under complex geological conditions, this study investigates the synthesis of artificial ground motions for bridges crossing strike-slip faults and analyzes their nonlinear seismic response. First, we develop a theoretical method for simulating high- and low-frequency seismic motions using a finite fault and an equivalent velocity pulse model. Next, using a specific long-span railway cable-stayed bridge as a case study, we construct a nonlinear finite element model with OpenSees software. Finally, we assess the seismic response of key bridge components considering various crossing angles, seismic amplitudes, fault rupture directivity, and fling-step effects. The results show that the crossing angle significantly influences the seismic response, with longitudinal and transverse responses exhibiting opposite patterns. Additionally, the scaling factor of seismic motion significantly affects bridge response. For bridges crossing strike-slip faults, the longitudinal response exhibits a sudden increase in displacement due to instantaneous velocity pulses, while the transverse response shows notable residual displacement influenced by the fling-step effect. However, the critical section curvatures of bridge towers and piers remain within the elastic range across all crossing angles, indicating that controlling large displacement deformations is crucial for the seismic design of bridges crossing strike-slip faults.

Regional study of site effects on the high-frequency spectral-decay parameter

The high-frequency spectral-decay parameter κ has been used in the field of engineering seismology for several decades. The zero-distance κ value (κ0) is a site parameter in ground motion models and strong motion synthesis. Using surface-borehole pairs of strong motion records from the strong-motion seismograph network KiK-net, we explored the effect of site soil on κ0, specifically the contribution from ground motion directionality. In the target area (138°E−143°E, 36°N–40°N), and records from earthquakes at 50 pairs of stations were collected. All horizontal orthogonal records were rotated from 0° to 180° with an increment of 10° to capture the difference between the average κ0 on two horizontal components and the median value of the rotated κ0 at surface and borehole stations. The same process was applied to Δκ0, the difference between surface station κ0 and borehole station κ0, to measure the contribution by soil column. The nonlinear behavior of soils led to large shear strains, reduced stiffness, and decreased resonance frequency. Thus, the effect of soil nonlinearity on κ is investigated. In the same area, strong ground motion records with peak ground acceleration >100 cm/s2 were selected to identify the soil nonlinearity by the moving time window deconvolution method, and the temporal variation of time-average shear wave velocity and κ at the 50 surface stations were assessed.

Pose‐to‐Motion: Cross‐Domain Motion Retargeting with Pose Prior

AbstractCreating plausible motions for a diverse range of characters is a long‐standing goal in computer graphics. Current learning‐based motion synthesis methods rely on large‐scale motion datasets, which are often difficult if not impossible to acquire. On the other hand, pose data is more accessible, since static posed characters are easier to create and can even be extracted from images using recent advancements in computer vision. In this paper, we tap into this alternative data source and introduce a neural motion synthesis approach through retargeting, which generates plausible motion of various characters that only have pose data by transferring motion from one single existing motion capture dataset of another drastically different characters. Our experiments show that our method effectively combines the motion features of the source character with the pose features of the target character, and performs robustly with small or noisy pose data sets, ranging from a few artist‐created poses to noisy poses estimated directly from images. Additionally, a conducted user study indicated that a majority of participants found our retargeted motion to be more enjoyable to watch, more lifelike in appearance, and exhibiting fewer artifacts. Our code and dataset can be accessed here.

Long‐term Motion In‐betweening via Keyframe Prediction

AbstractMotion in‐betweening has emerged as a promising approach to enhance the efficiency of motion creation due to its flexibility and time performance. However, previous in‐betweening methods are limited to generating short transitions due to growing pose ambiguity when the number of missing frames increases. This length‐related constraint makes the optimization hard and it further causes another constraint on the target pose, limiting the degrees of freedom for artists to use. In this paper, we introduce a keyframe‐driven approach that effectively solves the pose ambiguity problem, allowing robust in‐betweening performance on various lengths of missing frames. To incorporate keyframe‐driven motion synthesis, we introduce a keyframe score that measures the likelihood of a frame being used as a keyframe as well as an adaptive keyframe selection method that maintains appropriate temporal distances between resulting keyframes. Additionally, we employ phase manifolds to further resolve the pose ambiguity and incorporate trajectory conditions to guide the approximate movement of the character. Comprehensive evaluations, encompassing both quantitative and qualitative analyses, were conducted to compare our method with state‐of‐the‐art in‐betweening approaches across various transition lengths. The code for the paper is available at https://github.com/seokhyeonhong/long-mib

IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition

One of the primary challenges in the field of human activity recognition (HAR) is the lack of large labeled datasets. This hinders the development of robust and generalizable models. Recently, cross modality transfer approaches have been explored that can alleviate the problem of data scarcity. These approaches convert existing datasets from a source modality, such as video, to a target modality, such as inertial measurement units (IMUs). With the emergence of generative AI models such as large language models (LLMs) and text-driven motion synthesis models, language has become a promising source data modality as well - as shown in proof of concepts such as IMUGPT. In this work, we conduct a large-scale evaluation of language-based cross modality transfer to determine their effectiveness for HAR. Based on this study, we introduce two new extensions for IMUGPT that enhance its use for practical HAR application scenarios: a motion filter capable of filtering out irrelevant motion sequences to ensure the relevance of the generated virtual IMU data, and a set of metrics that measure the diversity of the generated data facilitating the determination of when to stop generating virtual IMU data for both effective and efficient processing. We demonstrate that our diversity metrics can reduce the effort needed for the generation of virtual IMU data by at least 50%, which opens up IMUGPT for practical use cases beyond a mere proof of concept.

A Two-Part Transformer Network for Controllable Motion Synthesis.

Although part-based motion synthesis networks have been investigated to reduce the complexity of modeling heterogeneous human motions, their computational cost remains prohibitive in interactive applications. To this end, we propose a novel two-part transformer network that aims to achieve high-quality, controllable motion synthesis results in real-time. Our network separates the skeleton into the upper and lower body parts, reducing the expensive cross-part fusion operations, and models the motions of each part separately through two streams of auto-regressive modules formed by multi-head attention layers. However, such a design might not sufficiently capture the correlations between the parts. We thus intentionally let the two parts share the features of the root joint and design a consistency loss to penalize the difference in the estimated root features and motions by these two auto-regressive modules, significantly improving the quality of synthesized motions. After training on our motion dataset, our network can synthesize a wide range of heterogeneous motions, like cartwheels and twists. Experimental and user study results demonstrate that our network is superior to state-of-the-art human motion synthesis networks in the quality of generated motions.

Machine Learning Approaches for 3D Motion Synthesis and Musculoskeletal Dynamics Estimation: A Survey.

The inference of 3D motion and dynamics of the human musculoskeletal system has traditionally been solved using physics-based methods that exploit physical parameters to provide realistic simulations. Yet, such methods suffer from computational complexity and reduced stability, hindering their use in computer graphics applications that require real-time performance. With the recent explosion of data capture (mocap, video) machine learning (ML) has started to become popular as it is able to create surrogate models harnessing the huge amount of data stemming from various sources, minimizing computational time (instead of resource usage), and most importantly, approximate real-time solutions. The main purpose of this paper is to provide a review and classification of the most recent works regarding motion prediction, motion synthesis as well as musculoskeletal dynamics estimation problems using ML techniques, in order to offer sufficient insight into the state-of-the-art and draw new research directions. While the study of motion may appear distinct to musculoskeletal dynamics, these application domains provide jointly the link for more natural computer graphics character animation, since ML-based musculoskeletal dynamics estimation enables modeling of more long-term, temporally evolving, ergonomic effects, while offering automated and fast solutions. Overall, our review offers an in-depth presentation and classification of ML applications in human motion analysis, unlike previous survey articles focusing on specific aspects of motion prediction.

Interactive Character Control with Auto-Regressive Motion Diffusion Models

Real-time character control is an essential component for interactive experiences, with a broad range of applications, including physics simulations, video games, and virtual reality. The success of diffusion models for image synthesis has led to the use of these models for motion synthesis. However, the majority of these motion diffusion models are primarily designed for offline applications, where space-time models are used to synthesize an entire sequence of frames simultaneously with a pre-specified length. To enable real-time motion synthesis with diffusion model that allows time-varying controls, we propose A-MDM (Auto-regressive Motion Diffusion Model). Our conditional diffusion model takes an initial pose as input, and auto-regressively generates successive motion frames conditioned on the previous frame. Despite its streamlined network architecture, which uses simple MLPs, our framework is capable of generating diverse, long-horizon, and high-fidelity motion sequences. Furthermore, we introduce a suite of techniques for incorporating interactive controls into A-MDM, such as task-oriented sampling, in-painting, and hierarchical reinforcement learning (See Figure 1). These techniques enable a pre-trained A-MDM to be efficiently adapted for a variety of new downstream tasks. We conduct a comprehensive suite of experiments to demonstrate the effectiveness of A-MDM, and compare its performance against state-of-the-art auto-regressive methods.

Machine Learning-Based Human Movements Mimicking System for Animation and Virtual Reality

A trend, which can be noticed over the last several years, is the increasing interest to the machine learning methods in the animation and virtual reality (VR) industries to process the human motion data. The present abstract studies the possible contribution of human like machine learning simulations on human motions, which aim to improve animation and motion synthesis personalization. In this research, we propose a novel framework that utilizes camera-captured human motion data. This technique, which takes advantage of comes of motion data to simulate these animated characters movements in real time, will allow animators to create more creature-like animated figures with subtle actions as seen in real life. Additionally, in virtual reality, it links users’ movements with their avatars so the whole interaction is more engaging and realistic.

Dimensional synthesis of motion generation of a planar four-bar mechanism

A motion generation method for a planar four-bar mechanism without prescribed timing is proposed in this article. A characteristic of coupler points is found: for the coupler points of a four-bar mechanism in a standard installation position rotated by the corresponding input angle clockwise around the origin of the Cartesian coordinate system xOy, the generated points lie on the feature coupler circles. Next, based on this, the design process is divided into three steps. In the first step, the relative input angles, the relative coupler angles and five geometric parameters of the desired four-bar mechanism are optimized. In the second step, the basic dimensional types and initial input angle are determined, and then, the input angles are obtained. In the third step, the real size of the desired linkage mechanism is calculated. Because the dimension of the design variables in each step is exceptionally small, the motion synthesis method can yield design solutions with high precision in a short time. Six comparison examples are presented to demonstrate the efficacy of the proposed method. In addition, the method is used to design a robot for lower limb rehabilitation for application to the human foot.

Improved methods for synthesizing near-fault ground motions based on specific response spectrum

An increasing number of civil structures are being built near or across earthquake faults, making them more susceptible to severe seismic damage due to the presence of large velocity pulses in near-fault ground motions (NFGMs). Because of limited availability of NFGMs records, synthesized NFGMs are necessary for seismic designing of structures located around near-fault areas. Based on the far-field synthesis of ground motions (GMs), various methods for synthesizing NFGMs by superimposing high-frequency and low-frequency components in the time domain have been developed. However, such techniques suffer from wider variation in the response spectrum between the synthesized NFGMs and the target spectrum. To address this issue, a new approach for synthesizing NFGMs has been proposed. This approach investigates key parameters of the response spectrum for NFGMs based on a comprehensive database. Similarly, the shortcomings in existing NFGM methods have been analyzed by evaluating a couple of criteria for the suitability of the synthesizing method. Two adjustment strategies have been developed to attain consistency in the response spectrum. The spectral characteristics can be maintained during the spectrum adjustment process to overcome limitations of existing methods. Synthesized specimen of NFGMs has been utilized to validate the effectiveness of the proposed approach.

Climbing Motion Synthesis using Reinforcement Learning

Climbing Motion Synthesis using Reinforcement Learning

MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model.

Human motion modeling is important for many modern graphics applications, which typically require professional skills. In order to remove the skill barriers for laymen, recent motion generation methods can directly generate human motions conditioned on natural languages. However, it remains challenging to achieve diverse and fine-grained motion generation with various text inputs. To address this problem, we propose MotionDiffuse, one of the first diffusion model-based text-driven motion generation frameworks, which demonstrates several desired properties over existing methods. 1) Probabilistic Mapping. Instead of a deterministic language-motion mapping, MotionDiffuse generates motions through a series of denoising steps in which variations are injected. 2) Realistic Synthesis. MotionDiffuse excels at modeling complicated data distribution and generating vivid motion sequences. 3) Multi-Level Manipulation. MotionDiffuse responds to fine-grained instructions on body parts, and arbitrary-length motion synthesis with time-varied text prompts. Our experiments show MotionDiffuse outperforms existing SoTA methods by convincing margins on text-driven motion generation and action-conditioned motion generation. A qualitative analysis further demonstrates MotionDiffuse's controllability for comprehensive motion generation.

Exploring the Synthesis of Self-Organization and Active Motion.

Proteins, genetic material, and membranes are fundamental to all known organisms, yet these components alone do not constitute life. Life emerges from the dynamic processes of self-organization, assembly, and active motion, suggesting the existence of similar artificial systems. Against this backdrop, our Perspective explores a variety of chemical phenomena illustrating how nonequilibrium self-organization and micromotors contribute to life-like behavior and functionalities. After explaining key terms, we discuss specific examples including enzymatic motion, diffusiophoretic and bubble-driven self-propulsion, pattern-forming reaction-diffusion systems, self-assembling inorganic aggregates, and hierarchically emergent phenomena. We also provide a roadmap for combining self-organization and active motion and discuss possible outcomes through biological analogs. We suggest that this research direction, deeply rooted in physical chemistry, offers opportunities for further development with broad impacts on related sciences and technologies.

The Review of Modeling and Rendering 3D Environment: A Survey

This paper explores several topics in the field of 3D Environments, including ray tracing, rendering techniques, algorithms and optimization, data processing, real-time rendering, modeling, physical simulation, and material modeling. The paper covers a large variety of applications and techniques that are widely used in computer graphics and visualization. This paper also points out that significant progress has been made in ray tracing, texture filtering, image processing, motion synthesis, geometric modeling, and physical simulation. These advances have improved the visual quality, performance, and fidelity of computer-generated images. In addition, the article mentions the trend of integrating machine learning, deep learning, and data analysis techniques into computer graphics and visualization, especially in data processing and analysis models, to improve the results of rendering and simulation. Overall, this article explores details of the current state of the computer graphics and visualization field, highlighting the various technologies and applications which shapes the field.

A Deep Learning Framework for Start-End Frame Pair-Driven Motion Synthesis.

A start-end frame pair and a motion pattern-based motion synthesis scheme can provide more control to the synthesis process and produce content-various motion sequences. However, the data preparation for the motion training is intractable, and concatenating feature spaces of the start-end frame pair and the motion pattern lacks theoretical rationality in previous works. In this article, we propose a deep learning framework that completes automatic data preparation and learns the nonlinear mapping from start-end frame pairs to motion patterns. The proposed model consists of three modules: action detection, motion extraction, and motion synthesis networks. The action detection network extends the deep subspace learning framework to a supervised version, i.e., uses the local self-expression (LSE) of the motion data to supervise feature learning and complement the classification error. A long short-term memory (LSTM)-based network is used to efficiently extract the motion patterns to address the speed deficiency reflected in the previous optimization-based method. A motion synthesis network consists of a group of LSTM-based blocks, where each of them is to learn the nonlinear relation between the start-end frame pairs and the motion patterns of a certain joint. The superior performances in action detection accuracy, motion pattern extraction efficiency, and motion synthesis quality show the effectiveness of each module in the proposed framework.

Research on the Vector Space and Its Linearity

A vector space is an abelian group over a field. From the viewpoint of group theory, the tangent space is a theoretical concept. Tangent space is similar to a vector space, particularly in terms of linearity. From a field's perspective, a subfield is a subset of the original field that shares the same properties as the original field. Subfield is logically similar to a subspace of a vector space. A vector is an element of a vector space. Weight vectors can be used to describe motion, which can be applied in pattern recognition. A linear combination of weight vectors can describe a sequence of motions. Kernel graph theory treats a graph as an element of its theoretical structure. Kernel graph theory can also be applied in pattern recognition. So, basically, this article presents a perspective on applying vector space in pattern recognition. The inherent logic of vector space can be applied to both motion synthesis and graph pattern recognition. The linearity of the vector space is the main focus of this article.

CT respiratory motion synthesis using joint supervised and adversarial learning

Objective. Four-dimensional computed tomography (4DCT) imaging consists in reconstructing a CT acquisition into multiple phases to track internal organ and tumor motion. It is commonly used in radiotherapy treatment planning to establish planning target volumes. However, 4DCT increases protocol complexity, may not align with patient breathing during treatment, and lead to higher radiation delivery. Approach. In this study, we propose a deep synthesis method to generate pseudo respiratory CT phases from static images for motion-aware treatment planning. The model produces patient-specific deformation vector fields (DVFs) by conditioning synthesis on external patient surface-based estimation, mimicking respiratory monitoring devices. A key methodological contribution is to encourage DVF realism through supervised DVF training while using an adversarial term jointly not only on the warped image but also on the magnitude of the DVF itself. This way, we avoid excessive smoothness typically obtained through deep unsupervised learning, and encourage correlations with the respiratory amplitude. Main results. Performance is evaluated using real 4DCT acquisitions with smaller tumor volumes than previously reported. Results demonstrate for the first time that the generated pseudo-respiratory CT phases can capture organ and tumor motion with similar accuracy to repeated 4DCT scans of the same patient. Mean inter-scans tumor center-of-mass distances and Dice similarity coefficients were 1.97 mm and 0.63, respectively, for real 4DCT phases and 2.35 mm and 0.71 for synthetic phases, and compares favorably to a state-of-the-art technique (RMSim). Significance. This study presents a deep image synthesis method that addresses the limitations of conventional 4DCT by generating pseudo-respiratory CT phases from static images. Although further studies are needed to assess the dosimetric impact of the proposed method, this approach has the potential to reduce radiation exposure in radiotherapy treatment planning while maintaining accurate motion representation. Our training and testing code can be found at https://github.com/cyiheng/Dynagan.

Impact of AI in the Animation Industry

Abstract: This manuscript develops into the transformative role of artificial intelligence (AI) in the field of animation. Over the years, AI has revolutionized various industries including healthcare, banking, e-commerce, agriculture, manufacturing, and others., and animation is no exception. This paper provides an overview of the applications of AI in animation, including character creation, motion synthesis, scene generation, generating facial expressions and body language for characters, planning out scene layouts and camera movements, and generating backgrounds and environments and more. It discusses the challenges and ethical considerations associated with AI-driven animation and presents insights into the future directions of AI in the animation industry