Motion Prediction Research Articles

A Motion Logic Network for Pedestrian Motion Prediction

Accurate and fast motion prediction such as pedestrian motion prediction (PMP) is crucial for safe autonomous driving. Much research effort has been devoted to studying the reactive behaviors of pedestrians, such as the interaction between pedestrians or the interaction between pedestrians and the environment. However, compared with behavioral logic, the current motion state of pedestrians has a greater influence on the future trajectory. In this work, we propose a motion logic network (MLN) to improve both the accuracy and efficiency of pedestrian motion prediction. Compared with the traditional data-driven neural networks, the concept of motion logic is directly introduced into the network so that the training of the network is not required. In particular, instead of fitting the network based on inputs and target outputs, the proposed method directly adopts motion logic to predict the future trajectory. To illustrate the performance of the proposed MLN, several experiments have been performed. For acceleration and deceleration motion in practical experiment, the average displacement error (ADE) of MLN has an improvement of as high as 6.25% than CVM’s, while the final displacement error (FDE) of MLN has an improvement of 11.1%. In terms of efficiency, MLN is a hundred times faster than LSTM. It indicates that motion logic plays an important role in the development of prediction algorithms for pedestrian motion. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This article is motivated by the effects of pedestrians’ different motion states on the future trajectory. For example, the pedestrians’ state may also change drastically in some situations such as dashing across the road or stopping in anticipation of a bicycle crossing the path. This work explores the use of pedestrians’ physical motion states to develop a network that emphasizes the logical features of pedestrian motion so as to directly estimate the future motion trajectory and trend based on the motion logic. For the first time, the concept of motion logic and the resultant training-free motion logic network (MLN) is proposed, which takes into account the accuracy and efficiency performance of the algorithm. Compared with the state-of-the-art pedestrian prediction methods, the proposed method can better predict the trajectory of pedestrians in more complex motion states. Moreover, a series of simulations and practical experiments with pseudo-constant velocity motion and acceleration/deceleration motion was taken to verify the performance of the proposed MLN.

A FRACTAL-FRACTIONAL TSUNAMI MODEL CONSIDERING NEAR-SHORE FRACTAL BOUNDARY

Every fluid problem is greatly affected by its boundary conditions, especially the near-shore seabed could produce an irrevocable harm when a tsunami wave is approaching, and a real-life mathematical model could stave off the worst effect. This paper assumes that the unsmooth seabed is a fractal surface, and fractal-fractional governing equations are established according to physical laws in the fractal space. The geometrical potential theory is used to explain the force produced by the wave surface, and Kong-He friction law is applied to further figuring out the local and memory properties of the friction along the fractal boundary. This paper aims at studying tsunami waves in a fractal space, rendering a reliable mathematical model for both prediction of the tsunami motion and the coastal protection.

Feature extraction method of EEG based on wavelet packet reconstruction and deep learning model of VR motion sickness feature classification and prediction.

The surging popularity of virtual reality (VR) technology raises concerns about VR-induced motion sickness, linked to discomfort and nausea in simulated environments. Our method involves in-depth analysis of EEG data and user feedback to train a sophisticated deep learning model, utilizing an enhanced GRU network for identifying motion sickness patterns. Following comprehensive data pre-processing and feature engineering to ensure input accuracy, a deep learning model is trained using supervised and unsupervised techniques for classifying and predicting motion sickness severity. Rigorous training and validation procedures confirm the model's robustness across diverse scenarios. Research results affirm our deep learning model's 84.9% accuracy in classifying and predicting VR-induced motion sickness, surpassing existing models. This information is vital for improving the VR experience and advancing VR technology.

Dynamic Motion Transition: A Hybrid Data-driven and Model-driven Method for Human Pose Transitions.

The rapid, accurate, and robust computation of virtual human figures' "in-between" pose transitions from available and sometimes sparse inputs is of fundamental significance to 3D interactive graphics and computer animation. Various methods have been proposed to produce natural lifelike transitions of human pose automatically in recent decades. Nevertheless, conventional pure model-driven methods require heuristic knowledge (e.g., least motion guided by physics laws) and ad-hoc clues (e.g., splines with non-uniform time warp) that are difficult to obtain, learn, and infer. With the fast emergence of large-scale datasets readily available to animators in the most recent years, deep models afford a powerful alternative to tackle the aforementioned challenges. However, pure data-driven methods still suffer from the remaining challenges such as unseen data in practice and less generative power in model/domain/data transfer, and the measurement of the generative power has always been omitted in these works. In essence, data-driven methods solely rely on the qualities and quantities of training datasets. In this paper, we propose a hybrid approach built upon the seamless integration of data-driven and model-driven methods, called Dynamic Motion Transition (DMT), with the following salient modeling advantages: (1) The data augmentation capability based on the limited human locomotion data capture and the concept of force-derived directly from physical laws; (2) Force learning by which skeleton joints are driven to move, and the Conditional Temporal Transformer (CTT) being trained to learn the force change in the local range, both at the fine level; and (3) At the coarse level, the effective and flexible creation of the subsequent step motion using Dynamic Movement Primitives (DMP) until the target is reached. Our extensive experiments have confirmed that our model can outperform the state-of-the-art methods under the newly devised metric by virtue of the least action loss function. In addition, our novel method and system are of immediate benefit to many other animation tasks such as motion synthesis and control, and motion tracking and prediction in this bigdata graphics era.

Motion Detection and Prediction Using Opencv, Resnet and Tkinter

Motion Detection and Prediction Using Opencv, Resnet and Tkinter

Method for Prediction of Motion Based on Recursive Least Squares Method with Time Warp Parameter and its Application to Physical Therapy

Method for Prediction of Motion Based on Recursive Least Squares Method with Time Warp Parameter and its Application to Physical Therapy

Multi-Semantics Aggregation Network Based on the Dynamic-Attention Mechanism for 3D Human Motion Prediction

Multi-Semantics Aggregation Network Based on the Dynamic-Attention Mechanism for 3D Human Motion Prediction

Considering Cyclists' Aggressiveness and Bounded Rationality: A Self-learning Motion Prediction Model for Cyclists

Considering Cyclists' Aggressiveness and Bounded Rationality: A Self-learning Motion Prediction Model for Cyclists

Multi-Condition Latent Diffusion Network for Scene-Aware Neural Human Motion Prediction.

Inferring 3D human motion is fundamental in many applications, including understanding human activity and analyzing one's intention. While many fruitful efforts have been made to human motion prediction, most approaches focus on pose-driven prediction and inferring human motion in isolation from the contextual environment, thus leaving the body location movement in the scene behind. However, real-world human movements are goal-directed and highly influenced by the spatial layout of their surrounding scenes. In this paper, instead of planning future human motion in a "dark" room, we propose a Multi-Condition Latent Diffusion network (MCLD) that reformulates the human motion prediction task as a multi-condition joint inference problem based on the given historical 3D body motion and the current 3D scene contexts. Specifically, instead of directly modeling joint distribution over the raw motion sequences, MCLD performs a conditional diffusion process within the latent embedding space, characterizing the cross-modal mapping from the past body movement and current scene context condition embeddings to the future human motion embedding. Extensive experiments on large-scale human motion prediction datasets demonstrate that our MCLD achieves significant improvements over the state-of-the-art methods on both realistic and diverse predictions.

FuzzyTrack: User Adaptive Cervical Spine Motion Prediction with Earable Inertial Sensing

FuzzyTrack: User Adaptive Cervical Spine Motion Prediction with Earable Inertial Sensing

Differentiable Integrated Motion Prediction and Planning With Learnable Cost Function for Autonomous Driving.

Predicting the future states of surrounding traffic participants and planning a safe, smooth, and socially compliant trajectory accordingly are crucial for autonomous vehicles (AVs). There are two major issues with the current autonomous driving system: the prediction module is often separated from the planning module, and the cost function for planning is hard to specify and tune. To tackle these issues, we propose a differentiable integrated prediction and planning (DIPP) framework that can also learn the cost function from data. Specifically, our framework uses a differentiable nonlinear optimizer as the motion planner, which takes as input the predicted trajectories of surrounding agents given by the neural network and optimizes the trajectory for the AV, enabling all operations to be differentiable, including the cost function weights. The proposed framework is trained on a large-scale real-world driving dataset to imitate human driving trajectories in the entire driving scene and validated in both open-loop and closed-loop manners. The open-loop testing results reveal that the proposed method outperforms the baseline methods across a variety of metrics and delivers planning-centric prediction results, allowing the planning module to output trajectories close to those of human drivers. In closed-loop testing, the proposed method outperforms various baseline methods, showing the ability to handle complex urban driving scenarios and robustness against the distributional shift. Importantly, we find that joint training of planning and prediction modules achieves better performance than planning with a separate trained prediction module in both open-loop and closed-loop tests. Moreover, the ablation study indicates that the learnable components in the framework are essential to ensure planning stability and performance. Code and Supplementary Videos are available at https://mczhi.github.io/DIPP/.

Renewing Iterative Self-Labeling Domain Adaptation With Application to the Spine Motion Prediction

The area of transfer learning comprises supervised machine learning methods that cope with the issue when the training and testing data have different input feature spaces or distributions. In this work, we propose a novel transfer learning algorithm called Renewing Iterative Self-labeling Domain Adaptation (Re-ISDA). This work is motivated by a cervical spine motion prediction problem, with the goal of predicting the cervical spine motion of different subjects using the measurements of “exterior features”. The joint distribution of the exterior features for each subject may vary with one’s distinct BMI, sex and other characteristics; the sample size of this problem is limited due to the high experimental cost. The proposed method is well suited for transfer learning problems with limited training samples. The learning problem is formulated as a dynamic programming model and the latter is then solved by an efficient greedy algorithm. Numerical studies show that the proposed method outperforms prevailing transfer learning methods. The proposed method also achieves high prediction accuracy for the cervical spine motion problem. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper is motivated by the problem of using human body exterior features (e.g., contours, surface markers) acquired by computer vision or other measurement means to determine the underlying physics or mechanics. The goal is to build a subject-specific model to predict cervical spine motion based on surface-based measurements. Conventional supervised learning and transfer learning methods fail to give an accurate prediction due to the subject-to-subject variation and/or the small sample size of the training data. This paper proposes a novel transfer learning method called renewing iterative self-labeling domain adaptation to modify the possible error of the pseudo labels and thus make the training process more stable. The proposed method improves the accuracy by over <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$35\%$</tex-math> </inline-formula> on the traditional transfer learning methods. This method is applicable to limited-sample problems with different training and testing input distributions, and is particularly suitable for time-series data.

Decision‐Making and Path Planning for Head‐On Collision Avoidance on Curved Roads

Deviating to the left on two‐way roads can result in fatal head‐on collisions. This article presents an intelligent decision‐making and path‐planning algorithm aimed at avoiding collision with a vehicle that has deviated from the opposing lane. The path‐planning process utilizes the model predictive control (MPC) approach, employing a linear kinematic prediction model with a horizon of 2 seconds. Considering that the deviated vehicle may abruptly return to its original lane at any moment, its motion is associated with significant uncertainty. To address this challenge, the path‐planning algorithm directs the ego vehicle (EV) under specific constraints to ensure that both the left and right sides of the road are symmetrically reachable in future time steps. This enables the decision‐making algorithm to select the safer direction for evasive maneuver at the appropriate moment. The motion prediction of the threat vehicle (TV) is conducted until the potential collision time, taking into account its motion history, and is utilized in the decision‐making process. Once the maneuver direction is determined, the collision‐free path planning continues using the MPC method. To evaluate the algorithm, six simulations are conducted, modeling various distant and close encounter states of the vehicles on roads with left‐ and right‐hand curves. The simulation results indicate the flexibility and appropriate performance of the algorithm in planning safe and maneuverable paths.

Modeling and Prediction of Nonstationary Ground Motions as Time–Frequency Images

Proper modeling of nonstationary characteristics of ground motions is critical in estimating the seismic performance of structures. The importance of ground motion nonstationarity is being increasingly recognized in seismic design codes and provisions. This paper develops a new method for predicting seismic ground motions in the joint time–frequency domain based on recorded ground motions. We treat wavelet energy maps of acceleration records as images and extract essential patterns from them using principal component analysis. These patterns are then linked to the seismic source, path, and site variables using general regression neural network. The resulting model predicts an image representing the expected evolutionary power spectrum conditioned on the input variables. An example application is presented using records from the Next Generation Attenuation of Ground Motions Database of the Pacific Earthquake Engineering Center. As opposed to conventional ground motion prediction models, the proposed approach retains the time domain characteristics of ground motions. The results show that the proposed model can predict significant patterns in the seismic energy distribution, acceleration time series, and 5% damped elastic spectral accelerations.

A Physical Human–Robot Interaction Framework for Trajectory Adaptation Based on Human Motion Prediction and Adaptive Impedance Control

A Physical Human–Robot Interaction Framework for Trajectory Adaptation Based on Human Motion Prediction and Adaptive Impedance Control

Satellite Videos Object Tracking Based on Enhanced Correlation Filter With Motion Prediction Network

Satellite Videos Object Tracking Based on Enhanced Correlation Filter With Motion Prediction Network

Motion-Aware Memory Network for Fast Video Salient Object Detection.

Previous methods based on 3DCNN, convLSTM, or optical flow have achieved great success in video salient object detection (VSOD). However, these methods still suffer from high computational costs or poor quality of the generated saliency maps. To address this, we design a space-time memory (STM)-based network that employs a standard encoder-decoder architecture. During the encoding stage, we extract high-level temporal features from the current frame and its adjacent frames, which is more efficient and practical than methods reliant on optical flow. During the decoding stage, we introduce an effective fusion strategy for both spatial and temporal branches. The semantic information of the high-level features is used to improve the object details in the low-level features. Subsequently, spatiotemporal features are methodically derived step by step to reconstruct the saliency maps. Moreover, inspired by the boundary supervision prevalent in image salient object detection (ISOD), we design a motion-aware loss that predicts object boundary motion, and simultaneously perform multitask learning for VSOD and object motion prediction. This can further enhance the model's capability to accurately extract spatiotemporal features while maintaining object integrity. Extensive experiments on several datasets demonstrate the effectiveness of our method and can achieve state-of-the-art metrics on some datasets. Our proposed model does not require optical flow or additional preprocessing, and can reach an impressive inference speed of nearly 100 FPS.

Dynamic Dense Graph Convolutional Network for Skeleton-Based Human Motion Prediction.

Graph Convolutional Networks (GCN) which typically follows a neural message passing framework to model dependencies among skeletal joints has achieved high success in skeleton-based human motion prediction task. Nevertheless, how to construct a graph from a skeleton sequence and how to perform message passing on the graph are still open problems, which severely affect the performance of GCN. To solve both problems, this paper presents a Dynamic Dense Graph Convolutional Network (DD-GCN), which constructs a dense graph and implements an integrated dynamic message passing. More specifically, we construct a dense graph with 4D adjacency modeling as a comprehensive representation of motion sequence at different levels of abstraction. Based on the dense graph, we propose a dynamic message passing framework that learns dynamically from data to generate distinctive messages reflecting sample-specific relevance among nodes in the graph. Extensive experiments on benchmark Human 3.6M and CMU Mocap datasets verify the effectiveness of our DD-GCN which obviously outperforms state-of-the-art GCN-based methods, especially when using long-term and our proposed extremely long-term protocol.

Robust Data Association Against Detection Deficiency for Semantic SLAM

Robust and accurate object association is essential for precise 3D object landmark inference in semantic Simultaneous Localization and Mapping (SLAM), and yet remains challenging due to the detection deficiency caused by high miss detection rate, false alarm, occlusion and limited field-of-view, etc. The 2D location of an object is a crucial complementary cue to the appearance feature, especially in the case of associating objects across frames under large viewpoint changes. However, motion model or trajectory pattern based methods struggle to infer object motion reliably with a moving camera. In this paper, by exploiting the local projective warping consistency, a local homography based 2D motion inference method is proposed to sequentially estimate the object location along with uncertainty. By integrating the deep appearance feature and semantic information, an object association method, named HOA, which is robust to detection deficiency is proposed. Experimental evaluations suggest that the proposed motion prediction method is capable of maintaining a low cumulative error over a long duration, which enhances the object association performance in both accuracy and robustness. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work aims to consistently associate 2D detection boxes corresponding to the same 3D object across images. In tasks of landmark-based navigation, collision avoidance, grasping and manipulation, objects in the task space are commonly simplified into 3D enveloping surfaces (e.g. cuboid or ellipsoid) by using 2D object detection boxes from multiple image views, and accurate data association is a prerequisite for precise enveloping surface reconstruction. This problem remains challenging considering the imperfect object detections, the appearance similarity of objects and the unpredictable trajectory of the moving camera. This work proposes a long-term reliable 2D location prediction algorithm that is capable of handling the complex motion of the target. Along with the appearance feature extracted by a retrain-free deep learning based model, this work proposes an object association method that can simultaneously deal with multiple objects with unknown object categories under the moving camera scenario.

LLMT: A Transformer-Based Multi-Modal Lower Limb Human Motion Prediction Model for Assistive Robotics Applications

LLMT: A Transformer-Based Multi-Modal Lower Limb Human Motion Prediction Model for Assistive Robotics Applications