Keypoint Trajectories Research Articles

Key point trajectory prediction method of human stochastic posture falls

ABSTRACT The human body will show very complex and diversified posture changes in the process of falling, including body posture, limb position, angle and movement trajectory, etc. The coordinates of the key points of the model are mapped to the three-dimensional space to form a three-dimensional model and obtain the three-dimensional coordinates of the key points; The construction decomposition method is used to calculate the rotation matrix of each key point, and the rotation matrix is solved to obtain the angular displacement data of the key points on different degrees of freedom. The method of curve fitting combined with the weight distribution kernel function based on self-organizing mapping theory is used to obtain the motion trajectory prediction equation of the human body falling in different degrees of freedom at random positions in three-dimensional space, determine the key point trajectory of human random fall behaviour. The experimental results show that the mapped 3D model is consistent with the real human body structure. This method can accurately determine whether the human body falls or squats randomly, and the prediction results of the key points of the human fall are consistent with the actions of the human body after the fall.

Human Motion Enhancement and Restoration via Unconstrained Human Structure Learning.

Human motion capture technology, which leverages sensors to track the movement trajectories of key skeleton points, has been progressively transitioning from industrial applications to broader civilian applications in recent years. It finds extensive use in fields such as game development, digital human modeling, and sport science. However, the affordability of these sensors often compromises the accuracy of motion data. Low-cost motion capture methods often lead to errors in the captured motion data. We introduce a novel approach for human motion reconstruction and enhancement using spatio-temporal attention-based graph convolutional networks (ST-ATGCNs), which efficiently learn the human skeleton structure and the motion logic without requiring prior human kinematic knowledge. This method enables unsupervised motion data restoration and significantly reduces the costs associated with obtaining precise motion capture data. Our experiments, conducted on two extensive motion datasets and with real motion capture sensors such as the SONY (Tokyo, Japan) mocopi, demonstrate the method's effectiveness in enhancing the quality of low-precision motion capture data. The experiments indicate the ST-ATGCN's potential to improve both the accessibility and accuracy of motion capture technology.

Three-dimensional surface motion capture of multiple freely moving pigs using MAMMAL

Understandings of the three-dimensional social behaviors of freely moving large-size mammals are valuable for both agriculture and life science, yet challenging due to occlusions in close interactions. Although existing animal pose estimation methods captured keypoint trajectories, they ignored deformable surfaces which contained geometric information essential for social interaction prediction and for dealing with the occlusions. In this study, we develop a Multi-Animal Mesh Model Alignment (MAMMAL) system based on an articulated surface mesh model. Our self-designed MAMMAL algorithms automatically enable us to align multi-view images into our mesh model and to capture 3D surface motions of multiple animals, which display better performance upon severe occlusions compared to traditional triangulation and allow complex social analysis. By utilizing MAMMAL, we are able to quantitatively analyze the locomotion, postures, animal-scene interactions, social interactions, as well as detailed tail motions of pigs. Furthermore, experiments on mouse and Beagle dogs demonstrate the generalizability of MAMMAL across different environments and mammal species.

Local Corner and Motion Key Point Trajectory Extraction for Facial Forgery Identification

At present, the development of deep forgery technology has brought new challenges to media content forensics, and the use of deep forgery identification methods to identify forged audio and video has become a significant focus of research and difficulty. Deep forgery technology and forensic technology play a mutual game and promote each other’s development. This paper proposes a spatiotemporal local feature abstraction (STLFA) framework for facial forgery identification to solve the media industry challenges of deep forgery technology. To adequately utilize local facial features, we combine facial key points, key point movement, and facial corner points to detect forgery content. This paper establishes a spatiotemporal relation, which realizes face forgery detection by identifying abnormalities of facial keypoints and corner points for interframe judgments. Meanwhile, we utilize RNNs to predict the sequences from facial key point movement abnormalities and corner points for interframe. Experimental results show that our method achieves better performance than some existing methods and good anticompression forgery face detection performance on FF++.

Toward Fine-Grained Talking Face Generation.

Talking face generation is the process of synthesizing a lip-synchronized video when given a reference portrait and an audio clip. However, generating a fine-grained talking video is nontrivial due to several challenges: 1) capturing vivid facial expressions, such as muscle movements; 2) ensuring smooth transitions between consecutive frames; and 3) preserving the details of the reference portrait. Existing efforts have only focused on modeling rigid lip movements, resulting in low-fidelity videos with jerky facial muscle deformations. To address these challenges, we propose a novel Fine-gRained mOtioN moDel (FROND), consisting of three components. In the first component, we adopt a two-stream encoder to capture local facial movement keypoints and embed their overall motion context as the global code. In the second component, we design a motion estimation module to predict audio-driven movements. This enables the learning of local key point motion in the continuous trajectory space to achieve smooth temporal facial movements. Additionally, the local and global motions are fused to estimate a continuous dense motion field, resulting in spatially smooth movements. In the third component, we devise a novel implicit image decoder based on an implicit neural network. This decoder recovers high-frequency information from the input image, resulting in a high-fidelity talking face. In summary, the FROND refines the motion trajectories of facial keypoints into a continuous dense motion field, which is followed by a decoder that fully exploits the inherent smoothness of the motion. We conduct quantitative and qualitative model evaluations on benchmark datasets. The experimental results show that our proposed FROND significantly outperforms several state-of-the-art baselines.

Local Trajectory Occurrence Patterns for Partial Action and Gesture Recognition

Action and gesture recognition is essential in computer vision because of their multiple and potential applications. Nowadays, in the literature, dramatic advances have been reported regarding recognizing gestures and actions under uncontrolled scenarios with significant appearance and motion variations. Nevertheless, much of these approaches still require manual segmentation of temporal action boundaries and complete processing of whole sequences to obtain a prediction. This work introduces a novel motion description that can recognize actions and gestures over partial sequences. The approach starts by representing video sequences as a set of key-point trajectories. Such trajectories are then hierarchically represented from a local and regional perspective, following a statistical counting process. Firstly, each trajectory is defined as a binary occurrence pattern that allows for standing out critical motions by neighborhood densities from a local perspective. Such occurrence patterns are involved in a regional bag-of-words representation of actions. Both representations could be obtained for any interval inside the video, achieving a partial recognition of motion, and regional representation is mapped to a support vector machine to obtain a prediction. The proposed approach was evaluated on academic action recognition datasets and a large gesture dataset used for sign recognition. Regarding partial video sequence recognition, the proposed approach achieves an accuracy rate of 63% using only 20% of frames. The proposed strategy achieved a very compact description, with only 400 scalar values, which ideal for online applications.

Video dataset of human demonstrations of folding clothing for robotic folding

General-purpose clothes-folding robots do not yet exist owing to the deformable nature of textiles, making it hard to engineer manipulation pipelines or learn this task. In order to accelerate research for the learning of the robotic clothes-folding task, we introduce a video dataset of human folding demonstrations. In total, we provide 8.5 hours of demonstrations from multiple perspectives leading to 1,000 folding samples of different types of textiles. The demonstrations are recorded in multiple public places, in different conditions with a diverse set of people. Our dataset consists of anonymized RGB images, depth frames, skeleton keypoint trajectories, and object labels. In this article, we describe our recording setup, the data format, and utility scripts, which can be accessed at https://adverley.github.io/folding-demonstrations.

Gesture Recognition System for Isolated Word Sign Language Based on Key-Point Trajectory Matrix

This paper suggests a new system to help the deaf and the hearing-impaired community improve their connection with the hearing world and communicate freely. The most important thing in this system is how to help the users be free and finally have a more natural way of communication. For this reason, we present a new process based on two levels: a static-level aiming to extract the most head/hands key points and a dynamic-level with the objective of accumulating the key-point trajectory matrix. Also our proposed approach takes into account the signer-independence constraint. A SIGNUM database is applied in the classification stage and our system performances have improved with a 94.3\% recognition rate. Furthermore, a reduction in time processing is obtained when the removing of redundant frame step is applied. The obtained results prove the superiority of our system compared to the state-of- the-art methods in terms of recognition rate and execution time.

Multi-modal learning for affective content analysis in movies

Affective content analysis is an important research topic in video content analysis, and has extensive applications in many fields. However, it is a challenging task to design a computational model for predicting emotions induced by videos, since the elicited emotions can be considered relatively subjective. Intuitively, several features of different modalities can depict the elicited emotions, but the correlation and influence of these features are still not well studied. To address this issue, we propose a multi-modal learning framework, which classifies affective contents in the valence-arousal space. In particular, we utilize the features extracted by the methods of motion keypoint trajectory and convolutional neural networks to depict the visual modality of elicited emotions, and extract a global audio feature by the openSMILE toolkit to describe the audio modality. Then, the linear support vector machine and support vector regression are employed to learn the affective models. By comparing these three features with five baseline features, we discover that the three features are significant for describing affective content. Experimental results also demonstrate that the three features complement each other. Moreover, the proposed framework obtains the state-of-the-art results on two challenging datasets of video affective content analysis.

Motion keypoint trajectory and covariance descriptor for human action recognition

Human action recognition from videos is a challenging task in computer vision. In recent years, histogram-based descriptors that are calculated along dense trajectories have shown promising results for human action recognition, but they usually ignore motion information of the tracking points, and the relationship between different motion variables is not well utilized. To address this issue, we propose a motion keypoint trajectory (MKT) approach and a trajectory-based covariance (TBC) descriptor, which is calculated along the motion keypoint trajectories. The proposed MKT approach tracks motion keypoints at multiple spatial scales and employs an optical flow rectification algorithm to reduce the influence of camera motions and thus achieves better performance than the improved dense trajectory (IDT) approach well known in the literature. In particular, MKT is faster than IDT, because MKT does not need to use human detection and extracts fewer trajectories than IDT. Furthermore, the TBC descriptor outperforms the classical histogram-based descriptors, such as the Histogram of Oriented Gradient, Histogram of Optical Flow and Motion Boundary Histogram. Experimental results on three challenging datasets (i.e., Olympic Sports, HMDB51 and UCF50) demonstrate that our approach is able to achieve better recognition performances than a number of state-of-the-art approaches.

Estimation of viewers’ ratings of TV programs based on behaviors in home environments

A system is described that can estimate a viewer's ratings of TV programs on the basis of his/her behaviors in a home environment. A Kinect sensor, a motion-sensing device developed by Microsoft for its Xbox game console, is used to measure various behavioral parameters. The system first detects whether a viewer is present by extracting keypoint trajectories in video sequences captured by the sensor's video camera. It then identifies whether the viewer is gazing at the TV screen or not by extracting head pose information. The extraction is carried out using two modules: a color-image-based module and a color- and depth-image-based module. The two modules share their parameters and complement each other's characteristics. The proposed system was evaluated by having 30 participants individually spend about 2 h watching 15 TV programs in a simulated home environment, capturing video images of their behaviors, and having them rate each program on a five-point scale. Comparison of the system's estimated ratings with the actual viewer ratings demonstrated that the system can robustly estimate a viewer's ratings of TV programs in a home environment.

Video Saliency Map Detection by Dominant Camera Motion Removal

We present a trajectory-based approach to detect salient regions in videos by dominant camera motion removal. Our approach is designed in a general way so that it can be applied to videos taken by either stationary or moving cameras without any prior information. Moreover, multiple salient regions of different temporal lengths can also be detected. To this end, we extract a set of spatially and temporally coherent trajectories of keypoints in a video. Then, velocity and acceleration entropies are proposed to represent the trajectories. In this way, long-term object motions are exploited to filter out short-term noises, and object motions of various temporal lengths can be represented in the same way. On the other hand, we are inspired by the observation that the trajectories in backgrounds, i.e., the nonsalient trajectories, are usually consistent with the dominant camera motion no matter whether the camera is stationary or not. We make use of this property to develop a unified approach to saliency generation for both stationary and moving cameras. Specifically, one-class SVM is employed to remove the consistent trajectories in motion. It follows that the salient regions could be highlighted by applying a diffusion process to the remaining trajectories. In addition, we create a set of manually annotated ground truth on the collected videos. The annotated videos are then used for performance evaluation and comparison. The promising results on various types of videos demonstrate the effectiveness and great applicability of our approach.

Recognizing Human Actions in Basketball Video Sequences on the Basis of Global and Local Pairwise Representation

A feature-representation method for recognizing actions in sports videos on the basis of the relationship between human actions and camera motions is proposed. The method involves the following steps: First, keypoint trajectories are extracted as motion features in spatio-temporal sub-regions called “spatio-temporal multiscale bags” (STMBs). Global representations and local representations from one sub-region in the STMBs are then combined to create a “glocal pairwise representation” (GPR). The GPR considers the co-occurrence of camera motions and human actions. Finally, two-stage SVM classifiers are trained with STMB-based GPRs, and specified human actions in video sequences are identified. An experimental evaluation of the recognition accuracy of the proposed method (by using the public OSUPEL basketball video dataset and broadcast videos) demonstrated that the method can robustly detect specific human actions in both public and broadcast basketball video sequences.

Weighted feature trajectories and concatenated bag-of-features for action recognition

Key-point trajectory based approaches to recognizing human actions in realistic videos have recently shown promising results. However, their coverage of the entire actor is not sufficient for describing human actions, and the trajectories often disappear due to partial occlusions. In this paper, we propose a novel approach based on weighted feature trajectories and concatenated bag-of-features (BOF) using weighted spatio-temporal descriptors to overcome these limitations. To capture spatio-temporal regions of interest (ROI), we first build scale-invariant feature transform (SIFT) trajectories based on matching SIFT descriptors. Then, we extract particle trajectories around the SIFT trajectories using dense optical flow to retain ROI subsidiarily and to give a sound coverage of the actors. For balancing the leverages between SIFT and particle trajectories, a weighting scheme is presented according to the accumulated global distribution of feature points. Furthermore, we propose the concatenated two distinct SIFT and particle BOFs with weighted spatio-temporal descriptors for action recognition. Experimental results demonstrate the effectiveness of this approach in challenging real-world action datasets such as KTH, UCF sports, and TV human interaction.

A novel framework for motion segmentation and tracking by clustering incomplete trajectories

In this paper, we present a framework for visual object tracking based on clustering trajectories of image key points extracted from an image sequence. The main contribution of our method is that the trajectories are automatically extracted from the image sequence and they are provided directly to a model-based clustering approach. In most other methodologies, the latter constitutes a difficult part since the resulting feature trajectories have a short duration, as the key points disappear and reappear due to occlusion, illumination, viewpoint changes and noise. We present here a sparse, translation invariant regression mixture model for clustering trajectories of variable length. The overall scheme is converted into a maximum a posteriori approach, where the Expectation–Maximization (EM) algorithm is used for estimating the model parameters. The proposed method detects the different objects in the input image sequence by assigning each trajectory to a cluster, and simultaneously provides their motion. Numerical results demonstrate the ability of the proposed method to offer more accurate and robust solutions in comparison with other tracking approaches, such as the mean shift tracker, the camshift tracker and the Kalman filter.

Action recognition using spatiotemporal features and hybrid generative/discriminative models

We propose a new method for human action recognition based on multiple features and a hybrid generative/discriminative model. Specifically, we propose a new action representation based on computing a rich set of descriptors from Affine-SIFT key point trajectories. A new hybrid generative/discriminative approach based on support vector machine and topic model is proposed using Fisher kernel method for action recognition. Fisher score for the topic model is evaluated by the variational inference algorithm. To obtain efficient and compact representations for actions, we develop a feature fusion method to combine spatial-temporal local motion descriptors and demonstrate how this kernel framework can be used to combine different types of features and models into a single classifier. Our experiments, conducted on a number of popular datasets, show performance improvements over the corresponding generative approach and are competitive with the best results reported in the literature.

Human gesture recognition system for TV viewing using time-of-flight camera

We developed a new device-free user interface for TV viewing that uses a human gesture recognition technique. Although many motion recognition technologies have been reported, no man---machine interface that recognizes a large enough variety of gestures has been developed. The difficulty was the lack of spatial information that could be acquired from normal video sequences. We overcame the difficulty by using a time-of-flight camera and novel action recognition techniques. The main functions of this system are gesture recognition and posture measurement. The former is performed using the bag-of-features approach, which uses key-point trajectories as features. The use of 4-D spatiotemporal trajectory features is the main technical contribution of the proposed system. The latter is obtained through face detection and object tracking technology. The interface is useful because it does not require any contact-type devices. Several experiments proved the effectiveness of our proposed method and the usefulness of the system.

A robotic human masticatory system: kinematics simulations

A robotic device is proposed by means of which the mastication process could be reproduced in a mechanically controllable way whilst the masticatory efficiency and/or food dynamics are assessed quantitatively. A robotic model of the mastication system is developed according to biomechanical findings about the jaw structure and muscles of mastication, where each of the major muscles, temporalis, masseter and pterygoid, are responsible for the masticatory movements is represented in a linear actuator and muscle origin and insertion are modelled as a spherical joint placed between the mandible and the skull. This forms a robotic platform model with the mandible being a moving plate and the skull a ground plate. To validate the robotic model, extensive simulations of forward kinematics, i.e., producing the mandible movements with given muscular actuations, and inverse kinematics, i.e., generating the muscular actuations required for a given human chewing pattern have been conducted using the SolidWorks and COSMOS/Motion. Results are given and discussed in terms of trajectories of key reference points, such as incisor and time-history of actuations.