Sequence Of Poses Research Articles

PoseBERT: A Generic Transformer Module for Temporal 3D Human Modeling.

Training state-of-the-art models for human pose estimation in videos requires datasets with annotations that are really hard and expensive to obtain. Although transformers have been recently utilized for body pose sequence modeling, related methods rely on pseudo-ground truth to augment the currently limited training data available for learning such models. In this paper, we introduce PoseBERT, a transformer module that is fully trained on 3D Motion Capture (MoCap) data via masked modeling. It is simple, generic and versatile, as it can be plugged on top of any image-based model to transform it in a video-based model leveraging temporal information. We showcase variants of PoseBERT with different inputs varying from 3D skeleton keypoints to rotations of a 3D parametric model for either the full body (SMPL) or just the hands (MANO). Since PoseBERT training is task agnostic, the model can be applied to several tasks such as pose refinement, future pose prediction or motion completion without finetuning. Our experimental results validate that adding PoseBERT on top of various state-of-the-art pose estimation methods consistently improves their performances, while its low computational cost allows us to use it in a real-time demo for smoothly animating a robotic hand via a webcam. Test code and models are available at https://github.com/naver/posebert.

<P Class="Elsarticletitle"><Span Lang="En-Us">Model-Based Gait Recognition Using Graph Network on Very Large Population Database<O:P></O:P></Span></P>

At present, the existing gait recognition systems are focusing on developing methods to extract robust gait feature from silhouette images and they indeed achieved great success. However, gait can be sensitive to appearance features such as clothing and carried items. Compared with appearance-based method, model-based gait recognition is promising due to the robustness against these variations. In recent years, with the development of human pose estimation, the difficulty of model-based gait recognition methods has been mitigated. In this paper, to resist the increase of subjects and views variation, local features are built and a siamese network is proposed to maximize the distance of samples from the same subject. We leverage recent advances in action recognition to embed human pose sequence to a vector and introduce Spatial-Temporal Graph Convolution Blocks (STGCB) which has been commonly used in action recognition for gait recognition. Experiments on the very large population dataset named OUMVLP-Pose and the popular dataset, CASIA-B, show that our method archives some state-of-the-art (SOTA) performances in model-based gait recognition. The code and models of our method are available at https://github.com/timelessnaive/Gait-for-Large-Dataset after being accepted.

Generalized Pose Decoupled Network for Unsupervised 3D Skeleton Sequence-Based Action Representation Learning.

Human action representation is derived from the description of human shape and motion. The traditional unsupervised 3-dimensional (3D) human action representation learning method uses a recurrent neural network (RNN)-based autoencoder to reconstruct the input pose sequence and then takes the midlevel feature of the autoencoder as representation. Although RNN can implicitly learn a certain amount of motion information, the extracted representation mainly describes the human shape and is insufficient to describe motion information. Therefore, we first present a handcrafted motion feature called pose flow to guide the reconstruction of the autoencoder, whose midlevel feature is expected to describe motion information. The performance is limited as we observe that actions can be distinctive in either motion direction or motion norm. For example, we can distinguish "sitting down" and "standing up" from motion direction yet distinguish "running" and "jogging" from motion norm. In these cases, it is difficult to learn distinctive features from pose flow where direction and norm are mixed. To this end, we present an explicit pose decoupled flow network (PDF-E) to learn from direction and norm in a multi-task learning framework, where 1 encoder is used to generate representation and 2 decoders are used to generating direction and norm, respectively. Further, we use reconstructing the input pose sequence as an additional constraint and present a generalized PDF network (PDF-G) to learn both motion and shape information, which achieves state-of-the-art performances on large-scale and challenging 3D action recognition datasets including the NTU RGB+D 60 dataset and NTU RGB+D 120 dataset.

3D human pose sequence estimation from point clouds combing temporal feature and joint learning strategy

3D human pose sequence estimation from point clouds combing temporal feature and joint learning strategy

Human dynamics from monocular video with dynamic camera movements

We propose a new method that reconstructs 3D human motion from in-the-wild video by making full use of prior knowledge on the laws of physics. Previous studies focus on reconstructing joint angles and positions in the body local coordinate frame. Body translations and rotations in the global reference frame are partially reconstructed only when the video has a static camera view. We are interested in overcoming this static view limitation to deal with dynamic view videos. The camera may pan, tilt, and zoom to track the moving subject. Since we do not assume any limitations on camera movements, body translations and rotations from the video do not correspond to absolute positions in the reference frame. The key technical challenge is inferring body translations and rotations from a sequence of 3D full-body poses, assuming the absence of root motion. This inference is possible because human motion obeys the law of physics. Our reconstruction algorithm produces a control policy that simulates 3D human motion imitating the one in the video. Our algorithm is particularly useful for reconstructing highly dynamic movements, such as sports, dance, gymnastics, and parkour actions.

Modeling clothing as a separate layer for an animatable human avatar

We have recently seen great progress in building photorealistic animatable full-body codec avatars, but generating high-fidelity animation of clothing is still difficult. To address these difficulties, we propose a method to build an animatable clothed body avatar with an explicit representation of the clothing on the upper body from multi-view captured videos. We use a two-layer mesh representation to register each 3D scan separately with the body and clothing templates. In order to improve the photometric correspondence across different frames, texture alignment is then performed through inverse rendering of the clothing geometry and texture predicted by a variational autoencoder. We then train a new two-layer codec avatar with separate modeling of the upper clothing and the inner body layer. To learn the interaction between the body dynamics and clothing states, we use a temporal convolution network to predict the clothing latent code based on a sequence of input skeletal poses. We show photorealistic animation output for three different actors, and demonstrate the advantage of our clothed-body avatars over the single-layer avatars used in previous work. We also show the benefit of an explicit clothing model that allows the clothing texture to be edited in the animation output.

View-Invariant, Occlusion-Robust Probabilistic Embedding for Human Pose

Recognition of human poses and actions is crucial for autonomous systems to interact smoothly with people. However, cameras generally capture human poses in 2D as images and videos, which can have significant appearance variations across viewpoints that make the recognition tasks challenging. To address this, we explore recognizing similarity in 3D human body poses from 2D information, which has not been well-studied in existing works. Here, we propose an approach to learning a compact view-invariant embedding space from 2D body joint keypoints, without explicitly predicting 3D poses. Input ambiguities of 2D poses from projection and occlusion are difficult to represent through a deterministic mapping, and therefore we adopt a probabilistic formulation for our embedding space. Experimental results show that our embedding model achieves higher accuracy when retrieving similar poses across different camera views, in comparison with 3D pose estimation models. We also show that by training a simple temporal embedding model, we achieve superior performance on pose sequence retrieval and largely reduce the embedding dimension from stacking frame-based embeddings for efficient large-scale retrieval. Furthermore, in order to enable our embeddings to work with partially visible input, we further investigate different keypoint occlusion augmentation strategies during training. We demonstrate that these occlusion augmentations significantly improve retrieval performance on partial 2D input poses. Results on action recognition and video alignment demonstrate that using our embeddings without any additional training achieves competitive performance relative to other models specifically trained for each task.

Velocity-to-velocity human motion forecasting

Forecasting human motion from a sequence of human poses is an important problem in the fields of computer vision and robotics. Most previous approaches merely consider learning the temporal dynamics of body joints or joint angles, while neglect derivatives of body joints (i.e., pose velocities) which could reasonably reduce noise impact and improve stability. To exploit the benefits of pose velocities, we propose the velocity-to-velocity learning paradigm for human motion prediction which attempts to directly build the sequence-to-sequence model in the velocity space. Two variant architectures based on recurrent encoder-decoder networks are introduced under this paradigm. Considering human motion as kinematics of rigid bodies, joint angles which denote transformation are the computations of inverse kinematics. Accordingly, a novel loss function in terms of rotation matrices is designed during training for human motion prediction through a rotation matrix transformation (RMT) layer. Finally, we present an effective training algorithm which exploits sequence transformation to improve model generalization. Our approaches substantially outperform state-of-the-art approaches on two large-scale datasets, Human3.6M and CMU Motion Capture, for both short-term prediction and long-term prediction. In particular, our model can competently forecast human-like and meaningful poses up to 1000 milliseconds. The code is available on GitHub: https://github.com/hongsong-wang/RNN_based_human_motion_prediction.

Dancing with the sound in edge computing environments

Conventional motion predictions have achieved promising performance. However, the length of the predicted motion sequences of most literatures are short, and the rhythm of the generated pose sequence has rarely been explored. To pursue high quality, rhythmic, and long-term pose sequence prediction, this paper explores a novel dancing with the sound task, which is appealing and challenging in computer vision field. To tackle this problem, a novel model is proposed, which takes the sound as an indicator input and outputs the dancing pose sequence. Specifically, our model is based on the variational autoencoder (VAE) framework, which encodes the continuity and rhythm of the sound information into the hidden space to generate a coherent, diverse, rhythmic and long-term pose video. Extensive experiments validated the effectiveness of audio cues in the generation of dancing pose sequences. Concurrently, a novel dataset of audiovisual multimodal sequence generation has been released to promote the development of this field.

Attention-Based 3D Human Pose Sequence Refinement Network.

Three-dimensional human mesh reconstruction from a single video has made much progress in recent years due to the advances in deep learning. However, previous methods still often reconstruct temporally noisy pose and mesh sequences given in-the-wild video data. To address this problem, we propose a human pose refinement network (HPR-Net) based on a non-local attention mechanism. The pipeline of the proposed framework consists of a weight-regression module, a weighted-averaging module, and a skinned multi-person linear (SMPL) module. First, the weight-regression module creates pose affinity weights from a 3D human pose sequence represented in a unit quaternion form. Next, the weighted-averaging module generates a refined 3D pose sequence by performing temporal weighted averaging using the generated affinity weights. Finally, the refined pose sequence is converted into a human mesh sequence using the SMPL module. HPR-Net is a simple but effective post-processing network that can substantially improve the accuracy and temporal smoothness of 3D human mesh sequences obtained from an input video by existing human mesh reconstruction methods. Our experiments show that the noisy results of the existing methods are consistently improved using the proposed method on various real datasets. Notably, our proposed method reduces the pose and acceleration errors of VIBE, the existing state-of-the-art human mesh reconstruction method, by 1.4% and 66.5%, respectively, on the 3DPW dataset.

Aggregated Multi-GANs for Controlled 3D Human Motion Prediction

Human motion prediction from historical pose sequence is at the core of many applications in machine intelligence. However, in current state-of-the-art methods, the predicted future motion is confined within the same activity. One can neither generate predictions that differ from the current activity, nor manipulate the body parts to explore various future possibilities. Undoubtedly, this greatly limits the usefulness and applicability of motion prediction. In this paper, we propose a generalization of the human motion prediction task in which control parameters can be readily incorporated to adjust the forecasted motion. Our method is compelling in that it enables manipulable motion prediction across activity types and allows customization of the human movement in a variety of fine-grained ways. To this aim, a simple yet effective composite GAN structure, consisting of local GANs for different body parts and aggregated via a global GAN is presented. The local GANs game in lower dimensions, while the global GAN adjusts in high dimensional space to avoid mode collapse. Extensive experiments show that our method outperforms state-of-the-art. The codes are available at https://github.com/herolvkd/AM-GAN.

Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers and Mixture Density Networks

Sign languages are multi-channel visual languages, where signers use a continuous 3D space to communicate. Sign language production (SLP), the automatic translation from spoken to sign languages, must embody both the continuous articulation and full morphology of sign to be truly understandable by the Deaf community. Previous deep learning-based SLP works have produced only a concatenation of isolated signs focusing primarily on the manual features, leading to a robotic and non-expressive production. In this work, we propose a novel Progressive Transformer architecture, the first SLP model to translate from spoken language sentences to continuous 3D multi-channel sign pose sequences in an end-to-end manner. Our transformer network architecture introduces a counter decoding that enables variable length continuous sequence generation by tracking the production progress over time and predicting the end of sequence. We present extensive data augmentation techniques to reduce prediction drift, alongside an adversarial training regime and a mixture density network (MDN) formulation to produce realistic and expressive sign pose sequences. We propose a back translation evaluation mechanism for SLP, presenting benchmark quantitative results on the challenging PHOENIX14T dataset and setting baselines for future research. We further provide a user evaluation of our SLP model, to understand the Deaf reception of our sign pose productions.

Simultaneous task placement and sequence optimization in an inspection robotic cell

AbstractIn this article, we improve the efficiency of a turbine blade inspection robotic workcell. The workcell consists of a stationary camera and a 6-axis serial robot that is holding a blade and presenting different zones of the blade to the camera for inspection. The problem at hand consists of a 6-DOF (degree of freedom) continuous optimization of the camera placement and a discrete combinatorial optimization of the sequence of inspection poses (images). For each image, all robot configurations (up to eight) are taken into consideration. A novel combined approach is introduced, called blind dynamic particle swarm optimization (BD-PSO), to simultaneously obtain the optimal design for both domains. The objective is to minimize the cycle time of the inspection process, while avoiding any collisions. Even though PSO is vastly used in engineering problems, the novelty of our combinatorial optimization method is in its ability to be used efficiently in traveling salesman problems where the distances between the cities are unknown and subject to change. This highly unpredictable environment is the case of the inspection cell where the cycle time between two images will change for different camera placements.

Follower: A Novel Self-Deployable Action Recognition Framework.

Deep learning technology has improved the performance of vision-based action recognition algorithms, but such methods require a large number of labeled training datasets, resulting in weak universality. To address this issue, this paper proposes a novel self-deployable ubiquitous action recognition framework that enables a self-motivated user to bootstrap and deploy action recognition services, called FOLLOWER. Our main idea is to build a “fingerprint” library of actions based on a small number of user-defined sample action data. Then, we use the matching method to complete action recognition. The key step is how to construct a suitable “fingerprint”. Thus, a pose action normalized feature extraction method based on a three-dimensional pose sequence is designed. FOLLOWER is mainly composed of the guide process and follow the process. Guide process extracts pose action normalized feature and selects the inner class central feature to build a “fingerprint” library of actions. Follow process extracts the pose action normalized feature in the target video and uses the motion detection, action filtering, and adaptive weight offset template to identify the action in the video sequence. Finally, we collect an action video dataset with human pose annotation to research self-deployable action recognition and action recognition based on pose estimation. After experimenting on this dataset, the results show that FOLLOWER can effectively recognize the actions in the video sequence with recognition accuracy reaching 96.74%.

Generation of Pedestrian Crossing Scenarios Using Ped-Cross Generative Adversarial Network

The safety of vulnerable road users is of paramount importance as transport moves towards fully automated driving. The richness of real-world data required for testing autonomous vehicles is limited and furthermore, available data do not present a fair representation of different scenarios and rare events. Before deploying autonomous vehicles publicly, their abilities must reach a safety threshold, not least with regards to vulnerable road users, such as pedestrians. In this paper, we present a novel Generative Adversarial Networks named the Ped-Cross GAN. Ped-Cross GAN is able to generate crossing sequences of pedestrians in the form of human pose sequences. The Ped-Cross GAN is trained with the Pedestrian Scenario dataset. The novel Pedestrian Scenario dataset, derived from existing datasets, enables training on richer pedestrian scenarios. We demonstrate an example of its use through training and testing the Ped-Cross GAN. The results show that the Ped-Cross GAN is able to generate new crossing scenarios that are of the same distribution from those contained in the Pedestrian Scenario dataset. Having a method with these capabilities is important for the future of transport, as it will allow for the adequate testing of Connected and Autonomous Vehicles on how they correctly perceive the intention of pedestrians crossing the street, ultimately leading to fewer pedestrian casualties on our roads.

DanceIt: Music-Inspired Dancing Video Synthesis.

Close your eyes and listen to music, one can easily imagine an actor dancing rhythmically along with the music. These dance movements are usually made up of dance movements you have seen before. In this paper, we propose to reproduce such an inherent capability of the human-being within a computer vision system. The proposed system consists of three modules. To explore the relationship between music and dance movements, we propose a cross-modal alignment module that focuses on dancing video clips, accompanied by pre-designed music, to learn a system that can judge the consistency between the visual features of pose sequences and the acoustic features of music. The learned model is then used in the imagination module to select a pose sequence for the given music. Such pose sequence selected from the music, however, is usually discontinuous. To solve this problem, in the spatial-temporal alignment module we develop a spatial alignment algorithm based on the tendency and periodicity of dance movements to predict dance movements between discontinuous fragments. In addition, the selected pose sequence is often misaligned with the music beat. To solve this problem, we further develop a temporal alignment algorithm to align the rhythm of music and dance. Finally, the processed pose sequence is used to synthesize realistic dancing videos in the imagination module. The generated dancing videos match the content and rhythm of the music. Experimental results and subjective evaluations show that the proposed approach can perform the function of generating promising dancing videos by inputting music.

Optimisation of the combined application planning and execution time utilising repeated PRM replanning for point-to-point sequences

Motion planning and collision avoidance are well studied topics in modern industrial robotics and they are already relatively simple to adopt using packages like ROS MoveIt or Battelle PathPlan. Using probabilistic planning algorithms, collision-free trajectories can be found in as little time as hundredths of a second. However, the obtained trajectories can vary significantly for each planning request. This can result in notably longer trajectories than necessary, increasing the total time required for planning and execution. Optimisation-based motion planners such as PRM* and RRT* can find shorter paths, at the cost of a higher planning time. Our focus is put on applications that require both short planning and short trajectory execution times using existing planning tools, this with the goal of minimising the total time required for the entire application.In this paper, an easy-to-implement approach is proposed to perform path optimisation for trajectory planning applications, without altering the used planner itself, which the authors dubbed repeated PRM. This optimisation is performed by planning, selecting and eliminating trajectories based on minimal motion time during the robot’s current movement. Testing is done in a virtual environment using ROS MoveIt with a 6-DOF Stäubli TX2-90XL, utilising the OMPL PRM motion planning algorithm, eight random pose targets and eight collision objects. Tests are performed on pose sequences of one to eight poses with 30 simulations each. The average of all total times required to perform the pose sequences with the repeated PRM approach are compared to these of sequential PRM and PRM* methods.After testing, the repeated PRM method shows an average impact on the total planning and execution time with a reduction of up to 26,35 % compared to sequential PRM depending on the length of the pose sequence, and up to 57,60 % compared to sequential PRM*, as the additional computational time required for PRM* significantly increases its total required time. The variation of the total required times of the found trajectory sequences also improves by an average of 56,16 % and 45,56 % compared to PRM and PRM* respectively.

TWINBOT: Autonomous Underwater Cooperative Transportation

Underwater Inspection, Maintenance, and Repair operations are nowadays performed using Remotely Operated Vehicles (ROV) deployed from dynamic-positioning vessels, having high daily operational costs. During the last twenty years, the research community has been making an effort to design new Intervention Autonomous Underwater Vehicles (I-AUV), which could, in the near future, replace the ROVs, significantly decreasing these costs. Until now, the experimental work using I-AUVs has been limited to a few single-vehicle interventions, including object search and recovery, valve turning, and hot stab operations. More complex scenarios usually require the cooperation of multiple agents, i.e., the transportation of large and heavy objects. Moreover, using small, autonomous vehicles requires consideration of their limited load capacity and limited manipulation force/torque capabilities. Following the idea of multi-agent systems, in this paper we propose a possible solution: using a group of cooperating I-AUVs, thus sharing the load and optimizing the stress exerted on the manipulators. Specifically, we tackle the problem of transporting a long pipe. The presented ideas are based on a decentralized Task-Priority kinematic control algorithm adapted for the highly limited communication bandwidth available underwater. The aforementioned pipe is transported following a sequence of poses. A path-following algorithm computes the desired velocities for the robots' end-effectors, and the on-board controllers ensure tracking of these setpoints, taking into account the geometry of the pipe and the vehicles' limitations. The utilized algorithms and their practical implementation are discussed in detail and validated through extensive simulations and experimental trials performed in a test tank using two 8 DOF I-AUVs.

Realistic Speech-Driven Talking Video Generation with Personalized Pose

In this work, we propose a method to transform a speaker’s speech information into a target character’s talking video; the method could make the mouth shape synchronization, expression, and body posture more realistic in the synthesized speaker video. This is a challenging task because changes of mouth shape and posture are coupled with audio semantic information. The model training is difficult to converge, and the model effect is unstable in complex scenes. Existing speech-driven speaker methods cannot solve this problem well. The method proposed in this paper first generates the sequence of key points of the speaker’s face and body postures from the audio signal in real time and then visualizes these key points as a series of two-dimensional skeleton images. Subsequently, we generate the final real speaker video through the video generation network. We take a random sampling of audio clips, encode audio contents and temporal correlations using a more effective network structure, and optimize and iterate network outputs using differential loss and attitude perception loss, so as to obtain a smoother pose key-point sequence and better performance. In addition, by inserting a specified action frame into the synthesized human pose sequence window, action poses of the synthesized speaker are enriched, making the synthesis effect more realistic and natural. Then, the final speaker video is generated by the obtained gesture key points through the video generation network. In order to generate realistic and high-resolution pose detail videos, we insert a local attention mechanism into the key point network of the generated pose sequence and give higher attention to the local details of the characters through spatial weight masks. In order to verify the effectiveness of the proposed method, we used the objective evaluation index NME and user subjective evaluation methods, respectively. Experiment results showed that our method could vividly use audio contentsto generate corresponding speaker videos, and its lip-matching accuracy and expression postures are better than those of previous work. Compared with existing methods in the NME index and user subjective evaluation, our method showed better results.

Predicting Intentions of Pedestrians from 2D Skeletal Pose Sequences with a Representation-Focused Multi-Branch Deep Learning Network

Understanding the behaviors and intentions of humans is still one of the main challenges for vehicle autonomy. More specifically, inferring the intentions and actions of vulnerable actors, namely pedestrians, in complex situations such as urban traffic scenes remains a difficult task and a blocking point towards more automated vehicles. Answering the question “Is the pedestrian going to cross?” is a good starting point in order to advance in the quest to the fifth level of autonomous driving. In this paper, we address the problem of real-time discrete intention prediction of pedestrians in urban traffic environments by linking the dynamics of a pedestrian’s skeleton to an intention. Hence, we propose SPI-Net (Skeleton-based Pedestrian Intention network): a representation-focused multi-branch network combining features from 2D pedestrian body poses for the prediction of pedestrians’ discrete intentions. Experimental results show that SPI-Net achieved 94.4% accuracy in pedestrian crossing prediction on the JAAD data set while being efficient for real-time scenarios since SPI-Net can reach around one inference every 0.25 ms on one GPU (i.e., RTX 2080ti), or every 0.67 ms on one CPU (i.e., Intel Core i7 8700K).