Pose Matching Research Articles

PFRNet: 3-D partial-to-full point cloud registration network for arbitrary pose matching

PFRNet: 3-D partial-to-full point cloud registration network for arbitrary pose matching

Analysis of the current situation and development countermeasures of college sports training and management based on the background of artificial intelligence

Abstract To solve the problems of low completeness of data collection and poor accuracy of evaluation of traditional sports training movement completion, this paper proposes an artificial intelligence-based DTW pose matching algorithm. Firstly, the sports postures are matched and evaluated for movements. Then define the limb angle deviation factor and calculate the deviation degree of each limb angle, output the limbs with a deviation degrees greater than the set standard, identify the deviated limbs that affect the overall movement standard, and realize the evaluation and result in the feedback of basic movements in sports training. Under the three feature extraction methods, the accuracy of the statistical value feature extraction algorithm, LR feature extraction algorithm and integrated feature extraction algorithm of the DTW poses matching algorithm were 91.12%, 96.84% and 96.54%, respectively. The accuracy rates of LDA, KNN, NB, CART and DTW algorithms were 25%, 43%, 32%, 33%, and 72%, respectively, when the other four models were used to compare the accuracy rates with the DTW pose matching algorithm. Among them, the accuracy of the DTW algorithm is significantly higher than the other four algorithms. Therefore, the artificial intelligence-based sports training management model proposed in this paper can significantly improve the current problems, such as difficulty in assessing the accuracy of sports movement training, inefficient supervision of teachers’ guidance to students, and difficulty for students to improve their movements directly, which has strong practical significance and application value.

A Joint Network Based CNN for Yoga Pose Classification and Scoring

Comparing to traditional rehabilitation, rehabilitation at home becomes a need during pandemic. The technique brought up in this paper allows patients and yoga fans exercise at home with low cost and comfort while can also evaluate their postures. Previous works focus either on classifying poses or scoring on the sameness between the two input branches of patients’ poses and normative poses, but they ignore the combination of them in one single network. In this study, a residual block based Siamese CNN network with classification and scoring modules is proposed, aiming at providing accurate pose matching scores and classify pose types on yoga postures simultaneously. The Siamese network takes two inputs of learner’s pose and standard pose, which are preprocessed skeleton images by OpenPose. With the addition of residual block on the first convolutional module, back propagation is facilitated, which boosts up the process of updating parameters and optimization. The model calculates total loss by summing up cosine embedding loss and cross entropy loss in which the weight parameter lambda could be modified based on need. As for the scoring module, cosine similarity is used to calculate pose resemblance on batch level. The improvement in model performance is obvious when comparing the loss and accuracy between the Siamese network with residual block and VGG-16. Experimental results indicate that the residual block based Siamese network achieves competitive performance compared to the VGG-16 and can provide scoring feedback to learner’s yoga poses.

Pose-guided matching based on deep learning for assessing quality of action on rehabilitation training

The application of pose assessment on rehabilitation training has gradually received attention in recent years. However, current evaluation indicators of these methods are mostly based on the score or scoring function that defined by users, which is too subjective and hard to be used by patients directly. In this paper, we conceptualized a new idea for pose matching, namely pose-guided matching that aims at providing objective and accurate score, feedback and guidance (i.e. guided) to the patients when the pose is compared to the standard pose. More specifically, we proposed a pair-based Siamese Convolutional Neural Network (SCNN) abbreviated ST-AMCNN to realize the idea of pose-guided matching on the eight-section brocade dataset which is one of the most representative traditional rehabilitation exercises in China. We simplified the multi-stages pose matching by merging two standalone modules (i.e. alignment and matching module) into a one-stage task. Such that, only one loss function is required to tune, which reduces the computational complexity. On top of the Spatial Transformer Networks (STN) employed as an alignment module, we proposed a new Attention-based Multi-Scale Convolution (AMC) to match different posture parts (i.e. multi-scale). Furthermore, the proposed AMC can assign more weight to useful pose features as opposed to other irrelevant features e.g. background features for performance gain. Finally, Gradient-weighted Class Activation Mapping (Grad-CAM) is adopted to visualize the matching result for the learner. Experimental results indicate that ST-AMCNN achieves a competitive performance than the state-of-the-art models and can provide accurate feedback for learners on rehabilitation training. Simultaneously, the proposed method is also deployed in client software for testing.

Fusion Key Frame Image Confidence Assessment of the Medical Service Robot Whole Scene Reconstruction

Facing the precise service and emergency rescue needs of medical service robots in irregular scene, in order to achieve better navigation and path planning for robots in service scenarios, for the whole reconstruction of the absolute scale service scenario, this article proposes a frame of whole scene three-dimensional (3D) point cloud reconstruction based on the fusion of scene depth estimation, confidence assessment, and pose tracking with monocular camera. The algorithm first collects the scene focus stack images under an initial viewing angle through the robot mobile terminal of camera. The absolute depth information of the scene is estimated on the server side, and the confidence level of the reconstructed image of the point cloud is evaluated, and non- uniform sampling is performed to reduce the influence of the error estimation. Based on the sparse key frame position information defined by monocular SLAM, the 3D reconstruction of the whole scene in absolute scale is realized through multi-perspective point cloud pose matching. It provides information of cloud reconstruction of scenic spots for target recognition and navigation of a medical service robot.

Multi-Target Multi-Camera Tracking With Optical-Based Pose Association

Multi-target multi-camera tracking (MTMCT) targets to generate trajectories of the object that appeared under multiple cameras automatically. MTMCT can be treated as a combination of intra-camera tracking and cross-camera tracking. The existing work only employs the global description to perform the tracklet generating. However, the global description cannot model the local similarity between targets, leading to existing methods not to be robust to occlusion and fast motion. To handle the mentioned problem, we propose an online Optical-based Pose Association (OPA) for multi-target multi-camera tracking. The proposed method utilizes local pose matching to solve the occlusion problem, and applies optical flow to reduce the distance caused by fast motion. For optical-based pose association, we firstly employ OpenPose to generate human pose for each proposal. Then, we utilize the optical flow generated by PWC-Net to adjust the estimated pose for the previous frame. Finally, the modified Object Keypoint Similarity is used to compute the similarity between the pose of the current frame and adjusted pose in the prior frame. Once obtaining the optical-based pose similarity, we combine it with the visual and bounding box spatial similarities to generate the final similarity matrix, and apply the Kuhn-Munkras algorithm for data association. The experiments on the MTMCT and MOT datasets verify the rationality of using human pose information and prove the superiority of the proposed method.

An analysis of RelaxedIK: an optimization-based framework for generating accurate and feasible robot arm motions

We present a real-time motion-synthesis method for robot manipulators, called RelaxedIK, that is able to not only accurately match end-effector pose goals as done by traditional IK solvers, but also create smooth, feasible motions that avoid joint-space discontinuities, self-collisions, and kinematic singularities. To achieve these objectives on-the-fly, we cast the standard IK formulation as a weighted-sum non-linear optimization problem, such that motion goals in addition to end-effector pose matching can be encoded as terms in the sum. We present a normalization procedure such that our method is able to effectively make trade-offs to simultaneously reconcile many, and potentially competing, objectives. Using these trade-offs, our formulation allows features to be relaxed when in conflict with other features deemed more important at a given time. We compare performance against a state-of-the-art IK solver and a real-time motion-planning approach in several geometric and real-world tasks on seven robot platforms ranging from 5-DOF to 8-DOF. We show that our method achieves motions that effectively follow position and orientation end-effector goals without sacrificing motion feasibility, resulting in more successful execution of tasks compared to the baseline approaches. We also empirically evaluate how our solver performs with different optimization solvers, gradient calculation methods, and choice of loss function in the objective function.

激光测量点云数据补偿和位姿匹配的研究

激光测量点云数据补偿和位姿匹配的研究

DeepIM: Deep Iterative Matching for 6D Pose Estimation

Estimating the 6D pose of objects from images is an important problem in various applications such as robot manipulation and virtual reality. While direct regression of images to object poses has limited accuracy, matching rendered images of an object against the observed image can produce accurate results. In this work, we propose a novel deep neural network for 6D pose matching named DeepIM. Given an initial pose estimation, our network is able to iteratively refine the pose by matching the rendered image against the observed image. The network is trained to predict a relative pose transformation using an untangled representation of 3D location and 3D orientation and an iterative training process. Experiments on two commonly used benchmarks for 6D pose estimation demonstrate that DeepIM achieves large improvements over state-of-the-art methods. We furthermore show that DeepIM is able to match previously unseen objects.

Research on pose point cloud matching error compensation method for confocal image assembly

High-precision matching of spatial poses is the key to ensuring the assembly performance of complex three-dimensional devices. The human eye cannot directly judge the three-dimensional pose of mesoscale devices, and high-precision observation instruments are needed to assist accurate judgment, ensuring smooth assembly. In this paper, based on the three-dimensional assembly system of micro-devices of confocal microscope, it is difficult to realize the problem of assembly alignment of spatial poses in the existing methods, this focuses on solving the problem of how to perform high-precision pose matching by using point cloud data measured through the confocal microscope.A new method for directly performing alignment of three-dimensional poses is proposed. The principle of point cloud registration ICP algorithm is studied, and the theory of point cloud axis hole registration theory is carried out.A neural network about the mapping relationship between the angle calculated by the RT matrix obtained by ICP algorithm and the theoretical angle is constructed. The comprehensive assembly accuracy of the whole method is analyzed to be 2.41μm. Finally, the actual experiment was carried out through the standard hole axis of 3μm for the single-edge gap and assembled successfully.

Cross-View Person Identification Based on Confidence-Weighted Human Pose Matching.

Cross-view person identification (CVPI) from multiple temporally synchronized videos taken by multiple wearable cameras from different, varying views is a very challenging but important problem, which has attracted more interest recently. Current state-of-the-art performance of CVPI is achieved by matching appearance and motion features across videos, while the matching of pose features does not work effectively given the high inaccuracy of the 3D pose estimation on videos/images collected in the wild. To address this problem, we first introduce a new metric of confidence to the estimated location of each human-body joint in 3D human pose estimation. Then, a mapping function, which can be hand-crafted or learned directly from the datasets, is proposed to combine the inaccurately estimated human pose and the inferred confidence metric to accomplish CVPI. Specifically, the joints with higher confidence are weighted more in the pose matching for CVPI. Finally, the estimated pose information is integrated into the appearance and motion features to boost the CVPI performance. In the experiments, we evaluate the proposed method on three wearable-camera video datasets and compare the performance against several other existing CVPI methods. The experimental results show the effectiveness of the proposed confidence metric, and the integration of pose, appearance, and motion produces a new state-of-the-art CVPI performance.

AR Oriented Pose Matching Mechanism from Motion Capture Data

Pose matching and skeletal mapping method are an integral part of Augmented Reality (AR) based learning technology. In this paper a mechanism for pose matching is presented based on extraction of skeletal data from the dance trainer’s physical movements in the form of color defined images snapped by Kinect, where each pose is modelled by a sequence of key movements and continues data frames. In order to extract the exact matched pose, the frame sequence is divided into pose feature frame and skeletal data frame by the use of pose matching dance training movement recognition algorithm (PMDTMR). This proposed algorithm is compared with other published methods in terms of frame level accuracy and learning time of dance session. The experimental results show that the proposed algorithm outperforms the state of art techniques for successful identification and recognition of matched pose between the dance trainer and the expert of the pre-recorded video through the Kinect sensor.

Cross-View Person Identification by Matching Human Poses Estimated With Confidence on Each Body Joint

Cross-view person identification (CVPI) from multiple temporally synchronized videos taken by multiple wearable cameras from different, varying views is a very challenging but important problem, which has attracted more interests recently. Current state-of-the-art performance of CVPI is achieved by matching appearance and motion features across videos, while the matching of pose features does not work effectively given the high inaccuracy of the 3D human pose estimation on videos/images collected in the wild. In this paper, we introduce a new metric of confidence to the 3D human pose estimation and show that the combination of the inaccurately estimated human pose and the inferred confidence metric can be used to boost the CVPI performance---the estimated pose information can be integrated to the appearance and motion features to achieve the new state-of-the-art CVPI performance. More specifically, the estimated confidence metric is measured at each human-body joint and the joints with higher confidence are weighted more in the pose matching for CVPI. In the experiments, we validate the proposed method on three wearable-camera video datasets and compare the performance against several other existing CVPI methods.

Synthesizing Obama

Given audio of President Barack Obama, we synthesize a high quality video of him speaking with accurate lip sync, composited into a target video clip. Trained on many hours of his weekly address footage, a recurrent neural network learns the mapping from raw audio features to mouth shapes. Given the mouth shape at each time instant, we synthesize high quality mouth texture, and composite it with proper 3D pose matching to change what he appears to be saying in a target video to match the input audio track. Our approach produces photorealistic results.

체감형 콘텐츠 개발을 위한 연속동작 매칭 방법에 관한 연구

The recording and matching method of pose and gesture based on PC-window platform is introduced in this paper. The method uses the gesture detection camera, Xtion which is for the Windows PC. To develop the method, the API is first developed which processes and compares the depth data, RGB image data, and skeleton data obtained using the camera. The pose matching method which selectively compares only valid joints is developed. For the gesture matching, the recognition method which can differentiate the wrong pose between poses is developed. The tool which records and tests the sample data to extract the specified pose and gesture is developed. 6 different pose and gesture were captured and tested. Pose was recognized 100% and gesture was recognized 99%, so the proposed method was validated.

Automatic extraction of relevant video shots of specific actions exploiting Web data

Video sharing websites have recently become a tremendous video source, which is easily accessible without any costs. This has encouraged researchers in the action recognition field to construct action database exploiting Web sources. However Web sources are generally too noisy to be used directly as a recognition database. Thus building action database from Web sources has required extensive human efforts on manual selection of video parts related to specified actions. In this paper, we introduce a novel method to automatically extract video shots related to given action keywords from Web videos according to their metadata and visual features. First, we select relevant videos among tagged Web videos based on the relevance between their tags and the given keyword. After segmenting selected videos into shots, we rank these shots exploiting their visual features in order to obtain shots of interest as top ranked shots. Especially, we propose to adopt Web images and human pose matching method in shot ranking step and show that this application helps to boost more relevant shots to the top. This unsupervised method of ours only requires the provision of action keywords such as “surf wave” or “bake bread” at the beginn ing. We have made large-scale experiments on various kinds of human actions as well as non-human actions and obtained promising results.

Depicting Structure and Function of Complex Skeleto‐Muscular Systems in 3D: The Singing Behavior of a Songbird, Cardinalis cardinalis, as a Model for Human Speech

Functional and kinetic analyses of complex skeleto‐muscular systems with multiple semi‐independent parts have traditionally been a challenge. We have used recent developments in x‐ray CT scanning, 3D reconstruction, imaging, computation, animation, and anatomy to analyze such a system: The vocal apparatus of a songbird, the Northern Cardinal, as a model for human speech. In a 3D reconstruction and visualization through the software Avizo of an x‐ray CT scan, the shoulders, neck, skull, upper jaw apparatus, mandible, hyoid apparatus, trachea, and laryngeal apparatus were segmented (i.e., individually marked). This 3D model was imported into the software Maya to be animated by the computer animation technique called “character rigging”. To match the movements of the animated 3D model to those of a living bird, proper kinematic constraints were applied to the rig. The 3D model was then animated frame by frame by pose matching to the image sequence of an x‐ray video of a singing bird. The correct position of the skeletal elements was checked in various views to account for variations in roll, yaw, and pitch. By adding the sound‐track of the x‐ray video, we matched the exact anatomical configuration of the complex 3D model to each syllable of the song. This method requires more investment at the front end, but allows the modeling of various movements once the 3D model has been constructed. Supported by NIH grant NINDS R01 NS029467 to RASGrant Funding Source: NIH grant NINDS R01 NS029467

Visual Sentences for Pose Retrieval Over Low-Resolution Cross-Media Dance Collections

We describe a system for matching human posture (pose) across a large cross-media archive of dance footage spanning nearly 100 years, comprising digitized photographs and videos of rehearsals and performances. This footage presents unique challenges due to its age, quality and diversity. We propose a forest-like pose representation combining visual structure (self-similarity) descriptors over multiple scales, without explicitly detecting limb positions which would be infeasible for our data. We explore two complementary multi-scale representations, applying passage retrieval and latent Dirichlet allocation (LDA) techniques inspired by the text retrieval domain, to the problem of pose matching. The result is a robust system capable of quickly searching large cross-media collections for similarity to a visually specified query pose. We evaluate over a cross-section of the UK National Research Centre for Dance's (UK-NRCD), and the Siobhan Davies Replay's (SDR) digital dance archives, using visual queries supplied by dance professionals. We demonstrate significant performance improvements over two base-lines: classical single and multi-scale bag of visual words (BoVW) and spatial pyramid kernel (SPK) matching .

3D Human Motion Tracking and Reconstruction Using DCT Matrix Descriptor

One of the most important issues in human motion analysis is the tracking and 3D reconstruction of human motion, which utilizes the anatomic points' positions. These points can uniquely define the position and orientation of all anatomical segments. In this work, a new method is proposed for tracking and 3D reconstruction of human motion from the image sequence of a monocular static camera. In this method, 2D tracking is used for 3D reconstruction, which a database of selected frames is used for the correction of tracking process. The method utilizes a new image descriptor based on discrete cosine transform (DCT), which is employed in different stages of the algorithm. The advantage of using this descriptor is the capabilities of selecting proper frequency regions in various tasks, which results in an efficient tracking and pose matching algorithms. The tracking and matching algorithms are based on reference descriptor matrixes (RDMs), which are updated after each stage based on the frequency regions in DCT blocks. Finally, 3D reconstruction is performed using Taylor’s method. Experimental results show the promise of the algorithm.