Pose Of Person Research Articles

A Markov Chain approach for video-based virtual try-on with denoising diffusion generative adversarial network

Video-based virtual try-ons have attracted unprecedented attention owing to the development of e-commerce. However, this problem is very challenging because of the arbitrary poses of persons and the demand for temporary consistency of frames, particularly when attempting to synthesize high-quality virtual try-on videos using single images. Specifically, there are two key challenges. 1) The existing video-based virtual try-on methods are based on generative adversarial networks (GAN), which are limited by unstable training and a lack of realism in generated details. 2) The explicit building of stronger constraints of generated frames, which aims to increase the coherence of generated videos. To address these challenges, this study proposed a novel framework, Extended Markov Chain Based Denoising Diffusion Generative Adversarial Network (EMC-DDGAN), which was derived from a denoising diffusion GAN, which is a diffusion model with efficient sampling. Moreover, we proposed an extended Markov chain that used a diffusion model to synthesize frames via sequential generation. With a carefully designed network and learning objects, the proposed approach achieved outstanding performance on public datasets. Rigorous experiments demonstrated that EMC-DDGAN could synthesize higher-quality videos compared to other state-of-the-art methods and validated the effectiveness of the proposed approach.

Heterogeneous Feature Fusion for Improving Performance of Action Detection

We present a novel framework aimed at improving video action detection through the integration of heterogeneous features. Conventional action detection methods which focus on modeling the relationships between person/object instances rely exclusively on video features and do not exploit valuable intra-instance heterogeneous features, such as person pose, positional information or object category, that can support action recognition. Our proposed framework, termed Heterogeneous Feature Fusion (HFF) framework, addresses this limitation by integrating such intra-instance heterogeneous features for person/object instances, and can improve existing action detection methods. To efficiently exploit each heterogeneous feature, which vary in importance depending on actions and/or scenes, we introduce an attention mechanism to dynamically enhance important heterogeneous features within an instance. Experiments on JHMDB and AVA v2.2 datasets show that our HFF significantly enhances the action detection performance of two existing methods.

Person Image Generation Guided by Posture, Expression and Illumination

Person pose transfer has shown great application prospects and potential in recent years. However, this research mainly focuses on posture transfer and does not pay much attention to other human body attributes. In addition, there are problems such as blurred facial expressions and inconsistent facial texture. In order to solve these problems and improve user experience, this paper proposes a multi-feature transfer person image generation model, which combines facial expression transfer and pose transfer and completes the lighting reconstruction of person images. Our model can simultaneously migrate and supplement facial expressions while migrating a person’s posture, ensuring the consistency of characters’ overall migration. Our model can freely switch personal body lighting to adapt to different scenes, greatly improving the user experience. The experimental results show that our method’s preservation rate for face identity is improved by nearly 35% compared with the benchmark, and the similarity index for the overall person body is improved by nearly 14%. The evaluation results indicate that our method performs well in person posture transfer. The experimental results show that the proposed method is superior to the existing algorithms in person posture transformation.

CONet: Crowd and occlusion-aware network for occluded human pose estimation

Human pose estimation has numerous applications in motion recognition, virtual reality, human–computer interaction, and other related fields. However, multi-person pose estimation in crowded and occluded scenes is challenging. One major issue about the current top-down human pose estimation approaches is that they are limited to predicting the pose of a single person, even when the bounding box contains multiple individuals. To address this problem, we propose a novel Crowd and Occlusion-aware Network (CONet) using a divide-and-conquer strategy. Our approach includes a Crowd and Occlusion-aware Head (COHead) which estimates the pose of both the occluder and the occluded person using two separate branches. We also use the attention mechanism to guide the branches for differentiated learning, aiming to improve feature representation. Additionally, we propose a novel interference point loss to enhance the model’s anti-interference ability. Our CONet is simple yet effective, and it outperforms the state-of-the-art model by +1.6 AP, achieving 71.6 AP on CrowdPose. Our proposed model has achieved state-of-the-art results on the CrowdPose dataset, demonstrating its effectiveness in improving the accuracy of human pose estimation in crowded and occluded scenes. This achievement highlights the potential of our model in many real-world applications where accurate human pose estimation is crucial, such as surveillance, sports analysis, and human–computer interaction.

LSG-GAN: Latent space guided generative adversarial network for person pose transfer

Person pose transfer is a popular subject in computer vision tasks and is widely used in image editing, virtual try-on, and game designing. Most of the state-of-the-art models are based on flow fields and parsing maps, focusing on improving the deforming of shapes and textures. However, variances in poses always cause spatial misalignments between the source and target images; therefore, complex textures cannot be generated, especially in invisible regions. To solve this issue, we build a latent space of features to model the implicit correlations between sources and targets, and propose a latent space guided generative adversarial network (LSG-GAN) for pose transfer. The proposed model maps source features to the latent space using a feature-mapping block (FMB), and the results of the FMB are introduced into the backbone using a content-based attention mechanism. The experimental results on the DeepFashion dataset and Market1501 dataset show that the proposed model is more advanced than the existing methods, and the qualitative comparisons indicate that invisible textures can also be recovered in high quality.

Unsupervised Domain Adaptation for Person Re-Identification Via Individual-Preserving and Environmental-Switching Cyclic Generation

Unsupervised domain adaptation for person re-identification (Re-ID) suffers severe domain discrepancies between source and target domains. To reduce the domain shift caused by the changes of context, camera style, or viewpoint, existing methods in this field fine-tune and adapt the Re-ID model with augmented samples, either translating source samples to the target style or assigning pseudo labels to the target. The former methods may lose identity details but keep redundant source background during translation. In contrast, the latter techniques may give noisy labels when the model meets the unseen background and person pose. We mitigate the domain shift in the former translation direction by cyclically decoupling environment and identity-related features. We propose a novel individual-preserving and environmental-switching cyclic generation network (IPES-GAN). Our network has the following distinct features: 1) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Decoupled features instead of fused features:</i> we encode the images into an individual part and an environmental part, which are proved beneficial to generation and adaptation; 2) <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Cyclic generation instead of one-step adaptive generation</i> . We swap source and target environment features to generate cross-domain images with preserved identity-related features conditioned with source (target) background features and then changed again to generate back the input image so that cyclic generation runs in a self-supervised way. Experiments carried out on two significant benchmarks: Market-1501 and DukeMTMC-Reid, reveal state-of-the-art performance.

Bipartite Graph Reasoning GANs for Person Pose and Facial Image Synthesis

We present a novel bipartite graph reasoning Generative Adversarial Network (BiGraphGAN) for two challenging tasks: person pose and facial image synthesis. The proposed graph generator consists of two novel blocks that aim to model the pose-to-pose and pose-to-image relations, respectively. Specifically, the proposed bipartite graph reasoning (BGR) block aims to reason the long-range cross relations between the source and target pose in a bipartite graph, which mitigates some of the challenges caused by pose deformation. Moreover, we propose a new interaction-and-aggregation (IA) block to effectively update and enhance the feature representation capability of both a person’s shape and appearance in an interactive way. To further capture the change in pose of each part more precisely, we propose a novel part-aware bipartite graph reasoning (PBGR) block to decompose the task of reasoning the global structure transformation with a bipartite graph into learning different local transformations for different semantic body/face parts. Experiments on two challenging generation tasks with three public datasets demonstrate the effectiveness of the proposed methods in terms of objective quantitative scores and subjective visual realness. The source code and trained models are available at https://github.com/Ha0Tang/BiGraphGAN.

Body orientation estimation through graph representation: Expanding accuracy with Data Augmentation and Gradient Boosting

Body Orientation Estimation (BOE) is important for a wide array of applications, including robotics, surveillance and consumer analysis. Although multi-sensor approaches are effective, they are not a viable option for in the wild scenarios; the usual approach in such cases is to use single camera images, with imprecise results. Some applications that deal with people benefit from obtaining 2D human skeletons for gesture recognition, and these skeletons bring valuable information about the person's pose. It is proposed to build a 2D skeleton via OpenPose and using its data as training data on XGBoost to detect BOE. To evaluate predictions considering real situations based on a single camera, the TUD Multiview Pedestrian dataset is used and extended considering that a single person is originally considered in images where more people were often identified. It is compared the proposed approach against various state-of-the-art methods and our results indicate better performance. Finally, it is proved that our method is viable for BOE in real-time scenarios by presenting case studies on simulated scenes.

Continuous human pose estimation by machine learning and computer vision

Pose estimation is an important topic that has drawn significant attention from the computer vision community (Luo, Wang, Wong, &amp; Cheng, 2020). The main task is to identify and track the pose of a person in an image or video. It can also be thought of as solving the problem of determining a set of coordinates that describe the person's pose.

A Deep Learning Approach for Biometric Security in Video Surveillance System Using Gait

Video surveillance systems and biometrics inclusion play a significant part in various applications like a criminal investigation, medical rehabilitation, virtual reality, etc. Human Gait is a popular biometric where the individual is differentiated by unique limb actions and special ground reaction force. The unique movement of limb actions is called gait, and a record of 2-D aggregate floor response force through one walking cycle is called Cumulative Foot Pressure Images (CFPI). Both gait and cumulative foot pressure images can be acquired simultaneously of the same person during walking under a surveillance system for human identification. Accurate gait recognition is highly impactful for most applications and a major challenge for researchers due to various external factors like different shoes, mood, clothes, injuries etc., affects the individual gait. The novel system addresses the accuracy issue and proposes two models using the Deep Convolution Neural Network (DCNN) architecture on a large standard database, CASIA-D, containing gait pose and CFPI images of the same person. First, the model of the DCNN is trained using unique Gait Energy Image (GEI) features which reduce the computational time compared to other types of feature sets. The second model is prepared using the CFPI features. Experimentation has been carried out to evaluate the performance of these models with different optimization methods and activation functions and has proven that the DCNN model is far superior in building an accurate gait recognition system on large standard datasets.

Unsupervised domain adaptation for clinician pose estimation and instance segmentation in the operating room.

The fine-grained localization of clinicians in the operating room (OR) is a key component to design the new generation of OR support systems. Computer vision models for person pixel-based segmentation and body-keypoints detection are needed to better understand the clinical activities and the spatial layout of the OR. This is challenging, not only because OR images are very different from traditional vision datasets, but also because data and annotations are hard to collect and generate in the OR due to privacy concerns. To address these concerns, we first study how joint person pose estimation and instance segmentation can be performed on low resolutions images with downsampling factors from 1x to 12x. Second, to address the domain shift and the lack of annotations, we propose a novel unsupervised domain adaptation method, called AdaptOR, to adapt a model from an in-the-wild labeled source domain to a statistically different unlabeled target domain. We propose to exploit explicit geometric constraints on the different augmentations of the unlabeled target domain image to generate accurate pseudo labels and use these pseudo labels to train the model on high- and low-resolution OR images in a self-training framework. Furthermore, we propose disentangled feature normalization to handle the statistically different source and target domain data. Extensive experimental results with detailed ablation studies on the two OR datasets MVOR+ and TUM-OR-test show the effectiveness of our approach against strongly constructed baselines, especially on the low-resolution privacy-preserving OR images. Finally, we show the generality of our method as a semi-supervised learning (SSL) method on the large-scale COCO dataset, where we achieve comparable results with as few as 1% of labeled supervision against a model trained with 100% labeled supervision. Code is available at https://github.com/CAMMA-public/HPE-AdaptOR.

Comparative Analysis of OpenPose, PoseNet, and MoveNet Models for Pose Estimation in Mobile Devices

Pose estimation is a significant strategy that has been actively researched in various fields. For example, the strategy has been adopted for motion capture in moviemaking, and character control in video games. It can also be applied to implement the user interfaces of mobile devices through human poses. Therefore, this paper compares and analyzes four popular pose estimation models, namely, OpenPose, PoseNet, MoveNet Lightning, and MoveNet Thunder, using pre-classified images. The results show that MoveNet Lightning was the fastest, and OpenPose was the slowest among the four models. But OpenPose was the only model capable of estimating the poses of multiple persons. The accuracies of OpenPose, PoseNet, MoveNet Lightning, and MoveNet Thunder were 86.2%, 97.6%, 75.1%, and 80.6%, respectively.

Hybrid Modality Metric Learning for Visible-Infrared Person Re-Identification

Visible-infrared person re-identification (Re-ID) has received increasing research attention for its great practical value in night-time surveillance scenarios. Due to the large variations in person pose, viewpoint, and occlusion in the same modality, as well as the domain gap brought by heterogeneous modality, this hybrid modality person matching task is quite challenging. Different from the metric learning methods for visible person re-ID, which only pose similarity constraints on class level, an efficient metric learning approach for visible-infrared person Re-ID should take both the class-level and modality-level similarity constraints into full consideration to learn sufficiently discriminative and robust features. In this article, the hybrid modality is divided into two types, within modality and cross modality. We first fully explore the variations that hinder the ranking results of visible-infrared person re-ID and roughly summarize them into three types: within-modality variation, cross-modality modality-related variation, and cross-modality modality-unrelated variation. Then, we propose a comprehensive metric learning framework based on four kinds of paired-based similarity constraints to address all the variations within and cross modality. This framework focuses on both class-level and modality-level similarity relationships between person images. Furthermore, we demonstrate the compatibility of our framework with any paired-based loss functions by giving detailed implementation of combing it with triplet loss and contrastive loss separately. Finally, extensive experiments of our approach on SYSU-MM01 and RegDB demonstrate the effectiveness and superiority of our proposed metric learning framework for visible-infrared person Re-ID.

Personalized motion kernel learning for human pose estimation

Estimating human poses from a video is at the foundation of many visual intelligent systems. Various convolutional neural networks have been proposed, achieving state-of-the-art performance on different image datasets. However, most existing approaches are image based, which deliver unreliable estimations on videos since they fail to model temporal consistency across video frames. Recently, another line of work leverages temporal cues for multi-frame person pose estimation, yet still in an instance-unaware fashion, disregarding the specific traits of different instances (persons) or different joints. In this paper, we propose a novel approach to learn specific keypoint motion representations for each person, termed Personalized Motion-Aware Network (PMAN). In the PMAN, we devise three components: (i) an Instance-Sensitive Extractor that adaptively computes the spatial features according to human physical characteristics; (ii) a Keypoint Motion Encoder that separately generates convolution kernels with fine-grained keypoint motion encoding; (iii) a Motion Driven Decoder that parses multi-frame spatial features of the same person to provide precise human pose estimations. Extensive experiments on PoseTrack2017 and PoseTrack2018 datasets demonstrate that our approach greatly improves the performance of multi-frame human pose estimation. It is worth mentioning that our approach surpasses the state-of-the-art method by +1.7 mAP and achieves 82.9 mAP on PoseTrack2017 dataset.

Yoga Pose Perfection using Deep Learning

Abstract&#x0D; Even with lots of attention and work in the computer vision and artificial intelligence field, human body pose detection is still a daunting task. The application of human pose detection is wide-ranging from health monitoring to public security. This paper focuses on the application in yoga, an art that has been performed for over a millennium. In modern society yoga has become a common method of exercise and there-in arises a demand for instructions on how to do yoga properly. Doing certain yoga postures improperly may lead to injuries and fatigue and hence the presence of a trainer becomes important. As many people don’t have the resources to have a yoga instructor or guide, artificial intelligence can act as a substitute and advise people on their poses. Currently, the research surrounding pose estimation for yoga mainly discusses the classification of yogic poses. In this work, we propose a method, using the Tensorflow MoveNet Thunder model, that allows real-time pose estimation to detect the error in a person's pose, thereby allowing them to correct it.

Person Re-Identification by Low-Dimensional Features and Metric Learning

Person re-identification (Re-ID) has attracted attention due to its wide range of applications. Most recent studies have focused on the extraction of deep features, while ignoring color features that can remain stable, even for illumination variations and the variation in person pose. There are also few studies that combine the powerful learning capabilities of deep learning with color features. Therefore, we hope to use the advantages of both to design a model with low computational resource consumption and excellent performance to solve the task of person re-identification. In this paper, we designed a color feature containing relative spatial information, namely the color feature with spatial information. Then, bidirectional long short-term memory (BLSTM) networks with an attention mechanism are used to obtain the contextual relationship contained in the hand-crafted color features. Finally, experiments demonstrate that the proposed model can improve the recognition performance compared with traditional methods. At the same time, hand-crafted features based on human prior knowledge not only reduce computational consumption compared with deep learning methods but also make the model more interpretable.

Monocular 3D multi-person pose estimation via predicting factorized correction factors

Despite the great achievement of 3D human pose estimation, recovering the 3D poses of multiple persons in a single image is still a challenging problem. In this paper, we focus on one specific problem in 3D multi-person pose estimation (3D-MPPE): estimating the absolute 3D human poses. We proposed a pipeline consists of human detection, absolute 3D human root localization, and root-relative 3D single-person pose estimation modules. For the absolute 3D human root localization task, we propose a decoupling dual-branch structure to reconstruct the height of the human body, and further output the depth and localization of the 3D human root in the camera coordinate system. Furthermore, a data augmentation strategy is presented to tackle occlusions, such that our model can effectively estimate the root localization with the incomplete bounding boxes. For the 3D human relative pose estimation task, we use the attention mechanism to capture the correlation between human joint coordinates and further improve the accuracy of relative pose estimation. Finally, we merge the absolute depth of human and the relative 3D pose to output the absolute 3D human pose.

Predictive Coding of Tracking and Intercepting Inferred Moving Target in Posterior Parietal Cortex

Person localization or segmentation in low resolution crowded scenes is important for person tracking and recognition, action detection and anomaly identification. Due to occlusion and lack of inter-person space, person localization becomes a difficult task. In this work, we propose a novel information fusion framework to integrate a Deep Head Detector and a body pose detector. A more accurate body pose showing limb positions will result in more accurate person localization. We propose a novel Deep Head Detector (DHD) to detect person heads in crowds. The proposed DHD is a fully convolutional neural network and it has shown improved head detection performance in crowds. We modify Deformable Parts Model (DPM) pose detector to detect multiple upper body poses in crowds. We efficiently fuse the information obtained by the proposed DHD and the modified DPM to obtain a more accurate person pose detector. The proposed framework is named as Fusion DPM (FDPM) and it has exhibited improved body pose detection performance on spectator crowds. The detected body poses are then used for more accurate person localization by segmenting each person in the crowd.

464 3D head visualization for mapping and tracking dermatological conditions

Tracking or quantifying changes in hair and skin conditions over time is a task usually left up to the clinician: the affected areas (or images thereof) are either compared to the photos taken during past visits or a standard reference scale is used to compare the condition to a limited set of images or drawings to determine the severity level of the current state. While automated skin detection algorithms are gaining traction for binary diagnostic tasks or to segment areas of interest at a particular time-point, tracking changes automatically remains challenging due to the lack of easily accessible and affordable systems that can be used to quantitatively compare the changes in a standardized way. Focusing on tracking conditions that occur on the head, we present a new method to recover a clinically relevant model of a person’s head, defined as the complete 3D head surface in the absence of hair volume, starting only with a video taken from a single hand-held camera. Using techniques from computer vision, more specifically structure-from-motion and multi-view stereo, we determine first the shape of each person's head and then the alignment of the fitted 3D head for all video frames to recover texture mapping information, irrespective of the person's pose. This alignment is then used to map and visualize hair or skin information, for example disease quantifications, onto the head model for tracking changes in the condition over time. We demonstrate that our approach recovers a consistent geometry for varying head shapes, from videos taken by different people, with different smartphones, and in a variety of uncontrolled environments such as outdoors, living rooms, and hallways, and hence, can be applied to the clinical setting as well. Furthermore, we show how, once the head geometry for the person has been recovered, it can also be used in augmented reality compatible smartphones to guide image capture and map dermatological quantifications for integration in the clinic.

AnimePose: Multi-person 3D pose estimation and animation

3D animation of human body movement is quite challenging as it involves using a huge setup with several motion trackers all over the persons body to track the movements of every limb. This is time- consuming and may cause the person discomfort in wearing exoskeleton body suits with motion sensors. In this work, we present a trivial yet effective solution to generate simple 3D animation of human movement of multiple persons from a 2D video using deep learning. Although significant improvement has been achieved recently in 3D human pose estimation, most of the prior works work well in case of single person pose estimation and multi-person pose estimation is still a challenging problem. In this work, we firstly propose a supervised multi-person 3D pose estimation and animation framework namely AnimePose for a given input RGB video sequence. The pipeline of the proposed system consists of various modules: i) Multi-Person 2D pose estimation, ii) Depth Map estimation, iii) Lifting 2D poses to 3D poses, iv) Person trajectory prediction and human pose tracking. Our proposed system produces comparable results on previous state-of-the-art 3D multi-person pose estimation methods on publicly available dataset MuPoTS-3D dataset and it also outperforms previous competing human pose tracking methods by a significant margin of 11.7% performance gain on MOTA score on Posetrack 2018 dataset.