Pose Deformation Research Articles

Deformable patch-based garment design in immersive virtual reality

PurposeThis study aims to create and deform 3D garment apparel in an immersive virtual reality using head-mounted display and controllers. For this, adequate design methods for immersive virtual environment were explored and developed in order to confirm the suitability of the developed methods.Design/methodology/approachAn immersive virtual environment was prepared using Unreal Engine (UE) version 5.1 and Meta Human Lexi to create template garment that corresponds to the sizes of a human model. Dual quaternion skinning was adopted for pose deformation. For size deformation, patches were constructed with the measurement lines defined on Lexi. This patch-based approach was adopted not only for automatic generation but also for flat pattern projection of the template garment.FindingsThe research found that garment-making process can be brought into immersive virtual reality. Free use of one's hands and body made apparel deformation in an immersive environment conform with the real garment draping process.Originality/valueSimulating garment making in an immersive virtual reality has not previously been explored in detail. This research discovered, implemented and tested methods that best suit the environment where head-mounted display and controllers are essential in detail.

Semi-automatic Framework for Voxel Human Deformation Modeling.

With the advancement of computer and medical imaging technologies, a number of high-resolution, voxel-based, full-body human anatomical models have been developed for medical education, industrial design, and physics simulation studies. However, these models are limited in many applications because they are often only in an upstanding posture. To quickly develop multi-pose human models for different applications. A semi-automatic framework for voxel deformation is proposed in the study. This paper describes a framework for human pose deformation based on three-dimensional (3D) medical images. The voxel model is first converted into a surface model using a surface reconstruction algorithm. Second, a deformation skeleton based on human bones is defined, and the surface model is bound to the skeleton. The bone Glow algorithm is used to assign weights to the surface vertices. Then, the model is deformed to the target posture by using the Smoothed Rotation Enhanced As-Rigid-As-Possible (SR-ARAP) algorithm. Finally, the volume-filling algorithm is applied to refill the tissues into the deformed surface model. The proposed framework is used to deform two standing human models, and the sitting and running models are developed. The results show that the framework can successfully develop the target pose. When compared to the results of the As-Rigid-As-Possible algorithm, SR-ARAP preserves local tissues better. The study proposes a frame for voxel human model deformation and improves the local tissue integrity during deformation.

Robots Autonomously Detecting People: A Multimodal Deep Contrastive Learning Method Robust to Intraclass Variations

Robotic detection of people in crowded and/or cluttered human-centered environments including hospitals, stores and airports is challenging as people can become occluded by other people or objects, and deform due to clothing or pose variations. There can also be loss of discriminative visual features due to poor lighting. In this paper, we present a novel multimodal person detection architecture to address the mobile robot problem of person detection under intraclass variations. We present a two-stage training approach using: 1) a unique pretraining method we define as Temporal Invariant Multimodal Contrastive Learning (TimCLR), and 2) a Multimodal YOLOv4 (MYOLOv4) detector for finetuning. TimCLR learns person representations that are invariant under intraclass variations through unsupervised learning. Our approach is unique in that it generates image pairs from natural variations within multimodal image sequences and contrasts crossmodal features to transfer invariances between different modalities. These pretrained features are used by the MYOLOv4 detector for finetuning and person detection from RGB-D images. Extensive experiments validate the performance of our DL architecture in both human-centered crowded and cluttered environments. Results show that our method outperforms existing unimodal and multimodal person detection approaches in detection accuracy when considering body occlusions and pose deformations in different lighting.

High-precision method for simultaneously measuring the six-degree-of-freedom relative position and pose deformation of satellites.

The high-precision measurement of the six degrees-of-freedom (6DoF) relative position and pose deformation of satellites on the ground in vacuum and high-/low-temperature environments plays a critical role in ensuring the on-orbit mapping accuracy of satellites. To meet the strict measurement requirements for a satellite of a high accuracy, high stability, and a miniaturized measurement system, this paper proposes a laser measurement method for simultaneously measuring 6DoF relative position and attitude. In particular, a miniaturized measurement system was developed and a measurement model was established. The problem of error crosstalk between the 6DoF relative position and pose measurements was solved by conducting a theoretical analysis and OpticStudio software simulation, and the measurement accuracy was improved. Laboratory experiments and field tests were then conducted. The experimental results revealed that the measurement accuracy of the developed system for the relative position and relative attitude reached 0.2 µm and 0.4", within the measurement ranges of 500 mm along the X axis, ±100 µm along Y and Z axes, and ±100", and the 24-h measurement stabilities were superior to 0.5 µm and 0.5", respectively, which meets the ground measurement requirements for the satellite. The developed system was successfully applied on site, and the 6Dof relative position and pose deformation of the satellite were obtained via a thermal load test. This novel measurement method and system provides an experimental means for satellite development, in addition to a method for the high-precision measurement of the relative 6DoF position and pose between two points.

Coarse X-ray Lumbar Vertebrae Pose Localization and Registration Using Triangulation Correspondence

Plain film X-ray scanners are indispensable for medical diagnostics and clinical procedures. This type of device typically produces two radiographic images of the human spine, including the anteroposterior and lateral views. However, these two photographs presented perspectives that were distinct. The proposed procedure consists of three fundamental steps. For automated cropping, the grayscale lumbar input image was initially projected vertically using its vertical pattern. Then, Delaunay triangulation was performed with the SURF features serving as the triangle nodes. The posture area of the vertebrae was calculated by utilizing the edge density of each node. The proposed method provided an automated estimation of the position of the human lumbar vertebrae, thereby decreasing the radiologist’s workload, computing time, and complexity in a variety of bone-clinical applications. Numerous applications can be supported by the results of the proposed method, including the segmentation of lumbar vertebrae pose, bone mineral density examination, and vertebral pose deformation. The proposed method can estimate the vertebral position with an accuracy of 80.32 percent, a recall rate of 85.37 percent, a precision rate of 82.36%, and a false-negative rate of 15.42 percent.

Bipartite Graph Reasoning GANs for Person Pose and Facial Image Synthesis

We present a novel bipartite graph reasoning Generative Adversarial Network (BiGraphGAN) for two challenging tasks: person pose and facial image synthesis. The proposed graph generator consists of two novel blocks that aim to model the pose-to-pose and pose-to-image relations, respectively. Specifically, the proposed bipartite graph reasoning (BGR) block aims to reason the long-range cross relations between the source and target pose in a bipartite graph, which mitigates some of the challenges caused by pose deformation. Moreover, we propose a new interaction-and-aggregation (IA) block to effectively update and enhance the feature representation capability of both a person’s shape and appearance in an interactive way. To further capture the change in pose of each part more precisely, we propose a novel part-aware bipartite graph reasoning (PBGR) block to decompose the task of reasoning the global structure transformation with a bipartite graph into learning different local transformations for different semantic body/face parts. Experiments on two challenging generation tasks with three public datasets demonstrate the effectiveness of the proposed methods in terms of objective quantitative scores and subjective visual realness. The source code and trained models are available at https://github.com/Ha0Tang/BiGraphGAN.

AdaptivePose: Human Parts as Adaptive Points

Multi-person pose estimation methods generally follow top-down and bottom-up paradigms, both of which can be considered as two-stage approaches thus leading to the high computation cost and low efficiency. Towards a compact and efficient pipeline for multi-person pose estimation task, in this paper, we propose to represent the human parts as points and present a novel body representation, which leverages an adaptive point set including the human center and seven human-part related points to represent the human instance in a more fine-grained manner. The novel representation is more capable of capturing the various pose deformation and adaptively factorizes the long-range center-to-joint displacement thus delivers a single-stage differentiable network to more precisely regress multi-person pose, termed as AdaptivePose. For inference, our proposed network eliminates the grouping as well as refinements and only needs a single-step disentangling process to form multi-person pose. Without any bells and whistles, we achieve the best speed-accuracy trade-offs of 67.4% AP / 29.4 fps with DLA-34 and 71.3% AP / 9.1 fps with HRNet-W48 on COCO test-dev dataset.

Driver Fatigue Detection Method Based on Human Pose Information Entropy

Driver fatigue detection (DFD) is an effective method to prevent traffic accidents. The existing research on DFD using facial features is an effective and noninvasive fatigue detection method. However, this approach is affected by facial occlusions (glasses, sunglasses, masks, etc.) and the large facial pose deformations in the extraction of effective fatigue features. In this paper, we introduce a novel DFD method using human pose information entropy. The method first estimates human pose from video sequences and then uses them as clues to extract multiple fatigue-related features which can reduce the influence of facial occlusion and head pose deformation. Information entropy and sliding window algorithm are applied to analyse and calculate sufficient consecutive video frames to obtain more robust and accurate fatigue-related values than by using a single frame. These information entropy values are combined resorting to the support vector machine (SVM) to recognize the driver fatigue state. Experimental results show that the method can achieve much higher accuracy and robustness, and the detection speed meets the requirements of real time.

Automatic Sitting Pose Generation for Ergonomic Ratings of Chairs.

Human poses play a critical role in human-centric product design. Despite considerable researches on pose synthesis and pose-driven product design, most of them adopt the simple stick figure model that captures only skeletons rather than real body geometries and do not link human poses to the environment (e.g., chairs for sitting). This paper focuses on user-tailored ergonomic design and rating of chairs using scanned human geometries. Fully utilizing the anthropometric information of the human models, our method considers more ergonomic guidelines of chair design (such as pressure distribution and support intensity) and links the geometry of 3D chair models and human-to-chair interactions into the pose deformation constraints of the human avatars. The core of our method is a pose generation algorithm which rigs the user's successive poses through coarse- and fine-level pose deformations. We define a non-linear energy function with contact, collision, and joint limit terms, and solve it using a hill-climbing algorithm. The fitting results allow us to quantitatively evaluate the chair model in terms of various ergonomic criteria. Our method is flexible and effective and can be applied to users with varying body shapes and a wide range of chairs. Moreover, the proposed technique can be easily extended to other furniture, such as desk, bed, and cabinet. Extensive evaluations and a user study demonstrate the efficiency and advantages of the proposed virtual fitting method. Given that our method avoids tedious on-site trying, facilitates the exploration/evaluation of various chair products, and provides valuable feedback for the designers and manufacturers to deliver customized products, it is ideal for online shopping of chairs.

Pose Flow Learning From Person Images for Pose Guided Synthesis.

Pose guided synthesis aims to generate a new image in an arbitrary target pose while preserving the appearance details from the source image. Existing approaches rely on either hard-coded spatial transformations or 3D body modeling. They often overlook complex non-rigid pose deformation or unmatched occluded regions, thus fail to effectively preserve appearance information. In this article, we propose a pose flow learning scheme that learns to transfer the appearance details from the source image without resorting to annotated correspondences. Based on such learned pose flow, we proposed GarmentNet and SynthesisNet, both of which use multi-scale feature-domain alignment for coarse-to-fine synthesis. Experiments on the DeepFashion, MVC dataset and additional real-world datasets demonstrate that our approach compares favorably with the state-of-the-art methods and generalizes to unseen poses and clothing styles.

RF-DCM: Multi-Granularity Deep Convolutional Model Based on Feature Recalibration and Fusion for Driver Fatigue Detection

Fatigue driving is one of the main causes of traffic accidents. For real-world driver fatigue detection, the large pose deformations exhibited by the captured global face significantly increase the difficulty of extracting effective features. Furthermore, previous fatigue detection methods have not achieved desired results in distinguishing actions with similar appearance, such as yawning and speaking. In this article, we propose a multi-granularity Deep Convolutional Model based on feature Recalibration and Fusion for driver fatigue detection (RF-DCM). Our deep model leverages cues from partial faces to alleviate the pose variations and obtains robust feature representations from both the global face and different local parts. The core innovative techniques are as follows: A multi-granularity extraction sub-network extracts more efficient multi-granularity features while compressing the parameters of the network. In order to match multi-granularity features, a feature rectification sub-network and a feature fusion sub-network are designed to adaptively recalibrate and fuse the multi-granularity features. A long short term memory network is used to explore the relationship among sequence frames to distinguish actions with similar appearances. Extensive experimental results on the public drowsy driver dataset from NTHU Driver Drowsy competition demonstrate significant performance improvements of our model over all published state-of-the-art methods.

Guided Cascaded Super-Resolution Network for Face Image

The image super-resolution algorithm can overcome the imaging system’s hardware limitation and obtain higher resolution and clearer images. Existing super-resolution methods based on convolutional neural networks(CNN) can learn the mapping relationship between high-resolution(HR) and low-resolution(LR) images. However, when the reconstruction target is a face image, the reconstruction results often have problems that the face area is too smooth and lacks details. We propose a guided cascaded face super-resolution network, called guided cascaded super-resolution network (GCFSRnet). GCFSRnet takes the LR image and a high-quality guided image as inputs, and it consists of a pose deformation module and a super-resolution network. Firstly, the pose deformation module converts the guide image’s posture into the same as the low-resolution face image based on 3D fitting and 3D morphable model (3DMM). Then, the LR image and the deformed guide image are used as input of the super-resolution network. The super-resolution networks are formed by a cascade of two layers of networks, which extract different features. During the reconstruction process, the guide image can provide real facial details and help generate subtle facial textures. The cascade structure of a super-resolution network can gradually extract features and restore different levels of image details. The experimental results on the CASIA Web Face and CelebA datasets show that the proposed method can generate facial images with clear outlines and rich details, which are superior to other state-of-the-art methods such as SRResNet, SRGAN, VDSR, DBPN, etc.

SwapGAN: A Multistage Generative Approach for Person-to-Person Fashion Style Transfer

Fashion style transfer has attracted significant attention because it both has interesting scientific challenges and it is also important to the fashion industry. This paper focuses on addressing a practical problem in fashion style transfer, person-to-person clothing swapping, which aims to visualize what the person would look like with the target clothes worn on another person instead of dressing them physically. This problem remains challenging due to varying pose deformations between different person images. In contrast to traditional nonparametric methods that blend or warp the target clothes for the reference person, in this paper we propose a multistage deep generative approach named SwapGAN that exploits three generators and one discriminator in a unified framework to fulfill the task end-to-end. The first and second generators are conditioned on a human pose map and a segmentation map, respectively, so that we can simultaneously transfer the pose style and the clothes style. In addition, the third generator is used to preserve the human body shape during the image synthesis process. The discriminator needs to distinguish two fake image pairs from the real image pair. The entire SwapGAN is trained by integrating the adversarial loss and the mask-consistency loss. The experimental results on the DeepFashion dataset demonstrate the improvements of SwapGAN over other existing approaches through both quantitative and qualitative evaluations. Moreover, we conduct ablation studies on SwapGAN and provide a detailed analysis about its effectiveness.

Salient Superpixel Visual Tracking with Graph Model and Iterative Segmentation

Visual object tracking is to locate an object of interest in a sequence of consecutive video frames, which is widely applied in many high-level computer vision tasks such as intelligent video surveillance and robotics. It is of great challenges for visual tracking methods to handle large target appearance variations caused by pose deformation, fast motion, occlusion, and surrounding environments in real-time videos. In this paper, inspired by human attention cognitive saliency model, we propose a visual tracking method based on salient superpixels which integrates the target appearance similarity and cognitive saliency, and helps to location inference and appearance model updating. The saliency of superpixel is detected by graph model and manifold ranking. We cluster the superpixels of the first four target boxes into a set corresponding to object foreground and model the target appearance with color descriptors. While tracking, the relevance is computed between the candidate superpixels and the target appearance set. We also propose an iterative threshold segmentation method to distinguish the foreground and background of superpixels based on saliency and relevance. To increase the accuracy of location inference, we explore particle filter in both confidence estimation and sampling procedures. We compared our method with the existing techniques in OTB100 dataset in terms of precision based on center location error and success rate based on overlap, and the experimental results show that our proposed method achieved substantially better performance. Promising results have shown that the proposed salient superpixel-based approach is effective to deformation, occlusion, and other challenges in object tracking.

Parametric 3D modeling of a symmetric human body

To realistically represent 3D human body shape in a mathematical way, the parametric model used should incorporate symmetry as displayed by real people. This paper proposes a symmetric parametric model called symmetricSCAPE. It successfully incorporates symmetry into a parametric model of the 3D body, formulating body geometric variations of both shape and pose using a triangular mesh representation. The symmetry constraint is imposed on each symmetrically-related triangle pair of the body mesh. Mathematically, symmetry-related constraint matrices are derived, and applied during shape and pose deformation of triangle pairs. By accurately registering a pre-designed symmetrization template mesh to the training dataset, we learn how the symmetricSCAPE model causes the body mesh to deform relative to the symmetry. Our experiments demonstrate that the symmetricSCAPE model results in a better, more parsimonious, and more accurate parametric model of the 3D human body than traditional non-symmetry-aware representations.

Visible Infrared Cross-Modality Person Re-Identification Network Based on Adaptive Pedestrian Alignment

Cross-modality person re-identification between the visible domain and infrared domain is important but extremely challenging for night-time surveillance. Besides the cross-modality discrepancies caused by different camera spectrums, visible infrared person re-identification (VI-REID) still suffers from much pedestrian misalignment as well as the variations caused by different camera viewpoints and various pedestrian pose deformations like traditional person re-identification. In this paper, we propose a multi-path adaptive pedestrian alignment network (MAPAN) to learn discriminative feature representations. The multi-path network learns features directly from the data in an end-to-end manner and aligns the pedestrians adaptively without any additional manual annotations. To alleviate the intra-modality discrepancies caused by image misalignment, we combine the aligned visible image features with the original visible image features and enhance the attention of the network towards pedestrians, resulting in significant improvements in distinguishability of the learning features. To mitigate the cross-modality discrepancies between the visible domain and the infrared domain, the discriminative features of the two modalities are mapped to the same feature embedding space, and the identity loss as well as triplet loss is incorporated as the overall loss. Extensive experiments demonstrate the superior performance of proposed method compared to the state-of-the-arts.

Toward More Accurate Matching of Contactless Palmprint Images Under Less Constrained Environments

Contactless personal identification using biometrics characteristics brings multifaceted advantages with improved hygiene, user security, and the convenience. Such imaging also generates deformation-free palmprint images which can lead to higher matching accuracy as the ground truth information is better preserved as compared with those from contact-based imaging. Advancement of palmprint identification technologies for new domains requires research using larger palmprint databases that are acquired from more realistic populations, under contactless, ambient, and indoor and outdoor environments. This paper presents such a new contactless palmprint database acquired from 600 different subjects, which is the largest to date and is also made available in the public domain. Unlike contactless fingerprints, contactless palmprint images often illustrate pose deformations along the optical axis of the camera, which also degrades the matching accuracy. This paper also introduces a new approach for matching contactless palmprint images using more accurate deformation alignment and matching. The experimental results are validated on three publicly available contactless palmprint databases. Comparative experimental results presented in this paper indicate consistently outperforming results over competing methods in the literature and validate the effectiveness of the investigated approach. These results also serve as baseline performance to advance much needed further research using the most challenging and largest database introduced from this paper.

Learning Reasoning-Decision Networks for Robust Face Alignment.

In this paper, we propose an end-to-end reasoning-decision networks (RDN) approach for robust face alignment via policy gradient. Unlike the conventional coarse-to-fine approaches which likely lead to bias prediction due to poor initialization, our approach aims to learn a policy by leveraging raw pixels to reason a subset of shape candidates, sequentially making plausible decisions to remove outliers for robust initialization. To achieve this, we formulate face alignment as a Markov decision process by defining an agent, which typically interacts with a trajectory of states, actions, state transitions and rewards. The agent seeks an optimal shape searching policy over the whole shape space by maximizing a discounted sum of the received values. To further improve the alignment performance, we develop an LSTM-based value function to evaluate the shape quality. During the training procedure, we adjust the gradient of our value function in directions of the policy gradient. This prevents our training goal from being trapped into local optima entangled by both the pose deformations and appearance variations especially in unconstrained environments. Experimental results show that our proposed RDN consistently outperforms most state-of-the-art approaches on four widely-evaluated challenging datasets.

On the Robustness and Reliability in the Pose Deformation System of Mobile Robots

A major vulnerability of mobile robots operating in more environments is their fragility in facing disturbances. A two-fold solution is proposed in this paper. First, a deformable structure was designed to reduce turbulence and to adapt to the uneven ground. Second, a novel control strategy is employed to avoid the limitations of the existing methods. In particular, among the existing solutions, the PID controller is known to be severely limited in handling disturbances and failures and the model-based designs all require detailed mathematical model, which may not be readily available, especially in the case of failures. To address these issues, a solution based on active disturbance rejection control is proposed in this paper, for its simplicity in design and tuning and its robustness against parameter variations and even failures in the pose deformation system. The proposed solution is systematically validated in the simulation, and the results are promising. The performance of the system was maintained in the presence of disturbances and uncertain dynamics, and the reliability of the robot is considerably improved when the unknown leg failures occurred.

Discovering Primary Objects in Videos by Saliency Fusion and Iterative Appearance Estimation

In this paper, we propose a new method for detecting primary objects in unconstrained videos in a completely automatic setting. Here, we define the primary object in a video as the object that presents saliently in most of the frames. Unlike previous works considering only local saliency detection or common pattern discovery, the proposed method integrates the local visual/motion saliency extracted from each frame, global appearance consistency throughout the video, and spatiotemporal smoothness constraint on object trajectories. We first identify a temporal coherent salient region throughout the whole video, and then explicitly learn a global appearance model to distinguish the primary object against the background. In order to obtain high-quality saliency estimations from both appearance and motion cues, we propose a novel self-adaptive saliency map fusion method by learning the reliability of saliency maps from labeled data. As a whole, our method can robustly localize and track primary objects in diverse video content, and handle the challenges such as fast object and camera motion, large scale and appearance variation, background clutter, and pose deformation. Moreover, compared with some existing approaches that assume the object is present in all the frames, our approach can naturally handle the case where the object is present only in part of the frames, e.g., the object enters the scene in the middle of the video or leaves the scene before the video ends. We also propose a new video data set containing 51 videos for primary object detection with per-frame ground-truth labeling. Quantitative experiments on several challenging video data sets demonstrate the superiority of our method compared with the recent state of the arts.