3D Human Shape Research Articles

A computer-vision based framework for virtual 3D garment reconstruction

AbstractExisting 3D garment reconstruction methods are difficult to implement for online fashion design and e-commerce or special applications. This paper proposes a novel computer-vision framework for 3D garment digital reconstruction, which aims to reconstruct high-quality and realistic virtual 3D garments with fabric mechanic properties for 3D virtual try-on. The new segmentation scheme is proposed to separate the 3D garment point clouds from background points, which is suitable for 3D human shapes and is adaptive for different 3D garment models in different scenes. The new Statistical Outlier Removal algorithm and the learning-based method PointCleanNet are combined to remove noise and outliers in 3D garment point clouds, which provides high-fidelity and high-quality 3D garment point clouds. The 3D garment meshes are then reconstructed from their corresponding point clouds with a modified rolling ball algorithm. Finally, the meshes are improved and converted into physics-based virtual try-on 3D garments with fabric mechanic properties added, which enables the assessment of different body shapes with varied sizes for the same reconstructed 3D garment. Comparison experiments demonstrate that our framework achieves high-quality and realistic 3D garment reconstruction and accurate 3D virtual try-on from 2D garment images. We also demonstrate the proposed framework on a large range of various garments to show this approach has a great potential for garment future technology, such as online garment shopping, garment design and manufacturing.

SMPLer: Taming Transformers for Monocular 3D Human Shape and Pose Estimation.

Existing Transformers for monocular 3D human shape and pose estimation typically have a quadratic computation and memory complexity with respect to the feature length, which hinders the exploitation of fine-grained information in high-resolution features that is beneficial for accurate reconstruction. In this work, we propose an SMPL-based Transformer framework (SMPLer) to address this issue. SMPLer incorporates two key ingredients: a decoupled attention operation and an SMPL-based target representation, which allow effective utilization of high-resolution features in the Transformer. In addition, based on these two designs, we also introduce several novel modules including a multi-scale attention and a joint-aware attention to further boost the reconstruction performance. Extensive experiments demonstrate the effectiveness of SMPLer against existing 3D human shape and pose estimation methods both quantitatively and qualitatively. Notably, the proposed algorithm achieves an MPJPE of 45.2mm on the Human3.6M dataset, improving upon the state-of-the-art approach (Lin et al., 2021) by more than 10% with fewer than one-third of the parameters.

A marker-less human motion analysis system for motion-based biomarker identification and quantification in knee disorders.

In recent years the healthcare industry has had increased difficulty seeing all low-risk patients, including but not limited to suspected osteoarthritis (OA) patients. To help address the increased waiting lists and shortages of staff, we propose a novel method of automated biomarker identification and quantification for the monitoring of treatment or disease progression through the analysis of clinical motion data captured from a standard RGB video camera. The proposed method allows for the measurement of biomechanics information and analysis of their clinical significance, in both a cheap and sensitive alternative to the traditional motion capture techniques. These methods and results validate the capabilities of standard RGB cameras in clinical environments to capture clinically relevant motion data. Our method focuses on generating 3D human shape and pose from 2D video data via adversarial training in a deep neural network with a self-attention mechanism to encode both spatial and temporal information. Biomarker identification using Principal Component Analysis (PCA) allows the production of representative features from motion data and uses these to generate a clinical report automatically. These new biomarkers can then be used to assess the success of treatment and track the progress of rehabilitation or to monitor the progression of the disease. These methods have been validated with a small clinical study, by administering a local anaesthetic to a small population with knee pain, this allows these new representative biomarkers to be validated as statistically significant (-value ). These significant biomarkers include the cumulative acceleration of elbow flexion/extension in a sit-to-stand, as well as the smoothness of the knee and elbow flexion/extension in both a squat and sit-to-stand.

Fast and Simple Statistical Shape Analysis of Pregnant Women Using Radial Deformation of a Cylindrical Template.

Non-rigid deformation of a template to fit 3D scans of human subjects is widely used to develop statistical models of 3D human shapes and poses. Complex optimization problems must be solved to use these models to parameterize scans of pregnant women, thus limiting their use in antenatal point-of-care tools in low-resource settings. Moreover, these models were developed using datasets that did not contain any 3D scans of pregnant women. In this study, we developed a statistical shape model of the torso of pregnant women at greater than 36 weeks of gestation using fast and simple vertex-based deformation of a cylindrical template constrained along the radial direction. The 3D scans were pre-processed to remove noisy outlier points and segment the torso based on anatomical landmarks. A cylindrical template mesh was then fitted onto the segmented scan of the torso by moving each vertex of in the direction of the radial vector. This process is computationally inexpensive taking only 14.80 seconds to deform a template with 9090 vertices. Principal component analysis (PCA) was performed on the deformed vertex co-ordinates to find the directions of maximum variance. The first 10 principal vectors of our model explained 79.03% of the total variance and reconstructed unseen scans with a mean error of 2.43 cm. We also used the PCA weights of the first 10 principal vectors to accurately predict anthropometric measurements of the pregnant women.

Object-Occluded Human Shape and Pose Estimation With Probabilistic Latent Consistency.

Occlusions between human and objects, especially for the activities of human-object interactions, are very common in practical applications. However, most of the existing approaches for 3D human shape and pose estimation require that human bodies are well captured without occlusions or with minor self-occlusions. In this paper, we focus on the problem of directly estimating the object-occluded human shape and pose from single color images. Our key idea is to utilize a partial UV map to represent an object-occluded human body, and the full 3D human shape estimation is ultimately converted as an image inpainting problem. We propose a novel two-branch network architecture to train an end-to-end regressor via a latent distribution consistency, which also includes a novel visible feature sub-net to extract the human information from object-occluded color images. To supervise the network training, we further build a novel dataset named as 3DOH50K. Several experiments are conducted to reveal the effectiveness of the proposed method. Experimental results demonstrate that the proposed method achieves state-of-the-art compared with previous methods. The dataset and codes are publicly available at https://www.yangangwang.com/papers/ZHANG-OOH-2020-03.html.

통계적 신체 외형 데이터베이스를 활용한 실루엣으로부터의 3차원 인체 외형 예측

In this paper, we present a method for estimating full 3D shapes from given 2D silhouette images of human bodies. Because the silhouette only consists of the partial information on the true shape, it is an ill-posed problem. To address the problem, we use the statistical human shape space obtained from the existing large 3D human shape database. The method consists of three steps. First, we extract the boundary pixels and their appropriate normal vectors from the input silhouette images. Then, we initialize the correspondences of each pixel to the vertex of the statistically-deformable 3D human model. Finally, we numerically optimize the parameters of the statistical model to fit best to the given silhouettes. The viability and the robustness of the method is demonstrated with various experiments.

Estimating 3D body mesh without SMPL annotations via alternating successive convex approximation

This paper focused on extracting effective human shape and pose information from solely joint annotations. With few training datasets and without using SMPL annotations from MoSh, we proposed a method based on alternating successive convex approximation (ASCA) to estimate the 3D human shape and pose. Previous methods tended to utilize a large number of mixed 2D and 3D datasets for training. These methods extract shape and pose information from SMPL annotations containing ground-truth shape and pose annotations. It is challenging to learn useful information of human shape and pose from solely joint annotations independently because 2D and 3D joint positions can be expressed as a non-convex function of coupled SMPL shape and pose parameters. The proposed method decouples the function into joint-shape and joint-pose functions making the training focus on shape and pose separately during each procedure. A minimum number of datasets are used to train the proposed method (InstaVariety and MPI-INF-3DHP) compare to previous methods. We make a comparison with other methods that have been trained with SMPL annotations. The result shows that the proposed method is competitive with other algorithms trained with additional SMPL annotations. What is more, when ground-truth SMPL annotations exist, ASAC can equally extract useful shape and pose information from image sequences. The result trained with a 3DPW training set outperforms most current video-based algorithms.

AvatarCLIP

3D avatar creation plays a crucial role in the digital age. However, the whole production process is prohibitively time-consuming and labor-intensive. To democratize this technology to a larger audience, we propose AvatarCLIP, a zero-shot text-driven framework for 3D avatar generation and animation. Unlike professional software that requires expert knowledge, AvatarCLIP empowers layman users to customize a 3D avatar with the desired shape and texture, and drive the avatar with the described motions using solely natural languages. Our key insight is to take advantage of the powerful vision-language model CLIP for supervising neural human generation, in terms of 3D geometry, texture and animation. Specifically, driven by natural language descriptions, we initialize 3D human geometry generation with a shape VAE network. Based on the generated 3D human shapes, a volume rendering model is utilized to further facilitate geometry sculpting and texture generation. Moreover, by leveraging the priors learned in the motion VAE, a CLIP-guided reference-based motion synthesis method is proposed for the animation of the generated 3D avatar. Extensive qualitative and quantitative experiments validate the effectiveness and generalizability of AvatarCLIP on a wide range of avatars. Remarkably, AvatarCLIP can generate unseen 3D avatars with novel animations, achieving superior zero-shot capability. Codes are available at https://github.com/hongfz16/AvatarCLIP.

Research and Application of 3D Clothing Design Based on Deep Learning

In order to fully explore the multi-scale features of 3D clothing design, at the same time, so as to realize the extraction of the clothing itself and the overall characteristics of the clothing, a research and application of 3D clothing design based on deep learning is proposed. Deep learning neural network model is applied to recognize 3D human shape features, the characteristic line is obtained by the threshold judgment, feature points are obtained based on the identified feature lines, the body circumference curve is fitted, and the measurement of human body size is realized. Simulation results show that for the training of convolutional neural networks in the research of feature recognition algorithms the final feature recognition accuracy is made to reach 92.56%. The training network based on the front projection has a good recognition effect for the neck, armpit, and waist, while the side projection has a good recognition effect for the back neck, shoulder, chest, waist, and buttocks. In general, the characteristic lines of the key parts of the human body have been effectively identified. The author uses convolutional neural networks in deep learning, carrying out 3D scanning human body feature recognition, a large number of experiments have verified the effectiveness and feasibility of the algorithm.

Hybrid optimization in loop: 3D human body reconstruction from basic information

3D human reconstruction is widely used for digital transformation in different industries such as e-retail, entertainment, health care, epidemiology. For practical applicability, the modeling process is required to be quick, affordable while still ensuring capabilities in reconstruction accuracy and reliability. To meet such business requirements, we propose a novel technique for producing an exact 3D human body using only basic anthropomorphic measurements. To begin, the paper refers to and summarizes core technologies of the three most common 3D human reconstruction approaches, including (1) Using Point Clouds, (2) Using Images, and (3) Using Anthropometric Measurements. Despite successfully recreating 3D human shapes, these methods face problems of long processing time and high investment cost, making the solution impractical for mass use. Moreover, in the human reconstruction sector particularly, the variety of human shapes, poses, and clothing poses a significant challenge to output accuracy. In this regard, our method combines (1) a local optimization model for determining hyperparameters for classifying different human shapes and (2) a global optimization for reconstructing 3D models, allowing reconstruction of both naked human bodies and clothed ones. The proposed method was evaluated quantitatively and qualitatively using a real dataset to demonstrate its feasibility and efficiency when used in real-world applications.

Action2video: Generating Videos of Human 3D Actions

We aim to tackle the interesting yet challenging problem of generating videos of diverse and natural human motions from prescribed action categories. The key issue lies in the ability to synthesize multiple distinct motion sequences that are realistic in their visual appearances. It is achieved in this paper by a two-step process that maintains internal 3D pose and shape representations, action2motion and motion2video. Action2motion stochastically generates plausible 3D pose sequences of a prescribed action category, which are processed and rendered by motion2video to form 2D videos. Specifically, the Lie algebraic theory is engaged in representing natural human motions following the physical law of human kinematics; a temporal variational auto-encoder is developed that encourages diversity of output motions. Moreover, given an additional input image of a clothed human character, an entire pipeline is proposed to extract his/her 3D detailed shape, and to render in videos the plausible motions from different views. This is realized by improving existing methods to extract 3D human shapes and textures from single 2D images, rigging, animating, and rendering to form 2D videos of human motions. It also necessitates the curation and reannotation of 3D human motion datasets for training purpose. Thorough empirical experiments including ablation study, qualitative and quantitative evaluations manifest the applicability of our approach, and demonstrate its competitiveness in addressing related tasks, where components of our approach are compared favorably to the state-of-the-arts.

TightCap: 3D Human Shape Capture with Clothing Tightness Field

In this article, we present TightCap, a data-driven scheme to capture both the human shape and dressed garments accurately with only a single three-dimensional (3D) human scan, which enables numerous applications such as virtual try-on, biometrics, and body evaluation. To break the severe variations of the human poses and garments, we propose to model the clothing tightness field—the displacements from the garments to the human shape implicitly in the global UV texturing domain. To this end, we utilize an enhanced statistical human template and an effective multi-stage alignment scheme to map the 3D scan into a hybrid 2D geometry image. Based on this 2D representation, we propose a novel framework to predict clothing tightness field via a novel tightness formulation, as well as an effective optimization scheme to further reconstruct multi-layer human shape and garments under various clothing categories and human postures. We further propose a new clothing tightness dataset of human scans with a large variety of clothing styles, poses, and corresponding ground-truth human shapes to stimulate further research. Extensive experiments demonstrate the effectiveness of our TightCap to achieve the high-quality human shape and dressed garments reconstruction, as well as the further applications for clothing segmentation, retargeting, and animation.

Anthropometric Indicators as a Tool for Diagnosis of Obesity and Other Health Risk Factors: A Literature Review.

Obesity is characterized by the accumulation of an excessive amount of fat mass (FM) in the adipose tissue, subcutaneous, or inside certain organs. The risk does not lie so much in the amount of fat accumulated as in its distribution. Abdominal obesity (central or visceral) is an important risk factor for cardiovascular diseases, diabetes, and cancer, having an important role in the so-called metabolic syndrome. Therefore, it is necessary to prevent, detect, and appropriately treat obesity. The diagnosis is based on anthropometric indices that have been associated with adiposity and its distribution. Indices themselves, or a combination of some of them, conform to a big picture with different values to establish risk. Anthropometric indices can be used for risk identification, intervention, or impact evaluation on nutritional status or health; therefore, they will be called anthropometric health indicators (AHIs). We have found 17 AHIs that can be obtained or estimated from 3D human shapes, being a noninvasive alternative compared to X-ray-based systems, and more accessible than high-cost equipment. A literature review has been conducted to analyze the following information for each indicator: definition; main calculation or obtaining methods used; health aspects associated with the indicator (among others, obesity, metabolic syndrome, or diabetes); criteria to classify the population by means of percentiles or cutoff points, and based on variables such as sex, age, ethnicity, or geographic area, and limitations.

Animated 3D human avatars from a single image with GAN-based texture inference

With the development of AR/VR technologies, a reliable and straightforward way to digitize a three-dimensional human body is in high demand. Most existing methods use complex equipment and sophisticated algorithms, but this is impractical for everyday users. In this paper, we propose a pipeline that reconstructs a 3D human shape avatar from a single image. Our approach simultaneously reconstructs the three-dimensional human geometry and whole body texture map with only a single RGB image as input. We first segment the human body parts from the image and then obtain an initial body geometry by fitting the segment to a parametric model. Next, we warp the initial geometry to the final shape by utilizing a silhouette-based dense correspondence. Finally, to infer invisible back texture from a frontal image, we propose a network called InferGAN. Based on human semantic information, we also propose a method to handle partial occlusion by reconstructing the occluded body parts separately. Comprehensive experiments demonstrate that our solution is robust and effective on both public and our own datasets. Our human avatars can be easily rigged and animated using MoCap data. We have developed a mobile application that demonstrates this capability for AR applications.

Deep authoring - an AI Tool set for creating immersive MultiMedia experiences

We introduce a fully automated 360° video processing pipeline using a hierarchical combination of Artificial Intelligence (AI) modules to create immersive volumetric XR experiences. Two critical productions tasks (person segmentation and depth estimation) are addressed with a parallel Deep Neural Network (DNN) pipeline that combines instance segmentation, person detection, pose estimation, camera stabilization, neural tracking, 3D face detection, hair masking, and monocular 360° depth computation in a single and robust tool set. To facilitate the rapid uptake of these techniques we provide a detailed review of AI-based methods to address these problems (complete with links to recommended open source implementations) as well as references to existing authoring tools in the market. Our key contributions include a method to create semi-synthetic data sets for data auto-augmentation and using this technique to generate over 3.8 m images as part of a concise evaluation and subsequent retraining of DNNs for person detection tasks. Furthermore, we apply the same techniques to develop a spherical DNN for monocular depth estimation with a Free Viewpoint Video (FVV) capture system and a novel method to generate 3D human shapes and pose mannequins for training. To evaluate the performance of our AI authoring tool set we address four challenging production tasks and demonstrate the practical use of our solution with videos showing processed output.

Image-Guided Human Reconstruction via Multi-Scale Graph Transformation Networks.

3D human reconstruction from a single image is a challenging problem. Existing methods have difficulties to infer 3D clothed human models with consistent topologies for various poses. In this paper, we propose an efficient and effective method using a hierarchical graph transformation network. To deal with large deformations and avoid distorted geometries, rather than using Euclidean coordinates directly, 3D human shapes are represented by a vertex-based deformation representation that effectively encodes the deformation and copes well with large deformations. To infer a 3D human mesh consistent with the input real image, we also use a perspective projection layer to incorporate perceptual image features into the deformation representation. Our model is easy to train and fast to converge with short test time. Besides, we present the D2Human (Dynamic Detailed Human) dataset, including variously posed 3D human meshes with consistent topologies and rich geometry details, together with the captured color images and SMPL models, which is useful for training and evaluation of deep frameworks, particularly for graph neural networks. Experimental results demonstrate that our method achieves more plausible and complete 3D human reconstruction from a single image, compared with several state-of-the-art methods. The code and dataset are available for research purposes at http://cic.tju.edu.cn/faculty/likun/projects/MGTnet.

Development of hip protectors for snowboarding utilizing 3D modeling and 3D printing

The purpose of this study was to develop a highly comfortable 3D male hip protector using 3D modeling and printing technologies. The hip protector pads and patterns were devised using 3D human body shapes, and three types of pads were chosen in consideration of snowboarding motions. The three types of pads were as follows: first, the original type with no hole; second, an inner open type with an incision on the inside; and third, an outer open type, with an incision on the outside. Another variable of the protective pads was the material: 3D printed thermoplastic polyurethane (TPU) pad + ethylene–vinyl acetate (EVA) foam or only EVA foam. Six types of pad prototypes were 3D printed and evaluated for subjective wearing comfort. Subjective comfort, fit, activity comfort, and shock absorption were evaluated on an 11-point Likert scale. The study results showed that protectors printed using TPU material were not different from the results of 3D modeling. The evaluation results revealed that comfort, fit, and motion comfort were all negatively evaluated by subjects when wearing the original pad. While fit, comfort, and motion comfort were all positively evaluated by subjects when wearing the outer open-type pad, and comfort and motion comfort were positively evaluated by subjects when wearing the inner open-type pad. With respect to materials, pads made with the 3D printing (TPU) and EVA foam combination provided the best results in terms of overall comfort, buttocks comfort, and activity comfort.

Pose Recognition of 3D Human Shapes via Multi-View CNN with Ordered View Feature Fusion

Rapid pose classification and pose retrieval in 3D human datasets are important problems in shape analysis. In this paper, we extend the Multi-View Convolutional Neural Network (MVCNN) with ordered view feature fusion for orientation-aware 3D human pose classification and retrieval. Firstly, we combine each learned view feature in an orderly manner to form a compact representation for orientation-aware pose classification. Secondly, for pose retrieval, the Siamese network is adopted to learn descriptor vectors, where their L2 distances are close for pairs of shapes with the same poses and are far away for pairs of shapes with different poses. Furthermore, we also construct a larger 3D Human Pose Recognition Dataset (HPRD) consisting of 100,000 shapes for the evaluation of pose classification and retrieval. Experiments and comparisons demonstrate that our method obtains better results than previous works of pose classification and retrieval on the 3D human datasets, such as SHREC’14, FAUST, and HPRD.

Single-Shot 3D Multi-Person Shape Reconstruction from a Single RGB Image.

Although the performance of the 3D human shape reconstruction method has improved considerably in recent years, most methods focus on a single person, reconstruct a root-relative 3D shape, and rely on ground-truth information about the absolute depth to convert the reconstruction result to the camera coordinate system. In this paper, we propose an end-to-end learning-based model for single-shot, 3D, multi-person shape reconstruction in the camera coordinate system from a single RGB image. Our network produces output tensors divided into grid cells to reconstruct the 3D shapes of multiple persons in a single-shot manner, where each grid cell contains information about the subject. Moreover, our network predicts the absolute position of the root joint while reconstructing the root-relative 3D shape, which enables reconstructing the 3D shapes of multiple persons in the camera coordinate system. The proposed network can be learned in an end-to-end manner and process images at about 37 fps to perform the 3D multi-person shape reconstruction task in real time.

Deep Neural Networks With Distance Distributions for Gender Recognition of 3D Human Shapes

Automatic human gender recognition is an important and classical problem in artificial intelligence. Most of the previous gender recognition works are based on vision appearance and biometric characteristics. However, there are fewer gender recognition approaches for 3D human shapes. In this article, we propose a novel deep neural network learning method for gender recognition of 3D human shapes. Firstly, we introduce effective descriptors to distinguish male and female of 3D human shapes via probability distributions of biharmonic distances among points. Secondly, the above distances-based low-level descriptors are fed into a fully connected neural network for gender recognition. Furthermore, we construct a larger 3D human shape dataset for evaluation of the proposed gender recognition method by collecting and labeling human shape models. Compared with previous works, our method obtains higher recognition accuracy and has more advantages, such as posture invariant, robust to noises, and no need of landmarks or pre-alignment process.