3D Human Body Reconstruction Research Articles

A Survey of 3D Human Body Reconstruction from Single and Multiple Camera Views

A Survey of 3D Human Body Reconstruction from Single and Multiple Camera Views

An iterative 3D human body reconstruction method driven by personalized dimensional prior knowledge

An iterative 3D human body reconstruction method driven by personalized dimensional prior knowledge

A single-image human body reconstruction method combining normal recovery and frequency domain completion

<p><span lang="EN-US">Reconstructing a 3D human body from a single image is convenient and efficient, but it faces challenges when the face is heavily occluded or the person is wearing loose clothing. In this paper, we propose a frequency domain-based method for completing missing parts of the human body using manifold harmonics and frequency domain analysis. Our approach involves linear interpolation of the incomplete 3D human body obtained through normal integration. The interpolated points are then projected onto the appropriate dimension of the frequency domain space using manifold harmonic bases. Through Laplacian mesh editing, the interpolated points are replaced, resulting in a refined and complete 3D human body. Our method surpasses the limitations of template human bodies and marching cubes algorithms, enabling more detailed feature reconstruction. By locally completing the body in a low-dimensional frequency domain space, our method avoids over smoothing and bulging issues, effectively filling the missing regions while maintaining mesh smoothness consistency. Experimental results demonstrate the effectiveness and superiority of our frequency domain-based completion method for accurate and detailed 3D human body reconstruction from single images.</span></p>

Gait recognition based on 3D human body reconstruction and multi-granular feature fusion

Gait recognition based on 3D human body reconstruction and multi-granular feature fusion

SHARP: Shape-Aware Reconstruction of People in Loose Clothing

Recent advancements in deep learning have enabled 3D human body reconstruction from a monocular image, which has broad applications in multiple domains. In this paper, we propose SHARP (SHape Aware Reconstruction of People in loose clothing), a novel end-to-end trainable network that accurately recovers the 3D geometry and appearance of humans in loose clothing from a monocular image. SHARP uses a sparse and efficient fusion strategy to combine parametric body prior with a non-parametric 2D representation of clothed humans. The parametric body prior enforces geometrical consistency on the body shape and pose, while the non-parametric representation models loose clothing and handles self-occlusions as well. We also leverage the sparseness of the non-parametric representation for faster training of our network while using losses on 2D maps. Another key contribution is 3DHumans, our new life-like dataset of 3D human body scans with rich geometrical and textural details. We evaluate SHARP on 3DHumans and other publicly available datasets, and show superior qualitative and quantitative performance than existing state-of-the-art methods.

Multi-View High Precise 3D Human Body Reconstruction Method for Virtual Fitting

Online shopping has experienced rapid development recently. However, compared to offline shopping, the return rate and complaint rate of online shopping are much higher, especially for online clothes shopping. In order to solve this problem, virtual fitting technology arises at the right moment, and 3D human modeling is a crucial part of virtual fitting technology. The reconstruction of the parametric 3D human models often faces challenges as long fitting time, low accuracy and fuzzy depth information. Although the reconstruction of nonparametric 3D human model has improved accuracy and detail to some extent, such models typically lack flexibility and controllability. Therefore, this paper reconstructs a high-precision parametric 3D human model by proposing a multi-view iterative registration strategy based on pose prior estimation, which is integrated with a nonparametric 3D human model based on implicit functions. The resulting model retains not only high-precision and detail but also high flexibility and controllability, which has achieved a good effect on the application of 3D virtual fitting. The proposed method is tested on the open-source 3D human body dataset multi-garment network (MGN). The Chamfer distance of the SMPL mannequin reconstructed from multiple views can reach 2.18[Formula: see text]cm and the per-vertical-error can reach 1.63[Formula: see text]cm.

Toward Clothing Personalized Customization Multi-Perspective Silhouettes 3D Human Body Rapid Reconstruction

The clothing personalized online customization is required to design around the user shape parameters, and the traditional anthropometry method obtains the user shape parameter with a low speed and a high error. To tackle this issue, a rapid 3D human body reconstruction method is proposed based on multi-perspective silhouettes for clothing personalized customization. The multilevel dilated convolution semantic network (MDS-Net) is leveraged to extract the global and local features in the human silhouettes to implement the semantic segmentation. The torso parameter extraction network (TPE-Net) is leveraged to extract the shape and pose parameters of the multi-perspective human body segmentation maps. The principal component analysis (PCA) is leveraged to extract the semantic features in the latent space of the 3D human body model, and the semantic features are mapped to the shape and posture parameters output by the torso parameter extraction network, thus, 3D human body model is reconstrued. This method was verified in a PyTorch environment with four datasets. The experiments demonstrated that the mIoU of the multilevel dilated convolution semantic network on the test set is 0.881, which can achieve overall segmentation and local detail preservation. The torso parameter extraction network has an accuracy of 0.74 in the shape parameter prediction on the test set, the prediction of joint offset distance is proportional to the index in the kinematic tree. The entire 3D human body reconstruction method is verified by the real cases.

3D Reconstruction of Human Body in Virtual Fitting Room Based on Kinect

3D Reconstruction of Human Body in Virtual Fitting Room Based on Kinect

Biharmonic 3D Parametric Human Model

3D human body parametric model is a kind of high-level semantic information, which can provide effective prior knowledge for 3D human body reconstruction. The existing human body parametric model does not consider the local similarity characteristics of parameters, and it is difficult to guarantee the smooth model. A 3D human body parametric model based on the BlendSCAPE model is proposed. The biharmonic constraint is considered in every deformation step of the model. Then, each parameter is trained in a data-driven way, and the whole training process is redesigned with the introduction of biharmonic constraint. The training of parameters is finished by solving some minimum energy functions. Finally, experiments are carried out on CAESAR datasets. The results show that the model has a 14.2% reduction in the real data fitting error compared with BlendSCAPE model. Through generating human shapes with different poses and shapes and comparing the mesh before and after Laplace smoothing, the smoothness of the model is improved by 7.3%. The pose estimation results of real data in A-pose show that the deviation of the body part of the model is reduced by 18.6%.

3D human body reconstruction based on SMPL model

3D human body reconstruction based on SMPL model

Hybrid optimization in loop: 3D human body reconstruction from basic information

3D human reconstruction is widely used for digital transformation in different industries such as e-retail, entertainment, health care, epidemiology. For practical applicability, the modeling process is required to be quick, affordable while still ensuring capabilities in reconstruction accuracy and reliability. To meet such business requirements, we propose a novel technique for producing an exact 3D human body using only basic anthropomorphic measurements. To begin, the paper refers to and summarizes core technologies of the three most common 3D human reconstruction approaches, including (1) Using Point Clouds, (2) Using Images, and (3) Using Anthropometric Measurements. Despite successfully recreating 3D human shapes, these methods face problems of long processing time and high investment cost, making the solution impractical for mass use. Moreover, in the human reconstruction sector particularly, the variety of human shapes, poses, and clothing poses a significant challenge to output accuracy. In this regard, our method combines (1) a local optimization model for determining hyperparameters for classifying different human shapes and (2) a global optimization for reconstructing 3D models, allowing reconstruction of both naked human bodies and clothed ones. The proposed method was evaluated quantitatively and qualitatively using a real dataset to demonstrate its feasibility and efficiency when used in real-world applications.

A High Precision 3d Human Body Reconstruction Scheme Based on Multi-View Image for Virtual Fitting

A High Precision 3d Human Body Reconstruction Scheme Based on Multi-View Image for Virtual Fitting

Syntactic model-based human body 3D reconstruction and event classification via association based features mining and deep learning.

The study of human posture analysis and gait event detection from various types of inputs is a key contribution to the human life log. With the help of this research and technologies humans can save costs in terms of time and utility resources. In this paper we present a robust approach to human posture analysis and gait event detection from complex video-based data. For this, initially posture information, landmark information are extracted, and human 2D skeleton mesh are extracted, using this information set we reconstruct the human 2D to 3D model. Contextual features, namely, degrees of freedom over detected body parts, joint angle information, periodic and non-periodic motion, and human motion direction flow, are extracted. For features mining, we applied the rule-based features mining technique and, for gait event detection and classification, the deep learning-based CNN technique is applied over the mpii-video pose, the COCO, and the pose track datasets. For the mpii-video pose dataset, we achieved a human landmark detection mean accuracy of 87.09% and a gait event recognition mean accuracy of 90.90%. For the COCO dataset, we achieved a human landmark detection mean accuracy of 87.36% and a gait event recognition mean accuracy of 89.09%. For the pose track dataset, we achieved a human landmark detection mean accuracy of 87.72% and a gait event recognition mean accuracy of 88.18%. The proposed system performance shows a significant improvement compared to existing state-of-the-art frameworks.

Detailed 3D human body reconstruction from multi-view images combining voxel super-resolution and learned implicit representation

The task of reconstructing detailed 3D human body models from images is interesting but challenging in computer vision due to the high freedom of human bodies. This work proposes a coarse-to-fine method to reconstruct detailed 3D human body from multi-view images combining Voxel Super-Resolution (VSR) based on learning the implicit representation. Firstly, the coarse 3D models are estimated by learning an Pixel-aligned Implicit Function based on Multi-scale Features (MF-PIFu) which are extracted by multi-stage hourglass networks from the multi-view images. Then, taking the low resolution voxel grids which are generated by the coarse 3D models as input, the VSR is implemented by learning an implicit function through a multi-stage 3D convolutional neural network. Finally, the refined detailed 3D human body models can be produced by VSR which can preserve the details and reduce the false reconstruction of the coarse 3D models. Benefiting from the implicit representation, the training process in our method is memory efficient and the detailed 3D human body produced by our method from multi-view images is the continuous decision boundary with high-resolution geometry. In addition, the coarse-to-fine method based on MF-PIFu and VSR can remove false reconstructions and preserve the appearance details in the final reconstruction, simultaneously. In the experiments, our method quantitatively and qualitatively achieves the competitive 3D human body models from images with various poses and shapes on both the real and synthetic datasets.

Deep Unsupervised 3D Human Body Reconstruction from a Sparse set of Landmarks

In this paper we propose the first deep unsupervised approach in human body reconstruction to estimate body surface from a sparse set of landmarks, so called DeepMurf. We apply a denoising autoencoder to estimate missing landmarks. Then we apply an attention model to estimate body joints from landmarks. Finally, a cascading network is applied to regress parameters of a statistical generative model that reconstructs body. Our set of proposed loss functions allows us to train the network in an unsupervised way. Results on four public datasets show that our approach accurately reconstructs the human body from real world mocap data.

Synthetic Humans for Action Recognition from Unseen Viewpoints

Although synthetic training data has been shown to be beneficial for tasks such as human pose estimation, its use for RGB human action recognition is relatively unexplored. Our goal in this work is to answer the question whether synthetic humans can improve the performance of human action recognition, with a particular focus on generalization to unseen viewpoints. We make use of the recent advances in monocular 3D human body reconstruction from real action sequences to automatically render synthetic training videos for the action labels. We make the following contributions: (1) we investigate the extent of variations and augmentations that are beneficial to improving performance at new viewpoints. We consider changes in body shape and clothing for individuals, as well as more action relevant augmentations such as non-uniform frame sampling, and interpolating between the motion of individuals performing the same action; (2) We introduce a new data generation methodology, SURREACT, that allows training of spatio-temporal CNNs for action classification; (3) We substantially improve the state-of-the-art action recognition performance on the NTU RGB+D and UESTC standard human action multi-view benchmarks; Finally, (4) we extend the augmentation approach to in-the-wild videos from a subset of the Kinetics dataset to investigate the case when only one-shot training data is available, and demonstrate improvements in this case as well.

DIMNet: Dense implicit function network for 3D human body reconstruction

In recent years, with the improvement of artificial intelligence technology, it has become possible to reconstruct high-precision 3D human body models based on ordinary RGB images. The current 3D human body reconstruction technology requires complex external equipment to scan all angles of the human body, which is complicated to be implemented and cannot be popularized. In order to solve this problem, this paper applies deep learning models on reconstructing 3D human body based on monocular images. First of all, this paper uses Stacked Hourglass network to perform convolution operations on monocular images collected from different views. Then Multi-Layer Perceptrons (MLPs) are used to decode the encoded high-level images. The feature codes in the two views(main and side) are fused, and the interior and exterior points are classified by the fusion features, so as to obtain the corresponding 3D occupancy field. At last, the Marching Cube algorithm is used for 3D reconstruction with a specific threshold and then we use Laplace smoothing algorithm to remove artifacts. This paper proposes a dense sampling strategy based on the important joint points of the human body, which has a certain optimization effect on the realization of high-precision 3D reconstruction. The performance of the proposed scheme has been validated on the open source datasets, MGN dataset and the THuman dataset, provided by Tsinghua University. The proposed scheme can reconstruct features such as clothing folds, color textures, and facial details,and has great potential to be applied in different applications.

3D Human Pose Estimation Based on a Fully Connected Neural Network With Adversarial Learning Prior Knowledge

3D human pose estimation is more and more widely used in the real world, such as sports guidance, limb rehabilitation training, augmented reality, and intelligent security. Most existing human pose estimation methods are designed based on an RGB image obtained by one optical sensor, such as a digital camera. There is some prior knowledge, such as bone proportion and angle limitation of joint hinge motion. However, the existing methods do not consider the correlation between different joints from multi-view images, and most of them adopt fixed spatial prior constraints, resulting in poor generalizations. Therefore, it is essential to build a multi-view image acquisition system using optical sensors and customized algorithms for a 3D reconstruction of the human pose in the image. Inspired by generative adversarial networks (GAN), we used a data-driven method to learn the implicit spatial prior information and classified joints according to the natural connection characteristics. To accelerate the proposed method, we proposed a fully connected network with skip connections and used the SMPL model to make the 3D human body reconstruction. Experimental results showed that compared with other state-of-the-art methods, the joints’ average error of the proposed method was the smallest, which indicated the best performance. Moreover, the running time of the proposed method was 1.3 seconds per frame, which may not meet real-time requirements, but is still much faster than most existing methods.

The impact of joint axis angle prior on the results of 3D human body reconstruction

The impact of joint axis angle prior on the results of 3D human body reconstruction

Detailed 3D Human Body Reconstruction From a Single Image Based on Mesh Deformation

In the research fields of 3D animation, virtual reality, etc., it is a very important task to reconstruct a 3D human body model with the surface textures. However, due to factors such as diversities of human body poses, uncertainty of depth, and a wide range of viewpoints, the reconstruction of the 3D human body from a single image is an enormous challenge. To solve the aforementioned challenge, this paper proposes a novel architecture to reconstruct the 3D human mesh with great surface details. The key point of the proposed architecture is to combine the topology of the SMPL (Skinned Multi-Person Linear) template mesh and the flexible mesh deformation. Firstly, the typical 2D CNN architecture, ResNet-50, extracts the perceptual features from the input image. The image features are fused into the 3D space via perceptual feature transformation. Our approach does not rely on the parameter space, although the mesh topology of the SMPL is preserved. The approach fully exploits the ability of the SMPL to represent various human body shapes, while avoiding the problem of the spatial loss caused by the parametric method. Secondly, to recover more details of the human body, a graph-based convolutional network is adopted to deform the 3D human body mesh. The network allows starting with fewer vertices and distributing vertex features to the representative positions in a coarse-to-fine manner, which fully utilizes the mesh topology. Experiments demonstrate that our approach can generate 3D human body mesh with great details qualitatively. Compared with the state-of-the-art methods, our approach also achieves better accuracy quantitatively.