Virtual Try-On With Garment Self-Occlusion Conditions

Abstract

Image-based virtual try-on focuses on changing the model's garment item to the target ones and preserving other visual features. To preserve the texture detail of the given in-shop garment, former methods use geometry-based methods (e.g., Thin-plate-spline interpolation) to realize garment warping. However, due to limited degree of freedom, geometry-based methods perform poorly when garment self-occlusion occurs, which is common in daily life. To address this challenge, we propose a novel occlusion-focused virtual try-on system. Compared to previous ones, our system contains three critical submodules, namely, Garment Part Modeling (GPM), a group of Garment Part Generators (GPGs), and Overlap Relation Estimator (ORE). GPM takes the pose landmarks as input, and progressively models the mask of body parts and garments. Based on these masks, GPGs are introduced to generate each garment part. Finally, ORE is proposed to model the overlap relationships between each garment part, and we bind the generated garments under the guidance of overlap relationships predicted by ORE. To make the most of extracted overlap relationships, we proposed an IoU-based hard example mining method for loss terms to handle the sparsity of the self-occlusion samples in the dataset. Furthermore, we introduce part affinity field as pose representation instead of landmark used widely by previous methods and achieve accuracy improvement on try-on layout estimation stage. We evaluate our model on the VITON dataset and found it can outperform previous approaches, especially on samples with garment self-occlusion.