ST-VTON: Self-supervised vision transformer for image-based virtual try-on

Abstract

Virtual try-on techniques are in the spotlight due to the promise of enhancing the user experience of online shopping in the apparel industry. Among these technologies, image-based virtual try-on methods have received more attention due to their efficiency and low cost. These methods commonly warp the specific clothing item to the desired shape and then fuse it with the target person image. However, existing approaches demand the laborious paired dataset of clothes and persons. While recent works alleviate the dependence on paired datasets by transferring clothing from one person to another, the pose information is still a rigid demand to guide the synthesis. To solve the above-mentioned problems, we propose ST-VTON, a two-stage self-supervised try-on approach that can synthesize photo-realistic try-on results with only person images and a standard ViT structure. Specifically, we first utilize massive person and clothing images to pre-train a powerful ViT encoder following the MAE approach to enhance the generalization ability and accelerate model training. Secondly, we propose the Clothing Recovery Module (CRM) to reconstruct in-shop clothes from person images and use the paired generated clothing and corresponding person images to train a new ViT decoder while optimizing the pre-trained encoder. Noted that, the CRM, as an independent module, can be used to effectively expand existing paired try-on datasets and enhance other try-on approaches in terms of training data. Extensive experiments conducted on the VITON dataset and UnPaired virtual Try-on (UPT) dataset with both paired and unpaired approaches demonstrate that try-on results rendered by our model can match or even outperform supervised methods. • We propose a novel ViT-backed try-on method with lightweight preprocessing and a standard ViT without elaborate modification. • .We introduce a two-stage self-supervised training method which can expand paired try-on datasets from separate person images. • Our model can achieve competitive results for both paired try-on and unpaired clothing transfer.