In-Shop Clothes Research Articles

Towards Improved Proxy-Based Deep Metric Learning via Data-Augmented Domain Adaptation

Deep Metric Learning (DML) plays an important role in modern computer vision research, where we learn a distance metric for a set of image representations. Recent DML techniques utilize the proxy to interact with the corresponding image samples in the embedding space. However, existing proxy-based DML methods focus on learning individual proxy-to-sample distance, while the overall distribution of samples and proxies lacks attention. In this paper, we present a novel proxy-based DML framework that focuses on aligning the sample and proxy distributions to improve the efficiency of proxy-based DML losses. Specifically, we propose the Data-Augmented Domain Adaptation (DADA) method to adapt the domain gap between the group of samples and proxies. To the best of our knowledge, we are the first to leverage domain adaptation to boost the performance of proxy-based DML. We show that our method can be easily plugged into existing proxy-based DML losses. Our experiments on benchmarks, including the popular CUB-200-2011, CARS196, Stanford Online Products, and In-Shop Clothes Retrieval, show that our learning algorithm significantly improves the existing proxy losses and achieves superior results compared to the existing methods. The code and Appendix are available at: https://github.com/Noahsark/DADA

OccluMix: Towards De-Occlusion Virtual Try-on by Semantically-Guided Mixup

Image Virtual try-on aims at replacing the cloth on a personal image with a garment image (in-shop clothes), which has attracted increasing attention from the multimedia and computer vision communities. Prior methods successfully preserve the character of clothing images, however, occlusion remains a pernicious effect for realistic virtual try-on. In this work, we first present a comprehensive analysis of the occlusions and categorize them into two aspects: i) Inherent-Occlusion: the ghost of the former cloth still exists in the try-on image; ii) Acquired-Occlusion: the target cloth warps to the unreasonable body part. Based on the in-depth analysis, we find that the occlusions can be simulated by a novel semantically-guided mixup module, which can generate semantic-specific occluded images that work together with the try-on images to facilitate training a de-occlusion try-on (DOC-VTON) framework. Specifically, DOC-VTON first conducts a sharpened semantic parsing on the try-on person. Aided by semantics guidance and pose prior, various complexities of texture are selectively blending with human parts in a copy-and-paste manner. Then, the Generative Module (GM) is utilized to take charge of synthesizing the final try-on image and learning to de-occlusion jointly. In comparison to the state-of-the-art methods, DOC-VTON achieves better perceptual quality by reducing occlusion effects.

Clothes retrieval based on ResNet and cluster triplet loss

Clothes image retrieval is a task that retrieves exact or very similar clothes from a large clothes image gallery according to a given query image. The high deformability of clothes and similar designs make clothes retrieval very difficult. Convolutional neural networks perform well in the feature extraction field, but cannot extract sufficiently fine-grained features for clothes retrieval tasks. Therefore, a model based on ResNet was designed in this article, with several improvements made to it to ensure better feature discrimination. On the other hand, pair-based loss functions that are commonly used in image retrieval suffer from insufficient discrimination ability, difficulty in convergence, and time-consuming during training. Cluster triplet loss function is proposed in this article, which reduces the computational complexity in the training phase and enables the model to have a certain ability to resist noise labels. Experiments are completed on In-shop clothes retrieval benchmark and Consumer-to-shop clothes retrieval benchmark in Deepfashion, and stanford online products (SOP) to supplement. Our method can have Recall@1 improvements of 1.3–1.7% on all three datasets, surpassing the latest state-of-the-art methods.

ST-VTON: Self-supervised vision transformer for image-based virtual try-on

Virtual try-on techniques are in the spotlight due to the promise of enhancing the user experience of online shopping in the apparel industry. Among these technologies, image-based virtual try-on methods have received more attention due to their efficiency and low cost. These methods commonly warp the specific clothing item to the desired shape and then fuse it with the target person image. However, existing approaches demand the laborious paired dataset of clothes and persons. While recent works alleviate the dependence on paired datasets by transferring clothing from one person to another, the pose information is still a rigid demand to guide the synthesis. To solve the above-mentioned problems, we propose ST-VTON, a two-stage self-supervised try-on approach that can synthesize photo-realistic try-on results with only person images and a standard ViT structure. Specifically, we first utilize massive person and clothing images to pre-train a powerful ViT encoder following the MAE approach to enhance the generalization ability and accelerate model training. Secondly, we propose the Clothing Recovery Module (CRM) to reconstruct in-shop clothes from person images and use the paired generated clothing and corresponding person images to train a new ViT decoder while optimizing the pre-trained encoder. Noted that, the CRM, as an independent module, can be used to effectively expand existing paired try-on datasets and enhance other try-on approaches in terms of training data. Extensive experiments conducted on the VITON dataset and UnPaired virtual Try-on (UPT) dataset with both paired and unpaired approaches demonstrate that try-on results rendered by our model can match or even outperform supervised methods. • We propose a novel ViT-backed try-on method with lightweight preprocessing and a standard ViT without elaborate modification. • .We introduce a two-stage self-supervised training method which can expand paired try-on datasets from separate person images. • Our model can achieve competitive results for both paired try-on and unpaired clothing transfer.

Deep Learning with Discriminative Margin Loss for Cross-Domain Consumer-to-Shop Clothes Retrieval.

Consumer-to-shop clothes retrieval refers to the problem of matching photos taken by customers with their counterparts in the shop. Due to some problems, such as a large number of clothing categories, different appearances of clothing items due to different camera angles and shooting conditions, different background environments, and different body postures, the retrieval accuracy of traditional consumer-to-shop models is always low. With advances in convolutional neural networks (CNNs), the accuracy of garment retrieval has been significantly improved. Most approaches addressing this problem use single CNNs in conjunction with a softmax loss function to extract discriminative features. In the fashion domain, negative pairs can have small or large visual differences that make it difficult to minimize intraclass variance and maximize interclass variance with softmax. Margin-based softmax losses such as Additive Margin-Softmax (aka CosFace) improve the discriminative power of the original softmax loss, but since they consider the same margin for the positive and negative pairs, they are not suitable for cross-domain fashion search. In this work, we introduce the cross-domain discriminative margin loss (DML) to deal with the large variability of negative pairs in fashion. DML learns two different margins for positive and negative pairs such that the negative margin is larger than the positive margin, which provides stronger intraclass reduction for negative pairs. The experiments conducted on publicly available fashion datasets DARN and two benchmarks of the DeepFashion dataset—(1) Consumer-to-Shop Clothes Retrieval and (2) InShop Clothes Retrieval—confirm that the proposed loss function not only outperforms the existing loss functions but also achieves the best performance.

SPG-VTON: Semantic Prediction Guidance for Multi-Pose Virtual Try-on

Image-based virtual try-on is challenging in fitting a target in-shop clothes onto a reference person under diverse human poses. Previous works focus on preserving clothing details ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">e.g.,</i> texture, logos, patterns) when transferring desired clothes onto a target person under a fixed pose. However, the performances of existing methods significantly dropped when extending existing methods to multi-pose virtual try-on. In this paper, we propose an end-to-end Semantic Prediction Guidance multi-pose Virtual Try-On Network (SPG-VTON), which can fit the desired clothing into a reference person under arbitrary poses. Specifically, SPG-VTON is composed of three sub-modules. First, a Semantic Prediction Module (SPM) generates the desired semantic map. The predicted semantic map provides more abundant guidance to locate the desired clothing region and produce a coarse try-on image. Second, a Clothes Warping Module (CWM) warps in-shop clothes to the desired shape according to the predicted semantic map and the desired pose. Specifically, we introduce a conductible cycle consistency loss to alleviate the misalignment in the clothing warping process. Third, a Try-on Synthesis Module (TSM) combines the coarse result and the warped clothes to generate the final virtual try-on image, preserving details of the desired clothes and under the desired pose. In addition, we introduce a face identity loss to refine the facial appearance and maintain the identity of the final virtual try-on result at the same time. We evaluate the proposed method on the most massive multi-pose dataset (MPV) and the DeepFashion dataset. The qualitative and quantitative experiments show that SPG-VTON is superior to the state-of-the-art methods and is robust to data noise, including background and accessory changes, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e.</i> , hats and handbags, showing good scalability to the real-world scenario.

Deep Metric Learning With Manifold Class Variability Analysis

In deep metric learning (DML) techniques, understanding both the local and global characteristics of embedding space is essential. However, conventional DML techniques have two limitations as follows: First, Euclidean distance-based metrics never imply global information such as class variability because they only depend on the physical distance of samples. Second, they assume that the embedding space is simply a vector space which cannot represent complex data features. Therefore, we propose a novel loss function which can fully utilize characteristics of embedding space by using discriminant analysis and nonlinear mapping. With theoretical analysis, the superior performance of the proposed method is verified for the fine-grained retrieval datasets such as Cars196, CUB200-2011, Stanford online products, and In-shop clothes. Source code is available at <uri>https://github.com/kdhht2334/MCVA</uri>.

Multi-level Distance Regularization for Deep Metric Learning

We propose a novel distance-based regularization method for deep metric learning called Multi-level Distance Regularization (MDR). MDR explicitly disturbs a learning procedure by regularizing pairwise distances between embedding vectors into multiple levels that represents a degree of similarity between a pair. In the training stage, the model is trained with both MDR and an existing loss function of deep metric learning, simultaneously; the two losses interfere with the objective of each other, and it makes the learning process difficult. Moreover, MDR prevents some examples from being ignored or overly influenced in the learning process. These allow the parameters of the embedding network to be settle on a local optima with better generalization. Without bells and whistles, MDR with simple Triplet loss achieves the-state-of-the-art performance in various benchmark datasets: CUB-200-2011, Cars-196, Stanford Online Products, and In-Shop Clothes Retrieval. We extensively perform ablation studies on its behaviors to show the effectiveness of MDR. By easily adopting our MDR, the previous approaches can be improved in performance and generalization ability.

Toward Realistic Virtual Try-on Through Landmark Guided Shape Matching

Image-based virtual try-on aims to synthesize the customer image with an in-shop clothes image to acquire seamless and natural try-on results, which have attracted increasing attentions. The main procedures of image-based virtual try-on usually consist of clothes image generation and try-on image synthesis, whereas prior arts cannot guarantee satisfying clothes results when facing large geometric changes and complex clothes patterns, which further deteriorates the afterwards try-on results. To address this issue, we propose a novel virtual try-on network based on landmark-guided shape matching (LM-VTON). Specifically, the clothes image generation progressively learns the warped clothes and refined clothes in an end-to-end manner, where we introduce a landmark-based constraint in Thin-Plate Spline (TPS) warping to inject finer deformation constraints around the clothes. The try-on process synthesizes the warped clothes with personal characteristics via a semantic indicator. Qualitative and quantitative experiments on two public datasets validate the superiority of the proposed method, especially for challenging cases such as large geometric changes and complex clothes patterns. Code will be available at https://github.com/lgqfhwy/LM-VTON.

Improving Deep Metric Learning by Divide and Conquer.

Deep metric learning (DML) is a cornerstone of many computer vision applications. It aims at learning a mapping from the input domain to an embedding space, where semantically similar objects are located nearby and dissimilar objects far from another. The target similarity on the training data is defined by user in form of ground-truth class labels. However, while the embedding space learns to mimic the user-provided similarity on the training data, it should also generalize to novel categories not seen during training. Besides user-provided groundtruth training labels, a lot of additional visual factors (such as viewpoint changes or shape peculiarities) exist and imply different notions of similarity between objects, affecting the generalization on the images unseen during training. However, existing approaches usually directly learn a single embedding space on all available training data, struggling to encode all different types of relationships, and do not generalize well. We propose to build a more expressive representation by jointly splitting the embedding space and the data hierarchically into smaller sub-parts. We successively focus on smaller subsets of the training data, reducing its variance and learning a different embedding subspace for each data subset. Moreover, the subspaces are learned jointly to cover not only the intricacies, but the breadth of the data as well. Only after that, we build the final embedding from the subspaces in the conquering stage. The proposed algorithm acts as a transparent wrapper that can be placed around arbitrary existing DML methods. Our approach significantly improves upon the state-of-the-art on image retrieval, clustering, and re-identification tasks evaluated using CUB200-2011, CARS196, Stanford Online Products, In-shop Clothes, and PKU VehicleID datasets.

Virtual Try-on Network With Attribute Transformation and Local Rendering

A virtual try-on network has gradually become a popular topic in recent years. It aims to transfer images of in-shop clothes onto the image of a target person. Owing to the diversity of clothing attributes, developing an image-based virtual try-on network is a complicated task for computers to perform and requires significant effort. Existing methods are unsatisfactory as they cannot preserve the characteristics of the clothes or the target person's identity well, thereby affecting the perception of the generated images; therefore, further research is required. To address this problem, we propose a novel try-on method that combines attribute transformation and local rendering. First, we employ pixel-level semantic segmentation to identify the try-on area and provide implementation conditions for local rendering. Second, we construct a learnable attribute transformation module to complete the try-on task for different attributes. Third, we use a learnable clothing warping module to fit the pose and figure of the target person well and establish a novel loss function, called modified style loss (M-SL), to handle clothes with rich details. Finally, we adopt a local rendering strategy, using which only renders the clothing area to ensure that the details of the non-target area are not lost. Extensive experiments are performed to test our method. The results demonstrate that our method outperforms other state-of-the-art methods.

Incomplete Descriptor Mining With Elastic Loss for Person Re-Identification

In this paper, we propose a novel person Re-ID model, Consecutive Batch DropBlock Network (CBDB-Net), to capture the attentive and robust person descriptor for the person Re-ID task. The CBDB-Net contains two novel designs: the Consecutive Batch DropBlock Module (CBDBM) and the Elastic Loss (EL). In the Consecutive Batch DropBlock Module (CBDBM), we firstly conduct uniform partition on the feature maps. And then, we independently and continuously drop each patch from top to bottom on the feature maps, which can output multiple incomplete feature maps. In the training stage, these multiple incomplete features can better encourage the Re-ID model to capture the robust person descriptor for the Re-ID task. In the Elastic Loss (EL), we design a novel weight control item to help the Re-ID model adaptively balance hard sample pairs and easy sample pairs in the whole training process. Through an extensive set of ablation studies, we verify that the Consecutive Batch DropBlock Module (CBDBM) and the Elastic Loss (EL) each contribute to the performance boosts of CBDB-Net. We demonstrate that our CBDB-Net can achieve the competitive performance on the three standard person Re-ID datasets (the Market-1501, the DukeMTMC-Re-ID, and the CUHK03 dataset), three occluded Person Re-ID datasets (the Occluded DukeMTMC, the Partial-REID, and the Partial iLIDS dataset), and a general image retrieval dataset (In-Shop Clothes Retrieval dataset).

DIABLO: Dictionary-based attention block for deep metric learning

Recent breakthroughs in representation learning of unseen classes and examples have been made in deep metric learning by training at the same time the image representations and a corresponding metric with deep networks. Recent contributions mostly address the training part (loss functions, sampling strategies, etc.), while a few works focus on improving the discriminative power of the image representation. In this paper, we propose DIABLO, a dictionary-based attention method for image embedding. DIABLO produces richer representations by aggregating only visually-related features together while being easier to train than other attention-based methods in deep metric learning. This is experimentally confirmed on four deep metric learning datasets (Cub-200-2011, Cars-196, Stanford Online Products, and In-Shop Clothes Retrieval) for which DIABLO shows state-of-the-art performances.

Hardness-Aware Deep Metric Learning

This paper presents a hardness-aware deep metric learning (HDML) framework for image clustering and retrieval. Most existing deep metric learning methods employ the hard negative mining strategy to alleviate the lack of informative samples for training. However, this mining strategy only utilizes a subset of training data, which may not be enough to characterize the global geometry of the embedding space comprehensively. To address this problem, we perform linear interpolation on embeddings to adaptively manipulate their hardness levels and generate corresponding label-preserving synthetics for recycled training so that information buried in all samples can be fully exploited and the metric is always challenged with proper difficulty. As a single synthetic for each sample may still not be enough to describe the unobserved distributions of the training data which is crucial for the generalization performance, we further extend HDML to generate multiple synthetics for each sample. We propose a randomly hardness-aware deep metric learning (HDML-R) method and an adaptively hardness-aware deep metric learning (HDML-A) method to sample multiple random and adaptive directions, respectively, for hardness-aware synthesis. Since the generated multiple synthetics might not all be useful and adaptive, we propose a synthetic selection method with three criteria for the selection of qualified synthetics that are beneficial to the training of the metric. Extensive experimental results on the widely used CUB-200-2011, Cars196, Stanford Online Products, In-Shop Clothes Retrieval, and VehicleID datasets demonstrate the effectiveness of the proposed framework.

Deep embedding learning with adaptive large margin N-pair loss for image retrieval and clustering

Deep embedding learning becomes more attractive for discriminative feature learning, but many methods still require hard-class mining, which is computationally complex and performance-sensitive. To this end, Adaptive Large Margin N-Pair loss (ALMN) is proposed to address the aforementioned issues. First, the class center is adopted as the anchor point to avoid the difficulty on anchor selection. Then instead of exploring hard example-mining strategy, we introduce the adaptive large margin constraint, where a novel geometrical Virtual Point Generating (VPG) method is proposed to convert a fixed margin into a local-adaptive angular margin, by automatically generating a boundary training sample in feature space. The effectiveness of our method is demonstrated on fine-grained image retrieval and clustering tasks using six popular databases, including CUB, CARS, Flowers, Aircraft, Stanford Online Products and In-Shop Clothes. The results show that the proposed method achieves better performance than other state-of-the-art methods, such as N-Pair loss, Lifted loss and Triplet loss.

Supervised Deep Feature Embedding With Handcrafted Feature.

Image representation methods based on deep convolutional neural networks (CNNs) have achieved the state-of-the-art performance in various computer vision tasks, such as image retrieval and person re-identification. We recognize that more discriminative feature embeddings can be learned with supervised deep metric learning and handcrafted features for image retrieval and similar applications. In this paper, we propose a new supervised deep feature embedding with a handcrafted feature model. To fuse handcrafted feature information into CNNs and realize feature embeddings, a general fusion unit is proposed (called Fusion-Net). We also define a network loss function with image label information to realize supervised deep metric learning. Our extensive experimental results on the Stanford online products' data set and the in-shop clothes retrieval data set demonstrate that our proposed methods outperform the existing state-of-the-art methods of image retrieval by a large margin. Moreover, we also explore the applications of the proposed methods in person re-identification and vehicle re-identification; the experimental results demonstrate both the effectiveness and efficiency of the proposed methods.

Deep Metric Learning with BIER: Boosting Independent Embeddings Robustly.

Learning similarity functions between image pairs with deep neural networks yields highly correlated activations of embeddings. In this work, we show how to improve the robustness of such embeddings by exploiting the independence within ensembles. To this end, we divide the last embedding layer of a deep network into an embedding ensemble and formulate the task of training this ensemble as an online gradient boosting problem. Each learner receives a reweighted training sample from the previous learners. Further, we propose two loss functions which increase the diversity in our ensemble. These loss functions can be applied either for weight initialization or during training. Together, our contributions leverage large embedding sizes more effectively by significantly reducing correlation of the embedding and consequently increase retrieval accuracy of the embedding. Our method works with any differentiable loss function and does not introduce any additional parameters during test time. We evaluate our metric learning method on image retrieval tasks and show that it improves over state-of-the-art methods on the CUB-200-2011, Cars-196, Stanford Online Products, In-Shop Clothes Retrieval and VehicleID datasets. Therefore, our findings suggest that by dividing deep networks at the end into several smaller and diverse networks, we can significantly reduce overfitting.