Product Image Search Research Articles

Multi-View Product Image Search with Deep ConvNets Representations

Multi-view product image queries can improve retrieval performance over single view queries significantly. In this paper, we investigated the performance of deep convolutional neural networks (ConvNets) on multi-view product image search. First, we trained a VGG-like network to learn deep ConvNets representations of product images. Then, we computed the deep ConvNets representations of database and query images and performed single view queries, and multi-view queries using several early and late fusion approaches. We performed extensive experiments on the publicly available Multi-View Object Image Dataset (MVOD 5K) with both clean background queries from the internet and cluttered background queries from a mobile phone. We compared the performance of ConvNets to the classical bag-of-visual-words (BoWs). We concluded that (1) multi-view queries with deep ConvNets representations perform significantly better than single view queries, (2) ConvNets perform much better than BoWs and have room for further improvement, (3) pre-training of ConvNets on a different image dataset with background clutter is needed to obtain good performance on cluttered product image queries obtained with a mobile phone.

Multi-Scale Deep Learning for Product Image Search

Multi-Scale Deep Learning for Product Image Search

Matryoshka Peek: Toward Learning Fine-Grained, Robust, Discriminative Features for Product Search

In sharp contrast to the traditional category/subcategory level image retrieval, product image search aims to find the images containing the exact same product. This is a challenging problem because in addition to being robust under different imaging conditions such as varying viewpoints and illumination changes, the features should also be able to distinguish the specific product among many similar products. Consequently, it is important to utilize a large dataset, containing many product classes, to learn a strongly discriminative representation. Building such a dataset requires laborious manual annotation. Toward learning fine-grained, robust, discriminative features for product image search, we present a novel paradigm that can construct the required dataset without any human annotation. Unlike other fine-grained recognition works that rely on high-quality annotated datasets and are very narrowly focused on a specific object category, our method handles multiple object classes and requires minimum human effort. First, an ImageNet pretrained model is used to generate product clusters. As the original features from ImageNet are not discriminative, the clusters generated by this unsupervised procedure contain much noise. We alleviate noise by explicitly modeling noise distribution and automatically detecting errors during learning. The proposed paradigm is general, requires minimum human efforts, and is applicable to any deep learning task where fine-grained discriminative features are desired. Extensive experiments on the ALISC dataset have demonstrated that our approach is sound and effective, surpassing the baseline GoogleNet model by 15.09%.

Large scale product search with spatial quantization and deep ranking

Product image search aims to retrieve similar product images based on a query image. While deep learning based features work well in retrieving images of the same category (e.g. “searching for T-shirts from all the clothing images”), they perform poorly when retrieving variants of images within the same category (e.g. “searching for uniform of Chelsea football club from all T-shirts image”), since it requires fine-grained matching on image details. In this paper, we present a spatial quantization approach that utilizes spatial pyramid pooling (SPP) and vector of locally aggregated descriptors (VLAD) to extract more discriminative features for instance-aware product search. By using the proposed spatial quantization, spatial information is encoded into the image feature to improve the fine grained product image search. We also present an triplet learning to rank method to finetune the deep learning model on product image search task. Finally, the experiments conducted on a large scale real world dataset provided by Alibaba large-scale image search challenge (ALISC) demonstrate the effectiveness of our method.

A Mobile Product Image Searching System Integrating Speeded Up Robust Features and Local Binary Pattern

In this paper, we present a novel method image retrieval model for mobile product image searching system. For feature extraction, a method integrating Speeded Up Robust Features (SURF) and Local Binary Pattern (LBP) is proposed. SURF is invariant to rotation, scaling, translation and have low calculated cost, so SURF is quite suitable for mobile product image search model. However, SURF is not effective with the noisy images, blur images, illuminated images. Because LBP operator is invariant to changes in illumination and contrast of images, we use LBP to supplement for disadvantages of the SURF features. Our proposed method can improve accuracy and speed of the system. For query, a query model using K-NN Search with vector quantization is used. This model improve performance and reduce the cost of computation of the mobile product image searching system. The experimental results show the feasibility of our proposal model.

Spatially-Constrained Similarity Measurefor Large-Scale Object Retrieval.

One fundamental problem in object retrieval with the bag-of-words model is its lack of spatial information. Although various approaches are proposed to incorporate spatial constraints into the model, most of them are either too strict or too loose so that they are only effective in limited cases. In this paper, a new spatially-constrained similarity measure (SCSM) is proposed to handle object rotation, scaling, view point change and appearance deformation. The similarity measure can be efficiently calculated by a voting-based method using inverted files. During the retrieval process, object localization in the database images can also be simultaneously achieved using SCSM without post-processing. Furthermore, based on the retrieval and localization results of SCSM, we introduce a novel and robust re-ranking method with the k-nearest neighbors of the query for automatically refining the initial search results. Extensive performance evaluations on six public data sets show that SCSM significantly outperforms other spatial models including RANSAC-based spatial verification, while k-NN re-ranking outperforms most state-of-the-art approaches using query expansion. We also adapted SCSM for mobile product image search with an iterative algorithm to simultaneously extract the product instance from the mobile query image, identify the instance, and retrieve visually similar product images. Experiments on two product image search data sets show that our approach can robustly localize and extract the product in the query image, and hence drastically improve the retrieval accuracy over baseline methods.

A new algorithm for product image search based on salient edge characterization

Visually assisted product image search has gained increasing popularity because of its capability to greatly improve end users' e‐commerce shopping experiences. Different from general‐purpose content‐based image retrieval (CBIR) applications, the specific goal of product image search is to retrieve and rank relevant products from a large‐scale product database to visually assist a user's online shopping experience. In this paper, we explore the problem of product image search through salient edge characterization and analysis, for which we propose a novel image search method coupled with an interactive user region‐of‐interest indication function. Given a product image, the proposed approach first extracts an edge map, based on which contour curves are further extracted. We then segment the extracted contours into fragments according to the detected contour corners. After that, a set of salient edge elements is extracted from each product image. Based on salient edge elements matching and similarity evaluation, the method derives a new pairwise image similarity estimate. Using the new image similarity, we can then retrieve product images. To evaluate the performance of our algorithm, we conducted 120 sessions of querying experiments on a data set comprised of around 13k product images collected from multiple, real‐world e‐commerce websites. We compared the performance of the proposed method with that of a bag‐of‐words method (Philbin, Chum, Isard, Sivic, & Zisserman, 2008) and a Pyramid Histogram of Orientated Gradients (PHOG) method (Bosch, Zisserman, & Munoz, 2007). Experimental results demonstrate that the proposed method improves the performance of example‐based product image retrieval.

When Amazon Meets Google

Product visualization is able to help users easily get knowledge about the visual appearance of a product. It is useful in many application and commercialization scenarios. However, the existing product image search on e-commerce Web sites or general search engines usually get insufficient search results or return images that are redundant and not relevant enough. In this article, we present a novel product visualization approach that automatically collects a set of diverse and relevant product images by exploring multiple Web sources. Our approach simultaneously leverages Amazon and Google image search engines, which represent domain-specific knowledge resource and general Web information collection, respectively. We propose a conditional clustering approach that is formulated as an affinity propagation problem regarding the Amazon examples as information prior. The ranking information of Google image search results is also explored. In this way, a set of exemplars can be found from the Google search results and they are provided together with the Amazon example images for product visualization. Experiments demonstrate the feasibility and effectiveness of our approach.