Mobile Visual Search Research Articles

Consumer Usage of Mobile Visual Search in China

As an emerging information retrieval technology, mobile visual search (MVS) has considerable development potential and broad market application prospects. Although many studies have explored user behaviors with various information technology artifacts, few have specifically investigated consumers' MVS intentions and behaviors. For filling the gap, this study is to integrate perceived contextual offer (PCO) and implementation intention into the theoretical model of UTAUT2, aiming to explore potential predictors of Chinese consumers' MVS usage intention and behavior. The results showed that consumers' perceptions of contextual offer, performance expectancy, hedonic motivation, and their using habit had significant effects on consumers' MVS usage intention. Habit, usage intention, and facilitating conditions showed significant predictive power for consumers' MVS implementation intention and their further usage behavior. Additionally, habit and usage intention indirectly influenced consumers' MVS usage through the complementary mediation of implementation intention.

A Study on the Shopping Life through Mobile Visual Search

Purpose - To examine the influence of mobile visual search as a strategic technology service on consumer perceived economic value and customer commitments, which in turn affect consumer's usage intention of mobile visual search. This study also explores the moderating effect of different levels of consumer online shopping orientation. Design/methodology/approach - One-by-one open-ended in-depth interview was first undertaken to 15 Korean consumers to figure the features of mobile visual search. Then a conceptual model was built to verify the hypotheses that indicate the impact of mobile visual search on consumer perceived economic value and customer commitment, which further influence consumer’s usage intention. Findings - The results show Convenience, Information quality, Personalization, Text-free search interface design and Visual communication of mobile visual search positively influence consumer perceived economic value and customer commitment and in turn positively affect consumer's usage intention. Moreover, the different levels of consumer online shopping orientation also found to have different effects on consumers’ perception and behavior of using mobile visual search in online fashion shopping. Research implications or Originality - The present study verified that mobile visual search is a service tool that consumers want to use in the online fashion shopping journey since it provides economic benefits.

A Novel Deep Learning-Based Visual Search Engine in Digital Marketing for Tourism E-Commerce Platforms

Visual search technology, because of its convenience and high efficiency, is widely used by major tourism e-commerce platforms in product search functions. This study introduces an innovative visual search engine model, namely CLIP-ItP, aiming to thoroughly explore the application potential of visual search in tourism e-commerce. The model is an extension of the CLIP (contrastive language-image pre-training) framework and is developed through three pivotal stages. Firstly, by training an image feature extractor and a linear model, the visual search engine labels images, establishing an experimental visual search engine. Secondly, CLIP-ItP jointly trains multiple text and image encoders, facilitating the integration of multimodal data, including product image labels, categories, names, and attributes. Finally, leveraging user-uploaded images and jointly selected product attributes, CLIP-ItP provides personalized top-k product recommendations.

Visual Data Mining: Digital Visual Activities In Network Society

This article describes scientific discoveries that drive the phenomenon of visual cultural communication media in networked societies. Many types of research on visual culture have focused more on visual arts and mass media. However, few have yet addressed the problem of people's digital activities in searching for visual information on the Internet ( Visual search engine as a resource for visual data mining The amount of digital information, including visual information, is increasing exponentially.This research uses a literature study method with a qualitative approach. The primary data used in this study are studies on visual culture and visual search engines published in international peer-reviewed journals. Analysis of digital behavior and visual culture in a networked society is described as a research topic in previous research results. The conclusions drawn indicate that there is still a relatively deep and broad analysis of the study of visual culture when using digital communication media. There are few research studies on visual culture and digital activities in connected communities, especially people's visual information search behavior.

Mobile visual search algorithm based on improved VGG-F and hash with application in IoT

Mobile visual search algorithm based on improved VGG-F and hash with application in IoT

Constructing a mobile visual search framework for Dunhuang murals based on fine-tuned CNN and ontology semantic distance

PurposeDunhuang murals are rich in cultural and artistic value. The purpose of this paper is to construct a novel mobile visual search (MVS) framework for Dunhuang murals, enabling users to efficiently search for similar, relevant and diversified images.Design/methodology/approachThe convolutional neural network (CNN) model is fine-tuned in the data set of Dunhuang murals. Image features are extracted through the fine-tuned CNN model, and the similarities between different candidate images and the query image are calculated by the dot product. Then, the candidate images are sorted by similarity, and semantic labels are extracted from the most similar image. Ontology semantic distance (OSD) is proposed to match relevant images using semantic labels. Furthermore, the improved DivScore is introduced to diversify search results.FindingsThe results illustrate that the fine-tuned ResNet152 is the best choice to search for similar images at the visual feature level, and OSD is the effective method to search for the relevant images at the semantic level. After re-ranking based on DivScore, the diversification of search results is improved.Originality/valueThis study collects and builds the Dunhuang mural data set and proposes an effective MVS framework for Dunhuang murals to protect and inherit Dunhuang cultural heritage. Similar, relevant and diversified Dunhuang murals are searched to meet different demands.

Mobile visual search model for Dunhuang murals in the smart library

PurposeThe purpose of this paper is to build a mobile visual search service system for the protection of Dunhuang cultural heritage in the smart library. A novel mobile visual search model for Dunhuang murals is proposed to help users acquire rich knowledge and services conveniently.Design/methodology/approachFirst, local and global features of images are extracted, and the visual dictionary is generated by the k-means clustering. Second, the mobile visual search model based on the bag-of-words (BOW) and multiple semantic associations is constructed. Third, the mobile visual search service system of the smart library is designed in the cloud environment. Furthermore, Dunhuang mural images are collected to verify this model.FindingsThe findings reveal that the BOW_SIFT_HSV_MSA model has better search performance for Dunhuang mural images when the scale-invariant feature transform (SIFT) and the hue, saturation and value (HSV) are used to extract local and global features of the images. Compared with different methods, this model is the most effective way to search images with the semantic association in the topic, time and space dimensions.Research limitations/implicationsDunhuang mural image set is a part of the vast resources stored in the smart library, and the fine-grained semantic labels could be applied to meet diverse search needs.Originality/valueThe mobile visual search service system is constructed to provide users with Dunhuang cultural services in the smart library. A novel mobile visual search model based on BOW and multiple semantic associations is proposed. This study can also provide references for the protection and utilization of other cultural heritages.

Mobile visual search algorithm based on improved VGG-F and hash with application in IOT

Mobile visual search algorithm based on improved VGG-F and hash with application in IOT

HoloCities: A Shared Reality application for Collaborative Tourism

Communication is a key arena for shared-reality application: our mixed reality application is developed on Microsoft Hololens and it has been designed to provide new engaging ways to discover the city using augmented reality. Most augmented reality tourism applications isolate the user, therefore, this application has been made multiplayer and collaborative to encourage shared experiences and socialization. In this project two scenarios are described: In the first scenario there is a real guide that exposes to the group, each equipped with Head Mounted Display, the peculiarities of the monument visited. Each user share the 3D models manipulated by the guide that can be in the same room or it can be a remote guide leveraging on the 5G network low latency. Moreover, the guide could highlight and label to convey relevant information about the objects. While in the second scenario there is no real guide, so the application automatically recognizes the framed monument thanks to a visual search engine running on a 5G-ready infrastructure and shows shared information to the whole group of tourists. Moreover, it uses the huge amount of multimedia data stored in the archives of the Italian Public Broadcaster RAI. Thanks to the fact that the new HMDs are able to track the position and orientation of the user, we have created a menu capable of following the user while exploring the city. Moreover, the shared-interaction with virtual objects is performed by hands or vocal commands. Both scenarios allow users to visit a city in a new and intelligent way, promoting participation and collaboration with others and with the guide, unlike previous approaches that involve only one tourist using the information provided by the app.

Analysis of Pelvic Organ Prolapse-related Content on Pinterest.

Our study is a cross-sectional, qualitative study analyzing pelvic organ prolapse-related content on Pinterest, a social media platform that also functions as a visual search engine. Primary outcomes assessed include quality, understandability, and actionability of content. Secondary outcomes include levels of misinformation and commercial bias. Our study further characterizes sources of prolapse content and treatment modalities discussed. A Pinterest search was conducted using the term "pelvic organ prolapse," analyzing the first 100 relevant results. We examined the publisher of each pin and the writer of linked content. Using validated tools including the Patient Education Materials Assessment Tool and DISCERN criteria, we assessed understandability and actionability of content. We further compared content to professional guidelines for discussing and treating prolapse, evaluating for misinformation. We also evaluated for commercial bias and subjectively rated pin quality. Of pins evaluated, low understandability was found in 16, low actionability in 30, and low-to-moderate quality in 69. The most common publisher of pins was health and wellness groups, and the most common writer of content was physical therapists. Pelvic floor muscle training (PFMT) is the treatment modality discussed most frequently on Pinterest. Pinterest content discussing pelvic organ prolapse shows good levels of understandability for patients at all levels of health literacy. However, the number of pins showing poor actionability raises concern that content may not encourage readers to take direct steps toward prolapse assessment and management. In all, a majority of prolapse-related content on Pinterest was of low-to-moderate quality.

Improvement of Knowledge Search Method for Numeral Terms in Power System Based on Multi-source Fusion

In the process of energy and power research, it is often necessary to find numeral data information from multiple ways based on searching words, so the domain information can better analyzed and understood, visually discover hidden relationships between information and ultimately improve search efficiency and experience. The article designs a set of numeral entity discovery methods based on NLP and improves a visual search engine. Using CRF for named entity pre-processing extraction, using TF-IDF, etc. to complete multi-source information combination search and information mining method establishment, the power system energy database is used as the information source of the search engine, enabling users to locate the document data of the searching words in the energy database and the context information associated with them quickly, generate summary information, extract the central words based on this, and image the information in the form of word cloud and chart demonstration. Experiments have shown that the searching result are of interest to users, quickly locate the literature and context-related information containing the numeral hotspots, and the search recall rate is significantly improved.

Visual Arts Search on Mobile Devices

Visual arts, especially paintings, appear everywhere in our daily lives. They are not only liked by art lovers but also by ordinary people, both of whom are curious about the stories behind these artworks and also interested in exploring related artworks. Among various methods, the mobile visual search has its merits in providing an alternative solution to text and voice searches, which are not always applicable. Mobile visual search for visual arts is far more challenging than the general image visual search. Conventionally, visual search, such as searching products and plant, focuses on locating images containing similar objects. Hence, approaches are designed to locate objects and extract scale-invariant features from distorted photos that are captured by the mobile camera. However, the objects are only part of the visual art piece; the background and the painting style are both important factors that are not considered in the conventional approaches. In this article, an empirical investigation is conducted to study issues in photos taken by mobile cameras, such as orientation variance and motion blur, and how they influence the results of the mobile visual arts search. Based on the empirical investigation results, a photo-rectification pipeline is designed to rectify the photos into perfect images for feature extraction. A new method is proposed to learn high discriminative features for visual arts, which considers both the content information and style information in visual arts. Apart from conducting solid experiments, a real-world system is built to prove the effectiveness of the proposed methods. To the best of our knowledge, this is the first article to solve problems for visual arts search on mobile devices.

Codebook-Free Compact Descriptor for Scalable Visual Search

The MPEG compact descriptors for visual search (CDVS) is a standard toward image matching and retrieval. To achieve high retrieval accuracy over a large scale image/video dataset, recent research efforts have demonstrated that employing extremely high-dimensional descriptors such as the Fisher vector (FV) and the vector of locally aggregated descriptors (VLAD) can yield good performance. Since the FV (or VLAD) possesses high discriminability but small visual vocabulary, it has been adopted by CDVS to construct a global compact descriptor. In this paper, we study the development of global compact descriptors in the completed CDVS standard and the emerging compact descriptors for video analysis (CDVA) standard, in which we formulate the FV (or VLAD) compression as a resource-constrained optimization problem. Accordingly, we propose a codebook-free aggregation method via dual selection to generate a global compact visual descriptor, which supports fast and accurate feature matching free of large visual codebooks, fulfilling the low memory requirement of mobile visual search at significantly reduced latency. Specifically, we investigate both sample-specific Gaussian component redundancy and bit dependency within a binary aggregated descriptor to produce compact binary codes. Our technique contributes to the scalable compressed Fisher vector (SCFV) adopted by the CDVS standard. Moreover, the SCFV descriptor is currently serving as the frame-level hand-crafted video feature, which inspires the inheritance of CDVS descriptors for the emerging CDVA standard. Furthermore, we investigate the positive complementary effect of our standard compliant compact descriptor and deep learning based features extracted from convolutional neural networks with significant mean average precision gains. Extensive evaluation over benchmark databases shows the significant merits of the codebook-free binary codes for scalable visual search.

Local Feature Reliability Measure Consistent with Match Conditions for Mobile Visual Search

Local Feature Reliability Measure Consistent with Match Conditions for Mobile Visual Search

DeepProduct

Features extracted by deep networks have been popular in many visual search tasks. This article studies deep network structures and training schemes for mobile visual search. The goal is to learn an effective yet portable feature representation that is suitable for bridging the domain gap between mobile user photos and (mostly) professionally taken product images while keeping the computational cost acceptable for mobile-based applications. The technical contributions are twofold. First, we propose an alternative of the contrastive loss popularly used for training deep Siamese networks, namely robust contrastive loss, where we relax the penalty on some positive and negative pairs to alleviate overfitting. Second, a simple multitask fine-tuning scheme is leveraged to train the network, which not only utilizes knowledge from the provided training photo pairs but also harnesses additional information from the large ImageNet dataset to regularize the fine-tuning process. Extensive experiments on challenging real-world datasets demonstrate that both the robust contrastive loss and the multitask fine-tuning scheme are effective, leading to very promising results with a time cost suitable for mobile product search scenarios.

The Replica Project: Building a visual search engine for art historians

From the time of prehistoric etchings on the walls of the Lascaux cave to the present day, people have always been creating art. With millions of artistic artifacts filling museums, churches, cultural institutions, and private collections across the globe, connecting to our shared cultural and artistic past is no longer impossible.

Mobile Visual Search Based on Histogram Matching and Zone Weight Learning

In this paper, we propose a novel image retrieval algorithm for mobile visual search. At first, a short visual codebook is generated based on the descriptor database to represent the statistical information of the dataset. Then, an accurate local descriptor similarity score is computed by merging the tf-idf weighted histogram matching and the weighting strategy in compact descriptors for visual search (CDVS). At last, both the global descriptor matching score and the local descriptor similarity score are summed up to rerank the retrieval results according to the learned zone weights. The results show that the proposed approach outperforms the state-of-the-art image retrieval method in CDVS.

Feature Extracting from Video Encoded for Searching Using Improved Methods

There is a need for the transportation of visual content from a particular node to a computation unit, on a network limited by bandwidth for distributed visual analysis applications, like mobile visual search or Visual Sensor Networks (VSNs). The proposed system uses the video compression technique to transfer information over a communication network. The system is used for key point detection, descriptor calculation and feature matching. The proposed system provides key points to the server as side data to retrieve only the relevant points from the encoded video. In the proposed encoding approach, the User chooses the video from the dataset, extracts frames from the video and then extracts the key point of the selected frame and encodes the string or the key point on the extracted frame to then reconstruct or compress the video and upload it on the server. For decoding, whenever the user enters the search query on the browser, the server receives the search query. The server then selects the video, which is stored on the server, and extracts frames from the video, then key point or encoded data from the selected frame using decoding, and then compares the search query and the decoded data. If it is true, the video is sent to the user. The results show that the proposed approach offers significantly improved feature matching and image retrieval performance at given bitrates.

Dehashing: Server-Side Context-Aware Feature Reconstruction for Mobile Visual Search

Due to the prevalence of mobile devices, mobile search becomes a more convenient way than desktop search. Different from the traditional desktop search, mobile visual search needs more consideration for the limited resources on mobile devices (e.g., bandwidth, computing power, and memory consumption). The state-of-the-art approaches show that the bag-of-words (BoW) model is robust for image and video retrieval; however, the large vocabulary tree might not be able to be loaded on the mobile device. We observe that recent works mainly focus on designing compact feature representations on mobile devices for bandwidth-limited network (e.g., 3G) and directly adopt feature matching on remote servers (cloud). However, the compact (binary) representation might fail to retrieve target objects (images and videos). Based on the hashed binary codes, we propose a dehashing process that reconstructs BoW by leveraging the computing power of remote servers. To mitigate the information loss from binary codes, we further utilize contextual information (e.g., GPS) to reconstruct a context-aware BoW for better retrieval results. Experiment results show that the proposed method can achieve competitive retrieval accuracy as BoW while only transmitting few bits from mobile devices.

Paper] Adaptive Substring Extraction and Modified Local NBNN Scoring for Binary Feature-based Local Mobile Visual Search without False Positives

Paper] Adaptive Substring Extraction and Modified Local NBNN Scoring for Binary Feature-based Local Mobile Visual Search without False Positives