Click-through Data Research Articles

Automatically generating effective search queries directly from community question-answering questions for finding related questions

Community Question-Answering platforms are massive knowledge bases of questions and answers pairs produced by their members. In other to provide a vibrant service, they are compelled to provide answers to new posted questions as soon as possible. However, since their dynamic requires their own users to answer questions, there is an inherent delay between posting time and the arrival of good answers. In fact, many of these new questions might be already asked and satisfactorily answered in the past. Ergo, one of the pressing needs of these services is capitalizing on good answers given to related resolved questions across their large-scale knowledge base. To that end, current approaches have studied the effectiveness of human-generated web queries across search logs in fetching related questions and potential good answers from these community archives. However, this kind of strategy is not suitable for questions without click-through data, in particular those recently posted, limiting their capability of providing them with real-time answers.In this paper, we propose an approach to find related questions across the cQA knowledge base, which automatically generate effective search strings directly from question titles and bodies. In so doing, we automatically construct a massive corpus of related questions on top of the relationships yielded by their click-through graph, and generated candidate queries by inspecting dependency paths across the title and body of each question afterwards. Then, we utilize this corpus for automatically annotating the retrieval power of each of these candidates. With this labelled corpus, we study the effectiveness of several learning to rank models enriched with assorted linguistically-motivated properties. Thus deducing the linguistic structure of automatically generated search strings that are effective in finding related questions. Since these models are inferred solely from each question itself, they can be used when search log data (i.e., web queries) is unavailable.Overall, our experiments underline the effectiveness of our approach, in particular our outcomes indicate that named entity recognition is instrumental in structuring and recognizing 2–5 terms effective queries. Furthermore, we carry out experiments considering and ignoring question bodies, and we show that profiting only from question titles is more promising, but most effective queries are harder to detect. Conversely, adding question bodies makes the retrieval of past related questions noisier, but their content helps to generalize models capable of identifying more effective candidates.

Marketing @ Microsoft: The Value of Customer Perception

Quantifying the efficacy of marketing is an age-old challenge. As John Wanamaker said a century ago, “Half the money I spend on advertising is wasted; the trouble is I don't know which half.” The big difference today, however, is that the Internet enables detailed tracking of marketing campaigns in real time, or near time. Exemplifies how to leverage the Internet to dramatically improve the efficacy of marketing. Centers upon the Microsoft Security Guidance marketing campaign, which was designed to change IT professionals' perception of Microsoft's software product security. The integrated marketing campaign involved print media, analyst relations, and online advertising. The advertising was designed to drive IT professionals to a Web site on security guidance, then sign them up for free in-person security training classes. Illustrates two important best practices for marketing in the Internet age: first, the campaign was designed to be measured, and second, agility was specifically designed into the campaign. In addition to tracking weekly click-through data from the print and online advertising, the campaign also used online pop-up customer perception surveys. Analyzing the click-though data, Microsoft realized it had a problem at the end of the first week of the campaign–there were far fewer signups for the training sessions than anticipated. By the end of the second week the campaign was changed, resulting in a huge improvement in efficacy. Creates a scorecard illustrating the pros and cons of the Microsoft approach compared to a more traditional campaign. Illustrates how, rather than creating big-bang campaigns, high-performing marketing organizations today are continually experimenting. They build flexibility into campaigns and design them to be measured.To learn how to leverage the Internet in marketing campaigns, analyze click-through data and online survey results acquired in near time, and learn how it is used to fine tune and dramatically improve a campaign. Furthermore, illustrates how nonfinancial metrics can be used to quantify marketing efficacy.

Heterogeneous Manifold Ranking for Image Retrieval

Graph-based ranking models, such as manifold ranking (MR), have been widely used in various image retrieval applications. To further improve such models, a current trend is to fuse the ranking results from multiple feature sets. Most of existing methods mainly concentrate on fusing the homogeneous feature sets derived from a single information channel, like the multiple modalities of image visual content, but little is known in fusing such heterogeneous feature sets derived from multiple information channels as the click-through data associated with images and their visual content. The primary challenge is how to effectively exploit the complementary properties of the heterogeneous feature sets. Another tough issue is the low-quality nature of the click-through data, which makes the exploration of such complementary properties more difficult. In this paper, we propose a heterogeneous MR (HMR) model, in which a couple of graphs built on the click and visual feature sets are fused to simultaneously encode the image ranking results. Specifically, our HMR model applies different solutions to fuse the heterogeneous feature sets in terms of whether the relevance feedback mechanism is available or not. In addition, we develop a click refinement technique to address the noiseness and sparseness problems inherent in the click-through data. Concretely, it prunes the inaccurate clicks from the click-through data using a neighbor voting strategy, and then enriches the pruned data with novel yet accurate clicks based on a novel collaborative filtering (CF) approach, which is devised by integrating the merits of three popularly used CF methods, thus called Tri-CF algorithm. Extensive experiments on the tasks of click refinement and image retrieval demonstrate the superior performance of the proposed algorithms over several representative methods, especially when the click-through data is highly noisy and sparse.

Click-boosted graph ranking for image retrieval

Graph ranking is one popular and successful technique for image retrieval, but its effectiveness is often limited by the well-known semantic gap. To bridge this gap, one of the current trends is to leverage the click-through data associated with images to facilitate the graph-based image ranking. However, the sparse and noisy properties of the image click-through data make the exploration of such resource challenging. Towards this end, this paper propose a novel click-boosted graph ranking framework for image retrieval, which consists of two coupled components. Concretely, the first one is a click predictor based on matrix factorization with visual regularization, in order to alleviate the sparseness of the click-through data. The second component is a soft-label graph ranker that conducts the image ranking by using the enriched click-through data noise-tolerantly. Extensive experiments for the tasks of click predicting and image ranking validate the effectiveness of the proposed methods in comparison to several existing approaches.

Automatic Generation of Social Event Storyboard From Image Click-Through Data

Recent studies have shown that a noticeable percentage of Web search traffic is about social events. While traditional Web sites can only show human-edited events, in this paper, we present a novel system to automatically detect events from search log data and generate storyboards where the events are arranged chronologically. We chose image search log as the resource for event mining, as search logs can directly reflect people’s interests. To discover events from log data, we present a smooth nonnegative matrix factorization framework, which combines the information of query semantics, temporal correlations, search logs, and time continuity. Moreover, we consider the time factor to be an important element, since different events will develop in different time tendencies. In addition, to provide a media-rich and visually appealing storyboard, each event is associated with a set of representative photos arranged along a timeline. These relevant photos are automatically selected from image search results by analyzing image content features. We use celebrities as our test domain, which takes a large percentage of image search traffic. Experiments consisting of Web search traffic on 200 celebrities, for a period of six months, show very encouraging results compared with handcrafted editorial storyboards.

Learning to rank with click-through features in a reinforcement learning framework

Purpose Learning to rank algorithms inherently faces many challenges. The most important challenges could be listed as high-dimensionality of the training data, the dynamic nature of Web information resources and lack of click-through data. High dimensionality of the training data affects effectiveness and efficiency of learning algorithms. Besides, most of learning to rank benchmark datasets do not include click-through data as a very rich source of information about the search behavior of users while dealing with the ranked lists of search results. To deal with these limitations, this paper aims to introduce a novel learning to rank algorithm by using a set of complex click-through features in a reinforcement learning (RL) model. These features are calculated from the existing click-through information in the data set or even from data sets without any explicit click-through information. Design/methodology/approach The proposed ranking algorithm (QRC-Rank) applies RL techniques on a set of calculated click-through features. QRC-Rank is as a two-steps process. In the first step, Transformation phase, a compact benchmark data set is created which contains a set of click-through features. These feature are calculated from the original click-through information available in the data set and constitute a compact representation of click-through information. To find most effective click-through feature, a number of scenarios are investigated. The second phase is Model-Generation, in which a RL model is built to rank the documents. This model is created by applying temporal difference learning methods such as Q-Learning and SARSA. Findings The proposed learning to rank method, QRC-rank, is evaluated on WCL2R and LETOR4.0 data sets. Experimental results demonstrate that QRC-Rank outperforms the state-of-the-art learning to rank methods such as SVMRank, RankBoost, ListNet and AdaRank based on the precision and normalized discount cumulative gain evaluation criteria. The use of the click-through features calculated from the training data set is a major contributor to the performance of the system. Originality/value In this paper, we have demonstrated the viability of the proposed features that provide a compact representation for the click through data in a learning to rank application. These compact click-through features are calculated from the original features of the learning to rank benchmark data set. In addition, a Markov Decision Process model is proposed for the learning to rank problem using RL, including the sets of states, actions, rewarding strategy and the transition function.

Direct-to-Consumer Advertising and Online Search

Beginning in 1997, the Food and Drug Administration (FDA) allowed television advertisements to make major statements about a prescription drug, while referring to detailed drug information on the internet (FDA 1997; 2015). The hope was that consumers would seek additional information online to fully understand the risks and benefits of taking the medication. To better understand the effects of the policy, we analyze direct-to-consumer advertising (DTCA) and search engine click-through data on a set of drugs over a three-year period. Regression analysis shows that advertising on a prescription drug serves to increase the frequency of online search and subsequent clicks for that drug, as well as search for other drugs in the same class. We find the relationship between DTCA and search is stronger for younger drugs, for those drugs that treat acute conditions, those drugs that are less likely to be covered by insurance, and those whose searcher population tends to be older. These findings suggest that DTCA motivates consumers to search online for drug information, but the magnitude of the effect is heterogeneous and potentially associated with clicks on websites that are more promotional in nature.

Web Image Search Re-Ranking With Click-Based Similarity and Typicality.

In image search re-ranking, besides the well-known semantic gap, intent gap, which is the gap between the representation of users' query/demand and the real intent of the users, is becoming a major problem restricting the development of image retrieval. To reduce human effects, in this paper, we use image click-through data, which can be viewed as the implicit feedback from users, to help overcome the intention gap, and further improve the image search performance. Generally, the hypothesis-visually similar images should be close in a ranking list-and the strategy-images with higher relevance should be ranked higher than others-are widely accepted. To obtain satisfying search results, thus, image similarity and the level of relevance typicality are determinate factors correspondingly. However, when measuring image similarity and typicality, conventional re-ranking approaches only consider visual information and initial ranks of images, while overlooking the influence of click-through data. This paper presents a novel re-ranking approach, named spectral clustering re-ranking with click-based similarity and typicality. First, to learn an appropriate similarity measurement, we propose click-based multi-feature similarity learning algorithm, which conducts metric learning based on click-based triplets selection, and integrates multiple features into a unified similarity space via multiple kernel learning. Then, based on the learnt click-based image similarity measure, we conduct spectral clustering to group visually and semantically similar images into same clusters, and get the final re-rank list by calculating click-based clusters typicality and within-clusters click-based image typicality in descending order. Our experiments conducted on two real-world query-image data sets with diverse representative queries show that our proposed re-ranking approach can significantly improve initial search results, and outperform several existing re-ranking approaches.

P125 “I googled it…”: what is recommended online for the management of vulvovaginal candidiasis?

<h3>Background</h3> Recently our centre encountered women reporting self-treatment of candidiasis with intravaginal applications of foodstuffs including garlic, vinegar and yoghurt. All patients had a unifying factor of reporting “googling” their therapy. <h3>Aim</h3> To establish which candidiasis management strategies female patients are most likely to encounter when searching via Google. <h3>Method</h3> Search history data was collated from Google Trends to identify the ten most popular search terms related to candidiasis in the UK between 15/03/15–06/03/16. These terms, along with term “thrush”, were assessed totalling 11 Google searches. All websites on the initial results page for each search term were accessed to review recommended therapies. Click-through data suggests the vast majority of Google users (&gt;90%) select their chosen website from this first results page. <h3>Results</h3> 116 search results included 97 (83.6%) advising women about vulvovaginal candidiasis. 96/97 (99%) recommended imidazole therapy first line, all reassuringly advising against oral therapy in pregnancy. Patients were recommended to seek treatment via a pharmacy (72, 74.2%) or their GP (54, 55.7%) rather than attending a genitourinary service (12, 12.2%). The recommendation of natural yoghurt for symptomatic relief was frequent (40, 41.2%), more than using emollients or soap substitutes (27, 27.8%). Unfounded treatments including eating probiotic yoghurts (9, 9.3%), vinegar (8, 8.2%), and treatment of sexual partners (8, 8.2%) were encountered. <h3>Conclusions</h3> Sensible evidence-based advice is the most prevalent online for vulvovaginal candidiasis. However a number of poorly evidenced therapies are encouraged. This information should be discussed and appropriately challenged during routine management of vulvovaginal candidiasis.

Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval

This paper develops a model that addresses sentence embedding, a hot topic in current natural language processing research, using recurrent neural networks with Long Short-Term Memory (LSTM) cells. Due to its ability to capture long term memory, the LSTM-RNN accumulates increasingly richer information as it goes through the sentence, and when it reaches the last word, the hidden layer of the network provides a semantic representation of the whole sentence. In this paper, the LSTM-RNN is trained in a weakly supervised manner on user click-through data logged by a commercial web search engine. Visualization and analysis are performed to understand how the embedding process works. The model is found to automatically attenuate the unimportant words and detects the salient keywords in the sentence. Furthermore, these detected keywords are found to automatically activate different cells of the LSTM-RNN, where words belonging to a similar topic activate the same cell. As a semantic representation of the sentence, the embedding vector can be used in many different applications. These automatic keyword detection and topic allocation abilities enabled by the LSTM-RNN allow the network to perform document retrieval, a difficult language processing task, where the similarity between the query and documents can be measured by the distance between their corresponding sentence embedding vectors computed by the LSTM-RNN. On a web search task, the LSTM-RNN embedding is shown to significantly outperform several existing state of the art methods. We emphasize that the proposed model generates sentence embedding vectors that are specially useful for web document retrieval tasks. A comparison with a well known general sentence embedding method, the Paragraph Vector, is performed. The results show that the proposed method in this paper significantly outperforms it for web document retrieval task.

Online training of concept detectors for image retrieval using streaming clickthrough data

Clickthrough data from image search engines provide a massive and continuously generated source of user feedback that can be used to model how the search engine users perceive the visual content. Image clickthrough data have been successfully used to build concept detectors without any manual annotation effort, although the generated annotations suffer from labeling errors. Previous research efforts therefore focused on modeling the sample uncertainty in order to improve concept detector effectiveness. In this paper, we study the problem in an online learning setting using streaming clickthrough data where each click is treated seperately when it becomes available; the concept detector model is therefore continuously updated without batch retraining. We argue that sample uncertainty can be incorporated in the online learning setting by exploiting the repetitions of incoming clicks at the classifier level, where these act as an implicit importance weighting mechanism. For online concept detector training we use the LASVM algorithm. The inferred weighting approximates the solution of batch trained concept detectors using weighted SVM variants that are known to achieve improved performance and high robustness to noise compared to the standard SVM. Furthermore, we evaluate methods for selecting negative samples using a small number of candidates sampled locally from the incoming stream of clicks. The selection criteria aim at drastically improving the performance and the convergence speed of the online concept detectors. To validate our arguments we conduct experiments for 30 concepts on the Clickture-Lite dataset. The experimental results demonstrate that: (a) the proposed online approach produces effective and noise resilient concept detectors that can take advantage of streaming clickthrough data and achieve performance that is equivalent to Fuzzy SVM concept detectors with sample weights and 78.6% improved compared to standard SVM concept detectors; and (b) the selection criteria speed up convergence and improve effectiveness compared to random negative sampling even for a small number of available clicks (up to 134% after 100 clicks).

Learning to rank: new approach with the layered multi-population genetic programming on click-through features

Users' click-through data is a valuable source of information about the performance of Web search engines, but it is included in few datasets for learning to rank. In this paper, inspired by the click-through data model, a novel approach is proposed for extracting the implicit user feedback from evidence embedded in benchmarking datasets. This process outputs a set of new features, named click-through features. Generated click-through features are used in a layered multi-population genetic programming framework to find the best possible ranking functions. The layered multi-population genetic programming framework is fast and provides more extensive search capability compared to the traditional genetic programming approaches. The performance of the proposed ranking generation framework is investigated both in the presence and in the absence of explicit click-through data in the utilized benchmark datasets. The experimental results show that click-through features can be efficiently extracted in both cases but that more effective ranking functions result when click-through features are generated from benchmark datasets with explicit click-through data. In either case, the most noticeable ranking improvements are achieved at the tops of the provided ranked lists of results, which are highly targeted by the Web users.

Image Tag Refinement With View-Dependent Concept Representations

Image tag refinement is the task of refining initial tags of an image such that the refined tags can better reflect the content of the image and, therefore, can help users better access that image. The quality of tag refinement depends on the quality of concept representations that build a mapping from concepts to visual images. While good progress was made in the past decade on tag refinement, the previous approaches only achieved a limited success due to their limited concept representations. In this paper, we show that the visual appearances of a concept consist of both a generic view and a specific view , and therefore we can comprehensively represent a concept by two components. To ensure a clean concept representation, this representation is learned on clean click-through data, where noises are greatly reduced. In the framework, a coarse-to-fine image tag refinement is proposed, which: 1) first generates an efficient star graph to find candidate tags but missing in the initial tag list of an input image and 2) guided by this view-dependent concept representation, formulates a probabilistic objective function to eliminate irrelevant tags. Extensive experiments on two widely used standard data sets (MIRFlickr-25K and NUS-WIDE-270K) demonstrate the effectiveness of our approach.

CF-Rank: Learning to rank by classifier fusion on click-through data

Ranking as a key functionality of Web search engines, is a user-centric process. However, click-through data, which is the source of implicit feedback of users, are not included in almost all of datasets published for the task of ranking. This limitation is also observable in the majority of benchmark datasets prepared for the learning to rank which is a new and promising trend in the information retrieval literature. In this paper, inspiring from the click-through data concept, the notion of click-through features is introduced. Click-through features could be derived from the given primitive dataset even in the absence of click-through data in the utilized benchmark dataset. These features are categorized into three different categories and are either related to the users’ queries, results of searches or clicks of users. With the use of click-through features, in this research, a novel learning to rank algorithm is proposed. By taking into account informativeness measures such as MAP, NDCG, InformationGain and OneR, at its first step, the proposed algorithm generates a classifier for each category of click-through features. Thereafter, these classifiers are fused together by using exponential ordered weighted averaging operators. Experimental results obtained from a plenty of investigations on WCL2R and LETOR4.0 benchmark datasets, demonstrate that the proposed method can substantially outperform well-known ranking methods in the presence of explicit click-through data based on MAP and NDCG criteria. Specifically, such an improvement is more noticeable on the top of ranked lists, which usually attract users’ attentions more than other parts of these lists. This betterment on WCL2R dataset is about 20.25% for P@1 and 5.68% for P@3 in comparison with SVMRank, which is a well-known learning to rank algorithm. CF-Rank can also obtain higher or comparable performance with baseline methods even in the absence of explicit click-through data in utilized primitive datasets. In this regard, the proposed method on the LETOR4.0 dataset has achieved an improvement of about 2.7% on MAP measure compared to AdaRank-NDCG algorithm.

Building effective SVM concept detectors from clickthrough data for large-scale image retrieval

Clickthrough data is a source of information that can be used for automatically building concept detectors for image retrieval. Previous studies, however, have shown that in many cases the resulting training sets suffer from severe label noise that has a significant impact in the SVM concept detector performance. This paper evaluates and proposes a set of strategies for automatically building effective concept detectors from clickthrough data. These strategies focus on: (1) automatic training set generation; (2) assignment of label confidence weights to the training samples and (3) using these weights at the classifier level to improve concept detector effectiveness. For training set selection and in order to assign weights to individual training samples three Information Retrieval (IR) models are examined: vector space models, BM25 and language models. Three SVM variants that take into account importance at the classifier level are evaluated and compared to the standard SVM: the Fuzzy SVM, the Power SVM, and the Bilateral-weighted Fuzzy SVM. Experiments conducted on the MM Grand Challenge dataset (consisting of 1M images and 82.3M unique clicks) for 40 concepts demonstrate that (1) on average, all weighted SVM variants are more effective than the standard SVM; (2) the vector space model produces the best training sets and best weights; (3) the Bilateral-weighted Fuzzy SVM produces the best results but is very sensitive to weight assignment and (4) the Fuzzy SVM is the most robust training approach for varying levels of label noise.

Visually Interpreting Names as Demographic Attributes by Exploiting Click-Through Data

Name of an identity is strongly influenced by his/her cultural background such as gender and ethnicity, both vital attributes for user profiling, attribute-based retrieval, etc. Typically, the associations between names and attributes (e.g., people named "Amy" are mostly females) are annotated manually or provided by the census data of governments. We propose to associate a name and its likely demographic attributes by exploiting click-throughs between name queries and images with automatically detected facial attributes. This is the first work attempting to translate an abstract name to demographic attributes in visual-data-driven manner, and it is adaptive to incremental data, more countries and even unseen names (the names out of click-through data) without additional manual labels. In the experiments, the automatic name-attribute associations can help gender inference with competitive accuracy by using manual labeling. It also benefits profiling social media users and keyword-based face image retrieval, especially for contributing 12% relative improvement of accuracy in adapting to unseen names.

Content and Location based Information Retrieval System

Personalized Web search is an effective means of providing precise results to different users when they submit the same query. As the amount of web information grows rapidly an efficient personalization approach that modifies the appearance of a website's content to satisfy a specific user's instructions or preferences is required. It is also essential to keep track of the change of interest of the user from time to time. An approach which involves a concept based user profiling strategy, along with the click-through data and keyword-based search, is developed. Concepts are split into content and location concepts and are maintained separately for monitoring the gradual transition in the interest of a user over the time. The user’s interest is captured from the clickthrough information. Depending upon the links clicked and the concepts returned users’ information access behavior is analyzed and re-ranking is performed to obtain the relevant results.

Automatic Deep Web Query Results User Satisfaction Evaluation with Click-through Data Analysis

We browse through hundreds of Deep web pages everyday to find information of interest. We feel happy when Deep web browsing operations provide us with necessary information; otherwise, we feel bitter. Now, the measurement of this user satisfaction has become a hot research topic. In this paper, we propose a click-through-data-based and unsupervised user satisfaction evaluation system, CNEITE, to evaluate the user satisfaction of Deep Web query result pages. It applies query type classifying, navigational query evaluating, informational/transactional query evaluating to solve the challenging tasks. We evaluated our CNEITE system on the AOL data sets, experimental results show that CNEITE achieves higher classify precision than a widely used classify method , Dtree, and higher annotate answer accuracy than method proposed in [17].

Image search reranking with query-dependent click-based relevance feedback.

Our goal is to boost text-based image search results via image reranking. There are diverse modalities (features) of images that we can leverage for reranking, however, the effects of different modalities are query-dependent. The primary challenge we face is how to fuse multiple modalities adaptively for different queries, which has often been overlooked in previous reranking research. Moreover, multimodality fusion without an understanding of the query is risky, and may lead to incorrect judgment in reranking. Therefore, to obtain the best fusion weights for the query, in this paper, we leverage click-through data, which can be viewed as an "implicit" user feedback and an effective means of understanding the query. A novel reranking algorithm, called click-based relevance feedback, is proposed. This algorithm emphasizes the successful use of click-through data for identifying user search intention, while leveraging multiple kernel learning algorithm to adaptively learn the query-dependent fusion weights for multiple modalities. We conduct experiments on a real-world data set collected from a commercial search engine with click-through data. Encouraging experimental results demonstrate that our proposed reranking approach can significantly improve the NDCG@10 of the initial search results by 11.62%, and can outperform several existing approaches for most kinds of queries, such as tail, middle, and top queries.

Discovering and understanding word level user intent in Web search queries

Identifying and interpreting user intent are fundamental to semantic search. In this paper, we investigate the association of intent with individual words of a search query. We propose that words in queries can be classified as either content or intent, where content words represent the central topic of the query, while users add intent words to make their requirements more explicit. We argue that intelligent processing of intent words can be vital to improving the result quality, and in this work we focus on intent word discovery and understanding. Our approach towards intent word detection is motivated by the hypotheses that query intent words satisfy certain distributional properties in large query logs similar to function words in natural language corpora. Following this idea, we first prove the effectiveness of our corpus distributional features, namely, word co-occurrence counts and entropies, towards function word detection for five natural languages. Next, we show that reliable detection of intent words in queries is possible using these same features computed from query logs. To make the distinction between content and intent words more tangible, we additionally provide operational definitions of content and intent words as those words that should match, and those that need not match, respectively, in the text of relevant documents. In addition to a standard evaluation against human annotations, we also provide an alternative validation of our ideas using clickthrough data. Concordance of the two orthogonal evaluation approaches provide further support to our original hypothesis of the existence of two distinct word classes in search queries. Finally, we provide a taxonomy of intent words derived through rigorous manual analysis of large query logs.