Click-through Data Research Articles

Does Online Fundraising Increase Charitable Giving? A Nationwide Field Experiment on Facebook

Does online fundraising increase charitable giving? Using the Facebook advertising tool, we implemented a natural field experiment across Germany, randomly assigning almost 8,000 postal codes to Save the Children fundraising videos or to a pure control. We studied changes in the donation revenue and frequency for Save the Children and other charities by postal code. Our geo-randomized design circumvented many difficulties inherent in studies based on click-through data, especially substitution and measurement issues. We found that (i) video fundraising increased donation revenue and frequency to Save the Children during the campaign and in the subsequent five weeks, that (ii) the campaign was profitable for the fundraiser, and that (iii) the effects were similar independent of video content and impression assignment strategy. However, we also found some crowding out of donations to other similar charities or projects. Finally, we demonstrated that click data may be an inappropriate proxy for donations and recommend that managers use careful experimental designs that can plausibly evaluate the effects of advertising on relevant outcomes. This paper was accepted by Yan Chen, behavioral economics and decision analysis. Funding: M. Adena acknowledges financial support from the Deutsche Forschungsgemeinschaft [Grants 417014946 and CRC TRR 190 (Project 280092119)]. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2020.00596 .

Automatic Skill-Oriented Question Generation and Recommendation for Intelligent Job Interviews

Job interviews are the most widely accepted method for companies to select suitable candidates, and a critical challenge is finding the right questions to ask job candidates. Moreover, there is a lack of integrated tools for automatically generating interview questions and recommending the right questions to interviewers. To this end, in this paper, we propose an intelligent system for assisting job interviews, namely, DuerQues. To build this system, we first investigate how to automatically generate skill-oriented interview questions in a scalable way by learning external knowledge from online knowledge-sharing communities. Along this line, we develop a novel distantly supervised skill entity recognition method to identify skill entities from large-scale search queries and web page titles with less need for human annotation. Additionally, we propose a neural generative model for generating skill-oriented interview questions. In particular, we introduce a data-driven solution to create high-quality training instances and design a learning algorithm to improve the performance of question generation. Furthermore, we exploit click-through data from query logs and design a recommender system for recommending suitable questions to interviewers. Specifically, we introduce a graph-enhanced algorithm to efficiently recommend suitable questions given a set of queried skills. Finally, extensive experiments on real-world datasets demonstrate the effectiveness of our DuerQues system in terms of the quality of generated skill-oriented questions and the performance of question recommendation.

Filtering objectionable information access based on click-through behaviours with deep learning methods

This study explores URL click-through behaviour to predict the category of users’ online information accesses and applies the results to progressively filter objectionable accesses during web surfing. Each clicked URL is represented by the embedding technique and fed into the Bidirectional Long Short-Term Memory neural network cascaded with a Conditional Random Field (BiLSTM-CRF) model to predict the category of a user’s access. Large-scale experiments on click-through data from nearly one million real users show that our proposed BiLSTM-CRF model achieves promising results. The proposed method outperforms related approaches by a high accuracy of 0.9492 (near 27% relative improvement) for context-aware category prediction and an F1-score of 0.8995 (about 29% relative improvement) for objectionable access identification. In addition, in real-time filtering simulations, our model gradually achieves a macro-averaging blocking rate of 0.9221, while maintaining a favourably low false-positive rate of 0.0041.

Unbiased Pairwise Approach toward Learning-to-Rank: An Empirical Study

With the bloom of information technology in recent decades, people are constantly being exposed to a huge amount of information. Learning-to-rank comes out as one of the solutions to ease out the mentioned obstacle by trying to rearrange objects according to their degrees of importance or relevance. This solution usually applies machine learning techniques to construct ranking models in information retrieval systems. The aim of this study is to explore and experiment the existing learning-to-rank approaches with real-life logs data. The study also includes estimating and minimizing the bias noise found in the click-through data of the recorded logs. Evaluation results have presented the advantage and disadvantage of the experimented approaches in realistic settings.

How to build high quality L2R training data: Unsupervised compression-based selective sampling for learning to rank

Learning to Rank (L2R) improves ranking quality but relies on the existence of manually labeled training sets, which are expensive and cumbersome to generate. Using automated labeling (e.g., clickthrough data) imposes its own challenges. Active learning (AL) can be used to gather high-quality training data by producing very informative yet small training sets. Cover, a method we have previously developed, allows for unsupervised sampling of training sets as good as those created using AL. In this paper we provide an extensive analysis of how and why Cover works. We revisit the method in a more formal way, with theorems and proofs, and provide additional empirical evidence of its practicality. We answer questions related to why Cover works so well and how its properties are related to AL methods. We show how certain characteristics of Cover’s clustering step allows it to more thoroughly explore the feature space by selecting query-document pairs that are representative and diverse, allowing L2R methods to produce effective models. The main novel contribution is a detailed analysis of the method’s inner workings and information-theoretic properties, allowing us to advance the understanding of L2R fundamentals through the lens of training set building.

Clarifying Ambiguous Keywords with Personal Word Embeddings for Personalized Search

Personalized search tailors document ranking lists for each individual user based on her interests and query intent to better satisfy the user’s information need. Many personalized search models have been proposed. They first build a user interest profile from the user’s search history, and then re-rank the documents based on the personalized matching scores between the created profile and candidate documents. In this article, we attempt to solve the personalized search problem from an alternative perspective of clarifying the user’s intention of the current query. We know that there are many ambiguous words in natural language such as “Apple.” People with different knowledge backgrounds and interests have personalized understandings of these words. Therefore, we propose a personalized search model with personal word embeddings for each individual user that mainly contain the word meanings that the user already knows and can reflect the user interests. To learn great personal word embeddings, we design a pre-training model that captures both the textual information of the query log and the information about user interests contained in the click-through data represented as a graph structure. With personal word embeddings, we obtain the personalized word and context-aware representations of the query and documents. Furthermore, we also employ the current session as the short-term search context to dynamically disambiguate the current query. Finally, we use a matching model to calculate the matching score between the personalized query and document representations for ranking. Experimental results on two large-scale query logs show that our designed model significantly outperforms state-of-the-art personalization models.

A Mouse-Trajectory Based Model for Predicting Query-URL Relevance

For the learning-to-ranking algorithms used in commercial search engines, a conventional way to generate the training examples is to employ professional annotators to label the relevance of query-url pairs. Since label quality depends on the expertise of annotators to a large extent, this process is time-consuming and labor-intensive. Automatically generating labels from click-through data has been well studied to have comparable or better performance than human judges. Click-through data present users’ action and imply their satisfaction on search results, but exclude the interactions between users and search results beyond the page-view level (e.g., eye and mouse movements). This paper proposes a novel approach to comprehensively consider the information underlying mouse trajectory and click-through data so as to describe user behaviors more objectively and achieve a better understanding of the user experience. By integrating multi-sources data, the proposed approach reveals that the relevance labels of query-url pairs are related to positions of urls and users’ behavioral features. Based on their correlations, query-url pairs can be labeled more accurately and search results are more satisfactory to users. The experiments that are conducted on the most popular Chinese commercial search engine (Baidu) validated the rationality of our research motivation and proved that the proposed approach outperformed the state-of-the-art methods.

Towards purchase prediction: A transaction-based setting and a graph-based method leveraging price information

Targeting at boosting business revenue, purchase prediction based on user behavior is crucial to e-commerce. However, it is not a well-explored topic due to a lack of relevant datasets. Specifically, no public dataset provides both price and discount information varying on time, which play an essential role in the user’s decision making. Besides, existing learn-to-rank methods cannot explicitly predict the purchase possibility for a specific user-item pair. In this paper, we propose a two-step graph-based model, where the graph model is applied in the first step to learn representations of both users and items over click-through data, and the second step is a classifier incorporating the price information of each transaction record. To evaluate the model performance, we propose a transaction-based framework focusing on the purchased items and their context clicks, which contain items that a user is interested in but fails to choose after comparison. Our experiments show that exploiting the price and discount information can significantly enhance prediction accuracy.

Trends in digital newspaper discovery: a natural experiment

In this article, the discovery and use of digital newspaper collections are explored by capitalizing on a natural experiment that arose when five California State University libraries activated the Primo newspapers search interface, and five other libraries with similar enrollment numbers and comparable demographic profiles did not. By analyzing Primo Analytics data, COUNTER R4 data, and A–Z database list click-through data collected from the ten libraries over the course of academic years (AY) 2018–2019 (pre-deployment) and AY 2019–2020 (post-deployment), the effects on usage and legible user behavior of introducing a specialized Primo newspaper scope are calculated. Researchers explore how this research method can be a model for libraries to investigate trends within their own organizations.

A new Query Reformulation Approach using Web Result Clustering and User Profile

Query reformulation techniques are essential for information retrieval systems. These techniques eliminate the bad queries (short and ambiguous queries) and minimize users search time. There are two kinds of techniques in literature; the first uses the query logs with the click-through data to identify relevant queries and propose them to the user. The second group uses relevant terms extracted from different sources such as Wikipedia, Wordnet, and pseudo-relevant documents to expand the initial query and increase the likelihood with relevant documents. Both groups generate false queries because they do not consider the user interests and due to the assumption that the user clicks only on relevant results (first group) and that the top-k retrieved documents (top 5 to 15) are relevant (second group).In this paper, we propose a novel approach to reformulate the user queries using his profile (which contains the user interests). The approach retrieves documents related to the query from four different data sources (≈ 1000 doc). Then, it extracts the topics of these documents using the Lingo clustering algorithm (cluster the document with the same topic) and Text razor API (extract the potential topics of each cluster). Finally, it generates queries based on these topics and the user profile. The results show that the proposed approach outperforms the existing solution in P@10, P@20, and MAP.

Detecting Effects of Misinformation Through Emotion and Trace Behavior

To explore the impact of misinformation on social media, this study employed a text analysis process to first identify topics within online rhetoric and emotion toward misinformation on Facebook. Then, emotion and advertising behavioral measures were used to examine the relationship between six discrete emotions and advertising costs. Facebook advertising click-through data were collected between the months of January and November 2020 totaling 44 weeks of data from a sample of 20 national U.S. businesses that advertised on Facebook during the analysis time frame. The regression revealed that fear, identified in the online discourse mentioning misinformation, had a positive and significant relationship with click-through rates. In sum, the results suggest that as fear of misinformation increases, so does the available attention of social media users. Policy implications are discussed.

Learn to rank Persian web content based on multi-layered genetic programming

Learning to rank (L2R) has emerged as a promising approach in handling the existing challenges of Web search engines. However, there are major drawbacks with the present learning to rank techniques. Current L2R algorithms do not take into account to the search behavior of the users embedded in their search sessions’ logs. On the other hand, machine-learning as a data-intensive process requires a large volume of data about users’ queries as well as Web documents. This situation has made the usage of L2R techniques questionable in the real-world applications. Recently, by the use of the click-through data model and based on the generation of click-through features, a novel approach is proposed, named as MGP-Rank. Using the layered genetic-programming model, MGP-Rank has achieved noticeable performance on the ranking of the English Web content. In this study, with respect to the specific characteristics of the Persian language, some suitable scenarios are presented for the generation of the click-through features. In this way, a customized version of the MGP-Rank is proposed of the Persian Web retrieval. The evaluation results of this algorithm on the dotIR dataset, indicate its considerable improvement in comparison with major ranking methods. The improvement of the performance is particularly more noticeable in the top part of the search results lists, which are most frequently visited by the Web users.

A personalized search model using online social network data based on a holonic multiagent system

Personalized search utilizes user preferences to optimize search results, and most existing studies obtain user preferences by analyzing user behaviors in search engines that provide click-through data. However, the behavioral data are noisy because users often clicked some irrelevant documents to find their required information, and the new user cold start issue represents a serious problem, greatly reducing the performance of personalized search. This paper attempts to utilize online social network data to obtain user preferences that can be used to personalize search results, mine the knowledge of user interests, user influence and user relationships from online social networks, and use this knowledge to optimize the results returned by search engines. The proposed model is based on a holonic multiagent system that improves the adaptability and scalability of the model. The experimental results show that utilizing online social network data to implement personalized search is feasible and that online social network data are significant for personalized search.

An Integrated Methods for Refined Facial-Image Retrieval using Sift and Click-Through Data

In the modern era, the emergence of social networking sites paved the way to the people to upload lot of images online. Social sites like Instagram and Flickr allow users to add semantic information to the images in the form of tags. Often these tags are the firsthand semantic data for retrieving the images from the Internet. When a user searches for images in the web, the images with tags relevant to user query are retrieved. Most of the time, these semantic data are not relevant to the content of the image and hence the user gets irrelevant images in contrast to their intended search. This is more common in facial image search. In this paper, we propose an integrated approach to refine the relevance of retrieved facial images. Using Scale Invariant Feature Extraction (SIFT) the facial features of semantically most relevant image and click-through data of all the retrieved images are used to rank and present a meaningful search result. Along with facial features and click through information, the co-occurrence of related tags is also considered. Also we propose the construction of inverted index structure to improve the search performance.

Constructing Click Model for Mobile Search with Viewport Time

A series of click models has been proposed to extract accurate and unbiased relevance feedback from valuable yet noisy click-through data in search logs. Previous works have shown that users search behavior in mobile and desktop scenarios are rather different in many aspects, therefore, the click models designed for desktop search may not be effective in the mobile context. To address this problem, we propose two novel click models for mobile search: (1) Mobile Click Model (MCM), which models click necessity bias and examination satisfaction bias; (2) Viewport Time Click Model (VTCM), which further extends MCM by utilizing the viewport time. Extensive experiments on large-scale real mobile search logs show that: (1) MCM and VTCM outperform existing models in predicting users’ clicks and estimating result relevance; (2) MCM and VTCM can extract richer information, such as the click necessity of search results and the probability of user satisfaction, from mobile click logs; (3) By modeling the viewport time distributions of heterogeneous results, VTCM can bring a significant improvement over MCM in click prediction and relevance estimation tasks. Our proposed click models can help better understand user behavior patterns in mobile search and improve the ranking performance of mobile search engines.

Query Expansion and Query Fuzzy with Large-Scale Click-through Data for Microblog Retrieval

Query Expansion and Query Fuzzy with Large-Scale Click-through Data for Microblog Retrieval

Direct-to-consumer advertising and online search

Beginning in 1997, the Food and Drug Administration (FDA) allowed television advertisements to make major statements about a prescription drug, while referring to detailed information on the internet. The hope was that consumers would seek additional information online to understand the risks and benefits of taking the medication. This policy motivates us to analyze direct-to-consumer advertising (DTCA) and search engine click-through data on a set of drugs over the period from 2008 to 2011.Regression analysis shows that DTCA on a prescription drug is associated with a higher frequency of online search and subsequent clicks for that drug, as well as search for other drugs in the same class. While the FDA’s policy was intended to direct consumers to the internet for drug information, the effects of DTCA are larger for paid clicks relative to organic clicks and for clicks on promotional websites relative to informational websites. We also find the relationship between DTCA and search is heterogeneous: the effects are stronger for younger drugs and drugs that treat acute conditions.

Personalized and Enhanced Hybridized Semantic Algorithm for web image retrieval incorporating ontology classification, strategic query expansion, and content-based analysis

Most of the existing web search systems are query-centric and are not user-centric. Mining images from the Web is a challenging task as it requires choosing the right methodology. A strategy that recommends images for homonyms and contextually similar terms have been proposed. The proposed system facilitates ontology modeling for homonyms and contextually related synonymous terms using description logics semantics and semantic similarity computation. An Enhanced Hybrid Semantic Algorithm that computes the semantic similarity and establishes dynamic OntoPath for easing the web image recommendation has been proposed. The proposed system classifies the ontologies using SVM and a Homonym LookUp directory. The methodology focuses on generating unique classes of images as an initial recommendation set. Based on the user click, strategic expansion of OntoPath takes place. Personalization is achieved by content-based analysis of the user click-through data. An overall accuracy of 95.87% is achieved by the proposed system.

Relevant Feedback Based Accurate and Intelligent Retrieval on Capturing User Intention for Personalized Websites

With the rapid growth of networking, cyber–physical–social systems (CPSSs) provide vast amounts of information. Aimed at the huge and complex data provided by networking, obtaining valuable information to meet precise search needs when capturing user intention has become a major challenge, especially in personalized websites. General search engines face difficulties in addressing the challenges brought by this exploding amount of information. In this paper, we use real-time location and relevant feedback technology to design and implement an efficient, configurable, and intelligent retrieval framework for personalized websites in CPSSs. To improve the retrieval results, this paper also proposes a strategy of implicit relevant feedback based on click-through data analysis, which can obtain the relationship between the user query conditions and retrieval results. Finally, this paper designs a personalized PageRank algorithm including modified parameters to improve the ranking quality of the retrieval results using the relevant feedback from other users in the interest group. Experiments illustrate that the proposed accurate and intelligent retrieval framework improves the user experience.

Accurately Interpreting Clickthrough Data as Implicit Feedback

This paper examines the reliability of implicit feedback generated from clickthrough data in WWW search. Analyzing the users' decision process using eyetracking and comparing implicit feedback against manual relevance judgments, we conclude that clicks are informative but biased. While this makes the interpretation of clicks as absolute relevance judgments difficult, we show that relative preferences derived from clicks are reasonably accurate on average.