TREC Web Research Articles

A bias–variance evaluation framework for information retrieval systems

In information retrieval (IR), the improvement of the effectiveness often sacrifices the stability of an IR system. To evaluate the stability, many risk-sensitive metrics have been proposed. Since the theoretical limitations, the current works study the effectiveness and stability separately, and have not explored the effectiveness–stability tradeoff. In this paper, we propose a Bias–Variance Tradeoff Evaluation (BV-Test) framework, based on the bias–variance decomposition of the mean squared error, to measure the overall performance (considering both effectiveness and stability) and the tradeoff between effectiveness and stability of a system. In this framework, we define generalized bias–variance metrics, based on the Cranfield-style experiment set-up where the document collection is fixed (across topics) or the set-up where document collection is a sample (per-topic). Compared with risk-sensitive evaluation methods, our work not only measures the effectiveness–stability tradeoff of a system, but also effectively tracks the source of system instability. Experiments on TREC Ad-hoc track (1993–1999) and Web track (2010–2014) show a clear effectiveness–stability tradeoff across topics and per-topic, and topic grouping and max–min normalization can effectively reduce the bias–variance tradeoff. Experimental results on TREC Session track (2010–2012) also show that the query reformulation and increase of user data are beneficial to both effectiveness and stability simultaneously.

ON THE USEFULNESS OF HTML META ELEMENTS FOR WEB RETRIEVAL

Web retrieval studies have mostly used URL, title, body, and anchor text fields to represent Web documents. On the other hand, HTML standards provide a rich set of elements to define different parts of a Web page. For example, meta elements are used to provide structured metadata about a Web page not to end users, but instead to browsers or crawlers. However, it is unclear whether meta tags are or are not useful for Web retrieval, as most of the previous studies leveraged URL, title, body, and anchor text fields. In this work, we examine the usefulness of two meta tags, namely keywords and description, based on ad-hoc tasks of previous TREC studies. Through experiments on the standard TREC Web datasets and several query sets, our results using the state-of-the-art term-weighting models show that the utilization of description field systematically increases the retrieval effectiveness, to a statistically significant degree most of the time. By contrast, the employment of keywords field may cause a significant deterioration in retrieval effectiveness for certain term-weighting models.

Relevance-based entity selection for ad hoc retrieval

Recent developments have shown that entity-based models that rely on information from the knowledge graph can improve document retrieval performance. However, given the non-transitive nature of relatedness between entities on the knowledge graph, the use of semantic relatedness measures can lead to topic drift. To address this issue, we propose a relevance-based model for entity selection based on pseudo-relevance feedback, which is then used to systematically expand the input query leading to improved retrieval performance. We perform our experiments on the widely used TREC Web corpora and empirically show that our proposed approach to entity selection significantly improves ad hoc document retrieval compared to strong baselines. More concretely, the contributions of this work are as follows: (1) We introduce a graphical probability model that captures dependencies between entities within the query and documents. (2) We propose an unsupervised entity selection method based on the graphical model for query entity expansion and then for ad hoc retrieval. (3) We thoroughly evaluate our method and compare it with the state-of-the-art keyword and entity based retrieval methods. We demonstrate that the proposed retrieval model shows improved performance over all the other baselines on ClueWeb09B and ClueWeb12B, two widely used Web corpora, on the NDCG@20, and ERR@20 metrics. We also show that the proposed method is most effective on the difficult queries. In addition, We compare our proposed entity selection with a state-of-the-art entity selection technique within the context of ad hoc retrieval using a basic query expansion method and illustrate that it provides more effective retrieval for all expansion weights and different number of expansion entities.

Low-cost, bottom-up measures for evaluating search result diversification

Search result diversification aims at covering different user intents by returning a diversified document list. Most existing diversity measures require a predefined set of intents for a given query, where it is assumed that there is no relationship across these intents. However, studies have shown that modeling a hierarchy of intents has some benefits over the standard measure of using a flat list of intents. Intuitively, having more layers in the intent hierarchy seems to imply that we can consider more intricate relationships between intents and thereby identify subtle differences between documents that cover different intents. On the other hand, manually building a rich intent hierarchy imposes extra cost and is probably not very practical. In light of these considerations, we first propose a measure to build a hierarchy of intents from a given set of flat intents by clustering per-intent relevant documents and thereby identifying subintents. Furthermore, in our second measure, we consider a variant of our first measure that clusters per-topic relevance documents rather than per-intent ones, which is also intent-free. In addition, we propose our third measure, a simple, completely intent-free measure to search result diversity evaluation, which leverages document similarities. Our experiments based on TREC Web Track 2009–2013 test collections show that our proposed measures have advantages over existing diversity measures despite their low annotation costs.

Estimating Gaussian mixture models in the local neighbourhood of embedded word vectors for query performance prediction

The study of query performance prediction (QPP) in information retrieval (IR) aims to predict retrieval effectiveness. The specificity of the underlying information need of a query often determines how effectively can a search engine retrieve relevant documents at top ranks. The presence of ambiguous terms makes a query less specific to the sought information need, which in turn may degrade IR effectiveness. In this paper, we propose a novel word embedding based pre-retrieval feature which measures the ambiguity of each query term by estimating how many ‘senses’ each word is associated with. Assuming each sense roughly corresponds to a Gaussian mixture component, our proposed generative model first estimates a Gaussian mixture model (GMM) from the word vectors that are most similar to the given query terms. We then use the posterior probabilities of generating the query terms themselves from this estimated GMM in order to quantify the ambiguity of the query. Previous studies have shown that post-retrieval QPP approaches often outperform pre-retrieval ones because they use additional information from the top ranked documents. To achieve the best of both worlds, we formalize a linear combination of our proposed GMM based pre-retrieval predictor with NQC, a state-of-the-art post-retrieval QPP. Our experiments on the TREC benchmark news and web collections demonstrate that our proposed hybrid QPP approach (in linear combination with NQC) significantly outperforms a range of other existing pre-retrieval approaches in combination with NQC used as baselines.

A selective approach to index term weighting for robust information retrieval based on the frequency distributions of query terms

A typical information retrieval (IR) system applies a single retrieval strategy to every information need of users. However, the results of the past IR experiments show that a particular retrieval strategy is in general good at fulfilling some type of information needs while failing to fulfil some other type, i.e., high variation in retrieval effectiveness across information needs. On the other hand, the same results also show that an information need that a particular retrieval strategy failed to fulfil could be fulfilled by one of the other existing retrieval strategies. The challenge in here is therefore to determine in advance what retrieval strategy should be applied to which information need. This challenge is related to the robustness of IR systems in retrieval effectiveness. For an IR system, robustness can be defined as fulfilling every information need of users with an acceptable level of satisfaction. Maintaining robustness in retrieval effectiveness is a long-standing challenge and in this article we propose a simple but powerful method as a remedy. The method is a selective approach to index term weighting and for any given query (i.e., information need) it predicts the “best” term weighting model amongst a set of alternatives, on the basis of the frequency distributions of query terms on a target document collection. To predict the best term weighting model, the method uses the Chi-square statistic, the statistic of the Chi-square goodness-of-fit test. The results of the experiments, performed using the official query sets of the TREC Web track and the Million Query track, reveal in general that the frequency distributions of query terms provide relevant information on the retrieval effectiveness of term weighting models. In particular, the results show that the selective approach proposed in this article is, on average, more effective and more robust than the most effective single term weighting model.

How sensitive are the term-weighting models of information retrieval to spam Web pages?

Many term-weighting models have been proposed for information retrieval. In this paper we investigate the extent to how retrieval effectiveness of term-weighting models is affected by the presence of spam Web pages. We perform retrieval experiments on the ClueWeb09-English (Category A) dataset – a substantial fraction of which are spam pages that are deliberately designed to manipulate commercial search engines – as well as the ClueWeb12 (Category A) dataset. Ad hoc tasks of TREC Web tracks 2009 through 2012 are completed to examine the spam sensitivity of the state-of-the-art retrieval models using Apache Lucene as the retrieval engine. Moreover, ad hoc tasks of two Web tracks and two Tasks tracks 2013 through 2016 are also included in a part of the experiment where the number of documents that are explicitly judged as spam in the search results returned by each retrieval model is inspected.Our experimental results show that hypergeometric models of information retrieval are more immune than other models to spam content. All the results presented in this article are fully repeatable and reproducible with data and code available online at a public GitHub repository.

Fusion-based methods for result diversification in web search

Search result diversification of text documents is especially necessary when a user issues a faceted or ambiguous query to the search engine. A variety of approaches have been proposed to deal with this issue in recent years. In this article, we propose a group of fusion-based result diversification methods with the aim to improve performance that considers both relevance and diversity. They are linear combinations of scores that are obtained from different component search systems. The weight of each search system is determined by considering three factors: performance, dissimilarity, and complementarity. There are two major contributions. Firstly, we find that all the three factors of performance and complementarity and dissimilarity are useful for effective weighting of linear combination. Secondly, we present the logarithmic function-based model for converting ranking information into scores. Experiments are carried out with four groups of results submitted to the TREC web diversity task. Experimental results show that some of the fusion methods that use the aforementioned techniques perform more effectively than the state-of-the-art fusion methods for result diversification.

Search Result Diversity Evaluation Based on Intent Hierarchies

Search result diversification aims at returning diversified document lists to cover different user intents of a query. Existing diversity measures assume that the intents of a query are disjoint, and do not consider their relationships. In this paper, we introduce intent hierarchies to model the relationships between intents, and present four weighing schemes. Based on intent hierarchies, we propose several hierarchical measures that take into account the relationships between intents. We demonstrate the feasibility of hierarchical measures by using a new test collection based on TREC Web Track 2009-2013 diversity test collections and by using NTCIR-11 IMine test collection. Our main experimental findings are: (1) Hierarchical measures are more discriminative and intuitive than existing measures. In terms of intuitiveness, it is preferable for hierarchical measures to use the whole intent hierarchies than to use only the leaf nodes. (2) The types of intent hierarchies used affect the discriminative power and intuitiveness of hierarchical measures. We suggest the best type of intent hierarchies to be used according to whether the nonuniform weights are available. (3) To measure the benefits of the diversification algorithms which use automatically mined hierarchical intents, it is important to use hierarchical measures instead of existing measures.

Exploiting entities for query expansion

A substantial fraction of web search queries contain references to entities, such as persons, organizations, and locations. This signi cant presence of named entities in queries provides an opportunity for web search engines to improve their understanding of the user's information need. In this thesis, we investigate the entity-oriented query expansion process. Particularly, we propose two novel and effective query expansion approaches that exploit semantic sources of evidence to devise discriminative term features, and machine learning techniques to effectively combine these features in order to rank candidate expansion terms. As a result, not only do we select effective expansion terms, but we also weigh these terms according to their predicted effectiveness. In addition, since our query expansion approaches consider Wikipedia infoboxes as a source of candidate expansion terms, a frequent obstacle is that only about 20% of Wikipedia articles have an infobox. To overcome this problem we propose WAVE, a self-supervised approach to autonomously generate infoboxes for Wikipedia articles. First, we propose UQEE, an unsupervised entity-oriented query expansion approach, which effectively selects expansion terms using taxonomic features devised by the semantic structure implicitly provided by infobox templates. We show that query expansion using infoboxes presents a better trade-off between retrieval performance and query latency. Moreover, we demonstrate that the automatically generated infoboxes provided by WAVE are as effective as manually generated infoboxes for query expansion. Lastly, we propose L2EE, a learning to rank approach for entity-oriented query expansion, which considers semantic evidence encoded in the content of Wikipedia article fields, and automatically labels training examples proportionally to their observed retrieval effectiveness. Experiments on three TREC web test collections attest the effectiveness of L2EE, with significant gains compared to UQEE and state-of-the-art pseudo-relevance feedback and entity-oriented pseudo-relevance feedback approaches.

Introduction to the special issue on search intents and diversification

This special issue of the Information Retrieval journal is focused on the topic of query intents and diversification within search engines. Indeed, information retrieval users have diverse search needs, and the level of detail given in the user’s search query can differ widely depending on how clear the underlying information need itself is, the user’s search environment (e.g. input device), and how difficult it is for the user to express the need in the form of a query. For these reasons, many queries are ambiguous and/or underspecified. In light of this, research in accommodating different search intents has received a lot of attention lately. For example, major conferences like SIGIR, WWW and WSDM have begun to see papers on search result diversification, which aims to capture different user needs within one entry-point search result page. This problem was discussed intensively at the ECIR 2011 and WSDM 2012 Diversity in Document Retrieval Workshops. Also, in recent years, the TREC Web (2009–2012) and TREC Blog (2010) tracks have measured the diversity of participating systems in retrieving web pages and blog posts, respectively. Moreover, the recent INTENT tasks at NTCIR-9 and NTCIR-10 tackled not only search result diversification but also the task of mining intents given a query. The aim of this special issue is to highlight the advances and clarify the future goals in the ares of search intents and diversification. In particular, the articles included in this special issue address two areas within this vision, namely the identification of query

Metasearch information fusion using linear programming

For a specific query merging the returned results from multiple search engines, in the form of a metasearch aggregation, can provide significant improvement in the quality of relevant documents. This paper suggests a minimax linear programming (LP) formulation for fusion of multiple search engines results. The paper proposes a weighting method to include the importance weights of the underlying search engines. This is a two-phase approach which in the first phase a new method for computing the importance weights of the search engines is introduced and in the second stage a minimax LP model for finding relevant search engines results is formulated. To evaluate the retrieval effectiveness of the suggested method, the 50 queries of the 2002 TREC Web track were utilized and submitted to three popular Web search engines called Ask, Bing and Google. The returned results were aggregated using two exiting approaches, three high-performance commercial Web metasearch engines and our proposed technique. The efficiency of the generated lists was measured using TREC-Style Average Precision (TSAP). The new findings demonstrate that the suggested model improved the quality of merging considerably

Empirical challenges and solutions in constructing a high‐performance metasearch engine

PurposeThis paper seeks to disclose the important role of missing documents, broken links and duplicate items in the results merging process of a metasearch engine in detail. It aims to investigate some related practical challenges and proposes some solutions. The study also aims to employ these solutions to improve an existing model for results aggregation.Design/methodology/approachThis research measures the amount of an increase in retrieval effectiveness of an existing results merging model that is obtained as a result of the proposed improvements. The 50 queries of the 2002 TREC web track were employed as a standard test collection based on a snapshot of the worldwide web to explore and evaluate the retrieval effectiveness of the suggested method. Three popular web search engines (Ask, Bing and Google) as the underlying resources of metasearch engines were selected. Each of the 50 queries was passed to all three search engines. For each query the top ten non‐sponsored results of each search engine were retrieved. The returned result lists of the search engines were aggregated using a proposed algorithm that takes the practical issues of the process into consideration. The effectiveness of the result lists generated was measured using a well‐known performance indicator called “TSAP” (TREC‐style average precision).FindingsExperimental results demonstrate that the proposed model increases the performance of an existing results merging system by 14.39 percent on average.Practical implicationsThe findings of this research would be helpful for metasearch engine designers as well as providing motivation to the vendors of web search engines to improve their technology.Originality/valueThis study provides some valuable concepts, practical challenges, solutions and experimental results in the field of web metasearching that have not been previously investigated.

Learning to rank query suggestions for adhoc and diversity search

Query suggestions have become pervasive in modern web search, as a mechanism to guide users towards a better representation of their information need. In this article, we propose a ranking approach for producing effective query suggestions. In particular, we devise a structured representation of candidate suggestions mined from a query log that leverages evidence from other queries with a common session or a common click. This enriched representation not only helps overcome data sparsity for long-tail queries, but also leads to multiple ranking criteria, which we integrate as features for learning to rank query suggestions. To validate our approach, we build upon existing efforts for web search evaluation and propose a novel framework for the quantitative assessment of query suggestion effectiveness. Thorough experiments using publicly available data from the TREC Web track show that our approach provides effective suggestions for adhoc and diversity search.

Diversified search evaluation: lessons from the NTCIR-9 INTENT task

The evaluation of diversified web search results is a relatively new research topic and is not as well-understood as the time-honoured evaluation methodology of traditional IR based on precision and recall. In diversity evaluation, one topic may have more than one intent, and systems are expected to balance relevance and diversity. The recent NTCIR-9 evaluation workshop launched a new task called INTENT which included a diversified web search subtask that differs from the TREC web diversity task in several aspects: the choice of evaluation metrics, the use of intent popularity and per-intent graded relevance, and the use of topic sets that are twice as large as those of TREC. The objective of this study is to examine whether these differences are useful, using the actual data recently obtained from the NTCIR-9 INTENT task. Our main experimental findings are: (1) The $$\hbox{D}\,\sharp$$ evaluation framework used at NTCIR provides more "intuitive" and statistically reliable results than Intent-Aware Expected Reciprocal Rank; (2) Utilising both intent popularity and per-intent graded relevance as is done at NTCIR tends to improve discriminative power, particularly for $$\hbox{D}\,\sharp$$ -nDCG; and (3) Reducing the topic set size, even by just 10 topics, can affect not only significance testing but also the entire system ranking; when 50 topics are used (as in TREC) instead of 100 (as in NTCIR), the system ranking can be substantially different from the original ranking and the discriminative power can be halved. These results suggest that the directions being explored at NTCIR are valuable.

The 1st international workshop on diversity in document retrieval

When an ambiguous query is received, a sensible approach is for the information retrieval (IR) system to diversify the results retrieved for this query, in the hope that at least one of the interpretations of the query intent will satisfy the user. Diversity is an increasingly important topic, of interest to both academic researchers (such as participants in the TREC Web and Blog track diversity tasks), as well as to search engines professionals. In this workshop, we solicited submissions of both technical papers and position papers on three themes: approaches and models for diversity; the evaluation of diverse search results; and on applications of diverse search results. In the workshop, two invited talks, along with ten refereed papers were presented before breakout groups considered questions and issues with the three workshop themes.

Automatic performance evaluation of web search engines using judgments of metasearch engines

PurposeThe purpose of this paper is to introduce two new automatic methods for evaluating the performance of search engines. The reported study uses the methods to experimentally investigate which search engine among three popular search engines (Ask.com, Bing and Google) gives the best performance.Design/methodology/approachThe study assesses the performance of three search engines. For each one the weighted average of similarity degrees between its ranked result list and those of its metasearch engines is measured. Next these measures are compared to establish which search engine gives the best performance. To compute the similarity degree between the lists two measures called the “tendency degree” and “coverage degree” are introduced; the former assesses a search engine in terms of results presentation and the latter evaluates it in terms of retrieval effectiveness. The performance of the search engines is experimentally assessed based on the 50 topics of the 2002 TREC web track. The effectiveness of the methods is also compared with human‐based ones.FindingsGoogle outperformed the others, followed by Bing and Ask.com. Moreover significant degrees of consistency – 92.87 percent and 91.93 percent – were found between automatic and human‐based approaches.Practical implicationsThe findings of this work could help users to select a truly effective search engine. The results also provide motivation for the vendors of web search engines to improve their technology.Originality/valueThe paper focuses on two novel automatic methods to evaluate the performance of search engines and provides valuable experimental results on three popular ones.

The meaning of structure

How can search engines use the hyperlinks between documents to determine which documents are the most relevant for a search query? Some search engines use links to determine popularity, where the underlying idea is that the number of links pointing to a document (Web page) is a measure of its popularity. Another aspect of links is they provide a signal that two documents have related content. After all, a link is a reference. If a document A is relevant for a search query, then documents linked to A are possibly also relevant. Link information could possibly contain evidence for the topical relevance of a document. The value of link evidence for ir was investigated through the trec Web Tracks of 1999--2004. First through the traditional ad hoc search methodology, with disappointing results. When ad hoc search was considered an inappropriate model for Web search, attention moved to moreWeb-centric search tasks such as navigational search, with great success. Home pages and other importantWeb pages tend to be pages with many incoming links, and link evidence such as indegree counts and PageRank proved highly effective. The question of why links are not useful for measuring topical relevance was never answered. The goal of this thesis is to give a more precise and complete account of the value of link evidence for information retrieval. This is first investigated using the English Wikipedia, because it is available in its entirety, including all the links, and comes with a large, highquality test collection in the form of the inex Ad Hoc collection of 2006--2007. As an encyclopedia, Wikipedia is a natural collection to study topical relevance search tasks. Link information can be derived from the link graph of the entire collection--giving global, query-independent information--or from a subset of the link graph derived from the top results for a given query produced by a text-based retrieval system, giving local, queryindependent link information. PageRank is often computed on the global link graph, while algorithms like hits are typically used on the local links of the top-ranked results. We find that for ad hoc search on the inex 2006 Wikipedia collection, local link evidence leads to significant performance improvements. Incoming and outgoing link information is equally effective; the direction of the links does not affect their value for ad hoc search. We compare this to Web-centric search tasks on the trec 2004 Web track collection and to the ad hoc task on the trec 2009 Web track collection. For Web-centric search tasks, such as home page and named page finding, incoming link evidence is more effective than outgoing link evidence, and global link evidence is more effective than local link evidence. The value of link evidence for Web-centric search is to identify popular and authoritative pages, and the link direction determines the value. For ad hoc search on the more recent ClueWeb09 collection, global outgoing link evidence is very effective, but turns out to favour Wikipedia pages, which are densely interlinked. If we remove Wikipedia from the ClueWeb09 collection, local link evidence is more effective than global link evidence, and incoming and outgoing link evidence are equally effective. We also find that site-internal links are more effective for ad hoc search than site-external links. To determine the impact of link density, we conduct filtering experiments, which show that link density has a minimal impact on the effectiveness of global link evidence. For local link evidence, removing links gradually reduces the impact of link evidence, but even with a small number of links, link evidence is still effective. We also experiment with filtering links based on the semantic relatedness of the link documents and find that links between semantically related pages are more effective for identifying topically relevant documents than links between unrelated pages. Further experiments on the inex Wikipedia collection show that the amount of local link evidence is related to the amount of relevant text in documents, and the fraction of local link evidence (the percentage of the global links incident to a document that are present in the local link graph) is related to the fraction of text in a document that is relevant. With these findings, the value of link information for ranking search results has become clearer and more complete. Link information can be evidence for both popularity and topical relevance. The meaning of information derived from the link structure is determined by the direction of the links, the topical relation between the linked documents and the selection of links that is used as evidence. Available at http://staff.science.uva.nl/~mhakoole/2011/kool:mean11.pdf.

Efficient diversification of web search results

In this paper we analyze the efficiency of various search results diversification methods. While efficacy of diversification approaches has been deeply investigated in the past, response time and scalability issues have been rarely addressed. A unified framework for studying performance and feasibility of result diversification solutions is thus proposed. First we define a new methodology for detecting when, and how, query results need to be diversified. To this purpose, we rely on the concept of "query refinement" to estimate the probability of a query to be ambiguous . Then, relying on this novel ambiguity detection method, we deploy and compare on a standard test set, three different diversification methods: IASelect, xQuAD, and OptSelect. While the first two are recent state-of-the-art proposals, the latter is an original algorithm introduced in this paper. We evaluate both the efficiency and the effectiveness of our approach against its competitors by using the standard TREC Web diversification track testbed. Results shown that OptSelect is able to run two orders of magnitude faster than the two other state-of-the-art approaches and to obtain comparable figures in diversification effectiveness.

Query-level loss functions for information retrieval

Many machine learning technologies such as support vector machines, boosting, and neural networks have been applied to the ranking problem in information retrieval. However, since originally the methods were not developed for this task, their loss functions do not directly link to the criteria used in the evaluation of ranking. Specifically, the loss functions are defined on the level of documents or document pairs, in contrast to the fact that the evaluation criteria are defined on the level of queries. Therefore, minimizing the loss functions does not necessarily imply enhancing ranking performances. To solve this problem, we propose using query-level loss functions in learning of ranking functions. We discuss the basic properties that a query-level loss function should have and propose a query-level loss function based on the cosine similarity between a ranking list and the corresponding ground truth. We further design a coordinate descent algorithm, referred to as RankCosine, which utilizes the proposed loss function to create a generalized additive ranking model. We also discuss whether the loss functions of existing ranking algorithms can be extended to query-level. Experimental results on the datasets of TREC web track, OHSUMED, and a commercial web search engine show that with the use of the proposed query-level loss function we can significantly improve ranking accuracies. Furthermore, we found that it is difficult to extend the document-level loss functions to query-level loss functions.