Retrieval Status Value Research Articles

A Relative Information Gain-based Query Performance Prediction Framework with Generated Query Variants

Query performance prediction (QPP) methods, which aim to predict the performance of a query, often rely on evidences in the form of different characteristic patterns in the distribution of Retrieval Status Values (RSVs). However, for neural IR models, it is usually observed that the RSVs are often less reliable for QPP because they are bounded within short intervals, different from the situation for statistical models. To address this limitation, we propose a model-agnostic QPP framework that gathers additional evidences by leveraging information from the characteristic patterns of RSV distributions computed over a set of automatically generated query variants, relative to that of the current query. Specifically, the idea behind our proposed method—Weighted Relative Information Gain (WRIG), is that a substantial relative decrease or increase in the standard deviation of the RSVs of the query variants is likely to be a relative indicator of how easy or difficult the original query is. To cater for the absence of human-annotated query variants in real-world scenarios, we further propose an automatic query variant generation method. This can produce variants in a controlled manner by substituting terms from the original query with new ones sampled from a weighted distribution, constructed either via a relevance model or with the help of an embedded representation of query terms. Our experiments on the TREC-Robust, ClueWeb09B, and MS MARCO datasets show that WRIG, by the use of this relative changes in QPP estimate, leads to significantly better results than a state-of-the-art baseline method that leverages information from (manually created) query variants by the application of additive smoothing [ 64 ]. The results also show that our approach can improve the QPP effectiveness of neural retrieval approaches in particular.

Burst-aware data fusion for microblog search

We consider the problem of searching posts in microblog environments. We frame this microblog post search problem as a late data fusion problem. Previous work on data fusion has mainly focused on aggregating document lists based on retrieval status values or ranks of documents without fully utilizing temporal features of the set of documents being fused. Additionally, previous work on data fusion has often worked on the assumption that only documents that are highly ranked in many of the lists are likely to be of relevance. We propose BurstFuseX, a fusion model that not only utilizes a microblog post’s ranking information but also exploits its publication time. BurstFuseX builds on an existing fusion method and rewards posts that are published in or near a burst of posts that are highly ranked in many of the lists being aggregated. We experimentally verify the effectiveness of the proposed late data fusion algorithm, and demonstrate that in terms of mean average precision it significantly outperforms the standard, state-of-the-art fusion approaches as well as burst or time-sensitive retrieval methods.

A probabilistic framework for information modelling and retrieval based on user annotations on digital objects

Annotations are a means to make critical remarks, to explain and comment things, to add notes and give opinions, and to relate objects. Nowadays, they can be found in digital libraries and collaboratories, for example as a building block for scientific discussion on the one hand or as private notes on the other. We further find them in product reviews, scientific databases and many Web 2.0 applications; even well-established concepts like emails can be regarded as annotations in a certain sense. Digital annotations can be (textual) comments, markings (i.e. highlighted parts) and references to other documents or document parts. Since annotations convey information which is potentially important to satisfy a user's information need, this thesis tries to answer the question of how to exploit annotations for information retrieval. It gives a first answer to the question if retrieval effectiveness can be improved with annotations. A survey of the universe reveals some facets of annotations; for example, they can be content level annotations (extending the content of the annotation object) or meta level ones (saying something about the annotated object). Besides the annotations themselves, other objects created during the process of annotation can be interesting for retrieval, these being the annotated fragments. These objects are integrated into an object-oriented model comprising digital objects such as structured documents and annotations as well as fragments. In this model, the different relationships among the various objects are reflected. From this model, the basic data structure for annotation-based retrieval, the structured annotation hypertext, is derived. In order to thoroughly exploit the information contained in structured annotation hypertexts, a probabilistic, object-oriented logical framework called POLAR is introduced. In POLAR, structured annotation hypertexts can be modelled by means of probabilistic propositions and four-valued logics. POLAR allows for specifying several relationships among annotations and annotated (sub)parts or fragments. Queries can be posed to extract the knowledge contained in structured annotation hypertexts. POLAR supports annotation-based retrieval, i.e. document and discussion search, by applying an augmentation strategy (knowledge augmentation, propagating propositions from subcontexts like annotations, or relevance augmentation, where retrieval status values are propagated) in conjunction with probabilistic inference, where P(d -> q), the probability that a document d implies a query q, is estimated. POLAR's semantics is based on possible worlds and accessibility relations. It is implemented on top of four-valued probabilistic Datalog. POLAR's core retrieval functionality, knowledge augmentation with probabilistic inference, is evaluated for discussion and document search. The experiments show that all relevant POLAR objects, merged annotation targets, fragments and content annotations, are able to increase retrieval effectiveness when used as a context for discussion or document search. Additional experiments reveal that we can determine the polarity of annotations with an accuracy of around 80%.

A merging strategy proposal: The 2-step retrieval status value method

A usual strategy to implement CLIR (Cross-Language Information Retrieval) systems is the so-called query translation approach. The user query is translated for each language present in the multilingual collection in order to compute an independent monolingual information retrieval process per language. Thus, this approach divides documents according to language. In this way, we obtain as many different collections as languages. After searching in these corpora and obtaining a result list per language, we must merge them in order to provide a single list of retrieved articles. In this paper, we propose an approach to obtain a single list of relevant documents for CLIR systems driven by query translation. This approach, which we call 2-step RSV (RSV: Retrieval Status Value), is based on the re-indexing of the retrieval documents according to the query vocabulary, and it performs noticeably better than traditional methods. The proposed method requires query vocabulary alignment: given a word for a given query, we must know the translation or translations to the other languages. Because this is not always possible, we have researched on a mixed model. This mixed model is applied in order to deal with queries with partial word-level alignment. The results prove that even in this scenario, 2-step RSV performs better than traditional merging methods.

Data-poor categorization and passage retrieval for gene ontology annotation in Swiss-Prot.

BackgroundIn the context of the BioCreative competition, where training data were very sparse, we investigated two complementary tasks: 1) given a Swiss-Prot triplet, containing a protein, a GO (Gene Ontology) term and a relevant article, extraction of a short passage that justifies the GO category assignement; 2) given a Swiss-Prot pair, containing a protein and a relevant article, automatic assignement of a set of categories.MethodsSentence is the basic retrieval unit. Our classifier computes a distance between each sentence and the GO category provided with the Swiss-Prot entry. The Text Categorizer computes a distance between each GO term and the text of the article. Evaluations are reported both based on annotator judgements as established by the competition and based on mean average precision measures computed using a curated sample of Swiss-Prot.ResultsOur system achieved the best recall and precision combination both for passage retrieval and text categorization as evaluated by official evaluators. However, text categorization results were far below those in other data-poor text categorization experiments The top proposed term is relevant in less that 20% of cases, while categorization with other biomedical controlled vocabulary, such as the Medical Subject Headings, we achieved more than 90% precision. We also observe that the scoring methods used in our experiments, based on the retrieval status value of our engines, exhibits effective confidence estimation capabilities.ConclusionFrom a comparative perspective, the combination of retrieval and natural language processing methods we designed, achieved very competitive performances. Largely data-independent, our systems were no less effective that data-intensive approaches. These results suggests that the overall strategy could benefit a large class of information extraction tasks, especially when training data are missing. However, from a user perspective, results were disappointing. Further investigations are needed to design applicable end-user text mining tools for biologists.

An entropy‐based interpretation of retrieval status value‐based retrieval, and its application to the computation of term and query discrimination value

AbstractThe concepts of Shannon information and entropy have been applied to a number of information retrieval tasks such as to formalize the probabilistic model, to design practical retrieval systems, to cluster documents, and to model texture in image retrieval. In this report, the concept of entropy is used for a different purpose. It is shown that any positive Retrieval Status Value (RSV)‐based retrieval system may be conceived as a special probability space in which the amount of the associated Shannon information is being reduced; in this view, the retrieval system is referred to as Uncertainty Decreasing Operation (UDO). The concept of UDO is then proposed as a theoretical background for term and query discrimination power, and it is applied to the computation of term and query discrimination values in the vector space retrieval model. Experimental evidence is given as regards such computation; the results obtained compare well to those obtained using vector‐based calculation of term discrimination values. The UDO‐based computation, however, presents advantages over the vector‐based calculation: It is faster, easier to assess and handle in practice, and its application is not restricted to the vector space model. Based on the ADI test collection, it is shown that the UDO‐based Term Discrimination Value (TDV) weighting scheme yields better retrieval effectiveness than using the vector‐based TDV weighting scheme. Also, experimental evidence is given to the intuition that the choice of an appropriate weighting scheme and similarity measure depends on collection properties, and thus the UDO approach may be used as a theoretical basis for this intuition.

A model of fuzzy linguistic IRS based on multi-granular linguistic information

An important question in IRSs is how to facilitate the IRS-user interaction, even more when the complexity of the fuzzy query language makes difficult to formulate user queries. The use of linguistic variables to represent the input and output information in the retrieval process of IRSs significantly improves the IRS-user interaction. In the activity of an IRS, there are aspects of different nature to be assessed, e.g., the relevance of documents, the importance of query terms, etc. Therefore, these aspects should be assessed with different uncertainty degrees, i.e., using several label sets with different granularity of uncertainty. In this contribution, an IRS based on fuzzy multi-granular linguistic information and a method to process the multi-granular linguistic information are proposed. The system accepts Boolean queries whose terms can be simultaneously weighted by means of ordinal linguistic values according to three semantics: a symmetrical threshold semantics, a relative importance semantics and a quantitative semantics. In the three semantics, the linguistic weights are represented by the linguistic variable “Importance”, but assessed on different label sets S 1, S 2 and S 3, respectively. The IRS evaluates weighted queries and obtains the linguistic retrieval status values of documents represented by the linguistic variable “Relevance” which is expressed on a different label set S ′.

From Retrieval Status Values to Probabilities of Relevance for Advanced IR Applications

Information Retrieval systems typically sort the result with respect to document retrieval status values (RSV). According to the Probability Ranking Principle, this ranking ensures optimum retrieval quality if the RSVs are monotonously increasing with the probabilities of relevance (as e.g. for probabilistic IR models). However, advanced applications like filtering or distributed retrieval require estimates of the actual probability of relevance. The relationship between the RSV of a document and its probability of relevance can be described by a “normalisation” function which maps the retrieval status value onto the probability of relevance (“mapping functions”). In this paper, we explore the use of linear and logistic mapping functions for different retrieval methods. In a series of upper-bound experiments, we compare the approximation quality of the different mapping functions. We also investigate the effect on the resulting retrieval quality in distributed retrieval (only merging, without resource selection). These experiments show that good estimates of the actual probability of relevance can be achieved, and that the logistic model outperforms the linear one. Retrieval quality for distributed retrieval is only slightly improved by using the logistic function.

Information retrieval method from uncertain requests and its optical implementation

We propose a high-speed information retrieval method which is tolerant to a user's uncertain request. To consider the degree of certainty of user's remembrance, a user's query is represented as a set of keywords and certainty factor pairs. Optical matrix-vector multiplication is applied to the computation of the retrieval status values to realize a high-speed retrieval system. An experimental result is also presented in this paper.

Combining the evidence of multiple query representations for information retrieval

We report on two studies in the TREC-2 program that investigated the effect on retrieval performance of combination of multiple representations of TREC topics. In one of the projects, five separate Boolean queries for each of the 50 TREC routing topics and 25 of the TREC ad hoc topics were generated by 75 experienced online searchers. Using the INQUERY retrieval system, these queries were both combined into single queries, and used to produce five separate retrieval results for each topic. In the former case, progressive combination of queries led to progressively improving retrieval performance, significantly better than that of single queries, and at least as good as the best individual single-query formulations. In the latter case, data fusion of the ranked lists also led to performance better than that of any single list. In the second project, two automatically produced vector queries and three versions of a manually produced P-norm extended Boolean query for each routing and ad hoc topic were compared and combined. This project investigated six different methods of combination of queries, and the combination of the same queries on different databases. As in the first project, progressive combination led to progressively improving results, with the best results, on average, being achieved by combination through summing of retrieval status values. Both projects found that the best method of combination often led to results that were better than the best performing single query. The combined results from the two projects have also been combined by data fusion. The results of this procedure show that combining evidence from completely different systems also leads to performance improvement.

The use of semantic links in hypertext information retrieval

Abstract Hypermedia links support the manual browsing through large hypertext or hypermedia collections; however, retrieving specific portions of information in such a collection cannot be achieved by browsing only. Retrieval mechanisms are necessary. In this article we highlight the notion of semantic links and show how the semantic content of hypertext links can be used for retrieval purposes. We present indexing and retrieval algorithms that exploit the link content in addition to the content of the nodes. Retrieval status values are obtained from a combination of conventional information retrieval and constrained spreading activation techniques. The results of some retrieval experiments in a hypertext test collection are presented: They are clearly superior to those obtained when the links are ignored. Since hypermedia collections basically have the same structure as hypertexts, the hope is that the same techniques can be applied to hypermedia information with the same effect of improving the retrieval results. Moreover, we hope that the results can be improved further by more sophisticated indexing algorithms.

The use of semantic links in hypertext information retrieval

Hypermedia links support the manual browsing through large hypertext or hypermedia collections; however, retrieving specific portions of information in such a collection cannot be achieved by browsing only. Retrieval mechanisms are necessary. In this article we highlight the notion of semantic links and show how the semantic content of hypertext links can be used for retrieval purposes. We present indexing and retrieval algorithms that exploit the link content in addition to the content of the nodes. Retrieval status values are obtained from a combination of conventional information retrieval and constrained spreading activation techniques. The results of some retrieval experiments in a hypertext test collection are presented: They are clearly superior to those obtained when the links are ignored. Since hypermedia collections basically have the same structure as hypertexts, the hope is that the same techniques can be applied to hypermedia information with the same effect of improving the retrieval results. Moreover, we hope that the results can be improved further by more sophisticated indexing algorithms.

Fuzzy information retrieval

Over the past decade, information retrieval has emerged as an active research area in the application of fuzzy set theory. Fuzzy information retrieval utilizes fuzzy sets to represent documents, membership degrees for query term relevance, fuzzy logical operators to define queries, and fuzzy compatibility measures to assess the retrieval status value of a document. This paper presents an overview of fuzzy relational databases and fuzzy information retrieval. A general description of the main components of fuzzy information retrieval are given: document representation, query representation, computer-aided query formulation, document retrieval status, and performance measures. Examples of areas currently being researched are provided. The relation between fuzzy information retrieval and fuzzy relational databases is examined.

A critical investigation of recall and precision as measures of retrieval system performance

Recall and precision are often used to evaluate the effectiveness of information retrieval systems. They are easy to define if there is a single query and if the retrieval result generated for the query is a linear ordering. However, when the retrieval results are weakly ordered, in the sense that several documents have an identical retrieval status value with respect to a query, some probabilistic notion of precision has to be introduced. Relevance probability, expected precision, and so forth, are some alternatives mentioned in the literature for this purpose. Furthermore, when many queries are to be evaluated and the retrieval results averaged over these queries, some method of interpolation of precision values at certain preselected recall levels is needed. The currently popular approaches for handling both a weak ordering and interpolation are found to be inconsistent, and the results obtained are not easy to interpret. Moreover, in cases where some alternatives are available, no comparative analysis that would facilitate the selection of a particular strategy has been provided. In this paper, we systematically investigate the various problems and issues associated with the use of recall and precision as measures of retrieval system performance. Our motivation is to provide a comparative analysis of methods available for defining precision in a probabilistic sense and to promote a better understanding of the various issues involved in retrieval performance evaluation.

Estimating effective display size in online retrieval systems

This article outlines a problem in commercial online retrieval systems, provides a review of the relevant literature, and presents a solution for a special case of the problem. Previous investigators have considered how to best determine, for a ranked list of records retrieved from an online retrieval system, whether or not the user should continue to display the output. This article examines the problem of how effective display size can be estimated as a means of assisting the users of commercial online retrieval systems. Although no experimental results are as yet available, the approach presented here will provide a guide to and prolegomenon for systematic study of the problem, as well as a method for providing the estimated number of relevant records remaining in a retrieved set ranked by a retrieval status value.

ABSTRACTS 25 (Chosen by Vo Raghavan from recent issues of journals in the retrieval area)

In this article, a general mathematical framework for information retrieval models is presented, giving more insight into the evaluation mechanisms that may be used in IR. In this framework a number of requirements for operators improving recall and precision are investigated. It is shown that these requirements can be satisfied by using descriptor weights and one combination operator only. Moreover, information items and queries - virtual items - can be treated in exactly the same way, reflecting the fact, that they both are descriptions of a number of concepts. Evaluation is performed by formulating similarity homomorphisms from query descriptions to retrieval status values that allow ranking items accordingly. For good comparisons, however, ranking must be done by the achieved proportion of perfect similarity rather than by similary itself. In term independence models, this normalization process may satisfy the homomorphism requirement of algebra under certain conditions, but it contradicts the requirement in the term dependence case. Nevertheless, by demanding separability for the unnormalized part of the evaluation measure, it can be guaranteed that the query is evaluated to each item descriptor by descriptor, and the normalization value must be calculated only once for the whole query, using the query as a whole. Also, for each real item, the normalization value must have been calculated only once, using the item as a whole.

Requirements for query evaluation in weighted information retrieval

In this article, a general mathematical framework for information retrieval models is presented, giving more insight into the evaluation mechanisms that may be used in IR. In this framework a number of requirements for operators improving recall and precision are investigated. It is shown that these requirements can be satisfied by using descriptor weights and one combination operator only. Moreover, information items and queries—virtual items—can be treated in exactly the same way, reflecting the fact, that they both are descriptions of a number of concepts. Evaluation is performed by formulating similarity homomorphisms from query descriptions to retrieval status values that allow ranking items accordingly. For good comparisons, however, ranking must be done by the achieved proportion of perfect similarity rather than by similarity itself. In term independence models, this normalization process may satisfy the homomorphism requirement of algebra under certain conditions, but it contradicts the requirement in the term dependence case. Nevertheless, by demanding separability for the unnormalized part of the evaluation measure, it can be guaranteed that the query is evaluated to each item descriptor by descriptor, and the normalization value must be calculated only once for the whole query, using the query as a whole. Also, for each real item, the normalization value must have been calculated only once, using the item as a whole.