Relevance Judgments Research Articles

Development of a text search engine for medicinal chemistry patents

Division of Medical Information Sciences (SIMED), University Hospitals of Geneva and University of Geneva, Geneva, Switzerland Bibliomics and Text-Mining Group (BiTeM), Information Science Department, University of Applied Sciences, Geneva, Switzerland Novartis Institute for BioMedical Research – Text Mining Services (NIBR-IT/TMS), Novartis Pharma AG, Basel, Switzerland e-mail: emilie.pasche@hcuge.ch (corresponding), julien.gobeill@hesge.ch, fatma.oezdemirzaech@novartis.com, therese.vachon@novartis.com, christian.lovis@hcuge.ch, patrick.ruch@hesge.ch

Obtaining High-Quality Relevance Judgments Using Crowdsourcing

The performance of information retrieval (IR) systems is commonly evaluated using a test set with known relevance. Crowdsourcing is one method for learning the relevant documents to each query in the test set. However, the quality of relevance learned through crowdsourcing can be questionable, because it uses workers of unknown quality with possible spammers among them. To detect spammers, the authors' algorithm compares judgments between workers; they evaluate their approach by comparing the consistency of crowdsourced ground truth to that obtained from expert annotators and conclude that crowdsourcing can match the quality obtained from the latter.

An eye‐tracking approach to the analysis of relevance judgments on the Web: The case of Google search engine

Eye movement data can provide an in‐depth view of human reasoning and the decision‐making process, and modern information retrieval (IR) research can benefit from the analysis of this type of data. The aim of this research was to examine the relationship between relevance criteria use and visual behavior in the context of predictive relevance judgments. To address this objective, a multimethod research design was employed that involved observation of participants’ eye movements, talk‐aloud protocols, and postsearch interviews. Specifically, the results reported in this article came from the analysis of 281 predictive relevance judgments made by 24 participants using the Google search engine. We present a novel stepwise methodological framework for the analysis of relevance judgments and eye movements on the Web and show new patterns of relevance criteria use during predictive relevance judgment. For example, the findings showed an effect of ranking order and surrogate components (Title, Summary, and URL) on the use of relevance criteria. Also, differences were observed in the cognitive effort spent between very relevant and not relevant judgments. We conclude with the implications of this study for IR research.

Ranking, relevance judgment, and precision of information retrieval on children's queries: Evaluation of Google, Yahoo!, Bing, Yahoo! Kids, and ask Kids

This study employed benchmarking and intellectual relevance judgment in evaluating Google, Yahoo!, Bing, Yahoo! Kids, and Ask Kids on 30 queries that children formulated to find information for specific tasks. Retrieved hits on given queries were benchmarked to Google's and Yahoo! Kids' top‐five ranked hits retrieved. Relevancy of hits was judged on a graded scale; precision was calculated using the precision‐at‐ten metric (P@10). Yahoo! and Bing produced a similar percentage in hit overlap with Google (nearly 30%), but differed in the ranking of hits. Ask Kids retrieved 11% in hit overlap with Google versus 3% by Yahoo! Kids. The engines retrieved 26 hits across query clusters that overlapped with Yahoo! Kids' top‐five ranked hits. Precision (P) that the engines produced across the queries was P = 0.48 for relevant hits, and P = 0.28 for partially relevant hits. Precision by Ask Kids was P = 0.44 for relevant hits versus P = 0.21 by Yahoo! Kids. Bing produced the highest total precision (TP) of relevant hits (TP = 0.86) across the queries, and Yahoo! Kids yielded the lowest (TP = 0.47). Average precision (AP) of relevant hits was AP = 0.56 by leading engines versus AP = 0.29 by small engines. In contrast, average precision of partially relevant hits was AP = 0.83 by small engines versus AP = 0.33 by leading engines. Average precision of relevant hits across the engines was highest on two‐word queries and lowest on one‐word queries. Google performed best on natural language queries; Bing did the same (P = 0.69) on two‐word queries. The findings have implications for search engine ranking algorithms, relevance theory, search engine design, research design, and information literacy.

Content-Based Analysis Improves Audiovisual Archive Retrieval

Content-based video retrieval is maturing to the point where it can be used in real-world retrieval practices. One such practice is the audiovisual archive, whose users increasingly require fine-grained access to broadcast television content. In this paper, we take into account the information needs and retrieval data already present in the audiovisual archive, and demonstrate that retrieval performance can be significantly improved when content-based methods are applied to search. To the best of our knowledge, this is the first time that the practice of an audiovisual archive has been taken into account for quantitative retrieval evaluation. To arrive at our main result, we propose an evaluation methodology tailored to the specific needs and circumstances of the audiovisual archive, which are typically missed by existing evaluation initiatives. We utilize logged searches, content purchases, session information, and simulators to create realistic query sets and relevance judgments. To reflect the retrieval practice of both the archive and the video retrieval community as closely as possible, our experiments with three video search engines incorporate archive-created catalog entries as well as state-of-the-art multimedia content analysis results. A detailed query-level analysis indicates that individual content-based retrieval methods such as transcript-based retrieval and concept-based retrieval yield approximately equal performance gains. When combined, we find that content-based video retrieval incorporated into the archive's practice results in significant performance increases for shot retrieval and for retrieving entire television programs. The time has come for audiovisual archives to start accommodating content-based video retrieval methods into their daily practice.

PubMed searches by Dutch‐speaking nursing students: The impact of language and system experience

This study analyzes the search behavior of Dutch‐speaking nursing students with a nonnative knowledge of English who searched for information in MEDLINE/PubMed about a specific theme in nursing. We examine whether and to what extent their search efficiency is affected by their language skills. Our task‐oriented approach focuses on three stages of the information retrieval process: need articulation, query formulation, and relevance judgment. The test participants completed a pretest questionnaire, which gave us information about their overall experience with the search system and their self‐reported computer and language skills. The students were briefly introduced to the use of PubMed and MeSH (medical subject headings) before they conducted their keyword‐driven subject search. We assessed the search results in terms of recall and precision, and also analyzed the search process. After the search task, a satisfaction survey and a language test were completed. We conclude that language skills have an impact on the search results. We hypothesize that language support might improve the efficiency of searches conducted by Dutch‐speaking users of PubMed.

An analysis of human factors and label accuracy in crowdsourcing relevance judgments

Crowdsourcing relevance judgments for the evaluation of search engines is used increasingly to overcome the issue of scalability that hinders traditional approaches relying on a fixed group of trusted expert judges. However, the benefits of crowdsourcing come with risks due to the engagement of a self-forming group of individuals--the crowd, motivated by different incentives, who complete the tasks with varying levels of attention and success. This increases the need for a careful design of crowdsourcing tasks that attracts the right crowd for the given task and promotes quality work. In this paper, we describe a series of experiments using Amazon's Mechanical Turk, conducted to explore the `human' characteristics of the crowds involved in a relevance assessment task. In the experiments, we vary the level of pay offered, the effort required to complete a task and the qualifications required of the workers. We observe the effects of these variables on the quality of the resulting relevance labels, measured based on agreement with a gold set, and correlate them with self-reported measures of various human factors. We elicit information from the workers about their motivations, interest and familiarity with the topic, perceived task difficulty, and satisfaction with the offered pay. We investigate how these factors combine with aspects of the task design and how they affect the accuracy of the resulting relevance labels. Based on the analysis of 960 HITs and 2,880 HIT assignments resulting in 19,200 relevance labels, we arrive at insights into the complex interaction of the observed factors and provide practical guidelines to crowdsourcing practitioners. In addition, we highlight challenges in the data analysis that stem from the peculiarity of the crowdsourcing environment where the sample of individuals engaged in specific work conditions are inherently influenced by the conditions themselves.

How do health care professionals select medical images they need?

Purpose – This paper aims to report the results of a study investigating the relevance criteria used by health care professionals when seeking medical images.Design/methodology/approach – Data were collected from 29 participants using a think‐aloud protocol and face‐to‐face interviews and analysed using the Straussian version of grounded theory (GT).Findings – The results show that participants made use of 15 relevance criteria, although they agreed on topicality being the most important. The findings suggest that users apply different criteria in different situations when evaluating the relevancy of medical images.Originality/value – To the best of the authors' knowledge, there have been few studies that investigate relevance judgments for visually orientated documents. Thus, this study helps to contribute to the understanding of medical image resources and the information needs of health care professionals. A clear understanding of the medical image information needs of health care professionals is also...

Evaluating retrieval models using retrievability measurement

Evaluation is the main driving force in research, development and applications related to information retrieval (IR). In the traditional IR evaluation paradigm a list of query topics along with their relevance judgments are given. The main limitation of this kind of evaluation paradigm is that it focuses almost exclusively on a small set of judged documents and does not consider what influence the given retrieval models have on accessing all the relevant information in the collection. This is particularly important for recall oriented retrieval applications where we want to ensure that that everything relevant has been found. In this thesis we analyze the effectiveness of retrieval models from the documents retrievability point of view. We focus particularly on the retrieval bias of different retrieval models, and try to examine to what extent this bias restricts the users in retrieving relevant information. We explore this research with the help of three factors. First, we analyze the relationship between different characteristics of queries and retrievability. This is important from the query generation point of view, since in case of exhaustive queries, it is practically infeasible to complete retrievability approximation in reasonable time. The strong correlation between retrievability and query characteristics allows us to approximate the retrievability score accurately with the help of a query subset without processing an exhaustive number of queries. After this, we examine to what extent the retrievability and other IR effectiveness measures are related to each other. This specifically helps us to understand to what extent it is possible to automatically rank the effectiveness of retrieval models on the basis of their retrieval bias. This also offers a basis for optimizing retrieval systems for specific collections without the need to provide manually annotated ground truth. This is particularly useful for those retrieval domains where it is difficult to obtain a sufficient amount of relevance judgments. At the end we investigate and devise different retrieval strategies for mitigating the effect of low retrievability of documents. These include collection partitioning and query expansion on the basis of improved pseudo relevance feedback selection. The work present in this thesis provides an a novel approach for the evaluation and optimization of retrieval models particularly for recall oriented retrieval domains, where the focus is on retrieving all relevant information but not just retrieving a subset of relevant information. Available online at: http://www.ifs.tuwien.ac.at/~bashir/shariqbashir_phd_thesis.pdf.

A split-list approach for relevance feedback in information retrieval

In this paper we present a new algorithm for relevance feedback (RF) in information retrieval. Unlike conventional RF algorithms which use the top ranked documents for feedback, our proposed algorithm is a kind of active feedback algorithm which actively chooses documents for the user to judge. The objectives are (a) to increase the number of judged relevant documents and (b) to increase the diversity of judged documents during the RF process. The algorithm uses document-contexts by splitting the retrieval list into sub-lists according to the query term patterns that exist in the top ranked documents. Query term patterns include a single query term, a pair of query terms that occur in a phrase and query terms that occur in proximity. The algorithm is an iterative algorithm which takes one document for feedback in each of the iterations. We experiment with the algorithm using the TREC-6, -7, -8, -2005 and GOV2 data collections and we simulate user feedback using the TREC relevance judgements. From the experimental results, we show that our proposed split-list algorithm is better than the conventional RF algorithm and that our algorithm is more reliable than a similar algorithm using maximal marginal relevance.

A novel term weighting scheme based on discrimination power obtained from past retrieval results

Term weighting for document ranking and retrieval has been an important research topic in information retrieval for decades. We propose a novel term weighting method based on a hypothesis that a term’s role in accumulated retrieval sessions in the past affects its general importance regardless. It utilizes availability of past retrieval results consisting of the queries that contain a particular term, retrieved documents, and their relevance judgments. A term’s evidential weight, as we propose in this paper, depends on the degree to which the mean frequency values for the relevant and non-relevant document distributions in the past are different. More precisely, it takes into account the rankings and similarity values of the relevant and non-relevant documents. Our experimental result using standard test collections shows that the proposed term weighting scheme improves conventional TF*IDF and language model based schemes. It indicates that evidential term weights bring in a new aspect of term importance and complement the collection statistics based on TF*IDF. We also show how the proposed term weighting scheme based on the notion of evidential weights are related to the well-known weighting schemes based on language modeling and probabilistic models.

A learning approach to optimizing exploration–exploitation tradeoff in relevance feedback

Relevance feedback is an effective technique for improving search accuracy in interactive information retrieval. In this paper, we study an interesting optimization problem in interactive feedback that aims at optimizing the tradeoff between presenting search results with the highest immediate utility to a user (but not necessarily most useful for collecting feedback information) and presenting search results with the best potential for collecting useful feedback information (but not necessarily the most useful documents from a user's perspective). Optimizing such an exploration---exploitation tradeoff is key to the optimization of the overall utility of relevance feedback to a user in the entire session of relevance feedback. We formally frame this tradeoff as a problem of optimizing the diversification of search results since relevance judgments on more diversified results have been shown to be more useful for relevance feedback. We propose a machine learning approach to adaptively optimizing the diversification of search results for each query so as to optimize the overall utility in an entire session. Experiment results on three representative retrieval test collections show that the proposed learning approach can effectively optimize the exploration---exploitation tradeoff and outperforms the traditional relevance feedback approach which only does exploitation without exploration.

Topic modelling of clickthrough data in image search

In this paper we explore the benefits of latent variable modelling of clickthrough data in the domain of image retrieval. Clicks in image search logs are regarded as implicit relevance judgements that express both user intent and important relations between selected documents. We posit that clickthrough data contains hidden topics and can be used to infer a lower dimensional latent space that can be subsequently employed to improve various aspects of the retrieval system. We use a subset of a clickthrough corpus from the image search portal of a news agency to evaluate several popular latent variable models in terms of their ability to model topics underlying queries. We demonstrate that latent variable modelling reveals underlying structure in clickthrough data and our results show that computing document similarities in the latent space improves retrieval effectiveness compared to computing similarities in the original query space. These results are compared with baselines using visual and textual features. We show performance substantially better than the visual baseline, which indicates that content-based image retrieval systems that do not exploit query logs could improve recall and precision by taking this historical data into account.

Do Financial Statement Users Judge Relevance Based on Properties of Reliability?

ABSTRACT Relevance and reliability (now referred to as “representational faithfulness”) are qualities of financial information that both the Financial Accounting Standards Board and the International Accounting Standards Board use in setting standards for financial reporting. Despite their importance, very little research has addressed how financial statement users apply these constructs. Via experiments set within the fair value context, we show that users do not view them as independent constructs. Instead, variations in properties that are associated with the reliability of a measurement influence users' assessments of the relevance of fair value. The relationship between assessed relevance and assessed reliability is unidirectional, in that factors underlying reliability influence judgments of relevance, but factors underlying relevance do not influence judgments of reliability. Our findings are important because inappropriate assessments of relevance can influence firm valuation. The results are particularly meaningful in the context of fair value because such measurements can vary widely in reliability. JEL Classifications: M41.

An Overview of Search Engine Evaluation Strategies

Basically evaluation of search engine is the process of making judgment about the value, importance and quality of search engine, after considering search engines carefully. The evaluation of search engines has not been keeping up with the advancement of their development. Web search engines work differently based on different mode of interface, features, coverage of the web, ranking methods, delivery of advertising and many more such factors. It is not easy to evaluate them on a single basis. There are many strategies for evaluating search engines such as automatic evaluation, human relevance judgment based evaluation. The purpose of this paper is to review the search engine evaluation strategies in order to propose an enhanced method for evaluating search engines.

Identifying top news using crowdsourcing

The influential Text REtrieval Conference (TREC) retrieval conference has always relied upon specialist assessors or occasionally participating groups to create relevance judgements for the tracks that it runs. Recently however, crowdsourcing has been championed as a cheap, fast and effective alternative to traditional TREC-like assessments. In 2010, TREC tracks experimented with crowdsourcing for the very first time. In this paper, we report our successful experience in creating relevance assessments for the TREC Blog track 2010 top news stories task using crowdsourcing. In particular, we crowdsourced both real-time newsworthiness assessments for news stories as well as traditional relevance assessments for blog posts. We conclude that crowdsourcing not only appears to be a feasible, but also cheap and fast means to generate relevance assessments. Furthermore, we detail our experiences running the crowdsourced evaluation of the TREC Blog track, discuss the lessons learned, and provide best practices.

Criteria of geographic relevance: an experimental study

The relevance of geographic information has become an emerging problem in geographic information science due to an enormous increase in volumes of data at high spatial, temporal, and semantic resolution, because of ever faster rates of new data capturing. At the same time, it is not clear whether the concept of relevance developed in information science and implemented for document-based information retrieval can be directly applied to this new, highly dynamic setting. In this study, we analyze the criteria users apply when judging the relevance of geographic entities in a given mobile usage context. Two different experiments have been set up in order to gather users' opinions on a set of possible criteria, and their relevance judgements in a given scenario. The importance ascribed to the criteria in both experiments clearly implies that a new concept of relevance is required when dealing with geographic entities instead of digital documents. This new concept of ‘Geographic Relevance’ is highly dependent on personal mobility and user's activity, whose understanding may in turn be refined by the assimilation of ‘Geographic Relevance’ itself.

GIS 프로그래밍을 이용한 도로중심선 평면선형 평가

The horizontal alignment of road is deeply related with the stability of the road and traffic capacity. It is necessary to analyze horizontal alignment of road accurately for efficient maintenance of the road and relevance judgment about the standard. Recently the study on horizontal alignment of road using Lidar data and GPS was concluded, but they were many problem analyzing horizontal alignment radius of curvature in wide area. In this study, the tool which the radius of curvature can evaluate the suitability about Rules about the Road Structure & Facilities Standards by using center lines of the road of the digital map tries to implement on GIS. The interface was designed and implemented which can automatically estimate the Road Centerline Horizontal Alignment by using .

Large-scale validation and analysis of interleaved search evaluation

Interleaving is an increasingly popular technique for evaluating information retrieval systems based on implicit user feedback. While a number of isolated studies have analyzed how this technique agrees with conventional offline evaluation approaches and other online techniques, a complete picture of its efficiency and effectiveness is still lacking. In this paper we extend and combine the body of empirical evidence regarding interleaving, and provide a comprehensive analysis of interleaving using data from two major commercial search engines and a retrieval system for scientific literature. In particular, we analyze the agreement of interleaving with manual relevance judgments and observational implicit feedback measures, estimate the statistical efficiency of interleaving, and explore the relative performance of different interleaving variants. We also show how to learn improved credit-assignment functions for clicks that further increase the sensitivity of interleaving.

Ad Hominem Fallacies, Bias, and Testimony

An ad hominem fallacy is committed when an individual employs an irrelevant personal attack against an opponent instead of addressing that opponent’s argument. Many discussions of such fallacies discuss judgments of relevance about such personal attacks, and consider how we might distinguish those that are relevant from those that are not. This paper will argue that the literature on bias and testimony can helpfully contribute to that analysis. This will highlight ways in which biases, particularly unconscious biases, can make ad hominem fallacies seem effective, even when the irrelevance is recognized.