Ad Hoc Track Research Articles

A bias–variance evaluation framework for information retrieval systems

In information retrieval (IR), the improvement of the effectiveness often sacrifices the stability of an IR system. To evaluate the stability, many risk-sensitive metrics have been proposed. Since the theoretical limitations, the current works study the effectiveness and stability separately, and have not explored the effectiveness–stability tradeoff. In this paper, we propose a Bias–Variance Tradeoff Evaluation (BV-Test) framework, based on the bias–variance decomposition of the mean squared error, to measure the overall performance (considering both effectiveness and stability) and the tradeoff between effectiveness and stability of a system. In this framework, we define generalized bias–variance metrics, based on the Cranfield-style experiment set-up where the document collection is fixed (across topics) or the set-up where document collection is a sample (per-topic). Compared with risk-sensitive evaluation methods, our work not only measures the effectiveness–stability tradeoff of a system, but also effectively tracks the source of system instability. Experiments on TREC Ad-hoc track (1993–1999) and Web track (2010–2014) show a clear effectiveness–stability tradeoff across topics and per-topic, and topic grouping and max–min normalization can effectively reduce the bias–variance tradeoff. Experimental results on TREC Session track (2010–2012) also show that the query reformulation and increase of user data are beneficial to both effectiveness and stability simultaneously.

Sub-document level information retrieval

XML is increasingly used to mark up content in present day information repositories. Over the last decade or so, retrieval from XML document collections has emerged as an area of active research. For the Information Retrieval community, XML retrieval poses a two-fold problem: finding effective techniques to retrieve appropriate or the most useful XML elements in response to a user query; and devising an appropriate evaluation methodology to measure the effectivity of such retrieval techniques. . This study examines both these issues. First, we revisited the pivoted length normalization scheme in the Vector Space Model using standard benchmark collections for XML retrieval. We reduced two parameters used in pivoted length normalization to a single combined parameter and experimentally found its optimum value, which works well at both the element and document levels for XML retrieval. We observed that a substantial number of focused queries used in XML retrieval clearly state, besides the information need, what the user does not want. We demonstrated that this negative information, if not handled properly, degrades retrieval performance. We proposed a solution for automatically removing negative information from XML queries. This led to significant improvements in retrieval results. On the evaluation of XML retrieval, we first studied the sensitivity & robustness of various evaluation metrics and reliability and reusability of the assessment pool that has been used at INEX Ad Hoc track since 2007. Specifically we investigated the behaviour of the metrics when assessments are incomplete, or when query sets are small. We observed that early precision metrics are more error-prone and less stable under both these conditions. Average metric, however, performs comparatively better in this respect. System rankings remain largely unaffected even when assessment effort is substantially (but systematically) reduced. We also found that the INEX collections remain usable when evaluating non-participating systems. For a fixed amount of assessment effort, judging shallow pools for many queries is found to be better than judging deep pools for a smaller set of queries. However, when judging only a random sample of a pool, it is better to completely judge fewer topics than to partially judge many topics. We also proposed a simple and pragmatic approach of creating assessment pool for evaluation of retrieval systems. Instead of using an apriori-fixed pool depth for all topics, the pool is incrementally built and judged interactively on a per query basis. When no new relevant document is found for a reasonably long run of pool-depths, pooling was stopped for the topic. Our proposed approach offers a trade-off between the available effort and required level of performance. Moreover, it is flexible to deep pooling for potential topics in order to ensure better estimate of recall. We demonstrated the effectivity of the technique by substantially reducing the assessment effort without seriously compromising on the reliability of evaluation. The approach provided good results in the evaluation of XML retrieval as well as traditional document retrieval. This doctoral work was done at and submitted to the Indian Statistical Institute, Kolkata, under the supervision of Dr. Mandar Mitra. Dr. Ellen Voorhees and Dr. Tetsuya Sakai served as reviewers; Dr. Soumen Chakrabarti was the examiner at the defence of the thesis. Available online at http://www.isical.ac.in/?mandar/sukomal-thesis.pdf.

Muon and cosmogenic neutron detection in Borexino

Borexino, a liquid scintillator detector at LNGS, is designed for the detection of neutrinos and antineutrinos from the Sun, supernovae, nuclear reactors, and the Earth. The feeble nature of these signals requires a strong suppression of backgrounds below a few MeV. Very low intrinsic radiogenic contamination of all detector components needs to be accompanied by the efficient identification of muons and of muon-induced backgrounds. Muons produce unstable nuclei by spallation processes along their trajectory through the detector whose decays can mimic the expected signals; for isotopes with half-lives longer than a few seconds, the dead time induced by a muon-related veto becomes unacceptably long, unless its application can be restricted to a sub-volume along the muon track. Consequently, not only the identification of muons with very high efficiency but also a precise reconstruction of their tracks is of primary importance for the physics program of the experiment. The Borexino inner detector is surrounded by an outer water-Cherenkov detector that plays a fundamental role in accomplishing this task. The detector design principles and their implementation are described. The strategies adopted to identify muons are reviewed and their efficiency is evaluated. The overall muon veto efficiency is found to be 99.992 % or better. Ad-hoc track reconstruction algorithms developed are presented. Their performance is tested against muon events of known direction such as those from the CNGS neutrino beam, test tracks available from a dedicated External Muon Tracker and cosmic muons whose angular distribution reflects the local overburden profile. The achieved angular resolution is ∼ 3°-5° and the lateral resolution is ∼ 35-50 cm, depending on the impact parameter of the crossing muon. The methods implemented to efficiently tag cosmogenic neutrons are also presented.

Topic Signature Language Models for Ad hoc Retrieval

Semantic smoothing, which incorporates synonym and sense information into the language models, is effective and potentially significant to improve retrieval performance. Previously implemented semantic smoothing models such as the translation model have shown good experimental results. However, these models are unable to incorporate contextual information. To overcome this limitation, we propose a novel context-sensitive semantic smoothing method that decomposes a document into a set of weighted context-sensitive topic signatures and then maps those topic signatures into query terms. The language model with such a context- sensitive semantic smoothing is referred to as the topic signature language model. In detail, we implement two types of topic signatures, depending on whether ontology exists in the application domain. One is the ontology-based concept and the other is the multiword phrase. The mapping probabilities from each topic signature to individual terms are estimated through the EM algorithm. Document models based on topic signature mapping are then derived. The new smoothing method is evaluated on the TREC 2004/ 2005 Genomics Track with ontology-based concepts, as well as the TREC Ad Hoc Track (Disks 1, 2, and 3) with multiword phrases. Both experiments show significant improvements over the two-stage language model, as well as the language model with context- insensitive semantic smoothing.

Multimedia retrieval at INEX 2006

Structured document retrieval allows for the retrieval of document fragments, i.e. XML elements, containing relevant information. The main INEX Ad Hoc track focuses on text-based XML element retrieval. Although text is dominantly present in most XML document collections, other types of media can also be found. Existing research on multimedia information retrieval has shown that it is far from trivial to determine the combined relevance of a document that contains several multimedia objects. The objective of the INEX multimedia track is to exploit the XML structure that provides a logical level at which multimedia objects are connected, to improve the retrieval performance of an XML-driven multimedia information retrieval system.

Report on the ad-hoc track of the INEX 2005 workshop

The INitiative for the Evaluation of XML retrieval (INEX) has, since 2002, been working towards the goal of establishing an infrastructure, in the form of a large XML test collection and appropriate scoring methods, for the evaluation of content-oriented XML retrieval systems. In 2005, 47 organizations registered to participate in INEX. Throughout the year a number of groups dropped out due to resource requirements, while 11 further groups joined. INEX 2005 concluded with a total of 41 active groups. In addition to the main ad-hoc retrieval track, six further research tracks were included in INEX 2005, each studying different aspects of XML information access: interactive, relevance feedback, heterogeneous, natural language processing, and two new tracks for 2005, multimedia and document mining tracks.