Information Retrieval Evaluation Research Articles

Query Expansion Using Proposed Location-Based Algorithm for Hindi–English CLIR: Analyzing Three Test Collections

The rapid growth of contents on the Web in different languages increases the demand of Cross-Lingual Information Retrieval (CLIR). The accuracy of result suffers due to many problems such as ambiguity and drift issue in query. Query Expansion (QE) offers reliable solution for obtaining suitable documents for user queries. In this paper, we proposed an architecture for Hindi–English CLIR system using QE for improving the relevancy of retrieved results. In this architecture, for the addition of term(s) at appropriate position(s), we proposed a location-based algorithm to resolve the drift query issue in QE. User queries in Hindi language have been translated into document language (i.e. English) and the accuracy of translation is improved using Back-Translation. Google search has been performed and the retrieved documents are ranked using Okapi BM25 to arrange the documents in the order of decreasing relevancy to select the most suitable terms for QE. We used term selection value (TSV) for QE and for retrieving the terms, we created three test collections namely the (i) description and narration of the Forum for Information Retrieval Evaluation (FIRE) dataset, (ii) Snippets of retrieved documents against each query and (iii) Nearest-Neighborhood (NN) words against each query word among the ranked documents. To evaluate the system, 50 queries of Hindi language are selected from the FIRE-2012 dataset. In this paper, we performed two experiments: (i) impact of the proposed location-based algorithm on the proposed architecture of CLIR; and (ii) analysis of QE using three datasets, i.e. FIRE, NN and Snippets. In the first case, result shows that the relevancy of Hindi–English CLIR is improved by performing QE using the location-based algorithm and a 12% of improvement is achieved as compared to the results of QE obtained without applying the location-based algorithm. In the second case, the location-based algorithm is applied on three datasets. The Mean Average Precision (MAP) values of retrieved documents after QE are 0.5379 (NN), 0.6018 (FIRE) and 0.6406 (Snippets) for the three test collections, whereas the MAP before QE is 0.37102. This clearly shows the significant improvement of retrieved results for all three test collections. Among the three test collections, QE has been found most effective along with Snippets as indicated by the results with the improvements of 6.48% and 19.12% over FIRE and NN test collections, respectively.

Information Retrieval Evaluation Measures Defined on Some Axiomatic Models of Preferences

Information retrieval (IR) evaluation measures are essential for capturing the relevance of documents to topics and determining the task performance efficiency of retrieval systems. The study of IR evaluation measures through their formal properties enables a better understanding of their suitability for a specific task. Some works have modeled the effectiveness of retrieval measures with axioms, heuristics, or desirable properties, leading to order relationships on the set where they are defined. Each of these ordering structures constitutes an axiomatic model of preferences (AMP), which can be considered as an “ideal” scenario of retrieval. Based on lattice theory and on the representational theory of measurement, this work formally explores numeric, metric, and scale properties of some effectiveness measures defined on AMPs. In some of these scenarios, retrieval measures are completely determined from the scores of a subset of document rankings: join-irreducible elements. All the possible metrics and pseudometrics, defined on these structures are expressed in terms of the join-irreducible elements. The deduced scale properties of the precision, recall, F -measure, RBP , DCG , and AP confirm some recent results in the IR field.

Recall, Precision, and F-Measure for Evaluating Information Retrieval System in Electronic Document Management Systems (EDMS)

The Legal Division of PT. ABC has implemented EDMS using the Integrated Legal Document Management Application, Centralized Legal Administration System (CLAS), to make the management of documents more accessible. The urgency is the system's accuracy in retrieving data or document input. The study employed a quantitative method with an experimental approach. Data collection techniques were observation, application search tests, documentation, and literature. Data analysis regarding the recall, precision, and F-measure calculation formulas will be compared with the standard information retrieval system effectiveness. The calculation results show that the average recall value for the search equals 1 or 100%. For the precision value that uses quotation marks, it is equal to 0.9713, while for searching without quotation marks, it is equal to 0.3920. From these results, the precision value is lower than the recall value. The precision value of 0.9713 is in the range of 0.68–1.00, which is placed as high precision, meaning that information retrieval is effective. The F-measure value is 0.98, close to 1, so the tracing system is sound. CLAS is quite effective in retrieving information according to user needs.

Summarization of Lengthy Legal Documents via Abstractive Dataset Building: An Extract-then-Assign Approach

Development of effective automatic summarization approaches for legal documents suffer from several challenges like extremely long document-summary pairs, lack of large scale training datasets with tractable document-summary token lengths, etc. In this work, we deal with the problem of legal document summarization by building a modified abstractive dataset from the original dataset. This ensures that the length of each document-summary pair is manageable and can be processed by the state of the art summarization approaches (such as BART). Secondly, we deal with the data scarcity problem by creating more number of training samples, from each of the original document-summary pair. This is done by creating multiple extractive summaries from each sample in the original dataset, following which ground-truth summary sentences are assigned to each of the extractive summary to generate new training samples. This results in a larger training dataset that can be utilized for fine-tuning summarization models. Our proposed approach has been evaluated on two different legal datasets- BillSum and Forum of Information Retrieval Evaluation (FIRE). With respect to the ROUGE metrics, the proposed approach is able to outperform pre-trained BART model fine-tuned on original dataset by (3−8)% for FIRE test sets, and by (1−3)% for the BillSum test sets. Considering the BERTScore metrics, the proposed approach obtains (1−2)% improvements on the FIRE test sets, while for the BillSum test sets (3−8)% improvements are observed. Such improvements suggest that the proposed dataset building approach can help achieve improved abstractive summarization of lengthy legal documents.

A Comparative Study of Convolutional Neural Networks and Recurrent Neural Networks for Chord Recognition

Using Mel-spectrograms, this study evaluates the effectiveness of Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNNs). Mel-spectrograms are justified by their non- linearity and similarity to the human hearing system. This study uses over 200 tracks by The Beatles and Queen collected through the Music Information Retrieval Evaluation Exchange. Data augmentation approaches are used to increase accuracy on unusual chords. This paper presents a 3-layer 2D CNN model trained on major and minor chords and then expanded to different types of chords. The dataset demonstrates that both models can recognize musical chords across various genres. We compare the proposed results to the existing literature and demonstrate the effectiveness of the proposed methodology. As a result of our analysis, we found that the CNN and RNN models were 79% and 76% accurate, respectively. The presented findings suggest that CNNs and RNNs are suitable models for chord recognition using Mel-spectrograms. Data augmentation can be an effective technique for improving accuracy on rare chords.

Capacity building to boost information and communication skills inside an institute of research

To enhance its visibility, the Library of the Istituto Superiore di Sanità (ISS) along with the Scientific Communication Unit of the same institution delivered a set of informal online training sessions, or webinars, on their fields of expertise: information retrieval, publication, effective communication, and research evaluation, specifically addressed to internal users. The collaboration was extremely useful in terms of improved knowledge on skills available among the personnel of these two services. It increased trust in the competencies of internal staff and at the same time it contributed to develop awareness of the value of the services rendered. Skills to use available online resources for training were improved as well as ISS staff cohesion favouring the development of new collaborations.

A Modular Efficiency Determination Formula for Information Retrieval Evaluations and Optimizations

A Modular Efficiency Determination Formula for Information Retrieval Evaluations and Optimizations

Modelling and Explaining IR System Performance Towards Predictive Evaluation

Information Retrieval (IR) systems play a fundamental role in many modern commodities, including search engines (SEs), digital libraries, recommender systems, and social networks. The IR task is particularly challenging because of the volatility of IR systems performance: users' information needs change daily, and so do the documents to be retrieved and the concept of what is relevant to a given information need. Therefore, the empirical offline evaluation of an IR system is a costly and slow post-hoc procedure, that happens after the system deployment. Given the challenges linked to empirical IR evaluation, predicting a system's performance before its deployment would add significant value to the development of an IR system. In this manuscript, we place the cornerstone for the prediction of IR performance, by considering two closely related areas: the modeling of IR systems performance and the Query Performance Prediction (QPP). The former area allows us to identify those features that impact the most on the performance and that can be used as predictors, while the latter provides us with a starting point to instantiate the predictive task in IR. Concerning the modeling of IR performance, we first investigate one of the most popular statistical tools, ANOVA. In particular, we compare traditional ANOVA with a recent approach, bootstrap ANOVA, and observe the different conclusions that can be achieved using these two different statistical tools [Faggioli and Ferro, 2021]. Secondly, using ANOVA, we study the concept of topic difficulty and observe that the topic difficulty is not an intrinsic property of the information need but stems from the formulation used to represent the topic [Culpepper et al., 2022]. Finally, we show how to use Generalized Linear Models (GLMs) as an alternative to the traditional linear modeling of IR performance [Faggioli et al., 2022]. We show how GLMs provide more powerful inference with comparable stability. Our analyses on the QPP domain start with developing a predictor used to select among a set of reformulations for the same information need, the best-performing one for the systematic review task [Di Nunzio and Faggioli, 2021]. Secondly, we investigate how to classify queries as either semantic or lexical to predict whether neural models will perform better than lexical ones. Finally, given the challenges shown in the evaluation of the previous approaches, we devise a new evaluation procedure, dubbed sMARE [Faggioli et al., 2021]. sMARE allows moving from a single point estimation of the performance to a distributional one, allowing us to achieve improved comparisons between QPP models and more precise analyses. Awarded by: University of Padova, Padova, Italy on 20 March 2023. Supervised by: Nicola Ferro. Available at: https://www.research.unipd.it/handle/11577/3472979.

Pattern-based hybrid book recommendation system using semantic relationships

In the fields of machine learning and artificial intelligence, recommendation systems (RS) or recommended engines are commonly used. In today's world, recommendation systems based on user preferences assist consumers in making the best decisions without depleting their cognitive resources. They can be applied to a variety of things, including search engines, travel, music, movies, literature, news, gadgets, and dining. A lot of people utilize RS on social media sites like Facebook, Twitter, and LinkedIn, and it has proven beneficial in corporate settings like those at Amazon, Netflix, Pandora, and Yahoo. There have been numerous proposals for recommender system variations. However, certain techniques result in unfairly recommended things due to biased data because there are no established connections between the items and consumers. In order to solve the challenges mentioned above for new users, we propose in this work to employ Content-based Filtering (CBF) and Collaborative Filtering (CF) with semantic relationships to capture the relationships as knowledge-based book recommendations to readers in a digital library. When proposing things, patterns are more discriminative than single phrases. To capture the similarity of the books that the new user had retrieved, the patterns were grouped in a semantically equivalent manner using the Clustering method. The effectiveness of the suggested model is examined through a series of extensive tests employing Information Retrieval (IR) evaluation criteria. Recall Precision and F-Measure, two of the three widely used performance measuring metrics, were employed. The findings demonstrate that the suggested model performs noticeably better than cutting-edge models.

Investigating the role of in-situ user expectations in Web search

Pre-adoption expectations often serve as an implicit reference point in users’ evaluation of information systems and are closely associated with their goals of interactions, behaviors, and overall satisfaction. Despite the empirically confirmed impacts, users’ search expectations and their connections to tasks, users, search experiences, and behaviors have been scarcely studied in the context of online information search. To address the gap, we collected 116 sessions from 60 participants in a controlled-lab Web search study and gathered direct feedback on their in-situ expected information gains (e.g., number of useful pages) and expected search efforts (e.g., clicks and dwell time) under each query during search sessions. Our study aims to examine (1) how users’ pre-search experience, task characteristics, and in-session experience affect their current expectations and (2) how user expectations are correlated with search behaviors and satisfaction. Our results with both quantitative and qualitative evidence demonstrate that: (1) user expectation is significantly affected by task characteristics, previous and in-situ search experience; (2) user expectation is closely associated with users’ browsing behaviors and search satisfaction. The knowledge learned about user expectation advances our understanding of users’ search behavioral patterns and their evaluations of interaction experience and will also facilitate the design, implementation, and evaluation of expectation-aware user models, metrics, and information retrieval (IR) systems.

The indexing policy in the practices of Brazilian institutional repositories: A diagnostic study from the perception of managers and indexers

Institutional repositories aim to gather, treat and disseminate information resources produced in a given institution. The adoption of an indexing policy aims to improve information retrieval by the user community of the institutional repository, by proposing elements that standardize the practices of librarians who work in subject cataloging. The objective of the research is to analyze the perception of managers and indexers of Brazilian institutional repositories about the indexing policy for information organization and representation, with the specific objectives of performing a compared evaluation of elements and variables (characteristics, processes, and instruments) of the indexing policy regarding indexing and retrieval, discussion of vocabulary control procedures, and obtaining an overview focusing on the perception of the need for an indexing policy in Brazilian institutional repositories. Semi-structured interviews were applied to nine participants from eight institutional repositories to investigate the requirements, elements and variables of the indexing policy in the practice of managers and librarians of institutional repositories. The results reveal the need for improvements in the practices of the analyzed institutional repositories regarding the activities of subject cataloging, standardization of subject metadata, and vocabulary control, queries, and in carrying out tests or trials for continuous evaluation of indexing and information retrieval with users. The study also revealed that this scenario can be more positive if institutional repositories adopt an indexing policy for the practices they perform in repositories to be understood by managers and librarians as a tool to improve information retrieval. La política de indización en las prácticas de los repositorios institucionales brasileños: un estudio de diagnóstico desde la percepción de gestores e indexadores

Report on the 2nd Joint Conference of the Information Retrieval Communities in Europe (CIRCLE 2022)

The Joint Conference of the Information Retrieval Communities in Europe (CIRCLE 2020) is the second joint conference of the French, Italian, and Spanish information retrieval communities. The three days of conference gathered interesting studies and research work on a wide range of topics on information retrieval, such as topic and document modeling, web search, information retrieval in e-government, social media, recommender systems, information retrieval evaluation, indexing and annotation, user profiling and interaction, frameworks and systems, and semantic extraction. It was hosted by Université de Toulouse, France in a holiday resort at Samatan. Date: 4--7 July, 2022. Website: https://www.irit.fr/CIRCLE/.

Toward Cranfield-inspired reusability assessment in interactive information retrieval evaluation

Re-using research resources is essential for advancing knowledge and developing repeatable, empirically solid experiments in scientific fields, including interactive information retrieval (IIR). Despite recent efforts on standardizing research re-use and documentation, how to quantitatively measure the reusability of IIR resources still remains an open challenge. Inspired by the reusability evaluations on Cranfield experiments, our work proactively explores the problem of measuring IIR test collection reusability and makes threefold contributions: (1) constructing a novel usefulness-oriented framework with specific analytical methods for evaluating the reusability of IIR test collections consisting of query sets, document/page sets, and sets of task-document usefulness (tuse); (2) explaining the potential impacts of varying IIR-specific factors (e.g. search tasks, sessions, user characteristics) on test collection reusability; (3) proposing actionable methods for building reusable test collections in IIR and thereby amortizing the true cost of user-oriented evaluations. The Cranfield-inspired reusability assessment framework serves as an initial step towards accurately evaluating the reusability of IIR research resources and measuring the reproducibility of IIR evaluation results. It also demonstrates an innovative approach to integrating the insights from individual heterogeneous user studies with the evaluation techniques developed in standardized ad hoc retrieval experiments, which will facilitate the maturation of IIR fields and eventually benefits both sides of research.

Prediction of the high-cost normalised discounted cumulative gain (nDCG) measure in information retrieval evaluation

Introduction. Information retrieval systems are vital to meeting daily information needs of users. The effectiveness of these systems has often been evaluated using the test collections approach, despite the high evaluation costs of this approach. Recent methods have been proposed that reduce evaluation costs through the prediction of information retrieval performance measures at the higher cut-off depths using other measures computed at the lower cut-off depths. The purpose of this paper is to propose two methods that addresses the challenge of accurately predicting the normalised discounted cumulative gain (nDCG) measure. Method. Data from selected test collections of the Text REtrieval Conference was used. The proposed methods employ the gradient boosting and linear regression models trained with topic scores of measures partitioned by TREC Tracks. Analysis. To evaluate the proposed methods, the coefficient of determination, Kendall's tau and Spearman correlations were used. Results. The proposed methods provide better predictions of the nDCG measure at the higher cut-off depths while using other measures computed at the lower cut-off depths. Conclusions. These proposed methods have shown improvement in the predictions of the nDCG measure while reducing the evaluation costs.

To Study the Influence of Social-Media on Consumer Buying Behavior

Abstract: The purpose of this research is to empirically explore the role of social media in the consumer decision-making process. Quantitative research looks at how using social media can change your experience. Today, outsiders in developed regions are overwhelmed by customer feedback on products and government, and as a result, influence conclusions in unrelated areas. Web-based social networks attract shoppers because advertisers have no control over the content, timing, or repetition of online discussions between shoppers. The results show that the use of social media influences consumer satisfaction during the information retrieval and alternative evaluation stages, and satisfaction increases as consumers follow the final purchase decision and post-purchase evaluation process. Indicates that it will be done.

ListMAP: Listwise learning to rank as maximum a posteriori estimation

Listwise learning to rank models, which optimize the ranking of a document list, are among the most widely adopted algorithms for finding and ranking relevant documents to user information needs. In this paper, we propose ListMAP, a new listwise learning to rank model with prior distribution that encodes the informativeness of training data and assigns different weights to training instances. The main intuition behind ListMAP is that documents in the training dataset do not have the same impact on training a ranking function. ListMAP formalizes the listwise loss function as a maximum a posteriori estimation problem in which the scoring function must be estimated such that the log probability of the predicted ranked list is maximized given a prior distribution on the labeled data. We provide a model for approximating the prior distribution parameters from a set of observation data. We implement the proposed learning to rank model using neural networks. We theoretically discuss and analyze the characteristics of the introduced model and empirically illustrate its performance on a number of benchmark datasets; namely MQ2007 and MQ2008 of the Letor 4.0 benchmark, Set 1 and Set 2 of the Yahoo! learning to rank challenge data set, and Microsoft 30k and Microsoft 10K datasets. We show that the proposed models are effective across different datasets in terms of information retrieval evaluation metrics NDCG and MRR at positions 1, 3, 5, 10, and 20.

Social media text analytics of Malayalam\u2013English code-mixed using deep learning

Zigzag conversational patterns of contents in social media are often perceived as noisy or informal text. Unrestricted usage of vocabulary in social media communications complicates the processing of code-mixed text. This paper accentuates two major aspects of code mixed text: Offensive Language Identification and Sentiment Analysis for Malayalam–English code-mixed data set. The proffered framework addresses 3 key points apropos these tasks—dependencies among features created by embedding methods (Word2Vec and FastText), comparative analysis of deep learning algorithms (uni-/bi-directional models, hybrid models, and transformer approaches), relevance of selective translation and transliteration and hyper-parameter optimization—which ensued in F1-Scores (model’s accuracy) of 0.76 for Forum for Information Retrieval Evaluation (FIRE) 2020 and 0.99 for European Chapter of the Association for Computational Linguistics (EACL) 2021 data sets. A detailed error analysis was also done to give meaningful insights. The submitted strategy turned in the best results among the benchmarked models dealing with Malayalam–English code-mixed messages and it serves as an important step towards societal good.

Evaluation of an interactive visualization tool to increase energy literacy in the building sector

The building sector, and especially residential households and office buildings, account for a large share of global emissions. Meanwhile, energy literacy is extremely low amongst residents and citizens in general, leading to insufficient evaluations of energy efficiency measures and technology equipment for buildings. To address this issue, we develop a research model and design an experiment to evaluate the ability of a website with interactive and vivid features to convey information in an engaging way, thus increasing the users’ enjoyment and their intention to (re)-use and recommend the website as well as the usefulness for information retrieval and technology evaluation. We conduct an experiment with two treatments in which the participants interact with an animated and a static website, respectively. While participants’ self-assessed knowledge improvement is statistically higher in the animated treatment, no difference was found in tested knowledge assessment or technology-specific knowledge. We find that the vividness of the website plays an important role for both the utilitarian and hedonic purpose of the website. However, somewhat contrasting to existing theories, interactivity did not increase enjoyment or diagnosticity.

Correlation and prediction of high-cost information retrieval evaluation metrics using deep learning

Introduction. To reduce cost of the evaluation of information retrieval systems, this study proposes a method that employs deep learning to predict the precision evaluation metric. It also aims to show why some of existing evaluation metrics correlate with each other while considering the varying distributions of relevance assessments. It aims to ensure reproducibility of all the presented experiments. Method. Using data from several test collections of the Text REetrieval Conference (TREC) we show why some evaluation metrics correlate with each other, through mathematical intuitions. In addition, regression models were used to investigate how the predictions of the evaluation metrics are affected by queries or topics with variations of relevance assessments. Lastly, the proposed prediction method employs deep learning. Analysis. We use coefficient of determination, Kendall's tau, Spearman and Pearson correlations. Results. This study showed that the proposed method performed better predictions than other recently proposed methods in retrieval research. It also showed why the correlation exists between precision and rank biased precision metrics, and why recall and average precision metrics have reduced correlation when the cut-off depth increases. Conclusions. The proposed method and the justifications for the correlations between some pairs of retrieval metrics will be valuable to researchers for the predictions of the evaluation metrics of information retrieval systems.

Vietnamese Information Retrieval Test Collection Development

Linguistic features processing can remarkably affect information retrieval effectiveness. Evaluation is an important phase in the development process of any information retrieval solutions. In the current state, due to the lack of test collections, it’s difficult to carry out research and share achieved results, since the development of Vietnamese test collections are required. This report will provide general information about information retrieval evaluation problems, the test collection development and evaluation processes. The achieved results show that information retrieval test collection development can take a lot of time and effort, but can be divided into many phases and can be accomplished incrementally.