Chapter 7 Personalised Search Engine Evaluation: Methodologies and Metrics

Abstract

Purpose — To develop methodologies to evaluate search engines according to an individual's preference in an easy and reliable manner, and to formulate user-oriented metrics to compare freshness and duplication in search results. Design/methodology/approach — A personalised evaluation model for comparing search engines is designed as a hierarchy of weighted parameters. These commonly found search engine features and performance measures are given quantitative and qualitative ratings by an individual user. Furthermore, three performance measurement metrics are formulated and presented as histograms for visual inspection. A methodology is introduced to quantitatively compare and recognise the different histogram patterns within the context of search engine performance. Findings — Precision and recall are the fundamental measures used in many search engine evaluations due to their simplicity, fairness and reliability. Most recent evaluation models are user oriented and focus on relevance issues. Identifiable statistical patterns are found in performance measures of search engines. Research limitations/implications — The specific parameters used in the evaluation model could be further refined. A larger scale user study would confirm the validity and usefulness of the model. The three performance measures presented give a reasonably informative overview of the characteristics of a search engine. However, additional performance parameters and their resulting statistical patterns would make the methodology more valuable to the users. Practical implications — The easy-to-use personalised search engine evaluation model can be tailored to an individual's preference and needs simply by changing the weights and modifying the features considered. A user is able to get an idea of the characteristics of a search engine quickly using the quantitative measure of histogram patterns that represent the search performance metrics introduced. Originality/value — The presented work is considered original as one of the first search engine evaluation models that can be personalised. This enables a Web searcher to choose an appropriate search engine for his/her needs and hence finding the right information in the shortest time with the least effort.