An investigation of the relative influences of database informativeness, query size and query term specificity on the effectiveness of Medline searching

Abstract

An investigation of search effectiveness in Medline used regression analysis to compare the relative influences on search performance (characterised by Recall, Precision and the MZ-metric) of the three variables: database inform ativeness, and size and mean specificity of the query. (A 'query' is defined as an unstructured set of search terms.) A reduction in random and/or systematic error was sought by means of three experimental devices: (i) error arising from the subjectivity of third-party relevance judges was minimised by using behaviourally-generated target sets of records; (ii) errors associated with the definitions of queries were avoided (or at least standardised) by using algorithmically derived queries, and (iii) error associated with uncontrolled variation in the logical structure of the search expression was minimised by using search expres sions that had been optimised separately against each of the three performance criteria referred to. It was found that the dominant variable influencing search performance was the number of query terms. Database informativeness had a statistically significant, but small, influence on search performance in almost all the data-sets, and this influence could be in either direction, depending on the data-set. The influence of term specificity was in general not statistically significant. Overall, the three predicting variables were able to explain up to 50% of the variation in search performance.