TOWARDS THE IDENTIFICATION AND CONTROL OF VARIABLES IN INFORMATION RETRIEVAL EXPERIMENTATION

Abstract

Experimentation in information science has suffered from lack of definition and deficiencies in experimental design and control. With the absence of such control it is often difficult to determine what has in fact been tested and under what conditions. The reproducibility of experiments, if desirable, is consequently open to doubt. Inadequate attention has been given to experimental design with even less care devoted to the provision of procedures for the successful execution of experiments. This paper identifies the essential elements necessary for valid and reliable experiments in the testing of retrieval systems. The variables (components and sub‐components) operating within the context of retrieval systems are enumerated, defined, and discussed. Models, experimental design, and methods of controlling variables in actual experimental situations are illustrated. The meaning of ‘control’ in retrieval experiments is discussed. Attention is given to sources of experimental bias which contaminate research findings. Specific points are discussed with reference to experimentation undertaken in the Comparative Systems Laboratory at Western Reserve. Variables and sources of bias are those identified and manipulated in experiments performed on the basis of an IR system model, incorporating components related to purpose: discipline, users, file size; and to functions: acquisition, input source, indexing language, coding, file organization, question analysis, search procedures, and dissemination. Each of these components is analysed separately and numerous variables operating within the components are grouped into those associated with system mechanics, human factors, or human‐system interaction. Since a multiplicity of variables associated with these components operates within any retrieval system it is argued that any experimental design, as a minimum, must take into account the formulation of adequate procedures regarding the control of all the enumerated variables.