Measurement, generalization, and publication: Sources of error in benefit transfers and their management

Abstract

Convergent validity tests of benefit transfer accuracy show errors to range from a few percentage points to 100% and more. This paper discusses three potential sources of errors that affect the accuracy of benefit transfers. (1) The measurement of values is subject to random errors and the caprices that arise from the many judgments and technical assumptions required by the researchers who conduct the primary studies. Measurement error occurs when researchers' decisions affect the transferability of measures of value or as the result of sampling. (2) Generalization error occurs when a measure of value is generalized to unstudied sites or resources. Generalization error is inversely related to the correspondence between study sites and policy sites. (3) Publication selection bias occurs when the objectives for publishing research limit benefit transfer applications of research outcomes. Criteria for selecting which research results are published may be at odds with the needs of benefit transfer practitioners. Several means for overcoming these sources of error are offered: standardized application of tested methods; closer adherence to benefit transfer protocol; the establishment of an e-journal with explicit criteria for fully recording, reporting, and disseminating research, which has the primary objective of estimating empirical measures of value.