Advancing the State of Policy Delphi Practice: A Systematic Review Evaluating Methodological Evolution, Innovation, and Opportunities

Abstract

The policy Delphi is a method that uses iterative stages of data collection to reveal positions on an issue within a panel of people with relevant knowledge. Policy Delphi surveys have become popular in a variety of disciplines since the method was first proposed in this journal in 1970. In this paper, we benchmark the state-of-the-art in policy Delphi methods, focusing on strengths and limitations, and on innovative ways of addressing key shortcomings. We report findings from a systematic review of 63 empirical studies conducted between 1971 and the end of 2014 that used the policy Delphi method. We found little consistency in how studies have been designed and executed. The inherent flexibility of the method is a strength, but a lack of consistency in how it is used undermines the ability of analysts to generate accessible insights. Specifically, our analysis reveals limited use of validity and reliability tests, a blurring of conventional and policy Delphi rationales, diverse data collection and analysis techniques, and mixed quality when reporting the approach, format, and results for individual studies. Indeed, potential new users of the method will struggle to understand what a policy Delphi survey actually is. We conclude with advice for addressing key shortcomings in current policy Delphi practice.