Survey Questionnaire Survey Pretesting Method: An Evaluation of Survey Questionnaire via Expert Reviews Technique

Abstract

Whereas the literature on questionnaire pretesting has revealed a paradox, questionnaire pretesting is a simple technique to measure in advance whether a questionnaire causes problems for respondents or interviewers. Consequently, experienced researchers and survey methodologists have declared questionnaire pretesting indispensable. All the same, published survey reports provide no information about whether a questionnaire was pretested and, if so, how and with what results. Moreover, until recently, there has been limited methodological research on questionnaire pretesting. The universally acknowledged importance of questionnaire pretesting has been honoured more in theory than in practice. As a result, we know very little about pretesting and the extent to which a pretest serves its intended purpose and leads to value-added on questionnaires. An expert review is a traditional method of questionnaire pretesting. Expert reviews can be conducted with varying levels of organisation and rigor. On the lower end of the spectrum, an experienced subject matter expert, or survey methodologist reviews a draft questionnaire to identify issues with question wording or administration that may lead to measurement error. On the more rigorous end of the spectrum, as employed in this study is the Questionnaire Appraisal Scheme method, a standardized instrument review containing 28 problem types that allow experienced researchers and/or coders to code, analyse and compare the results of questionnaire problems reported by the independent expert reviewers for consistency and agreement across the expert reviewers. However, in spite of the wider use of the expert review as a pretest method, fewer empirical evaluations of this method exist. Specifically, there is little evidence as to whether different expert reviews consistently identified similar questionnaire problems. Similarly, there has been no reasonable level of agreements across the expert reviewers in their evaluation of questionnaire problems. This paper addresses these shortcomings. The protocols employed in the paper would contribute to reducing the shortfall in pretesting guidelines and encourage roundtable discussions in academia and management practice.