Measurement agreement of the self-administered questionnaire of the Belgian Health Interview Survey: Paper-and-pencil versus web-based mode.

Elise Braekman,Johan Van Der Heyden,Rana Charafeddine,Lydia Gisle,Jean Tafforeau,Sabine Drieskens,Geert Molenberghs,Stefaan Demarest,Guido Van Hal,Finaba Berete,Arsham Alamian

doi:10.1371/journal.pone.0197434

Abstract

Before organizing mixed-mode data collection for the self-administered questionnaire of the Belgian Health Interview Survey, measurement effects between the paper-and-pencil and the web-based questionnaire were evaluated. A two-period cross-over study was organized with a sample of 149 employees of two Belgian research institutes (age range 22–62 years, 72% female). Measurement agreement was assessed for a diverse range of health indicators related to general health, mental and psychosocial health, health behaviors and prevention with kappa coefficients and intraclass correlation (ICC). The quality of the data collected by both modes was evaluated by quantifying the missing, ‘don’t know’ and inconsistent values and data entry mistakes. Good to very good agreement was found for all categorical indicators with kappa coefficients superior to 0.60, except for two mental and psychosocial health indicators namely the presence of a sleeping disorder and of a depressive disorder (kappa≥0.50). For the continuous indicators high to acceptable agreement was observed with ICC superior to 0.70. Inconsistent answers and data-entry mistakes were only occurring in the paper-and-pencil mode. There were no less missing values in the web-based mode compared to the paper-and-pencil mode. The study supports the idea that web-based modes provide, in general, equal responses to paper-and-pencil modes. However, health indicators based upon factual and objective items tend to have higher measurement agreement than indicators requiring an assessment of personal subjective feelings. A web-based mode greatly facilitates the data-entry process and guides the completing of a questionnaire. However, item non-response was not positively affected.

Highlights

Population surveys have traditionally used paper-and-pencil self-administered questionnaires to collect information on sensitive questions
With the growth of internet use, webbased questionnaires have become an important alternative to paper-and-pencil questionnaires due to their many advantages [1;2]
The continuous indicator vitality index had an intraclass correlation (ICC) value of 0.79 which indicates that the agreement was acceptable (Table 4)

Summary

Introduction

Population surveys have traditionally used paper-and-pencil self-administered questionnaires to collect information on sensitive questions. Web-based questionnaires cannot,be the sole mode of data collection for population surveys, as even in countries with high internet penetration, internet access and skills vary among demographic groups [5;6] To overcome this limitation, mixed-mode data collection including a web-based and paper-and-pencil mode can be used. A web-based mode offers a greater opportunity to multitask since respondents are more likely to be engaged in several other activities while completing the questionnaire [10;11] This might lead to “satisficing” behavior; respondents provide a satisfactory answer (e.g. answering don’t know or skipping the question) because an optimal response requires a substantial amount of cognitive effort [12;13]. It can create an illusion of privacy

Objectives

Methods

Conclusion