Improving the reliability of a nonprobability web survey: application to measuring gender pay gap

Abstract

ABSTRACT Nonprobability web surveys suffer from selection and coverage biases and are generally not representative of the target population. To carry out statistical modelling in a nonprobability web survey, we explore different methods of statistical adjustments to compensate for biases through the use of a probability-based reference sample. We also show that we need to account for these biases when imputing missing data. The methods for statistical adjustments include propensity score weighting with post-stratification and a technique called sample matching. For the substantive study, we use a nonprobability online web-survey taken from the WageIndicator (WI) programme (www.wageindicator.org) to estimate the gender pay gap (GPG) using the 2016 WI survey data from the Netherlands. We use the 2016 EU-SILC data as the probability-based reference sample. Based on the study of GPG, we show that using propensity score weighting with post-stratification outperforms sample matching with respect to compensating for biases and improves the outcome of the Blinder–Oaxaca decomposition model in terms of the degree of similarity relative to patterns found in representative probability samples in the Netherlands.