Evaluating the Quality of Online Survey Data Collected in 2018 in the USA: Univariate, Bivariate, and Multivariate Analyses

Abstract

Collecting survey data in Japan and the USA is difficult for many reasons, including the difficulty of creating complete sampling frames and low response rates. Online surveys, which collect data through online access panels and recruit members using nonprobability sampling, is becoming a popular alternative to traditional survey techniques, both in market and social science research. However, sociological researchers are hesitant to adopt online survey data collection techniques as it is unclear how representative such data are. This article examines the quality of online survey data by comparing online surveys with the two datasets collected using probability sampling, the American Community Survey and the General Social Survey. We conducted univariate, bivariate, and multivariate analyses and investigated whether similarities improve from the univariate analyses to bivariate relationships, and to bivariate relationships with controls. The results show that similarities improved from univariate to bivariate and multivariate analyses, but the results of bivariate and multivariate analyses did not differ. We concluded that data collected via volunteer‐based access panels yielded similar results to the benchmark data, especially when the relationships among variables are examined. We argue that automatically discrediting the data collected via volunteer‐based access panels is not good practice. With careful considerations of data quality, online survey data with nonprobability sampling could advance sociological research by giving researchers more freedom to ask questions at a relatively low cost, with a quick turnaround time, and more opportunities to conduct cross‐national research.