Paradata and dashboards in PIAAC

Abstract

PurposeThis paper aims to provides a brief review of the dashboard literature, an account of the development of performance dashboards for field data collection at Westat, and more specifically for the first cycle of the Programme for the International Assessment of Adult Competencies (PIAAC). A brief concluding section offers suggestions for improvements in survey dashboards for PIAAC’s next cycle.Design/methodology/approachTo manage field work at Westat, the authors create views into various paradata databases and present them in a dashboard, showing key performance indicators at a glance. Users can drill down from the dashboard into underlying databases to investigate potential problems. The US PIAAC dashboard is a monitoring system that supports daily review of many activities. For example, it provides overnight alerts to the field supervisor when global positioning system (GPS) data from an interviewer’s smartphone shows the interview occurred far from the respondent’s home.FindingsPerformance dashboards may represent best practice for monitoring field activities. Paradata sources and systems vary greatly across the PIAAC countries, but a multitude of process data exists in every country and can be used to create quality indicators and a monitoring system. PIAAC can establish standards/guidelines to improve visualization of quality metrics and management data, regardless of the local survey infrastructure.Originality/valueThe core of the paper is a case study of the experiences on the US PIAAC implementation of dashboards to monitor survey quality, production and costs, with special attention to the issue of fabrication.