A Low-cost Method to Try to Improve Panel Survey Representation

Abstract

Survey researchers are perpetually faced with the challenge of trying to balance quality, time, and cost. However, it is essentially impossible for surveys to achieve high quality, a quick turnaround time, and low costs. So, most often researchers settle for trying to achieve two of these three. As survey costs have escalated, funders have become unwilling to keep spending more on surveys, and what often happens is that quality is compromised. It is within this context that we propose a low-cost approach to improve panel survey quality by increasing a panel’s representation of its target population. This includes both the quality of the initial sample recruited to join the panel and the sample that remains active within the panel. Our approach addresses a way to raise the representativeness of initial and on-going panel samples essentially without raising on-going costs. We present a case study of this approach, using the Life in Australia™ panel. We began by asking 1,557 panel members an open-ended question about why they joined and remained in the panel. We then content analyzed the responses to create quantitative data which could be analyzed statistically. Also, we gathered all the communications used with these panel members and performed a qualitative content analysis to identify the themes used to try to persuade the sampled panelists to join and stay active in the panel. We then compared the two sets of findings. There were six motivations that panelists reported that were not included in the recruitment and maintenance communications that were used with potential panelists and currently inactive panelists. We also found that panelists with certain background characteristics were more likely to report certain motivations for being in the panel. We acknowledge that our approach is not a panacea but believe it adds to the “toolbox” of panel companies.