Lessons Learned from Large-Scale User Studies

Abstract

User studies with mobile devices have typically been cumbersome, since researchers have had to recruit participants, hand out or configure devices, and offer incentives and rewards. The increasing popularity of application stores has allowed researchers to use such mechanisms to recruit participants and conduct large-scale studies in authentic settings with relatively little effort. Most researchers who use application stores do not consider the side-effects or biases that such an approach may introduce. The authors summarize prior work that has reported experiences from using application stores as a recruiting, distribution and study mechanism, and also present a case study of a 4-week long study using the Android Market to deploy an application to over 4000 users that collected data on their mobile phone charging habits. The authors synthesize their own experiences with prior reported findings to discuss the challenges, advantages, limitations and considerations of using application stores as a recruitment and distribution approach for conducting large-scale studies.