Automated summative usability studies

Abstract

This paper evaluates a method for summative usability testing using an automated data collection system. We found automated summative testing to be a simple and effective alternative to lab-based summative testing and could be successfully conducted remotely. In our study, a web-based control window led participants through the summative study, provided tasks to perform, and asked follow up questions about the user experience. Using a within-group comparison, we found no major differences between data collected by a usability engineer and that collected through an automated testing system for performance metrics. Using a between-group comparison, we found automated summative studies could be conducted remotely with minor but acceptable differences in time on task and likelihood to give up on a task compared to lab-based testing. Task success and task satisfaction ratings were not different between remote and lab-based summative testing. Written comments provided by participants through the testing system were sufficient to identify the major usability problems that led to task failure but did not reveal as comprehensive a set of issues as did a usability engineer observing the sessions.