Advancing the theory and practice of system evaluation: a case study in geovisual analytics of social media

Abstract

ABSTRACT This paper advances the state-of-the-art in methodology design for empirical evaluation of (geo)visual analytics software. Specifically, we describe the process of design, development and application of a prototypical user study tailored to the evaluation of complex geovisual analytics tools that focus on social media analysis. We fist perform a synthesis of existing theory and best practices for software evaluation of comparable systems. We then demonstrate how the product of said synthesis – a methodological ‘check list’ – can be used to inform a proof-of-concept user study of an actual geovisual analytics software system. The resulting user study design accommodates for the use of real geographic social media datasets, the complexity of the intended analytical process, and for the learning challenges faced by the participants working with a fully-functional and mature geovisual analytics application, and is likely representative of a wide range of evaluation scenarios in (geo)visual analytics. A complete summary of all the study instruments is included to encourage their scrutiny, reuse and modification by others. Finally, we have discovered that participants’ curiosity and desire for autonomy played a noticeable role in the evaluation process – something not previously reported.