Evaluating municipal websites: A methodological comparison of three think-aloud variants

Abstract

Usability methods have received relatively little methodological attention within the field of E-Government. This paper aims to address this gap by reporting on a usability test of the municipal website of Deventer (the Netherlands), carried out by means of three variants of the think-aloud method (concurrent/retrospective think-aloud protocols and constructive interaction). These three methods had proved successful in a previous evaluation of a different municipal website, yet we decided to replicate our study in order to investigate whether the three methods would reveal different results when applied to another municipal website with a different information architecture. The results of our study showed that, as in the previous municipal website evaluation, the three evaluation methods were largely comparable in terms of output. Nevertheless, we did find a number of differences between the present and previous municipal website evaluation regarding the workings of the three methods—differences that could be explained by the different information architectures of the municipal websites tested. This suggests that the three evaluation methods might indeed work differently depending on the nature of the website that is being evaluated, and calls for more research into the effect of task type on the validity of evaluation methods.