Questionnaire experience of the pictorial usability inventory (PUI) – a comparison of pictorial and hybrid usability scales

Abstract

In recent years, alternative types of usability questionnaires using graphical elements (pictorial scales) or a combination of graphical and verbal elements (hybrid scales) have been introduced. Previous research indicates that these questionnaires have advantages, such as increased respondent motivation, and drawbacks, such as extended questionnaire completion time. This study aimed to systematically investigate the psychometric properties and the respondents’ experience of two versions of a recently developed questionnaire, the Pictorial Usability Inventory (PUI), consisting of a hybrid and pictorial version. Given that questionnaire length is a crucial factor for the usefulness of a scale, the study tested long and short versions (8 items vs 3 items) of both questionnaire types. The study involved an online usability test with 777 participants, who were asked to complete one of the four PUI versions and an established verbal usability scale after solving three tasks on a webpage. The results demonstrated high sensitivity, high convergent validity, and good internal consistency for all four PUI versions. While the long pictorial scale achieved the best psychometric properties overall, participants preferred the hybrid scales, particularly the short version. The study's findings are in line with previous research on pictorial and hybrid instruments and suggest that hybrid instruments, particularly short ones, may be superior to purely pictorial instruments in terms of respondent-centred aspects conceptualised in the term ‘questionnaire experience’.