Usability Evaluation of TV Interfaces: Subjective Evaluation Vs. Objective Evaluation

Abstract

ABSTRACT This study aims to design a usability test enhanced with subjective usability evaluation methods (concurrent think-aloud, post-task and posttest questionnaires) and objective usability evaluation methods (eye-tracking, emotion recognition and logging technologies). Furthermore, the relationships between the subjective metrics such as task difficulty level, PSSUQ (Post-Study System Usability Questionnaire) scores and objective metrics such as task completion time, eye tracking metrics (fixation count, fixation duration, average fixation duration, saccade count, saccade duration, scanpath length, blink count), logging metrics (keying and back keying count) and emotion recognition metrics (negative emotions count) are aimed to be analyzed. Therefore, a user testing study with 38 participants was conducted to evaluate the usability level of the TV interface of Digiturk, which is one of the digital TV broadcasting platforms in Turkey. The participants completed ten tasks related to the features of the TV interface, such as VOD (video-on-demand) watching, channel locking, and recording for future watches. Whether there is a significant difference between tasks in terms of subjective and objective metrics is investigated by using one-way ANOVA. The results show that while task completion time and task difficulty increase, values of all objective metrics increase. The relationships between the subjective and objective metrics are measured with Pearson correlation and the results show that there is a significant relationship between every subjective and objective metric except PSSUQ scores, which shows the perceived satisfaction level of the participants. Furthermore, exploratory factor analysis is performed to explore the latent structure of the usability measures. As a result of the factor analysis, a two-factor structure in which objective and subjective metrics load on two separate factors is obtained.