Optimizing Heuristic Evaluation Process in E-Commerce: Use of the Taguchi Method

Abstract

The heuristic evaluation method, a commonly used usability evaluation method, suffers from variability in evaluation results (Nielsen, 1993). Taguchi's quality-control method was applied to the heuristic evaluation process, and the robust condition of conducting heuristic evaluation that was insensitive to differences in an evaluator's cognitive styles was derived. Three control factors—task type, heuristic set, and evaluation mode—were used in the Taguchi analysis. The noise factor studied was the evaluator's cognitive style. The result of Taguchi analysis showed that if evaluators conducted the evaluations in pairs with the help of e-commerce-specific heuristics, the variability in the result caused by the noise factor was the minimum. The optimized evaluation process produced a 17.6% improvement in evaluation effectiveness over the traditional heuristic evaluation method (Nielsen, 1994) and reduced the variance among the participant's evaluation effectiveness at the same time.