Chapter 8 - Combined and Comparative Metrics

Abstract

Chapter 8 presents some techniques for combining basic user experience (UX) metrics, such as task success, time on task, and subjective ratings, to form new metrics or an overall “usability score.” An easy way to combine different usability metrics is to determine the percentage of users who achieve a combination of goals. This tells you the overall percentage of users who had a good experience with your product (based on the target goals). This method can be used with any set of metrics and is easily understood by management. One way of combining different metrics into an overall “usability score” is to convert each of the metrics to a percentage and then average them together. This requires being able to specify, for each metric, an appropriate minimum and maximum value. Another way to combine different metrics is to convert each metric to a z score and then average them together. Using z scores, each metric gets equal weight when they are combined, but the overall average of the z scores will always be 0. This metric is useful in comparing different subsets of the data to each other, such as data from different iterations, different groups, or different conditions. The Single Usability Metric (SUM) technique is another method for combining different metrics, specifically task completion, task time, errors, and task-level satisfaction rating. The method requires entry of individual task and participant data for the four metrics. The calculations yield a SUM score, as a percentage, for each task and across all tasks, including confidence intervals. Various types of graphs and charts can be useful for summarizing the results of a usability test in a “usability scorecard.” A combination line and column chart is useful for summarizing the results of two metrics for the tasks in a test. Radar charts are useful for summarizing the results of three or more metrics overall. A comparison chart using Harvey Balls to represent different levels of the metrics can summarize the results effectively for three or more metrics at the task level. Perhaps the best way to determine the success of a usability test is to compare the results to a set of predefined usability goals. Typically these goals address task completion, time, accuracy, and satisfaction. The percentage of users whose data met the stated goals can be a very effective summary. A reasonable alternative to comparing to predefined goals, especially for time data, is to compare actual performance data to the data for experts. The closer the actual performance is to expert performance, the better.