Determining the number of participants needed for the usability evaluation of e‐learning resources: A Monte Carlo simulation

Abstract

AbstractThe usability of computer interfaces has a major influence on learning. Optimising the usability of e‐learning resources is therefore essential. However, this may be neglected because of time and monetary constraints. User testing is a common approach to usability evaluation and involves studying typical end‐users interacting with the application being tested. Determining the minimum number of users that are required for such an evaluation is important as it has a direct bearing on the costs and time requirements. This issue has long been a subject of debate and the widely cited recommendation of five users being enough has been questioned. We conducted a usability evaluation of an e‐learning resource for electrolyte and acid‐base disorders by studying the interaction of medical doctors with the application. A total of 15 serious usability problems were detected, most of these related to an interactive treatment simulation. With this report, we are making available the data on the detection of serious usability problems by each of our participants. We have used these data to run a Monte Carlo simulation and examine how many users would be sufficient to test our application. The simulation revealed that we would have required 10 participants to detect 80% of the serious usability problems. The MATLAB® code for running the simulation is supplied, as are our calculations of problem discovery rates. The e‐learning resource which was evaluated is freely available, together with a revised version with all identified usability problems addressed.