Prioritizing Design Goals for a Medical Simulator Using Pairwise Comparisons

Abstract

A simulator is being developed to teach psychomotor skills for the lateral canthotomy and cantholysis procedure to medical providers, filling a current training gap at military medical training facilities. Subject Matter Experts (SMEs) agree on training system design goals; however, there is a lack of consensus on the relative order and magnitude of the priorities of each goal. The designs and implementations to achieve each goal have several interdependencies and tradeoffs, which require a comparison between different sets of design goals. A survey of pairwise comparisons was distributed to SMEs and two methods (a simple method and the Bradley-Terry model) were used to develop a unidimensional scale of prioritizations for each design goal. There was agreement in results across the simple method (calculating the proportion of times an alternative was chosen as preferable) and the Bradley-Terry model. The Bradley-Terry method offered a means to calculate measures of uncertainty, showing nuances where scores may overlap. This prioritization method enabled the research and development team to reconcile differing SME opinions. This also allowed for informed design decisions to ensure that the training system met expectations of SMEs and end users. Future efforts will involve developing scores for relevant sets of demographics to determine if there is concurrence and dissent in priorities.