Physicians found an interactive tool displaying structured evidence summaries for multiple comparisons understandable and useful: A qualitative user-testing study

Abstract

ObjectiveTo evaluate and improve MATCH-IT – a digital, interactive decision support tool displaying structured evidence summaries for multiple comparisons – to help physicians interpret and apply evidence from network meta-analysis (NMA) for their clinical decision-making. Study design and SettingWe conducted a qualitative user-testing study, applying principles from user centred design in an iterative development process. We recruited a convenience sample of practicing physicians in Norway, Belgium and Canada, and asked them to interpret structured evidence summaries for multiple comparisons – linked to clinical guideline recommendations – displayed in MATCH-IT. User-testing included (a) introduction of a clinical scenario, (b) a think aloud session with participant-tool-interaction and (c) a semi-structured interview. We video recorded, transcribed, and analysed user-tests using directed content analysis. The results informed new updates in MATCH-IT. ResultsDistributed across five development cycles we tested MATCH-IT with 26 physicians. Of these, 24 (94%) reported either no or sparse prior experience with interpretation of NMA. Physicians perceived MATCH-IT as easy to interpret, navigate and appreciated its ability to provide an overview of the evidence. Visualization of effects in pictograms and inclusion of information on burden of treatment (“practical issues”) were highlighted as potentially useful features in interacting with patients. We also identified problems, including undiscovered functionalities (drag and drop), suboptimal tutorial, and cumbersome navigation of the tool. In addition, physicians wanted definition/explanation of key terms (e.g., outcomes and “certainty”) and there were concerns that overwhelming evidence from a large NMA would complicate applicability to clinical practice. This led to several updates with development of a new start page, tutorial, updated user-interface for more efficient manoeuvring, solutions to display definition of key terms and a FAQ section. To facilitate interpretation of large networks, we improved categorization of results using colour coding and added filtering functionality. These modifications allowed physicians to focus on interventions of interest and reduce information overload. ConclusionsThis study provides proof of concept that physicians can use MATCH-IT to understand NMA evidence. Key features of MATCH-IT in a clinical context include providing an overview of the evidence, visualization of effects and the display of information on burden of treatments. However, unfamiliarity with GRADE concepts, time constraints and accessibility at the point of care may be challenges for use. To what extent our results are transferable to real-world clinical contexts remains to be explored.