TOPIC: Education, Research, and Quality Improvement TYPE: Original Investigations PURPOSE: Risk-stratification in pulmonary artery hypertension (PAH) is essential to determine prognosis, monitor disease progression, and response to treatment. To support accurate risk prediction, machine learning (ML) can “phenotype” patients with complex cardiovascular pathology more accurately than probabilistic risk-models derived methods or expert opinion. However, the output of ML algorithms is often less interpretable/‘user-friendly’, which is a barrier to its integration with best practices. METHODS: To understand the perspective of the provider when using a ML-based decision support tool, we interviewed physicians (n=30) with PAH expertise at academic, community, and international hospitals. Using a semi-structured interview protocol, we elicited feedback about how results from PHORA – a tree-augmented naïve Bayes model to predict 1-year survival in PAH patients – should be incorporated into a decision support tool for PAH risk prediction. Our interviews incorporated an interactive prototype of a user interface of the decision support tool. RESULTS: Using qualitative methods and thematic analysis, we extracted a cohesive set of design guidelines from our interviews. Highlights include: 1) the importance of showing the impact of the influence of specific variables into the overall risk prediction to help the user understand the logical basis for the prognosis 2) situating the model in context with more traditional risk calculators so physicians can calibrate their trust 3) presenting a temporal trajectory of the patients' risk over time, so they can see which variables are responsible for improved or declining health, and 4) the ability to add clinical notes to help keep track of useful situational context of the patient that might not be present in the collected variables alone. A common, unsolicited suggestion by several physicians interviewed was to consider using the interface as a communication tool to show patients, rather than simply as a prognosis tool to guide treatment. CONCLUSIONS: We assessed the needs of physicians for employing a ML-based decision support tool in the practice of determining PAH risk. Our interviews suggest physicians require interpretable models and should be contextualized with traditional statistical calculators and the changes of patient risk over time. By visualizing such information, physicians also believe, in addition to prognosis, such tools can serve as a useful communication tool for patients. CLINICAL IMPLICATIONS: Uptake of clinical decision support tools employing ML models will depend on the manner in which algorithmic output is presented to clinicians and patients. Our interviews suggest that by involving physicians in the design process, we can effectively design clinical decision support tools. DISCLOSURES: No relevant relationships by James Antaki, source=Web Response Grants relationship with Actelion Please note: 2019-2021 Added 04/29/2021 by Raymond Benza, source=Web Response, value=Grant/Research Support Grants relationship with Bayer AG Please note: 2019-2021 Added 04/29/2021 by Raymond Benza, source=Web Response, value=Grant/Research Support Grants relationship with Bellerophon Please note: 2019-2021 Added 04/29/2021 by Raymond Benza, source=Web Response, value=Grant/Research Support Grants relationship with EIGER Please note: 2019-2021 Added 04/29/2021 by Raymond Benza, source=Web Response, value=Grant/Research Support No relevant relationships by Aditi Dhabalia, source=Web Response Advisory Committee Member relationship with Bayer Please note: 2016-2019 Added 04/30/2021 by Manreet Kanwar, source=Web Response, value=Consulting fee No relevant relationships by Mia Manavalan, source=Web Response Advisory Committee Member relationship with Blue Spark Technologies Please note: 2020-Present Added 04/16/2021 by Adam Perer, source=Web Response, value=Consulting fee Consultant relationship with Johnson & Johnson Please note: 2021-Present Added 04/16/2021 by Adam Perer, source=Web Response, value=Consulting fee No relevant relationships by Zhirou Xin, source=Web Response