Prototype Clinical Decision Support System Research Articles

Formative evaluation of the accuracy of a clinical decision support system for cervical cancer screening

ObjectivesWe previously developed and reported on a prototype clinical decision support system (CDSS) for cervical cancer screening. However, the system is complex as it is based on multiple guidelines and free-text processing. Therefore, the system is susceptible to failures. This report describes a formative evaluation of the system, which is a necessary step to ensure deployment readiness of the system.Materials and methodsCare providers who are potential end-users of the CDSS were invited to provide their recommendations for a random set of patients that represented diverse decision scenarios. The recommendations of the care providers and those generated by the CDSS were compared. Mismatched recommendations were reviewed by two independent experts.ResultsA total of 25 users participated in this study and provided recommendations for 175 cases. The CDSS had an accuracy of 87% and 12 types of CDSS errors were identified, which were mainly due to deficiencies in the system's guideline rules. When the deficiencies were rectified, the CDSS generated optimal recommendations for all failure cases, except one with incomplete documentation.Discussion and conclusionsThe crowd-sourcing approach for construction of the reference set, coupled with the expert review of mismatched recommendations, facilitated an effective evaluation and enhancement of the system, by identifying decision scenarios that were missed by the system's developers. The described methodology will be useful for other researchers who seek rapidly to evaluate and enhance the deployment readiness of complex decision support systems.

From an expert-driven paper guideline to a user-centred decision support system: A usability comparison study

To assess whether a user-centred prototype clinical decision support system (CDSS) providing patient-specific advice better supports healthcare practitioners in terms of (a) types of usability problems detected and (b) effective and efficient retrieval of childhood cancer survivor's follow-up screening procedures compared to an expert-driven paper-based guideline. A user-centred design (UCD) process was employed to design a prototype CDSS. Usability problems in information retrieval with the paper-based guideline were assessed by think-aloud analysis with 13 participants. Both simple and more complex tasks were applied. The analysis provided input for the UCD process of the prototype. The usability of the prototype CDSS was subsequently evaluated by think-aloud analysis with the same participants. Usability problems of the paper-based guideline and the prototype CDSS were compared by using the classification of usability problems scheme. In addition, efficiency (time to complete task) and effectiveness (completeness of retrieved screening procedures) of information retrieval of participants in the expert-driven paper-based guideline and the user-centred prototype CDSS were compared. Usability problems in both the paper-based guideline and the CDSS prototype were mainly classified as 'incongruent with participants' mental model'. The prototype CDSS reduced this type of problem from 17 to 6 problems. The time to perform simple information retrieval tasks increased by 58 s when using the prototype CDSS, however, it resulted in a 58% improvement in task completeness compared to the paper-based guideline. The time to perform complex scenarios decreased by 3:50 min with the prototype CDSS, with 17% higher completeness compared to the paper-based guideline. Analysis showed that usability problems experienced by healthcare practitioners when using a paper-based guideline could be overcome by implementing the guideline in a user-centred CDSS design. Although different types of usability problems were experienced with the prototype CDSS, they did not inhibit effective and efficient performance of tasks in the system. The usability problem analysis of the paper-based guideline effectively supported comparison of usability problems found in the two information retrieval systems and it supported the UCD of the CDSS.

Observer Study of a Prototype Clinical Decision Support System for Breast Cancer Diagnosis Using Dynamic Contrast-Enhanced MRI

The purpose of this article is to evaluate the performance of radiologists using a prototype clinical decision support system to diagnose and manage patients with breast cancer based on dynamic contrast-enhanced MRI studies. The study was conducted with three breast radiologists and two breast imaging fellows who gave patient treatment recommendations and confidence ratings, both without and with computer aid. The computer aid presented similar cases from a retrieval database of 192 lesions (96 malignant and 96 benign) for a test set of 97 mass lesions (46 malignant and 51 benign). The performance of each observer was quantified by receiver operating characteristic analysis. The radiologists' confidence in their recommendations was analyzed with respect to the query case pathologic diagnosis, perceived usefulness of the similar cases, and the accuracy of the computer in retrieving cases of the correct diagnosis. The statistical significance in the performance measure differences was determined by using a two-tailed Student t test for paired data. For each observer, the area under the receiver operating characteristic curve did not change significantly with the use of the computer aid (from a mean of 0.8 to a mean of 0.8; p = 0.61). The average confidence of three of the five observers increased significantly with the computer aid (from 5.9 to 6.3 [p < 0.001], from 7.0 to 7.2 [p = 0.04], and from 4.4 to 5.4 [p < 0.001], respectively). The confidence change of the radiologists was more frequent and larger for malignant lesions where the computer was correct. However, for benign lesions, even when the computer was correct, the confidence of the radiologists did not necessarily change. The presentation of similar cases reinforced radiologists' confidence rating in the diagnosis of malignant lesions; however, it did not change their confidence rating for benign lesions or reduce the number of unnecessary biopsies in managing patients with breast cancer using dynamic contrast-enhanced MRI under the limited study conditions.

Clinical Decision Support System (CDSS) for the Classification of Atypical Cells in Pleural Effusions

The objective of this research is to develop a prototype Clinical Decision Support System (CDSS) to aid pathologists in correctly discriminating between reactive mesothelial cells and malignant epithelial cells. Currently, there is great difficulty in visually discriminating between cells that are malignant and cells that are otherwise reactive to antigens present in the effusion. Features have been identified, which can correctly discriminate between benign epithelial cells and malignant epithelial cells with a validation AZ accuracy of ∼ 0.934, training AZ of ∼ 0.937. Using these features, the system trained on visually known cases was shown to find discriminating information in the feature subset of the atypical cases by examining probabilities generated from subjecting the system to atypical cells. While these results are preliminary, they do demonstrate that an intelligent CDSS, which has the potential to discriminate between reactive mesothelial cells and malignant epithelial cells, designed using newly developed and/or revised statistical learning theory (SLT) algorithms, has the potential to be used as a second opinion diagnostic aid by physicians, as they deem appropriate.

A belief rule-based decision support system for clinical risk assessment of cardiac chest pain

This paper describes a prototype clinical decision support system (CDSS) for risk stratification of patients with cardiac chest pain. A newly developed belief rule-based inference methodology-RIMER was employed for developing the prototype. Based on the belief rule-based inference methodology, the prototype CDSS can deal with uncertainties in both clinical domain knowledge and clinical data. Moreover, the prototype can automatically update its knowledge base via a belief rule base (BRB) learning module which can adjust BRB through accumulated historical clinical cases. The domain specific knowledge used to construct the knowledge base of the prototype was learned from real patient data. We simulated a set of 1000 patients in cardiac chest pain to validate the prototype. The belief rule-based prototype CDSS has been found to perform extremely well. Firstly, the system can provide more reliable and informative diagnosis recommendations than manual diagnosis using traditional rules when there are clinical uncertainties. Secondly, the diagnostic performance of the system can be significantly improved after training the BRB through accumulated clinical cases.

Developing a decision support system for tobacco use counselling using primary care physicians

Clinical decision support systems (CDSS) have the potential to improve adherence to guidelines, but only if they are designed to work in the complex environment of ambulatory clinics as otherwise physicians may not use them. To gain input from primary care physicians in designing a CDSS for smoking cessation to ensure that the design is appropriate to a clinical environment before attempts to test this CDSS in a clinical trial. This approach is of general interest to those designing similar systems. Design and approach: We employed an iterative ethnographic process that used multiple evaluation methods to understand physician preferences and workflow integration. Using results from our prior survey of physicians and clinic managers, we developed a prototype CDSS, validated content and design with an expert panel, and then subjected it to usability testing by physicians, followed by iterative design changes based on their feedback. We then performed clinical testing with individual patients, and conducted field tests of the CDSS in two primary care clinics during which four physicians used it for routine patient visits. The CDSS prototype was substantially modified through these cycles of usability and clinical testing, including removing a potentially fatal design flaw. During field tests in primary care clinics, physicians incorporated the final CDSS prototype into their workflow, and used it to assist in smoking cessation interventions up to eight times daily. A multi-method evaluation process utilising primary care physicians proved useful for developing a CDSS that was acceptable to physicians and patients, and feasible to use in their clinical environment.

Involving users in the design and usability evaluation of a clinical decision support system

Aim: To design and evaluate a clinical decision support system (CDSS) to support cardiovascular risk prevention in type 2 diabetes. Methods: A preliminary requirements specification and three prototype CDSS interface designs were developed. Seven patients and seven clinicians conducted ‘usability tests’ on five different task scenarios with the CDSS prototypes to test its effectiveness, efficiency and ‘user-friendliness’. Structured, qualitative questions explored their preferences for the different designs and overall impressions of clinical usefulness. Results: Patients and clinicians were enthusiastic about the CDSS and used it confidently after a short learning period. Some patients had difficulty interpreting clinical data, but most were keen to see the CDSS used to help them understand their diabetes, provided a clinician explained their results. Clinicians’ main concern was that the CDSS would increase consultation times. Changes suggested by users were incorporated into the final interface design. Conclusion: We have successfully incorporated patients’ and clinicians’ views into the design of a CDSS, but it was an arduous process.

A clinical decision support system prototype for cardiovascular intensive care.

This paper describes the development and validation of a decision-support system prototype that can help manage hypovolemic hypotension in the Cardiovascular Intensive Care Unit (CVICU). The prototype uses physiologic pattern-matching, therapeutic protocols, computational drug-dosage response modeling and expert reasoning heuristics in its selection of intervention strategies and choices. As part of model testing, the prototype simulated real-time operation by processing historical physiologic and intervention data on a patient sequentially, generating alerts on questionable data, critiques of interventions instituted and recommendations on preferred interventions. Bench-testing with 399 interventions from 13 historical cases showed therapies for bleeding and fluid replacement proposed by the prototype were significantly more consistent (p < 0.0001) than those instituted by the staff when compared against expert critiques (80% versus 44%). This study has demonstrated the feasibility of formalizing hemodynamic management of CVICU patients in a manner that may be implemented and evaluated in a clinical setting.