Feedback Dashboard Research Articles

The Trial Enrollment Diversity Dashboard for Acute Leukemia Clinical Research: Intervention Development and Cohort Analysis.

Participation in acute leukemia clinical trials is inequitable across multiple sociodemographic categories. Tools that provide researchers with performance feedback on the representativeness of the patients they enroll are limited. We aimed to develop an electronic health record (EHR)-based dashboard to provide such feedback and to describe any enrollment inequities uncovered. We created a visual dashboard linking leukemia clinical trial registration and EHR data at the Dana-Farber Cancer Institute. Accuracy of a patient inclusion and assignment algorithm was tested with a target area under the receiver-operator curve (AUROC) of >0.90 against manual review. Demographic metric identification, visualization construction, and dashboard refinement were performed through stakeholder cognitive testing. Analysis of a recent 5-year cohort generated by the final algorithm assessed bivariate associations between enrollment and demographic metrics. Multivariable logistic regression included significant bivariate results. The final algorithm assignment AUROC was 0.98. Metrics were identified and visualizations successfully constructed. Fourteen individuals participated in testing and identified areas for revision: category mergers, denominator filters, and data delivery preferences. In the initial cohort of 1,315 patients, 1,020 (77.6%) had enrolled in any study protocol: 553 (42.1%) in a treatment trial and 936 (71.2%) in a biobanking study. In a multivariable model, older age (odds ratio [OR], 0.83 [95% CI, 0.73 to 0.94]) and Non-Hispanic Black race-ethnicity (OR, 0.38 [95% CI, 0.18 to 0.82]) were associated with lower enrollment, and English primary language with higher enrollment (OR, 2.50 [95% CI, 1.30 to 4.79]). We developed a research participation equity performance feedback dashboard for clinical researchers, and we identified actionable inequities. Next steps include feasibility and efficacy testing as well as implementation.

Exploration of medical students’ approach to progress test preparation

Introduction: Medical students are expected to develop competencies in applying clinical knowledge. Progress testing is an assessment for learning and is intended to assist in knowledge acquisition and to promote ongoing recall. This study explored student preparation for progress tests (PTs), relationships between approach and performance, and patterns that could assist with targeting learning support. Methods: A cross-sectional survey exploring study approach and learning context, comprising multi-choice and open-ended responses, was administered to students during their clinical years (n = 297). Quantitative data was analysed using descriptive statistics and comparative analysis, and qualitative data was categorised. Results: One hundred twenty-nine students responded (43.4%), with most demonstrating a stable PT performance over time. Twenty-two students had dynamic changes in their aggregate grades. Poor early PT performance was associated with developing an improvement strategy (χ2 6.954, p = 0.008). Students never falling below satisfactory were less likely to have a strategy (χ2 10.084, p = 0.001). All poorly performing students were using practice question banks, but this was not true for students who scored satisfactorily. The need for pastoral care was associated with poorer performance (χ2 4.701, p = 0.030). Conclusions: Student approaches to PT preparation are diverse, however there are correlations between preparation factors and performance outcomes. Students with poor early performance demonstrated improved results over time that may be attributed to targeted support and preparation strategies. Widespread use of practice questions may not be sufficient to impact results without additional strategies. External impacts to performance were common, and barriers to accessing pastoral care were evident. The feedback dashboard was underutilised, suggesting a need to improve feedback literacy and ensure this is fit for purpose.

The impact of real-time feedback on ventilation quality during out-of-hospital cardiac arrest: A before-and-after study

IntroductionVentilations are a critical component of cardiopulmonary resuscitation (CPR). There is conflicting evidence, however, on the most appropriate method of ventilation during cardiac arrest management. Recent evidence has suggested that regardless of the optimal ventilation strategy, ventilations are often not delivered compliant with guideline recommendations. Recent technological advancements have allowed for accurate measurement and real-time feedback of ventilation rate and volume during resuscitation. Simulation studies have found significant improvements in ventilations with the use of real-time feedback during simulated cardiac arrest. The use of feedback has not been studied in clinical practice.The objective of this study was to determine whether the use of real-time feedback improves compliance with pre-defined targets for ventilation rate and volume during out-of-hospital cardiac resuscitation. MethodsThis was a before-and-after study with four paramedic services in Ontario, Canada. We enrolled adult, out-of-hospital cardiac arrest (OHCA) patients where the ZOLL Accuvent® device was utilized to measure ventilation rate and volume. In the before phase (without feedback), the Accuvent® was used to measure ventilations, however, providers were blinded to the real-time feedback. In the after phase (with feedback), the feedback dashboard was activated and providers used the real-time feedback to guide their ventilations. All other aspects of resuscitation remained consistent throughout the study. The main objective of the study was to compare the proportion of each case that was compliant with pre-defined ventilation targets with real-time feedback and without real-time feedback. We also examined the use of advanced airways on ventilation quality and examined for associations between ventilation parameters and return of circulation. ResultsWe enrolled 412 patients in the study (191 in the before phase without feedback and 221 in the after phase with feedback). Overall, we found significant improvements in both ventilation rate and volume in the after phase (with real-time feedback) compared to the before phase (without real-time feedback). We did not find any differences in ventilation compliance with or without advanced airways, or intra-arrest or post-cardiac arrest. ConclusionThe use of real-time feedback was associated with an increased proportion of ventilations that were compliant with pre-defined targets during cardiac resuscitation. Further work is required to improve the use of real-time ventilation feedback, and to determine the impact of ventilations on patient outcomes.

A Closer Look at Instructor Use and Sensemaking Processes of Analytics Dashboards

There is a growing interest in the research and use of automated feedback dashboards that display classroom analytics; yet little is known about the detailed processes instructors use to make sense of these tools, and to determine the impact on their teaching practices. This research was conducted at a public Midwestern university within the context of an automated classroom observation and feedback implementation project. Fifteen engineering instructors engaged in this research. The overarching goal was to investigate instructor teaching beliefs, pedagogical practices, and sensemaking processes regarding dashboard use. A grounded theory approach was used to identify categories related to instructor perceptions. Results revealed that instructor experiences inform both their present use of the dashboard and consequential future actions. A model is presented that illustrates categories included in instructor pre-use, use, and post-use of an automated feedback dashboard. An extension to this model is presented and accompanied by recommendations for a more effective future use of automated dashboards. The model’s practical implications inform both instructors and designers on effective design and use of dashboards, ultimately paving a way to improve pedagogical practices and instruction.

A Closer Look at Instructor Use and Sensemaking Processes of Analytics Dashboards

There is a growing interest in the research and use of automated feedback dashboards that display classroom analytics; yet little is known about the detailed processes instructors use to make sense of these tools, and to determine the impact on their teaching practices. This research was conducted at a public Midwestern university within the context of an automated classroom observation and feedback implementation project. Fifteen engineering instructors engaged in this research. The overarching goal was to investigate instructor teaching beliefs, pedagogical practices, and sensemaking processes regarding dashboard use. A grounded theory approach was used to identify categories related to instructor perceptions. Results revealed that instructor experiences inform both their present use of the dashboard and consequential future actions. A model is presented that illustrates categories included in instructor pre-use, use, and post-use of an automated feedback dashboard. An extension to this model is presented and accompanied by recommendations for a more effective future use of automated dashboards. The model’s practical implications inform both instructors and designers on effective design and use of dashboards, ultimately paving a way to improve pedagogical practices and instruction.

Development of clinically meaningful quality indicators for contemporary lung cancer care, and piloting and evaluation in a retrospective cohort; experiences of the Embedding Research (and Evidence) in Cancer Healthcare (EnRICH) Program

ObjectivesLung cancer continues to be the most common cause of cancer-related death and the leading cause of morbidity and burden of disease across Australia. There is an ongoing need to...

Clinicians’ Evaluation of Lung Cancer Clinical Quality Indicators and Comparative Performance Data in Practice

Objective. Lung cancer is commonly diagnosed and is the leading cause of cancer-related death, morbidity, and burden of disease globally. There is an ongoing need to ensure patients receive optimal evidence-based care and to identify and reduce unwarranted clinical variation to achieve best possible outcomes. The EnRICH program has developed evidence-based clinical quality indicators to measure processes and outcomes of lung cancer care, and a feedback dashboard to report comparative performance data, which highlight variation in both care and outcomes. The aims of this study were to evaluate the acceptability and utility of the quality indicators and feedback dashboard and identify benchmarks for performance monitoring and priorities for future quality improvement interventions to address observed clinical variation. Method. Clinicians from lung cancer multidisciplinary teams (MDTs) at six tertiary clinical sites across regional and metropolitan NSW were invited to participate in evaluation interviews. Interviews were conducted via videoconference and recorded with consent. Data were analysed thematically using framework methods. Results. Thirteen clinicians participated in interviews, with representation from each clinical site and specialty. All participants considered the quality indicators to be clinically meaningful. Three main themes were identified: (i) the importance of timely, local, quality data; (ii) implementable versus nonimplementable clinical practice changes; and (iii) the need for ongoing performance monitoring. Clinicians prioritised two areas of unwarranted clinical variation that could be immediately addressed through easily implementable quality improvement interventions to positively impact patient care: (i) a process to ensure that all stage III patients are discussed by a multidisciplinary team prior to commencing treatment; (ii) a referral pathway to palliative care within eight weeks for patients diagnosed with stage IV disease. The importance of lung cancer nurse specialists for improved care coordination was highlighted. Conclusion. Clinicians would like to continue to receive close-to-real-time quality data for ongoing performance monitoring to identify and address unwarranted clinical variation.

Developing an Audit and Feedback Dashboard for Family Physicians: User-Centered Design Process.

Audit and feedback (A&F), the summary and provision of clinical performance data, is a common quality improvement strategy. Successful design and implementation of A&F-or any quality improvement strategy-should incorporate evidence-informed best practices as well as context-specific end user input. We used A&F theory and user-centered design to inform the development of a web-based primary care A&F dashboard. We describe the design process and how it influenced the design of the dashboard. Our design process included 3 phases: prototype development based on A&F theory and input from clinical improvement leaders; workshop with family physician quality improvement leaders to develop personas (ie, fictional users that represent an archetype character representative of our key users) and application of those personas to design decisions; and user-centered interviews with family physicians to learn about the physician's reactions to the revised dashboard. The team applied A&F best practices to the dashboard prototype. Personas were used to identify target groups with challenges and behaviors as a tool for informed design decision-making. Our workshop produced 3 user personas, Dr Skeptic, Frazzled Physician, and Eager Implementer, representing common users based on the team's experience of A&F. Interviews were conducted to further validate findings from the persona workshop and found that (1) physicians were interested in how they compare with peers; however, if performance was above average, they were not motivated to improve even if gaps compared to other standards in their care remained; (2) burnout levels were high as physicians are trying to catch up on missed care during the pandemic and are therefore less motivated to act on the data; and (3) additional desired features included integration within the electronic medical record, and more up-to-date and accurate data. We found that carefully incorporating data from user interviews helped operationalize generic best practices for A&F to achieve an acceptable dashboard that could meet the needs and goals of physicians. We demonstrate such a design process in this paper. A&F dashboards should address physicians' data skepticism, present data in a way that spurs action, and support physicians to have the time and capacity to engage in quality improvement work; the steps we followed may help those responsible for quality improvement strategy implementation achieve these aims.

Effect of a Performance Feedback Dashboard on Hospitalist Laboratory Test Utilization

Background: Healthcare spending continues to be an area of improvement across all forms of medicine. Overtreatment or low-value care, including overutilization of laboratory testing, has an estimated annual cost of waste of $75.7–$101.2 billion annually. Providing performance feedback to hospitalists has been shown to be an effective way to encourage the practice of quality-improvement-focused medicine. There remains limited data regarding the implementation of performance feedback and direct results on hospital laboratory testing spending in the short term. Objective: The objective of this project was to identify whether performance-based feedback on laboratory utilization between both hospitalists and resident teams results in more conservative utilization of laboratory testing. Design, Setting, Participants: This quality improvement project was conducted at a tertiary academic medical center, including both direct-care and house-staff teams. Intervention or Exposure: A weekly performance feedback report was generated and distributed to providers detailing laboratory test utilization by all hospitalists in a ranked system, normalized by the census of patients, for 3 months. Main Outcomes and Measures: The outcome measure was cumulative laboratory utilization during the intervention period compared to baseline utilization during the corresponding 3 months in the year prior and the weekly trend in laboratory utilization over 52 weeks. The aggregate laboratory utilization rate during intervention and control time periods was defined as the total number of laboratory tests ordered divided by the total number of patient encounters. Additionally, the cost difference was averaged per quarter and reported. The week-by-week trend in laboratory utilization was evaluated using a statistical process control (SPC) chart. Results: We found that following intervention during January–March 2020, the cumulative complete blood count utilization rate decreased from 5.54 to 4.83 per patient encounter and the basic metabolic panels/CMP utilization rate decreased from 6.65 to 6.11 per patient encounter compared with January–March 2019. This equated to cost savings of ~$42,700 in total for the quarter. Nonrandom variation was seen on SPC charts in weekly laboratory utilization rates for common laboratory tests during the intervention period. Conclusions: We found that our intervention did result in a decrease in laboratory test utilization rates across direct-care and house-staff teams. This study lays promising groundwork for one tool that can be used to eliminate a source of hospital waste and improve the quality and efficiency of patient care.

Implementing shared decision making for early-stage breast cancer treatment using a coproduction learning collaborative: the SHAIR Collaborative protocol

BackgroundShared decision making (SDM) in breast cancer care improves outcomes, but it is not routinely implemented. Results from the What Matters Most trial demonstrated that early-stage breast cancer surgery conversation aids, when used by surgeons after brief training, improved SDM and patient-reported outcomes. Trial surgeons and patients both encouraged using the conversation aids in routine care. We will develop and evaluate an online learning collaborative, called the SHared decision making Adoption Implementation Resource (SHAIR) Collaborative, to promote early-stage breast cancer surgery SDM by implementing the conversation aids into routine preoperative care. Learning collaboratives are known to be effective for quality improvement in clinical care, but no breast cancer learning collaborative currently exists.Our specific aims are to (1) provide the SHAIR Collaborative resources to clinical sites to use with eligible patients, (2) examine the relationship between the use of the SHAIR Collaborative resources and patient reach, and (3) promote the emergence of a sustained learning collaborative in this clinical field, building on a partnership with the American Society of Breast Surgeons (ASBrS).MethodsWe will conduct a two-phased implementation project: phase 1 pilot at five sites and phase 2 scale up at up to an additional 32 clinical sites across North America. The SHAIR Collaborative online platform will offer free access to conversation aids, training videos, electronic health record and patient portal integration guidance, a feedback dashboard, webinars, support center, and forum. We will use RE-AIM for data collection and evaluation. Our primary outcome is patient reach. Secondary data will include (1) patient-reported data from an optional, anonymous online survey, (2) number of active sites and interviews with site champions, (3) Normalization MeAsure Development questionnaire data from phase 1 sites, adaptations data utilizing the Framework for Reporting Adaptations and Modifications-Extended/-Implementation Strategies, and tracking implementation facilitating factors, and (4) progress on sustainability strategy and plans with ASBrS.DiscussionThe SHAIR Collaborative will reach early-stage breast cancer patients across North America, evaluate patient-reported outcomes, engage up to 37 active sites, and potentially inform engagement factors affecting implementation success and may be sustained by ASBrS.

Audit, Feedback, and Education to Improve Quality and Outcomes in Transurethral Resection and Single-Instillation Intravesical Chemotherapy for Nonmuscle Invasive Bladder Cancer Treatment: Protocol for a Multicenter International Observational Study With an Embedded Cluster Randomized Trial.

Nonmuscle invasive bladder cancer (NMIBC) accounts for 75% of bladder cancers. It is common and costly. Cost and detriment to patient outcomes and quality of life are driven by high recurrence rates and the need for regular invasive surveillance and repeat treatments. There is evidence that the quality of the initial surgical procedure (transurethral resection of bladder tumor [TURBT]) and administration of postoperative bladder chemotherapy significantly reduce cancer recurrence rates and improve outcomes (cancer progression and mortality). There is surgeon-reported evidence that TURBT practice varies significantly across surgeons and sites. There is limited evidence from clinical trials of intravesical chemotherapy that NMIBC recurrence rate varies significantly between sites and that this cannot be accounted for by differences in patient, tumor, or adjuvant treatment factors, suggesting that how the surgery is performed may be a reason for the variation. This study primarily aims to determine if feedback on and education about surgical quality indicators can improve performance and secondarily if this can reduce cancer recurrence rates. Planned secondary analyses aim to determine what surgeon, operative, perioperative, institutional, and patient factors are associated with better achievement of TURBT quality indicators and NMIBC recurrence rates. This is an observational, international, multicenter study with an embedded cluster randomized trial of audit, feedback, and education. Sites will be included if they perform TURBT for NMIBC. The study has four phases: (1) site registration and usual practice survey; (2) retrospective audit; (3) randomization to audit, feedback, and education intervention or to no intervention; and (4) prospective audit. Local and national ethical and institutional approvals or exemptions will be obtained at each participating site. The study has 4 coprimary outcomes, which are 4 evidence-based TURBT quality indicators: a surgical performance factor (detrusor muscle resection); an adjuvant treatment factor (intravesical chemotherapy administration); and 2 documentation factors (resection completeness and tumor features). A key secondary outcome is the early cancer recurrence rate. The intervention is a web-based surgical performance feedback dashboard with educational and practical resources for TURBT quality improvement. It will include anonymous site and surgeon-level peer comparison, a performance summary, and targets. The coprimary outcomes will be analyzed at the site level while recurrence rate will be analyzed at the patient level. The study was funded in October 2020 and began data collection in April 2021. As of January 2023, there were 220 hospitals participating and over 15,000 patient records. Projected data collection end date is June 30, 2023. This study aims to use a distributed collaborative model to deliver a site-level web-based performance feedback intervention to improve the quality of endoscopic bladder cancer surgery. The study is funded and projects to complete data collection in June 2023. ClinicalTrials.org NCT05154084; https://clinicaltrials.gov/ct2/show/NCT05154084. DERR1-10.2196/42254.

Energy audits and eco-feedback: Exploring the barriers and facilitators of agricultural energy efficiency improvements on Australian farms

This paper aims to understand the farm-level barriers and opportunities to greater energy efficiency in the Australian agricultural sector. Using a mixed methods approach informed by social practice theory, we detail the experiences of 12 farmers who received an energy audit and energy use feedback dashboard (eco-feedback) through government-supported energy efficiency initiatives. Findings suggest on-farm energy use is dictated by a number of heterogeneous factors and practices, where despite an overall positive reception to the interventions, a lack of available time, funds, complicated electricity tariffs and inadaptability of practices represent barriers to energy efficient actions. In response, this paper makes suggestions for the design of energy efficiency initiatives, policy and farm-specific eco-feedback, including: (1) Increased support around tariff navigation, (2) consideration of farmers’ financial circumstances in audit recommendations, (3) a re-ordering of program instruments and (4) designing eco-feedback specifically for farmers, including hardware that supports multiple meters, and real-time water use feedback.

Quantified factory worker: field study of a web application supporting work well-being and productivity

The work on the factory floor is gradually changing to resemble knowledge work due to highly automated manufacturing machines. In the increasingly automated work environment, the machine operator’s task is to keep the production running and to solve possible problems quickly. This work is expected to become more autonomous, which raises the importance of supporting the workers’ well-being. An important aspect of that is giving concrete feedback of success at work as well as feedback on physical and mental load. We implemented a smartphone optimized web application, Worker Feedback Dashboard that offers feedback to machine operators about their well-being at work and personally relevant production data as well as their connections to each other. The feedback is personal and based on objective, near real-time measurements. We present the results of a field study, in which ten machine operators used the application for 2–3 months. We studied the operators’ user experience, usage activity, perceived benefits and concerns for the application with questionnaires, interviews and application log data. The operators found the feedback interesting and beneficial, and used the application actively. The perceived benefits indicate impacts on well-being as well as on work performance. Based on the results, we highlight three design implications for quantified worker applications: presenting meaningful overviews, providing guidance to act based on the feedback and refraining from too pervasive quantification not to narrow down the meaningful aspects in one’s work.

Mixed methods evaluation of a computerised audit and feedback dashboard to improve patient safety through targeting acute kidney injury (AKI) in primary care

BackgroundReducing the harms associated with acute kidney injury (AKI) requires addressing a wide range of patient safety issues, including polypharmacy and transitions of care, particularly for vulnerable patient groups. Computerised audit and feedback can transform the way healthcare organisations measure, analyse and learn from quality and safety data across different care settings, potentially improving patient safety. ObjectiveTo implement and evaluate an audit and feedback dashboard targeting AKI to improve patient safety, focusing on factors affecting a range of user characteristics in primary care. MethodsWe performed a mixed methods study in three stages. Semi-structured interviews were initially performed with both primary (n = 10) and secondary care (n = 5) staff to gather user requirements for six quality indicators extracted from national guidance on post-discharge AKI care. Modified indicators were implemented in the Performance Improvement plaN GeneratoR (PINGR) audit and feedback dashboard for six months, across 45 general practices in Salford. Primary care professionals were then interviewed again (n = 7) and completed usability questionnaires. This was triangulated with an interrupted time series analysis on indicator performance, alongside software usage statistics. ResultsImprovements were observed for the indicators for medication review (+9.01 %; 95 % Confidence Interval (CI), +6.95 % to +11.06 %) and blood pressure measurement (+5.20 %; 95 % CI + 3.61 % to +6.78 %). Variable performance and engagement were observed for other indicators including AKI coding (+0.39 %; 95 % CI −1.88 % to +2.65 %), serum creatinine (−3.40 %; 95 % CI −7.66 % to +0.85 %), proteinuria (−1.08 %; 95 % CI −1.47 % to +0.32 %) and providing patient information (+0.16 %; 95 % CI −0.41 % to +0.73 %). A key facilitator to engagement was the development of ‘champions of change’, achieved through a raised awareness of high-risk patients, guidelines, inconsistencies in coding practice and evidence for quality and safety performance. Barriers related to the specificity and perceived achievability of indicators, and limitations in resources. ConclusionIn a six-month, quasi-experimental evaluation of an electronic audit and feedback dashboard targeting AKI, we found improvements for two out of six quality indicators. While information technology can facilitate improvements in patient safety, further allocation of protected staff time and investment into shared learning are needed to realise those improvements in practice.

An Innovative Feedback Tool Leading to Improved Faculty Feedback and Positive Reception by Residents.

IntroductionIn 2012 the Accreditation Council for Graduate Medical Education implemented trainee milestones as tools for clinical competency committees to use for evaluation, feedback, remediation, and promotion purposes. Prior to this innovation, there has not been an adequate method to capture, organize, and graphically illustrate the evaluations by attendings in a simple, fast and organized fashion.MethodsWe created a novel, web-based, mobile-friendly evaluation tool to help fill this identified gap. The survey-based program creates a milestone-based evaluation, takes only a few minutes to complete, and easily collates the results in a graphic format creating an individualized “dashboard.” The dashboard is then used by both trainees and their evaluators as a feedback platform.ResultsWith the implementation of the dashboard, educational leadership has noted an increase in the number of submitted evaluations of residents and the amount of face-to-face feedback given by attendings to residents. A post-implementation survey of the residents revealed that they found the dashboard-provided feedback more helpful than prior modes of feedback, although the number of evaluations was still too few.ConclusionThe use of our feedback dashboard is useful to multiple targeted end-users, including general faculty evaluators, program leadership, and the residents themselves for gathering formative feedback that is specific and timely. This tool is adaptable and likely generalizable to other residency programs and specialties.

Involving end-users in the design of an audit and feedback intervention in the emergency department setting \u2013 a mixed methods study

BackgroundLong length of stays (LOS) in emergency departments (ED) negatively affect quality of care. Ordering of inappropriate diagnostic tests contributes to long LOS and reduces quality of care. One strategy to change practice patterns is to use performance feedback dashboards for physicians. While this strategy has proven to be successful in multiple settings, the most effective ways to deliver such interventions remain unknown. Involving end-users in the process is likely important for a successful design and implementation of a performance dashboard within a specific workplace culture. This mixed methods study aimed to develop design requirements for an ED performance dashboard and to understand the role of culture and social networks in the adoption process.MethodsWe performed 13 semi-structured interviews with attending physicians in different roles within a single public ED in the U.S. to get an in-depth understanding of physicians’ needs and concerns. Principles of human-centered design were used to translate these interviews into design requirements and to iteratively develop a front-end performance feedback dashboard. Pre- and post- surveys were used to evaluate the effect of the dashboard on physicians’ motivation and to measure their perception of the usefulness of the dashboard. Data on the ED culture and underlying social network were collected. Outcomes were compared between physicians involved in the human-centered design process, those with exposure to the design process through the ED social network, and those with limited exposure.ResultsKey design requirements obtained from the interviews were ease of access, drilldown functionality, customization, and a visual data display including monthly time-trends and blinded peer-comparisons. Identified barriers included concerns about unintended consequences and the veracity of underlying data. The surveys revealed that the ED culture and social network are associated with reported usefulness of the dashboard. Additionally, physicians’ motivation was differentially affected by the dashboard based on their position in the social network.ConclusionsThis study demonstrates the feasibility of designing a performance feedback dashboard using a human-centered design approach in the ED setting. Additionally, we show preliminary evidence that the culture and underlying social network are of key importance for successful adoption of a dashboard.

Improving Community Health Worker performance by using a personalised feedback dashboard for supervision: a randomised controlled trial.

BackgroundCountries across sub-Saharan Africa are scaling up Community Health Worker (CHW) programmes, yet there remains little high-quality research assessing strategies for CHW supervision and performance improvement. This randomised controlled trial aimed to determine the effect of a personalised performance dashboard used as a supervision tool on the quantity, speed, and quality of CHW care.MethodsWe conducted a randomised controlled trial in a large health catchment area in peri-urban Mali. One hundred forty-eight CHWs conducting proactive case-finding home visits were randomly allocated to receive individual monthly supervision with or without the CHW Performance Dashboard from January to June 2016. Randomisation was stratified by CHW supervisor, level of CHW experience, and CHW baseline performance for monthly quantity of care (number of household visits). With regression analysis, we used a difference-in-difference model to estimate the effect of the intervention on monthly quantity, timeliness (percentage of children under five treated within 24 hours of symptom onset), and quality (percentage of children under five treated without protocol error) of care over a six-month post-intervention period relative to a three-month pre-intervention period.ResultsUse of the Dashboard during monthly supervision significantly increased the mean number of home visits by 39.94 visits per month (95% CI = 3.56-76.3; P = 0.031). Estimated effects on secondary outcomes of timeliness and quality were positive but not statistically significant. Across both study arms, CHW quantity, timeliness, and quality of care significantly improved over the study period, during which time all CHWs received dedicated monthly supervision, although effects plateaued over time.ConclusionsOur findings suggest that dedicated monthly supervision and personalised feedback using performance dashboards can increase CHW productivity. Further operational research is needed to understand how to sustain the performance improvements over time.Trial registrationClinicalTrials.gov (NCT03684551).

Radiology Resident Assessment and Feedback Dashboard.

Assessment of residents is optimally performed through processes and platforms that provide daily feedback, which can be immediately acted on. Given the documentation required by the Accreditation Council for Graduate Medical Education (ACGME), effective data management, integration, and presentation are crucial to ease the burden of manual documentation and increase the timeliness of actionable information. To this end, the authors modeled the learning activities of residents using the Experience Application Programming Interface (xAPI) framework, which is a standard framework for the learning community. On the basis of the xAPI framework and using open-source software to extend their existing infrastructure, the authors developed a Web-based dashboard that provides residents with a more holistic view of their educational experience. The dashboard was designed around the ACGME radiology milestones and provides real-time feedback to residents using various assessment metrics derived from multiple data sources. The purpose of this article is to describe the dashboard's architecture and components, the design and technical considerations, and the lessons learned in implementing the dashboard. ©RSNA, 2018.

Health professionals\u2019 perceptions about their clinical performance and the influence of audit and feedback on their intentions to improve practice: a theory-based study in Dutch intensive care units

BackgroundAudit and feedback aims to guide health professionals in improving aspects of their practice that need it most. Evidence suggests that feedback fails to increase accuracy of professional perceptions about clinical performance, which likely reduces audit and feedback effectiveness. This study investigates health professionals’ perceptions about their clinical performance and the influence of feedback on their intentions to change practice.MethodsWe conducted an online laboratory experiment guided by Control Theory with 72 intensive care professionals from 21 units. For each of four new pain management indicators, we collected professionals’ perceptions about their clinical performance; peer performance; targets; and improvement intentions before and after receiving first-time feedback. An electronic audit and feedback dashboard provided ICU’s own performance, median and top 10% peer performance, and improvement recommendations. The experiment took place approximately 1 month before units enrolled into a cluster-randomised trial assessing the impact of adding a toolbox with suggested actions and materials to improve intensive care pain management. During the experiment, the toolbox was inaccessible; all participants accessed the same version of the dashboard.ResultsWe analysed 288 observations. In 53.8%, intensive care professionals overestimated their clinical performance; but in only 13.5%, they underestimated it. On average, performance was overestimated by 22.9% (on a 0–100% scale). Professionals similarly overestimated peer performance, and set targets 20.3% higher than the top performance benchmarks. In 68.4% of cases, intentions to improve practice were consistent with actual gaps in performance, even before professionals had received feedback; which increased to 79.9% after receiving feedback (odds ratio, 2.41; 95% CI, 1.53 to 3.78). However, in 56.3% of cases, professionals still wanted to improve care aspects at which they were already top performers. Alternatively, in 8.3% of cases, they lacked improvement intentions because they did not consider indicators important; did not trust the data; or deemed benchmarks unrealistic.ConclusionsAudit and feedback helps health professionals to work on aspects for which improvement is recommended. Given the abundance of professionals’ prior good improvement intentions, the limited effects typically found by audit and feedback studies are likely predominantly caused by barriers to translation of intentions into actual change in clinical practice.Trial registrationClinicalTrials.govNCT02922101. Registered 26 September 2016.