Physician Performance Evaluation Research Articles

Evaluation of physician performance using a concurrent-read artificial intelligence system to support breast ultrasound interpretation

PurposeThe purpose of this study was to compare the diagnostic performance and the interpretation time of breast ultrasound examination between reading without and with the artificial intelligence (AI) system as a concurrent reading aid. Material and methodsA fully crossed multi-reader and multi-case (MRMC) reader study was conducted. Sixteen participating physicians were recruited and retrospectively interpreted 172 breast ultrasound cases in two reading scenarios, once without and once with the AI system (BU-CAD™, TaiHao Medical Inc.) assistance for concurrent reading. Interpretations of any given case set with and without the AI system were separated by at least 5 weeks. These reading results were compared to the reference standard and the area under the LROC curve (AUCLROC), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated for performance evaluations. The interpretation time was also compared between the unaided and aided scenarios. ResultsWith the help of the AI system, the readers had higher diagnostic performance with an increase in the average AUCLROC from 0.7582 to 0.8294 with statistically significant. The sensitivity, specificity, PPV, and NPV were also improved from 95.77%, 24.07%, 44.18%, and 93.50%–98.17%, 30.67%, 46.91%, and 96.10%, respectively. Of these, the improvement in specificity reached statistical significance. The average interpretation time was significantly reduced by approximately 40% when the readers were assisted by the AI system. ConclusionThe concurrent-read AI system improves the diagnostic performance in detecting and diagnosing breast lesions on breast ultrasound images. In addition, the interpretation time is effectively reduced for the interpreting physicians.

It’s not you, It’s me: The influence of patient and surgeon gender on patient satisfaction scores

BackgroundSurgeons face the unique challenge of being responsible for both clinical encounters and surgical outcomes. We aim to explore how patient evaluations of surgeons may be influenced by patient and provider factors. MethodsPatient responses from the 2016 CGCAHPS survey at a single institution were identified. A Poisson regression model was used to identify patient/provider factors associated with ratings. Results11,007 surveys of 134 surgeons were included. After adjustment, higher overall surgeon ratings were associated with older patient age (p < 0.001) and male patient gender (p = 0.001). Lower ratings were associated with higher patient education (p < 0.001) and lower patient self-health ratings (p < 0.001). Although female surgeons tended to have higher communication scores, overall scores did not differ based on any surgeon factors. ConclusionsPatient satisfaction scores of surgeons are more closely correlated with patient variables than surgeon factors. This may have implications for physician performance evaluation in value-based care models.

Perceptions of people with respiratory problems on physician performance evaluation-A qualitative study.

BackgroundDespite increasing calls for patient and public involvement in health‐care quality improvement, the question of how patient evaluations can contribute to physician learning and performance assessment has received scant attention.ObjectiveThe objective of this study was to explore, amid calls for patient involvement in quality assurance, patients' perspectives on their role in the evaluation of physician performance and to support physicians’ learning and decision making on professional competence.DesignA qualitative study based on semi‐structured interviews.Setting and ParticipantsThe study took place in a secondary care setting in the Netherlands. The authors selected 25 patients from two Dutch hospitals and through the Dutch Lung Foundation, using purposive sampling.MethodsData were analysed according to the principles of template analysis, based on an a priori coding framework developed from the literature about patient empowerment, feedback and performance assessment.ResultsThe analysis unearthed three predominant patient perspectives: the proactive perspective, the restrained perspective and the outsider perspective. These perspectives differed in terms of perceived power dynamics within the doctor‐patient relationship, patients' perceived ability, and willingness to provide feedback and evaluate their physician's performance. Patients' perspectives thus affected the role patients envisaged for themselves in evaluating physician performance.Discussion and conclusionAlthough not all patients are equally suitable or willing to be involved, patients can play a role in evaluating physician performance and continuing training through formative approaches. To involve patients successfully, it is imperative to distinguish between different patient perspectives and empower patients by ensuring a safe environment for feedback.

Postnatal development of organ systems: a species comparison

To evaluate obstetrics and gynecology resident physicians' performance following a simulation curriculum on dilation and evacuation (D&E) procedures.This study included two phases: simulation curriculum development and resident physician performance evaluation following training on a D&E simulator. Trainees participated in two evaluations. Simulation training evaluated participants performing six cases on a D&E simulator, measuring procedural time and a 26-step checklist of D&E steps. The operative training portion evaluated residents' performance after training on the simulator using mastery learning techniques. Intra-operative evaluation was based on a 21-step checklist score, Objective Structured Assessment of Technical Skills (OSATS), and percentage of cases completed.Twenty-two residents participated in simulation training, demonstrating improved performance from cases one and two to cases five and six, as measured by checklist score and procedural time (p < .001 and p = .001, respectively). Of 10 participants in the operative training, all performed at least three D&Es, while seven performed at least six cases. While checklist scores did not change significantly from the first to sixth case (mean for first case: 18.3; for sixth case: 19.6; p = .593), OSATS ratings improved from case one (19.7) to case three (23.5; p = .001) and to case six (26.8; p = .005). Trainees completed approximately 71.6% of their first case (range: 21.4–100%). By case six, the six participants performed 81.2% of the case (range: 14.3–100%).D&E simulation using a newly-developed uterine model and simulation curriculum improves resident technical skills. Simulation training with mastery learning techniques transferred to high level of performance in OR using checklist. The OSATS measured skills and showed improvement in performance with subsequent cases.Implementation of a D&E simulation curriculum offers potential for improved surgical training and abortion provision.

TOPLAM KALİTE YÖNETİMİNİN İÇ VE DIŞ MÜŞTERİLERİN PERFORMANS DEĞERLENDİRMESİ ÜZERİNE ETKİSİ

Total quality management is basically an understanding of continuous improvement of all kinds of activities to ensure satisfaction of both internal and external customers. For this reason, a right total quality management applications will affect the performance of internal and external customers positively. In this study, the evaluation of the effect of total quality management on the performance of patients and physicians, which are internal and external customers, was investigated. Factors such as "physical environment, information, cost, quality, trust, transaction, transportation and speed" were determined by using various scales to measure performance for patient and physician performance measurement. These factors constitute the purpose of studying how they affect the performance of the physician and how they relate to each other. Reliability, "KMO and Bartlett's test, factor analysis and correlation analysis" methods were used statistically to reveal the effects of patient and physician performance evaluation. As a result of the analysis identified that the 8 factors have a strong and positive relationship with each other within themselves.

Implementation and evaluation of a dilation and evacuation simulation training curriculum

ObjectivesTo evaluate obstetrics and gynecology resident physicians' performance following a simulation curriculum on dilation and evacuation (D&E) procedures. Study designThis study included two phases: simulation curriculum development and resident physician performance evaluation following training on a D&E simulator. Trainees participated in two evaluations. Simulation training evaluated participants performing six cases on a D&E simulator, measuring procedural time and a 26-step checklist of D&E steps. The operative training portion evaluated residents' performance after training on the simulator using mastery learning techniques. Intra-operative evaluation was based on a 21-step checklist score, Objective Structured Assessment of Technical Skills (OSATS), and percentage of cases completed. ResultsTwenty-two residents participated in simulation training, demonstrating improved performance from cases one and two to cases five and six, as measured by checklist score and procedural time (p<.001 and p=.001, respectively). Of 10 participants in the operative training, all performed at least three D&Es, while seven performed at least six cases. While checklist scores did not change significantly from the first to sixth case (mean for first case: 18.3; for sixth case: 19.6; p=.593), OSATS ratings improved from case one (19.7) to case three (23.5; p=.001) and to case six (26.8; p=.005). Trainees completed approximately 71.6% of their first case (range: 21.4–100%). By case six, the six participants performed 81.2% of the case (range: 14.3–100%). ConclusionsD&E simulation using a newly-developed uterine model and simulation curriculum improves resident technical skills. Simulation training with mastery learning techniques transferred to high level of performance in OR using checklist. The OSATS measured skills and showed improvement in performance with subsequent cases. ImplicationsImplementation of a D&E simulation curriculum offers potential for improved surgical training and abortion provision.

Practice patterns for peer review in radiation oncology

PurposePhysician peer review seeks to improve the quality of care through the evaluation of physician performance, specifically medical decision making and technical expertise. To establish current peer review practice patterns, evaluate interest in recommendations for peer review, and establish a framework for future recommendations, the American Society for Radiation Oncology (ASTRO) surveyed its physician members. Methods and materialsA radiation oncology-specific peer review survey instrument was developed, formally tested, and found to meet established levels of reliability and validity. The final instrument was delivered using a web-based survey platform including reminders. All ASTRO physician-members and members-in-training worldwide were invited by email to participate. ResultsA total of 5674 physicians were contacted starting in January 2013. A total of 572 physicians participated (10%) yielding a ±4% margin of error. Those responding were split evenly between academic providers and private practice and others. The median time since training=16years, median number of new patients per year=215, and median practice size=6 physicians; 83% of respondents were involved in peer review and 75% were comfortable with their program. Of those involved, 65% report doing some review before radiation begins. Of patients treated by these physicians, 56% are reviewed before treatment. Peer review elements reviewed include overall treatment strategy (86%), dose and fractionation (89%), contouring (59%), and isodose or dose-volume histogram (75%). Ninety percent of physicians have changed radiation plans because of peer review. These providers make changes in 7%-10% of cases. Seventy-four percent of physicians agree that ASTRO should make formal peer review recommendations, with 7% in opposition. ConclusionsThis survey suggests that peer review in radiation oncology is common and leads to changes in management in a meaningful fraction of cases. There is much variation in the manner of conducting, and reported utility of, peer review. The majority of ASTRO physician members support formal recommendations and guidance on peer review.

Physician performance assessment using a composite quality index

Assessing physician performance is important for the purposes of measuring and improving quality of service and reducing healthcare delivery costs. In recent years, physician performance scorecards have been used to provide feedback on individual measures; however, one key challenge is how to develop a composite quality index that combines multiple measures for overall physician performance evaluation. A controversy arises over establishing appropriate weights to combine indicators in multiple dimensions, and cannot be easily resolved. In this study, we proposed a generic unsupervised learning approach to develop a single composite index for physician performance assessment by using non-negative principal component analysis. We developed a new algorithm named iterative quadratic programming to solve the numerical issue in the non-negative principal component analysis approach. We conducted real case studies to demonstrate the performance of the proposed method. We provided interpretations from both statistical and clinical perspectives to evaluate the developed composite ranking score in practice. In addition, we implemented the root cause assessment techniques to explain physician performance for improvement purposes.

Commentary

There is a substantial international body of research that supports the value of the use of multisource feedback (MSF) for physician performance assessment. In this issue, Wright and colleagues add to this body of literature by examining two instruments designed to collect MSF from large cohorts of patients and colleagues in the context of physician performance evaluation in the United Kingdom. In the United States there is a reticence by leading physician organizations to adopt the use of patient and peer feedback in key evaluations such as maintenance of specialty board certification or licensure. This commentary explores this incongruity in light of the 50th anniversary of Thomas Kuhn's seminal work, The Structure of Scientific Revolutions. It is concluded that, although the paradigm of using patient and peer evaluation in the United States appears to be shifting, the profession, by way of not providing MSF to the public, is losing its role in being the unique provider of public information on physician performance.

Peer Review in Diagnostic Radiology: Current State and a Vision for the Future

Over the past decade, the level of interest in improving the quality of healthcare in the United States has increased. New requirements established by regulatory organizations require the ongoing practice-based evaluation of physician performance. Peer review, a key process in physician performance evaluation, is geared primarily toward measuring diagnostic accuracy. Accuracy may be measured in terms of interpretive agreement or disagreement during a blinded double reading or in workstation-integrated evaluations. Each method of assessing diagnostic accuracy has strengths and weaknesses that should be carefully considered before it is implemented in a particular departmental or institutional setting.

Prospective time study derivation of emergency physician workload predictors

A reliable emergency department (ED) workload measurement tool would provide a method of quantifying clinical productivity for performance evaluation and physician incentive programs; it would enable health administrators to measure ED outputs; and it could provide the basis for an equitable formula to estimate ED physician staffing requirements. Our objectives were to identify predictors that correlate with physician time needed to treat patients and to develop a multivariable model to predict physician workload. During 31 day, evening, night and weekend shifts, a research assistant (RA) shadowed 20 emergency physicians, documenting time spent performing clinical and non-clinical functions for 585 patient visits. The RA recorded key predictors including patient gender, age, vital signs and Glasgow Coma Scale (GCS) score, and the mode of arrival, triage level assigned, comorbidity and procedures performed. Multiple linear regression was used to describe the associations between predictor variables and total physician time per patient visit (TPPV), and to derive an equation for physician workload. Model derivation was based on 16 shifts and 314 patient visits; model validation was based on 15 shifts and 271 additional patient visits. The strongest predictor variables were: procedure required, triage level, arrival by ambulance, GCS, age, any comorbidity, and number of prior visits. The derived regression equation is: TPPV = 29.7 + 8.6 (procedure required [Yes]) - 3.8 (triage level [1-5]) + 7.1 (ambulance arrival) - 1.1 (GCS [3-15]) + 0.1 (age in years) - 0.05 (n of previous visits) + 3.1 (any comorbidity). This model predicted 31.3% of the variance in physician TPPV (F [12, 29] = 13.2; p < 0.0001). This study clarifies important determinants of emergency physician workload. If validated in other settings, the predictive formula derived and internally validated here is a potential alternative to current simplistic models based solely on patient volume and perceived acuity. An evidence-based workload estimation tool like that described here could facilitate ED productivity measurement, benchmarking, physician performance evaluation, and provide the substrate for an equitable formula to estimate ED physician staffing requirements.

Patient satisfaction profiling of individual physicians: impact of panel status.

Evaluation of physician performance is increasingly based on patient satisfaction. However, few data are available regarding the extent to which individual physician profiles might be influenced by factors such as whether a physician's practice is open or closed. We evaluated whether panel status (whether or not a physician is accepting new patients) is associated with patient satisfaction with their primary care physician (PCP). Cross-sectional analysis of patient satisfaction surveys. Surveys were available for 1,750 patients cared for by 69 PCPs. Patient satisfaction with their PCP was determined based on a composite of six questions derived from the Medical Outcomes Study. We used Generalized Estimating Equations to adjust for physician level variation. Patients of closed-panel physicians were more likely to rate their satisfaction with the provider as 'Excellent' or 'Very Good' compared to patients of open-panel physicians (78% vs. 69%, P <0.0001). After adjusting for satisfaction with the practice site, provider years in practice, managed care coverage, provider productivity, and patient race, the association between a closed panel and satisfaction remained significant (odds ratio 1.60, 95% confidence interval 1.10-2.31). Individual physicians' patient satisfaction data are confounded by factors not likely to be adjusted for in available profiles. After adjusting for other variables, physicians with closed panels still had better patient satisfaction compared to physicians with open panels. Further research is necessary to determine if panel status might also confound patient satisfaction.

Development, implementation and reliability assessment of an emergency physician performance evaluation tool

Evaluation of physician practice is necessary, both to provide feedback for self-improvement and to guide department heads during yearly evaluations. To develop and implement a peer-based performance evaluation tool and to measure reliability and physician satisfaction. Each emergency physician in an urban emergency department evaluated their peers by completing a survey consisting of 21 questions on effectiveness in 4 categories: clinical practice, interaction with coworkers and the public, nonclinical departmental responsibilities, and academic activities. A sample of emergency nurses evaluated each emergency physician on a subset of 5 of the questions. Factor analysis was used to assess the reliability of the questions and categories. Intra-class correlation coefficients were calculated to determine inter-rater reliability. After receiving their peer evaluations, each physician rated the process's usefulness to the individual and the department. 225 surveys were completed on 16 physicians. Factor analysis did not distinguish the nonclinical and academic categories as distinct; therefore, the survey questions fell into 3 domains, rather than the 4 hypothesized. The overall intra-class correlation coefficient was 0.43 for emergency physicians, indicating moderate, but far from perfect, agreement. This suggests that variability exists between physician evaluators, and that multiple reviewers are probably required to provide a balanced physician evaluation. The intra-class correlation coefficient for emergency nurses was 0.11, suggesting poor reliability. Overall, 11 of 15 physicians reported the process valuable or mostly valuable, 3 of 15 were unsure and 1 of 15 reported that the process was definitely not valuable. Physician evaluation by a single individual is probably unreliable. A useful physician peer evaluation tool can be developed. Most physicians view a personalized, broad-based, confidential peer review as valuable.

Is Price a Signal of Service Quality? Challenging Conventional Wisdom in the Health Care Industry

ABSTRACT A survey of 133 patients/consumers is used to assess their (1) level of satisfaction with health care treatment and (2) evaluation of physician performance across a variety of health care coverage options, ranging from entitlements to traditional indemnity health care plans. Following conventional wisdom, the research hypothesizes a positive relationship between the amount of out-of-pocket expenses to the patient/consumer (price) and both satisfaction and performance appraisals (quality). The results, however, show the opposite to be true. Findings indicate that the assumed price-quality relation-ship may not hold for health care. Patients/consumers don't necessarily feel that they only “get what they pay for” in terms of the quality of health care treatment and physician providers.

Achieving Effective Medical Information Security: Understanding the Culture

Cet article decrit, du point de vue d'un clinicien generaliste, une solution pour l'implementation de dossiers concernant les patients en toute securite, ainsi que les strategies mises en place pour cette implementation. Cet article souligne donc l'importance de la securite pour la confidentialite de ces dossiers, mais aussi la difficulte de mettre en place un systeme de protection de l'information sur

Assessment of patient satisfaction as part of a physician performance evaluation: the Fallon Clinic experience.

Patient satisfaction surveys were used as part of the physician evaluation process in a 300-member physician group practice with a largely health maintenance organization (HMO) patient population. The surveys were incorporated into the review process at the request of physicians who wanted to be evaluated by patients. Results were shared with physicians to identify problem areas for improving patient care. One year later, follow-up surveys showed improvement in patient ratings of participating physicians. In addition, physicians indicated that the surveys significantly improved the evaluation process.

Client Evaluation of Physician Performance

Client Evaluation of Physician Performance