Evaluate Physician Performance Research Articles

The Ability of Patients to Correctly Identify Their Hospitalist on a Patient Satisfaction Survey and the Impact of This Factor on Hospitalist Scores

Background: Patient evaluations of their physician have been utilized to evaluate physician performance and calculate hospital reimbursement. Despite the routine use of these evaluations, little research has been done to evaluate the accuracy of these evaluations. We assessed patient’s ability to correctly identify the hospitalist providing their care and whether identification of the hospitalist affected the cumulative satisfaction scores. Methods: This prospective observational study was conducted at a suburban voluntary hospital. Hospitalists were evaluated anonymously by their patients over a 4-month period on a questionnaire designed to assess patient satisfaction. Results: One hundred fifty-eight patients evaluated 16 hospitalists yielding an average of 9.9 evaluations per hospitalist. Forty Nine (31%) patients were able to identify their hospitalist. The top score (3) accounted for 76% of patient responses across all questions. Significant differences (p = .0007) existed in the distribution of these scores when patients who could identify their hospitalist was compared with those who couldn’t. The variables for effective communication and hospitalist introduction were scored significantly higher by patients who could identify their hospitalist (2.77 vs 2.55; p = .027 and 2.90 vs 2.67; p = .005). Patients of younger age and female gender were more likely identify their hospitalist. Conclusions: Less than one-third of patients were able to correctly identify their hospitalist. The variables for effective communication and hospitalist introduction to the patient were scored significantly higher by patients who could identify their hospitalist. The elimination of scores by patients who could not correctly identify their provider resulted in marked changes in the percentile ranking of these providers.

Should Patients Use Online Reviews to Pick Their Doctor? Lessons from Illinois and Indiana

How accurate and useful are online reviews of physicians? At a basic level, do online reviews meaningfully differentiate physicians with problematic medical malpractice (med mal) and disciplinary records from physicians with clean records? We compare the online reviews of 221 “Questionable physicians” from Illinois and Indiana (each of whom had multiple paid med mal claims and disciplinary sanctions) with matched control physicians with clean med mal and disciplinary records. Across five prominent online rating services, we find small, mostly insignificant differences in the star ratings and written reviews of Questionable versus control physicians, and only modest correlation in the ratings of the same physicians across different rating services. Online rating services also fail to capture the information provided by physicians’ med mal and disciplinary records; the only rating service that reports on med mal claims and disciplinary actions (Healthgrades) misses more than 90% of these actions. Patients that want to avoid Questionable physicians should not rely on online rating services to help them do so. Our findings also raise serious questions about the utility of patient-generated quality measures in evaluating physician performance.

Perceptions of people with respiratory problems on physician performance evaluation-A qualitative study.

BackgroundDespite increasing calls for patient and public involvement in health‐care quality improvement, the question of how patient evaluations can contribute to physician learning and performance assessment has received scant attention.ObjectiveThe objective of this study was to explore, amid calls for patient involvement in quality assurance, patients' perspectives on their role in the evaluation of physician performance and to support physicians’ learning and decision making on professional competence.DesignA qualitative study based on semi‐structured interviews.Setting and ParticipantsThe study took place in a secondary care setting in the Netherlands. The authors selected 25 patients from two Dutch hospitals and through the Dutch Lung Foundation, using purposive sampling.MethodsData were analysed according to the principles of template analysis, based on an a priori coding framework developed from the literature about patient empowerment, feedback and performance assessment.ResultsThe analysis unearthed three predominant patient perspectives: the proactive perspective, the restrained perspective and the outsider perspective. These perspectives differed in terms of perceived power dynamics within the doctor‐patient relationship, patients' perceived ability, and willingness to provide feedback and evaluate their physician's performance. Patients' perspectives thus affected the role patients envisaged for themselves in evaluating physician performance.Discussion and conclusionAlthough not all patients are equally suitable or willing to be involved, patients can play a role in evaluating physician performance and continuing training through formative approaches. To involve patients successfully, it is imperative to distinguish between different patient perspectives and empower patients by ensuring a safe environment for feedback.

Non-Financial Performance Measures and Physician Compensation

This study utilizes a national survey of physicians in the United States, administered four times between 1996 and 2005, to examine the use of non-financial performance measures in physician compensation contracts. Consistent with agency theory, we find that non-financial measures are used more frequently when the measures are more informative, when alternative control mechanisms are complements rather than substitutes, and when external pressures for quality of care and cost-containment are greater. Further, we find that contractual relationships in the healthcare value chain are inter-related; non-financial measures are more likely to be used to evaluate physician performance when the physicians’ practice is compensated based on fixed rate payments (i.e., capitation). We also find that physicians’ compensation contracts are more likely to incorporate non-financial performance measures when productivity in revenue generation is also used to evaluate performance. Taken together, the results suggest that non-financial performance measures play a significant role in physician compensation, acting to balance incentives tied to individual physician productivity.

Nonfinancial Performance Measures and Physician Compensation

ABSTRACT: This study utilizes a national survey of physicians in the United States, administered four times between 1996 and 2005, to examine the use of nonfinancial performance measures in physician compensation contracts. Consistent with agency theory, we find that nonfinancial measures are used more frequently when the measures are more informative; when alternative control mechanisms are complements rather than substitutes; and when external pressures for quality of care and cost containment are greater. Further, we find that contractual relationships in the health care value chain are interrelated; nonfinancial measures are more likely to be used to evaluate physician performance when the physicians’ practice is compensated based on fixed rate payments (i.e., capitation). We also find that physicians’ compensation contracts are more likely to incorporate nonfinancial performance measures when productivity in revenue generation is also used to evaluate performance. Taken together, the results suggest that nonfinancial performance measures play a significant role in physician compensation, acting to balance incentives tied to individual physician productivity.

Evaluating Physician Performance at Individualizing Care: A Pilot Study Tracking Contextual Errors in Medical Decision Making

Clinical decision making requires 2 distinct cognitive skills: the ability to classify patients' conditions into diagnostic and management categories that permit the application of research evidence and the ability to individualize or-more specifically-to contextualize care for patients whose circumstances and needs require variation from the standard approach to care. The purpose of this study was to develop and test a methodology for measuring physicians' performance at contextualizing care and compare it to their performance at planning biomedically appropriate care. First, the authors drafted 3 cases, each with 4 variations, 3 of which are embedded with biomedical and/or contextual information that is essential to planning care. Once the cases were validated as instruments for assessing physician performance, 54 internal medicine residents were then presented with opportunities to make these preidentified biomedical or contextual errors, and data were collected on information elicitation and error making. The case validation process was successful in that, in the final iteration, the physicians who received the contextual variant of cases proposed an alternate plan of care to those who received the baseline variant 100% of the time. The subsequent piloting of these validated cases unmasked previously unmeasured differences in physician performance at contextualizing care. The findings, which reflect the performance characteristics of the study population, are presented. This pilot study demonstrates a methodology for measuring physician performance at contextualizing care and illustrates the contribution of such information to an overall assessment of physician practice.

Measuring resident physicians' performance of preventive care. Comparing chart review with patient survey.

The Accreditation Council for Graduate Medical Education has suggested various methods for evaluation of practice-based learning and improvement competency, but data on implementation of these methods are limited. To compare medical record review and patient surveys on evaluating physician performance in preventive services in an outpatient resident clinic. Within an ongoing quality improvement project, we collected baseline performance data on preventive services provided for patients at the University of Alabama at Birmingham (UAB) Internal Medicine Residents' ambulatory clinic. Seventy internal medicine and medicine-pediatrics residents from the UAB Internal Medicine Residency program. Resident- and clinic-level comparisons of aggregated patient survey and chart documentation rates of (1) screening for smoking status, (2) advising smokers to quit, (3) cholesterol screening, (4) mammography screening, and (5) pneumonia vaccination. Six hundred and fifty-nine patient surveys and 761 charts were abstracted. At the clinic level, rates for screening of smoking status, recommending mammogram, and for cholesterol screening were similar (difference <5%) between the 2 methods. Higher rates for pneumonia vaccination (76% vs 67%) and advice to quit smoking (66% vs 52%) were seen on medical record review versus patient surveys. However, within-resident (N=70) comparison of 2 methods of estimating screening rates contained significant variability. The cost of medical record review was substantially higher ($107 vs $17/physician). Medical record review and patient surveys provided similar rates for selected preventive health measures at the clinic level, with the exception of pneumonia vaccination and advising to quit smoking. A large variation among individual resident providers was noted.

An automated scoring algorithm for computerized clinical vignettes: Evaluating physician performance against explicit quality criteria

Objective To evaluate the accuracy of an automated algorithm for scoring physicians’ responses to open-ended clinical vignettes against explicit, evidence-based quality criteria. Methods One hundred sixteen physicians completed a total of 915 computerized clinical vignettes at 4 sites. Each vignette simulated an outpatient primary care visit for one of 8 different clinical cases. The automated algorithm scored disease-specific quality criterion as done or not done by recognizing the presence or absence of predefined patterns in the physician's text response to the vignette. Scores generated by the automated algorithm for each criterion were compared to scores generated by trained human abstractors. Vignette responses were divided into development and test sets. Percentage agreement between automated and manual scores was computed separately for the development and test sets. Sensitivity and specificity were calculated. Costs of automated and manual scoring were compared. Results Accuracy of the algorithm exceeds 90% for both the development and test sets, and is high for care items that were deemed either necessary or unnecessary, across diverse clinical cases, and for all domains of the outpatient clinical encounter. The sensitivity of the automated scoring algorithm is 89.0%, and specificity is 93.5%. Automated scoring is approximately 84% less expensive than manual scoring. Conclusion Automated scoring of computerized vignettes appears feasible and accurate. Computerized vignettes incorporating accurate automated scoring offer the promise of a highly standardized but relatively inexpensive measurement tool for a wide range of quality assessments within and across health systems.

Why Should Pediatricians Use RBRVS-based Fee Schedules?

Why Should Pediatricians Use RBRVS-based Fee Schedules?

Why Should Pediatricians Use RBRVS-based Fee Schedules?

Why Should Pediatricians Use RBRVS-based Fee Schedules?

Interexaminer agreement on the american board of emergency medicine oral certification examination

Study objective: One of the principal criticisms of performance assessments, particularly those using an oral format to evaluate physician performance, is the lack of interrater reliability. The objective of this study was to assess interexaminer agreement in the scoring of candidates taking the American Board of Emergency Medicine (ABEM) oral certification examination. Methods: Examiner-observer pairs independently scored individual candidates on each of the 6 simulated cases selected for the October 1999 examination. Simple proportionate agreement for “acceptable” and “unacceptable” scores on critical actions and performance ratings were used to measure interexaminer (examiner-observer) agreement. Further analyses were performed to assess interexaminer agreement at the level of the raw scores of all performance ratings. The candidate's actual score was determined entirely by the examiner administering the case. The study examiner was present only to observe and independently score the candidate. Examiners were blinded to each other's scoring. All examiner-observer, examiner-candidate, and observer-candidate pairings were uniquely randomized and occurred only once during the examination. Results: Of 564 candidates, 186 were randomly selected for scoring by examiner-observer pairs drawn from a pool of 107 trained oral examiners. Interexaminer agreement for 47 critical actions was 97% (95% confidence interval [CI] 96% to 98%), and interexaminer agreement for 68 performance ratings was 95% (95% CI 94% to 96%). Further analyses of raw scores of performance ratings indicated that 94% (95% CI 93% to 95%) of 2,648 pairs of ratings differed by 1 point or less. Conclusion: High interexaminer agreement on the scoring of the ABEM oral certification examination supports the reliability of this final step in the certification process. On the basis of these findings, ABEM has incorporated this methodology into the administration of each oral examination as an ongoing quality control measure. [Ann Emerg Med. 2003;41:859-864.]

Why Should Pediatricians Use RBRVS-based Fee Schedules?

Why Should Pediatricians Use RBRVS-based Fee Schedules?

Why Pediatricians Should Use RBRVS-based Fee Schedules

Why Pediatricians Should Use RBRVS-based Fee Schedules

Using Standardized Patients to Measure Quality: Evidence from the Literature and a Prospective Study

Use of standardized patients for evaluating the clinical skills of medical students and medical trainees is commonplace. This has encouraged the use of standardized patients to evaluate the quality of physician practice in outpatient settings. However, there may be substantive differences between observing student performance and evaluating whether the provision of care meets defined quality criteria. This study had two primary objectives: (1) to review studies that use standardized patients to evaluate physician performance and (2) to ascertain directly whether standardized patients could be useful in assessing quality of outpatient care. A comprehensive literature review of studies that used standardized patients to assess physician performance was conducted. A prospective study that included 20 physicians at two outpatient settings and 27 actor patients assessed quality of care using eight clinical cases divided into five clinical domains, each of which had explicit criteria checklists based on widely accepted guidelines. The literature review identified five important issues: developing scenarios, selecting explicit criteria, standardizing standardized patient training, creating subterfuges, and ensuring reliability and validity of measures. In the study, trained standardized patients were able to assess physician practice accurately for common medical conditions, using proven criteria linked to health outcomes. The detection rate was 3%. There was no performance variation between actors for seven of the eight cases. Using standardized patients to measure the quality of care is practical and feasible. The major methodological challenge is incorporating observable evidence-based criteria into realistic scripts and objective checklists. The major logistical challenge is obtaining and maintaining undetected entry into physicians' offices.

Use of Peer Ratings to Evaluate Physician Performance

RAMSEY, PAUL G.; WEINRICH, MARJORIE D.; CARLINE, JAN D.; INUL, THOMAS N.; LARSON, ERIC B.; LOGERFO, JAMES P. Author Information

Use of Peer Ratings to Evaluate Physician Performance

Use of Peer Ratings to Evaluate Physician Performance

Use of Peer Ratings to Evaluate Physician Performance

To assess the feasibility and measurement characteristics of ratings completed by professional associates to evaluate the performance of practicing physicians. The clinical performance of physicians was evaluated using written questionnaires mailed to professional associates (physicians and nurses). Physician-associates were randomly selected from lists provided by both the subjects and medical supervisors, and detailed information was collected concerning the professional and social relationships between the associate and the subject. Responses were analyzed to determine factors that affect ratings and measurement characteristics of peer ratings. Physician-subjects were selected from among practicing internists in New York, New Jersey, and Pennsylvania who received American Board of Internal Medicine certification 5 to 15 years previously. Physician performance as assessed by peers. Peer ratings are not biased substantially by the method of selection of the peers or the relationship between the rater and the subject. Factor analyses suggest a two-dimensional conceptualization of clinical skills: one factor represents cognitive and clinical management skills and the other factor represents humanistic qualities and management of psychosocial aspects of illness. Ratings from 11 peer physicians are needed to provide a reliable assessment in these two areas. These findings suggest that it is feasible to obtain assessments from professional associates of practicing physicians in areas such as clinical skills, humanistic qualities, and communication skills. Using a shorter version of the questionnaire used in this study, peer ratings provide a practical method to assess clinical performance in areas such as humanistic qualities and communication skills that are difficult to assess with other measures.

Use of peer ratings to evaluate physician performance

Use of peer ratings to evaluate physician performance

Evaluation of substance-abusing adolescents by primary care physicians

Physicians are in a unique position to identify substance abusing adolescents. To evaluate physician performance from the patient's perspective, we interviewed 54 substance-abusing adolescents and their parents about previous medical encounters. Although nearly all patients had seen a physician during the time they were using drugs or alcohol, 43% did not recall being asked by a doctor about alcohol or drug use. Of the 26 patients who recalled being asked, 12 (46%) stated that they responded dishonestly, usually because a parent was present. Of the 23 who didn't recall being asked, five (18%) wished they had been asked. Physicians tended to ask about substance abuse more often ( p = .08) when they had previously discussed the problem with the parents, but many parents did not initiate such discussions. This survey suggests that physicians may not adequately assess high-risk adolescents for substance abuse because of physician-, patient- and/or parent-dependent factors.

The medical record as a basis for assessing physician competence.

The medical record can be used for evaluating physician performance. Within the limitations described, it is an essential source of information on the delivery of care as well as a measure of proficiency. The process of care should be particularly emphasized, without neglecting appropriate outcome measures. The confounding of outcome measures by compliance factors, natural history of the disease process, severity of illness, status of defense mechanisms, performance of ancillary services, and personnel does not make this an attractive single measure of physician performance. Knowledge assessment and performance assessment play a complementary role; both are necessary to identify competency. The potential for improving medical care and the recording of essential findings by such an emphasis on individual assessment based on medical records is enormous. Technologic progress in recording and retrieving medical information is facilitating use of the record to measure proficiency. A carefully structured blend of cognitive examination to establish a knowledge base and review of the medical record to measure performance will provide better measures of competency.