Standardized Patient Scores Research Articles

Analysis of virtual standardized patients for assessing clinical fundamental skills of medical students: a prospective study

BackgroundHistory-taking is an essential clinical competency for qualified doctors. The limitations of the standardized patient (SP) in taking history can be addressed by the virtual standardized patient (VSP). This paper investigates the accuracy of virtual standardized patient simulators and evaluates the applicability of the improved system’s accuracy for diagnostic teaching support and performance assessment.MethodsData from the application of VSP to medical residents and students were gathered for this prospective study. In a human–machine collaboration mode, students completed exams involving taking SP histories while VSP provided real-time scoring. Every participant had VSP and SP scores. Lastly, using the voice and text records as a guide, the technicians will adjust the system’s intention recognition accuracy and speech recognition accuracy.ResultsThe research revealed significant differences in scoring across several iterations of VSP and SP (p < 0.001). Across various clinical cases, there were differences in application accuracy for different versions of VSP (p < 0.001). Among training groups, the diarrhea case showed significant differences in speech recognition accuracy (Z = -2.719, p = 0.007) and intent recognition accuracy (Z = -2.406, p = 0.016). Scoring and intent recognition accuracy improved significantly after system upgrades.ConclusionVSP has a comprehensive and detailed scoring system and demonstrates good scoring accuracy, which can be a valuable tool for history-taking training.

Patient satisfaction and its health provider-related determinants in primary health facilities in rural China

BackgroundPatient satisfaction is an important outcome measure of health service and is one of the main reasons for the gradual deterioration of doctor–patient relationships in China. This study used the standardized patient (SP) method to explore patient satisfaction and its health provider-related determinants among primary health facilities in rural China.MethodsThe dataset comprised 1138 clinic cases in 728 rural primary health facilities in 31 counties, spread across four provinces. Information regarding the consultation interaction between the unannounced SPs and primary physicians was recorded. Patient satisfaction was gathered from the feedback of SPs after the visit.ResultsThe overall average score of SP satisfaction with rural primary health facilities was only 13.65 (SD = 3.22) out of 20. The SP scores were found to be consistent with those of real patients. After controlling variances in patient population via the SP method, the regression analysis demonstrated that health provider-related factors, such as physician-level characteristics, consultation process, affordability, and convenience, have a significant correlation with patient satisfaction among primary physicians. Among factors relating to physician-level characteristics, affordability, convenience and the consultation process of the visit, the quality of the consultation process (e.g., consultation time, proactively providing necessary instructions and other crucial information) were found to be the prominent determinants.ConclusionsThis study revealed the need to improve patient satisfaction in primary health facilities in rural China. To solve this issue, we recommend that policies to increase medical service quality be implemented in rural primary healthcare systems.

Checklist-based Training Improves Consenting Skills Among Radiology Residents

This study aimed to demonstrate whether a curriculum based on an informed consent conversation checklist led to improvement in a resident's ability to obtain patient centered and legally sound consent. In this prospective, IRB approved study, Radiology residents from a single institution were asked to obtain informed consent for an imaging study or image-guided procedure from a standardized patient (SP). Encounters were scored by an attending radiologist on a 20-point checklist as well as by the SPs on four consent related questions. Residents were then provided reading material, a lecture, and a consent checklist pocket card. Residents participated in a post-intervention SP encounter. Wilcoxon Signed-Ranks Test was performed to determine if there was a significant improvement in scores after intervention. Twenty-one residents completed all aspects of the program. There was statistically significant improvement in consenting skills as measured by the attending scores and the SP scores after receiving formal education. Pre-intervention scores had a mean of 9.29 of 20 (SD 1.39), while post-intervention scores had a mean of 16.95 of 20 (SD 1.83). 95.2% (20/21) of residents found the training useful and stated they would recommend it to future trainees. Checklist-based consent training improved radiology residents' ability to obtain informed consent.

Augmenting physician examiner scoring in objective structured clinical examinations: including the standardized patient perspective.

In Canada, high stakes objective structured clinical examinations (OSCEs) administered by the Medical Council of Canada have relied exclusively on physician examiners (PEs) for scoring. Prior research has looked at using SPs to replace PEs. This paper reports on two studies that implement and evaluate a standardized patient (SP) scoring tool to augment PE scoring. The unique aspect of this study is that it explores the benefits of combining SP and PE scores. SP focus groups developed rating scales for four dimensions they labelled: Listening, Communication, Empathy/Rapport, and Global Impression. In Study I, 43 SPs from one site of a national PE-scored OSCE rated 60 examinees with the initial SP rating scales. In Study II, 137 SPs used slightly revised rating scales with optional narrative comments to score 275 examinees at two sites. Examinees were blinded to SP scoring and SP ratings did not count. Separate PE and SP scoring was examined using descriptive statistics and correlations. Combinations of SP and PE scoring were assessed using pass-rates, reliability, and decision consistency and accuracy indices. In Study II, SP and PE comments were examined. SPs showed greater variability in their scoring, and rated examinees lower than PEs on common elements, resulting in slightly lower pass rates when combined. There was a moderate tendency for both SPs and PEs to make negative comments for the same examinee but for different reasons. We argue that SPs and PE assess performance from different perspectives, and that combining scores from both augments overall reliability of scores and pass/fail decisions. There is potential to provide examinees with feedback comments from each group.

A study on evaluator factors affecting physician-patient interaction scores in clinical performance examinations: a single medical school experience.

BackgroundThis study is an analysis of evaluator factors affecting physician-patient interaction (PPI) scores in clinical performance examination (CPX). The purpose of this study was to investigate possible ways to increase the reliability of the CPX evaluation. MethodsThe six-item Yeungnam University Scale (YUS), four-item analytic global rating scale (AGRS), and one-item holistic rating scale (HRS) were used to evaluate student performance in PPI. A total of 72 fourth-year students from Yeungnam University College of Medicine in Korea participated in the evaluation with 32 faculty and 16 standardized patient (SP) raters. The study then examined the differences in scores between types of scale, raters (SP vs. faculty), faculty specialty, evaluation experience, and level of fatigue as time passes. ResultsThere were significant differences between faculty and SP scores in all three scales and a significant correlation among raters’ scores. Scores given by raters on items related to their specialty were lower than those given by raters on items out of their specialty. On the YUS and AGRS, there were significant differences based on the faculty’s evaluation experience; scores by raters who had three to ten previous evaluation experiences were lower than others’ scores. There were also significant differences among SP raters on all scales. The correlation between the YUS and AGRS/HRS declined significantly according to the length of evaluation time. ConclusionIn CPX, PPI score reliability was found to be significantly affected by the evaluator factors as well as the type of scale.

Comparison of Resident Self-Evaluation to Standardized Patient Evaluators in a Multi-Institutional Objective Structured Clinical Examination: Objectively Measuring Residents' Communication and Counseling Skills.

For the past 30 years, there has been a growing emphasis on communication and self-evaluation skills training in graduate medical education. This is reflected in the Next Accreditation System. The Objective Structured Clinical Examination (OSCE) is widely used in graduate medical education for assessing dimensions of interpersonal communication and counseling skills. The OSCEs may be developed to target challenging clinical scenarios difficult to capture in clinical practice and can be used as a medium for resident self-evaluation. The aims of the study were to evaluate residents' interpersonal, communication, and counseling skills using Kalamazoo Essential Elements Communication Checklist in 4 clinically challenging scenarios and to compare standardized patient (SP) evaluations to residents' self-evaluation by category of medical school. South East Michigan Center for Medical Education is a consortium of teaching hospitals. Member residents participate in 4 OSCEs as part of their postgraduate 1 curriculum. The OSCEs were developed to evaluate clinically relevant but difficult to capture scenarios including: (a) error disclosure/counseling an angry patient; (b) delivering bad news/end of life; (c) domestic violence; and (d) counseling a patient with colon cancer requesting alternative treatments. At the conclusion of each OSCE, SPs evaluated and residents self-evaluated their performance. Once evaluations were completed, SPs provided residents with feedback. Six member institutions and 344 residents participated during the 2014, 2015, and 2016 academic years. There were more international medical graduates (59%) than graduates of Liaison Committee for Medical Education-accredited medical schools. There were more males (62.2%) than females. Standardized patients rated residents higher than residents rated themselves in 2014 (<0.001), but not in 2015 or 2016. When combining all years and all residents, there was no correlation of SP and resident scores. Standardized patients rated female residents higher than female residents rated themselves (P < 0.0001). Male residents scored themselves similarly to the SPs, but male residents rated themselves higher than female residents rated themselves (P < 0.001). Standardized patient scores for male and female residents were not significantly different. Targeted OSCEs provide an objective format to evaluate residents in challenging clinical scenarios. Resident self-evaluations did not correlate with SPs. In addition, female residents rated themselves lower than male residents and lower than SPs. There is need to develop interventions and curricula to improve resident's self-evaluation skills and in particular address lower self-evaluation by female trainees.

Practice Makes Perfect: Training Residents in Difficult Encounters.

A significant number of patient encounters are perceived to be difficult. Residents receive little training in managing difficult clinical encounters, and lectures or traditional simulated patient encounters may not provide the opportunity to practice learned skills. Deliberate practice has been shown to be effective in training clinical skills. We used simulation with deliberate practice and feedback to train residents in difficult patient encounters. Twelve second-year residents in the University of Missouri Family and Community Medicine residency program participated in simulated patient encounters with difficult patients. The patients represented challenging personalities identified in a resident focus group. Resident performance was scored by the standardized patient, resident observers, and faculty instructor. Following debriefing with feedback, the residents repeated the clinical encounter incorporating the feedback. The sessions were scored again by the same individuals. All scores improved from the first to second clinical encounter, except those that were at 100% for both encounters. The most improvement seen was in standardized patient scores. The smallest improvement was in provider self-scores. Resident performance improved according to all observers and their own self-assessments. These results are consistent with other studies that have demonstrated the effectiveness of deliberate practice in improving skills in other areas of medical education. Simulation-based learning with deliberate practice has the potential to improve resident management of difficult patient encounters.

Differences in Faculty and Standardized Patient Scores on Professionalism for Second-Year Podiatric Medical Students During a Standardized Simulated Patient Encounter

This study examined the differences between faculty and trained standardized patient (SP) evaluations on student professionalism during a second-year podiatric medicine standardized simulated patient encounter. Forty-nine second-year podiatric medicine students were evaluated for their professionalism behavior. Eleven SPs performed an assessment in real-time, and one faculty member performed a secondary assessment after observing a videotape of the encounter. Five domains were chosen for evaluation from a validated professionalism assessment tool. Significant differences were identified in the professionalism domains of "build a relationship" ( P = .008), "gather information" ( P = .001), and share information ( P = .002), where the faculty scored the students higher than the SP for 24.5%, 18.9%, and 26.5% of the cases, respectively. In addition, the faculty scores were higher than the SP scores in all of the "gather information" subdomains; however, the difference in scores was significant only in the "question appropriately" ( P = .001) and "listen and clarify" ( P = .003) subdomains. This study showed that professionalism scores for second-year podiatric medical students during a simulated patient encounter varied significantly between faculty and SPs. Further consideration needs to be given to determine the source of these differences.

Comparison of Patient-Physician Interaction Scores of Clinical Practice Examination between Checklists and Rating Scale.

BackgroundThe clinical practice examination (CPX) was introduced in 2010, and the Seoul-Gyeonggi CPX Consortium developed the patient-physician interaction (PPI) assessment tool in 2004. Both institutions use rating scales on classified sections of PPI but differ in their scoring of key components. This study investigated the accuracy of standardized patient scores across rating scales by comparing checklist methods and verified the concurrent validity of two comparable PPI rating tools.MethodsAn educational CPX module dyspepsia case was administered to 116 fourth-year medical students at Hanyang University College of Medicine. One experienced standardized patient rated exams using two different PPI scales. She scored checklists composed of 43 items related to the two original PPI scales through video clips of the same students. From these checklists, we calculated Pearson's correlation coefficient.ResultsThe correlations of total PPI score between the checklist and rating scale methods were 0.29 for the Korean Medical Licensing Examination (KMLE) tool and 0.30 for the consortium tool. The correlations between the KMLE and consortium tools were 0.74 for checklists and 0.83 for rating scales. In terms of section scores, the consortium tool showed only three significant correlations between the two methods out of seven sections and the KMLE tool showed only two statistically significant correlations out of five sections.ConclusionThe rating scale and checklist methods exhibited a weak relationship in the PPI assessment, but a high correlation between assessment tools using the same method. However, the current rating scale requires modification by reorganizing key scoring components through factor analysis.

Evaluation of a Medical Spanish Elective for Senior Medical Students: Improving Outcomes through OSCE Assessments.

There is no standardized curriculum or assessment method that addresses Spanish skills in U.S. medical schools. The authors implemented a 10-week Clinical Medical Spanish elective for fourth-year medical students. Data were collected for 58 students from 2013 to 2015 pertaining to pre- and post-course self-evaluation of Medical Spanish comfort level and Objective Structured Clinical Examinations. Student comfort level with Spanish-speaking patients improved for performing history and examination skills (P < 0.05). Standardized Patient (SP) and faculty-rated fluency were each higher than the student self-rated pre-course fluency (P < 0.001). Fluency levels were higher than the faculty rating of student ability to perform the medical interview (P < .001). Differences in student self-rating, SP scores, and faculty assessment illustrate the complexities of assessing medical skills in a second language, and highlight the importance of training and evaluating medical students and hospital staff in the appropriate use of Medical Spanish.

Assessing Nursing Students' Clinical Competencies Using a Problem-Focused Objective Structured Clinical Examination.

The objectives of this study were to develop, implement, and evaluate an innovative modified Objective Structured Clinical Examination (OSCE) model, and to compare students' performance of different clinical skills as assessed by standardized patients and OSCE examiners. Data were obtained from final year undergraduate students undergoing the modified OSCE as a graduation examination. Seventy-seven students rotated through four stations (nine substations). Standardized patients scored students higher than examiners in history taking (9.14 ± 0.92 vs. 8.42 ± 0.85), response to emergency event (8.88 ± 1.12 vs. 7.62 ± 1.54), executive medical orders (8.77 ± 0.96 vs. 8.25 ± 1.43), technical operation (18.21 ± 1.26 vs. 16.91 ± 1.35), nursing evaluation (4.53 ± 0.28 vs. 4.29 ± 0.52), and health education stations (13.79 ± 1.31 vs. 11.93 ± 2.25; p < .01). In addition, the results indicated that the difference between standardized patient and examiner scores for physical examination skills was nonsignificant (8.70 ± 1.18 vs. 8.80 ± 1.27; p > .05). The modified, problem-focused, and nursing process-driven OSCE model effectively assessed nursing students' clinical competencies, and clinical and critical thinking.

Validity Evidence for a Patient Note Scoring Rubric Based on the New Patient Note Format of the United States Medical Licensing Examination

This study examines validity evidence for the Patient Note Scoring Rubric, which was developed for a local graduation competency exam (GCE) to assess patient notes written in the new United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills format. The rubric was designed to measure three dimensions: Documentation, justified differential diagnosis (DDX), and Workup. Analyses used GCE data from 170 fourth-year medical students who completed five standardized patient (SP) cases in May 2012. Five physician raters each scored all responses for one case. Internal structure was examined using correlations between dimensions and between cases; a generalizability study was also conducted. Relationship to other variables was examined by correlating patient note scores with SP encounter scores. Consequence was assessed by comparing pass-fail rates between the rubric and the previous global rating. Response process was examined using rater feedback. Correlations between scores from different dimensions ranged between 0.33 and 0.44. Reliability of scores based on the phi coefficient was 0.43; 15 cases were required to reach a phi coefficient of 0.70. Evidence of case specificity was found. Documentation scores were moderately correlated with SP scores for data gathering (r = 0.47, P < .001). There was no meaningful change in pass-fail rates. Raters' feedback indicated that they required more training for scoring the DDX and Workup dimensions. There is initial validity evidence for use of this rubric to score local clinical exams that are based on the new USMLE patient note format.

An Evaluation of Interactive Web-Based Curricula for Teaching Code Status Discussions

Teaching resuscitation discussions to medical students and residents is time intensive and should be taught by teachers with competence in this area of clinical practice. There are plenty of data that these discussions are often inadequate, and that communication skills training, while time and faculty intensive, improves these conversations. The role of online instruction in teaching communication skills, such as resuscitation discussions, is not established. The study objective was to evaluate the effectiveness of an interactive online curriculum in teaching code status discussions to third-year medical students at one medical school. In this study we block randomized third-year medical students to one of two web-based curricula versus a similar written curriculum and used standardized patient scores on a communication exercise to measure differences in performance. We used student surveys to measure student satisfaction with the curricula. Of 121 students participating in the study between April 2007 and March 2008, 88 were randomized to one of two website formats and 33 to the written curriculum. We compared average scores on content and communication checklists between groups. There were no differences between the three groups in the primary outcome of student performance. We could not analyze data on satisfaction with or completion rates for the three curricula due to poor response rates to the student surveys. This block randomized study of web-based curricula versus a written curriculum did not show differences in student performance in code status discussions. The optimal use of online communication training remains unclear and requires further investigation.

Reliability of standardized patients used in a communication study on international nurses in the United States of America

As an evaluation method, standardized patients have a long history in medical education and research yet are less established in nursing. This paper explores the reliability of using standardized patients as the evaluative method in a communication competence pilot study with international nurses. Standardized patients and second raters scored the same encounters. We examined the scores by intraclass correlation coefficients. Anecdotal comments by the two types of raters were assessed qualitatively to highlight similarities and areas of difference between them. The results of reliability analysis for standardized patients scores for the composite variables of Establishing Communicative Rapport, Therapeutic Communication, Non-Verbal Communication, and Overall Satisfaction ranged from 0.755 (P < 0.01) to 0.42 (P = 0.09). In this study, the results showed standardized patient evaluation has moderate to substantial reliability when compared to second raters of the same set of clinical encounters. This is similar to the reliability established over many decades of medical research. Greater use of this dynamic and interactive technique may be beneficial to nursing education and research.

Do faculty show the ‘halo effect’ in rating students compared with standardized patients during a clinical examination?

BackgroundWe compared standardized patient (SP) and faculty ratings of student performance to test the hypothesis that faculty systematically overestimate student clinical skills compared with SPs.MethodsWe conducted a study of three successive classes of second year medical students (n=303). Students were assessed by SPs duing an Objective Structured Clinical Examination and faculty simultaneously rated student performance using the modified mini-clinical evaluation exercise (miniCEX). The Patient Physician Interaction scale (PPI), a communication scale designed for use by SPs and the mini-CEX measure designed for use by faculty were each analyzed for internal consistency reliability. Within the SP and faculty measures scores for similar skills of history-taking, physical examination, humanism/communication and global score were analyzed in aggregate for concordance. Range and patterns of SP and faculty scores were examined and compared.ResultsThe PPI and mini-CEX each demonstrated high internal consistency (Cronbach’s α .86, .89). Correlation between clinical skill (history, examination, humanism/communication and global) score as rated by SPs and faculty was modest (Pearson’s r 0.213 to 0.288). SPs and faculty both used the entire range of scores within each measure with a normal distribution.ConclusionsThe PPI and mini-CEX have high internal consistency reliability. There is only modest correlation between SP and faculty ratings of student clinical skills. Differences should be recognized when giving student immediate feedback on their performance in a standardized examination. Faculty rate students without a ‘halo’ effect.

Using Standardized Patients' Marks in Scoring Postgraduate Psychiatry OSCEs

Standardized patients (SPs) do not contribute scores in postgraduate psychiatry objective structured clinical examinations (OSCEs) in the United Kingdom. However, this may change in the near future. The primary aim of this study was to measure the degree of agreement between scores given by examiners and those given by SPs in an OSCE. The authors measured the degree of agreement in two consecutive postgraduate OSCEs for psychiatric residents on a London training scheme. Fifty-five candidates participated in the two OSCEs. There was a moderate degree of agreement between examiner and SP scores for communication and for the overall mark. However, there was a stronger relationship between the examiner score for communication and the candidates' overall mark. Examiners and SPs scored candidates differently. Therefore, the decision to include SP scores in the marking scheme for postgraduate OSCEs would be a significant development.

Initial Reliability of The Standardized Orthopedic Assessment Tool (SOAT)

Orthopaedic assessment skills are critical to the success of athletic therapists and trainers. The Standardized Orthopedic Assessment Tool (SOAT) has been content validated. To establish interrater reliability of the SOAT. Thirty-two college students, 10 raters, and 2 standardized patients (SPs) from Calgary, Alberta, Canada. Randomized observational study. Students were allowed 30 minutes to complete a mock orthopaedic assessment of an SP with an injury specific to a region of the body (shoulder, knee, or ankle). Using the region-specific SOAT, raters and SPs evaluated students' orthopaedic assessment skills. The sum totals of the SOAT for 2 raters and 1 SP were used to calculate each student's performance scores for respective scenarios. Scale reliability analysis (Cronbach alpha) was completed on the SOAT for each of the 3 body-region examinations. The mean overall reliability of 3 SOATs (ie, ankle, knee, and shoulder) was positive: alpha = .85 with the SP scores factored into the equation and alpha = .86 without the SP scores factored into the equation. Reliability for the ankle region was highest (alpha = .91), followed by the knee (alpha = .83) and the shoulder (alpha = .82). The study sample size was small, but the results will enable further study with generalization to a broader audience of athletic therapists and athletic trainers. Because a baseline measure of reliability was established using a robust statistical analysis, future researchers can employ more stringent statistical analysis and focus on the effects of various pedagogical techniques to teach and learn the underlying construct of clinical competence in orthopaedic assessment.

Ratings of Physician Communication by Real and Standardized Patients

Patient ratings of physician's patient-centered communication are used by various specialty credentialing organizations and managed care organizations as a measure of physician communication skills. We wanted to compare ratings by real patients with ratings by standardized patients of physician communication. We assessed physician communication using a modified version of the Health Care Climate Questionnaire (HCCQ) among a sample of 100 community physicians. The HCCQ measures physician autonomy support, a key dimension in patient-centered communication. For each physician, the questionnaire was completed by roughly 49 real patients and 2 unannounced standardized patients. Standardized patients portrayed 2 roles: gastroesophageal disorder reflux symptoms and poorly characterized chest pain with multiple unexplained symptoms. We compared the distribution, reliability, and physician rank derived from using real and standardized patients after adjusting for patient, physician, and standardized patient effects. There were real and standardized patient ratings for 96 of the 100 physicians. Compared with standardized patient scores, real-patient-derived HCCQ scores were higher (mean 22.0 vs 17.2), standard deviations were lower (3.1 vs 4.9), and ranges were similar (both 5-25). Calculated real patient reliability, given 49 ratings per physician, was 0.78 (95% confidence interval [CI], 0.71-0.84) compared with the standardized patient reliability of 0.57 (95% CI, 0.39-0.73), given 2 ratings per physician. Spearman rank correlation between mean real patient and standardized patient scores was positive but small to moderate in magnitude, 0.28. Real patient and standardized patient ratings of physician communication style differ substantially and appear to provide different information about physicians' communication style.

Can standardized patients replace physicians as OSCE examiners?

BackgroundTo reduce inter-rater variability in evaluations and the demand on physician time, standardized patients (SP) are being used as examiners in OSCEs. There is concern that SP have insufficient training to provide valid evaluation of student competence and/or provide feedback on clinical skills. It is also unknown if SP ratings predict student competence in other areas. The objectives of this study were: to examine student attitudes towards SP examiners; to compare SP and physician evaluations of competence; and to compare predictive validity of these scores, using performance on the multiple choice questions examination (MCQE) as the outcome variable.MethodsThis was a cross-sectional study of third-year medical students undergoing an OSCE during the Internal Medicine clerkship rotation. Fifty-two students rotated through 8 stations (6 physician, 2 SP examiners). Statistical tests used were Pearson's correlation coefficient, two-sample t-test, effect size calculation, and multiple linear regression.ResultsMost students reported that SP stations were less stressful, that SP were as good as physicians in giving feedback, and that SP were sufficiently trained to judge clinical skills. SP scored students higher than physicians (mean 90.4% +/- 8.9 vs. 82.2% +/- 3.7, d = 1.5, p < 0.001) and there was a weak correlation between the SP and physician scores (coefficient 0.4, p = 0.003). Physician scores were predictive of summative MCQE scores (regression coefficient = 0.88 [0.15, 1.61], P = 0.019) but there was no relationship between SP scores and summative MCQE scores (regression coefficient = -0.23, P = 0.133).ConclusionThese results suggest that SP examiners are acceptable to medical students, SP rate students higher than physicians and, unlike physician scores, SP scores are not related to other measures of competence.

University of Massachusetts Medical School.

University of Massachusetts Medical School.