Standardized Patient Examination Research Articles

University of Nevada School of Medicine

University of Nevada School of Medicine

University of Hawaii at Mãnoa John A. Burns School of Medicine

University of Hawaii at Mãnoa John A. Burns School of Medicine

The Impact of Repeat Information on Examinee Performance for a Large-Scale Standardized-Patient Examination

The United States Medical Licensing Examination series Step 2 Clinical Skills (CS) examination is a high-stakes performance assessment that uses standardized patients (SPs) to assess the clinical skills of physicians. Each Step 2 CS examination form involves 12 SPs, each of whom portrays a different clinical scenario or case. Examinees who fail and repeat the examination may encounter repeat information--the same SP, the same case, or the same SP portraying the same case. The goal of this study was twofold: to investigate score gains for all repeat examinees, regardless of whether they experienced repeat information, and to perform additional analyses for only those examinees who did encounter repeat information. The dataset consisted of 3,045 Step 2 CS repeat examinees who initially tested between April 2005 and December 2007. The authors used paired t tests and analysis of variance models to assess mean score gains (first attempt versus second attempt) and to determine standardized mean differences between encounters with repeat information and those without. The authors ran each set of analyses by test score component and by examinee subgroup. The authors observed significant mean score increases on second attempt examinations for the entire group of repeat examinees. However, they observed no significant score increases for the subgroup of examinees who encountered repeat information. Examinees taking Step 2 CS for the second time improve on average, and those with prior exposure to exam information do not appear to benefit unfairly from this exposure.

An investigation of construct validity of humanistic clinical skills on a medical licensure examination

Objective To test construct validity of humanistic clinical skills measured by a medical licensure performance examination using multitrait-multimethod confirmatory factor analysis. Methods Two hundred and twenty-seven third- and fourth-year undergraduate osteopathic medical students in 2007–2008 were randomly sampled. The Global Patient Assessment Tool (GPAT), designed to assess professionalism, interpersonal relationship, and doctor–patient communication was tested under two measurement methods: standard examination ratings and peer performance ratings of the same examinee. Two concurrent validity factors (data gathering and written patient notes) were included. Results Convergent validity was supported under the two methods of scoring. Moderate to strong correlations among trait factors indicated weak discriminate validity. Method effects were indicated. The relationship between GPAT and two concurrent validity factors indicate measures of relatively different constructs. Conclusion Evidence of construct validity for the GPAT indicate scores should be interpreted as measuring a humanistic clinical skills construct consisting of homogeneous measures of professionalism, interpersonal relationship, and physician–patient communication. Findings are consistent with the interpretation and use of the GPAT as an important part of medical licensure examinations. Practical implications The implications to medical licensure standardized-patient examinations are discussed in terms of the GPAT as a valid measure of medical students’ humanistic clinical skills.

Impact of an In-Person Versus Web-Based Practice Standardized Patient Examination on Student Performance on a Subsequent High-Stakes Standardized Patient Examination

Background: Optimal methods of preparing students for high-stakes standardized patient (SP) examinations are unknown. Purposes: The purpose is to compare the impact of two formats of a formative SP examination (Web-based vs. in-person) on scores on a subsequent high-stakes SP examination and to compare students' satisfaction with each formative examination format. Methods: Clustered randomized trial comparing a Web-based module versus in-person formative SP examination. We compared scores on a subsequent high-stakes SP examination and satisfaction. Results: Scores on the subsequent high-stakes SP examination did not differ between the two formative formats but were higher after the formative assessment than without (p < .001). Satisfaction was higher with the in-person than Web-based formative assessment format (4.00 vs. 3.62 on a 5-point scale, p = .01). Conclusions: Two formats of a formative SP examination led to equivalent improvement in scores on a subsequent high-stakes examination. Students preferred an in-person formative examination to online but were satisfied with both.

Improvisational Exercises to Improve Pharmacy Students' Professional Communication Skills

To create and implement improvisational exercises to improve first-year pharmacy students' communication skills. Twelve 1-hour improvisational sessions were developed and added to an existing/established patient communication course to improve 3 basic skills: listening, observing and responding. Standardized patient examinations were used to evaluate student communication skills, and course evaluations and reflective journaling were used to evaluate students' perceptions of the improvisational exercises. The improvisational exercises markedly improved the students' performance in several aspects of standardized patient examination. Additionally, course evaluations and student comments reflected their perception that the improvisational exercises significantly improved their communication skills. Improvisational exercises are an effective way to teach communication skills to pharmacy students.

Current Methods in Use for Assessing Clinical Competencies: What Works?

An online survey was used to capture qualitative descriptions of methods used by a veterinary college to assess clinical competencies in its students. Each college was specifically asked about use of the methods detailed in the Toolbox of Assessment Methods developed by the Accreditation Council for Graduate Medical Education and the American Board of Medical Specialties. Additionally, each college was asked to detail the methods used to ensure competency in each of the nine areas specified by the American Veterinary Medical Association Council on Education. Associate deans of academic affairs or their equivalents at veterinary colleges in the United States, the United Kingdom, Canada, and the Caribbean were contacted by e-mail and asked to complete the survey. Responses were obtained from 24 of 32 colleges. The methods most often used were review of students' medical records (16), checklist evaluation of must-learn skills (16), procedural logs (11), multiple-choice skill examinations (11), case simulations using role-playing (7), short-answer skill examinations (7), global rating of live or recorded performance (7), case simulations using computerized case simulations (7), 360-degree evaluation of clinical performance (4), and standardized patient or client examination (3). Additional methods used included medical record portfolio review, paper-and-pencil branching problems, chart-stimulated oral exams, externship mentor evaluation, performance rubrics for clinical rotations, direct observation and query on cases, video evaluation, case correlation tasks, and an employer survey. Non-realistic models were used more often for skill evaluation than realistic models. One college used virtual-reality models for testing.

Objective structured clinical examinations as an assessment method in residency training: practical considerations.

Objective structured clinical examinations as an assessment method in residency training: practical considerations.

Can simulations measure empathy? Considerations on how to assess behavioral empathy via simulations

Standardized patient simulations have been used as an assessment tool, providing teachers an opportunity to observe learner clinical and communication skills while eliminating the fear of harm to patients. Yet the vices and virtues of these simulations in measuring clinical and communication skills have been deliberated. Based on our standardized patient examination experience, we believe standardized patient simulations can be used to assess certain forms of learners’ empathic behaviors. We advocate that, in properly designed and conducted simulations, the scores and feedback comments from standardized patients to learners regarding their empathic behaviors can identify learners with important deficiencies. We conclude our discussion by recommending that reflective practice, challenging cases, decision moments, and raters training to provide feedback can supplement and enrich the use of standardized patient simulations in evaluating empathy.

Using the Integrated Standardized Patient Examination to Assess Clinical Competence in Physical Therapist Students

Background and Purpose. To test the success of integrated curricula and “clinical readiness” of students in Doctor of Physical Therapy (DPT) programs, meaningful measurements of student competence are necessary. Competence testing must include an authentic clinical encounter. Standardized patients (SP) have long shown to be reliable teaching and assessment tools in health professional education and should be explored with DPT students. Method/Model Description and Evaluation. This paper analyzes a new performance assessment tool, the Integrated Standardized Patient Examination (ISPE), specifically designed for assessing clinical competence in DPT students. Students performed 2 ISPE cases; one portraying a herniated lumbar disc (HD) and the other a status post-cerebral vascular accident (CVA). A maximum of 30 minutes was allotted to conduct a history, to respond to 4 “integration” questions asked by the SP, and to perform a targeted examination. Expert physical therapist (PT) evaluators, the investigator, and students scored each section and an overallencounter quality section. The SP also scored the overall-encounter section. All participants filled out a feedback survey about the experience. Outcomes. Acceptable reliability was demonstrated via interrater agreement on items that used a dichotomous scale, whereas the items requiring the use of the 4-point rubric were somewhat less reliable. For the entire scale, both cases had a significant correlation between the expert-investigator pair of raters; for the CVA case (r = .547, P < .05); and for the HD case (r = .700, P < .01). On the overall encounter section, the SPs scored students higher than the other raters. Students' self-assessments were most closely aligned with the investigator. Content validity was gathered in the process of developing cases and patient scenarios. Discussion and Conclusion. Future studies should examine the effect of rater training upon reliability. Criterion or predictive validity could be further studied by comparing students' performances on the ISPE with other estimates of students' competence. The unique integration questions of the ISPE were judged by experts and students to have good content validity; confirming that integration—a most crucial element of clinical competence—can be practiced, assessed, and used to determine student clinical competence.

Patient Assessment and Management Examination: lack of correlation between faculty assessment and resident self-assessment

Background The Patient Assessment and Management Examination (PAME) is a standardized patient examination designed to assess management skills of senior residents. This study explored the relationship between faculty and resident self-evaluation by using PAME. Methods Nine postgraduate year (PGY) 4 and PGY5 residents were examined with a 5 case PAME. Faculty rated interactions between residents and standardized patients and residents rated themselves based on review of audio-video recordings of their interactions. We examined correlations between faculty and resident self-assessments. Results Faculty and resident ratings of physical examination skills was the only competency that correlated significantly. Correlations were not significant for the other 15 competencies (Pearson r, –.197 to .262). Correlation was no better when examined within each case. Conclusions Although PAME may be a useful tool, this study suggests that even senior residents do not assess their performance as clinicians similarly to faculty. Further research is needed to better understand the source of these disagreements.

Crohn's disease at the small bowel imaging by the ultrasound-enteroclysis

Summary Crohn's disease is more likely a systemic disease governed by a shift in the immune response, thus affecting the whole MALT system. Its treatment should be as conservative as possible and surgery is usually taking place after complications like indolent fistulations, stenoses, bleeding, or bowel perforation started. Standard radiological methods to check the extent of the disease are loaded either with certain radiation exposure (enteroclysis, CT) or lack standardization (ultrasound). The aim of this small study was to evaluate the worth of ultrasound-enteroclysis in detecting the extent and complications of the Crohn's disease in surgically treated patients. Material and methods Since 1997, when we started with the ultrasound-enteroclysis, 246 surgical performances were involved into our study. Out of them, 181 had conventional abdominal intestinal ultrasound as well as conventional enteroclysis within 1 week. Remaining 65 cases were diagnosed by the ultrasound-enteroclysis. Intestinal ultrasound was performed on the Ultramark 3000 HDI device with autofocussable convex 5 MHz and linear 7.5 MHz probes or nowadays ATL 5000 HDI, 7–12 MHz linear probe. No contrast enhancement was used. Enteroclysis was done with the Micropaque suspension diluted 1:1 with HP-7000 300 ml with its application rate up to 75 ml/min followed by HP-7000 solution 2000 ml, application rate of 120 ml/min. The patients with ultrasound-enteroclysis were applied HP 7000 solution only (2000 ml, rate 100 ml/s) via an enteroclysis catheter. All investigations were video-recorded. Results Consent with the per-operative finding was reached in 162 from 181 enteroclyses and in 169 of 181 ultrasounds. Ultrasound-enteroclysis was precise in 61 cases from 65. Among these, 60 patients had the recurrence during the treatment proved by clinical and laboratory results. This re-activation was clearly revealed in 38 from 43 cases by enteroclysis, 41 from 43 by US and in 16 from 17 by ultrasound-enteroclysis. From 30 patients that developed acute complication non-responding to the conservative therapy (abscesses, fistulas and intestinal obstructions) there were 18 from 20 accurately diagnosed by enteroclysis, only 12 from 20 by US and 9 from 10 by US-enteroclysis. The differences were either statistically non-significant or there were too small numbers to give sensible statistical results, but low sensitivity of ultrasound in complications ( p = 0.05). Conclusion US-enteroclysis seems to became the standard examination of patients with Crohn's disease mainly in those with unclear conventional ultrasound. The most important fact is that this examination significantly decreases the radiation load when maintaining high sensitivity. This is very important namely in patients with Crohn's disease that require life-long observation and repeated examinations. This examination is much more easy to standardize than the conventional US.

The Relationship Between Experience With Standardized Patient Examinations and Subsequent Standardized Patient Examination Performance

Standardized patient (SP) exams are becoming an important part of testing clinical skills. The dynamics of this test format warrant exploration. This study is designed to measure the relationship between experience with SPs and performance on a SP examination. A general linear model is used to determine if a relationship exists and the pattern of that relationship as experience increases. There is a linear association between increased experience and better performance in technical domains. In interpersonal skill domains, there is a curvilinear association between experience and performance with improved performance with moderate levels of experience but not in the most experienced group. Different levels of experience with SPs are associated with differences in performance. The varied pattern of associations suggests a possible practice effect or test-taking behavior. Additional studies are needed to assess this potential effect on SP exam validity.

Sexual history inquiry and HIV counseling: Improving clinical skills and medical knowledge through an interactive workshop utilizing standardized patients

Sexual history and HIV counseling (SHHIVC) are essential clinical skills. Our project's purpose was to evaluate a standardized patient educational intervention teaching third-year medical students SHHIVC. A four-hour standardized patient workshop was delivered to one-half of the class each of three consecutive years at one medical school. Approximately 3.5 weeks later, all students engaged in a standardized patient examination including one station assessing SHHIVC, answered an open-ended written exercise following the standardized patient encounter, and completed a written examination including sexual history and HIV-related questions. Workshop participants scored higher than non-participants on SHHIVC items on the standardized patient station (P < .0001), written exam (P < .0001), and open-ended written exercise after the standardized patient encounter (P = .024). Our SHHIVC curriculum was associated with students demonstrating better clinical skills on a SHHIVC standardized patient examination station and more SHHIVC knowledge on two measures of medical knowledge than students not participating in the SHHIVC educational intervention.

Acting on Reflection: the Effect of Reflection on Students’ Clinical Performance on a Standardized Patient Examination

BackgroundLittle evidence exists to support the value of reflection in the clinical setting.ObjectiveTo determine whether reflecting and revisiting the “patient” during a standardized patient (SP) examination improves junior medical students’ performance and to analyze students’ perceptions of its value.DesignStudents completed a six-encounter clinical skills examination, writing a guided assessment after each encounter to trigger reflection. SPs evaluated the students with Medical Skills and Patient Satisfaction checklists. During the last three encounters, students could opt to revisit the SP and be reevaluated with identical checklists.ParticipantsOne hundred and forty-nine third year medical students.MeasurementsChanges in scores in the Medical Skills and Patient Satisfaction checklists between first visit and revisit were tested separately per case as well as across cases.ResultsOn the medical skills and patient satisfaction checklists, mean revisit scores across cases were significantly higher than mean first visit scores [12.6 vs 12.2 (pooled SD = 2.4), P = .0001; 31.2 vs 31.0 (pooled SD = 3.5), P = .0001)]. Sixty-five percent of the time, students rated “reflect–revisit” positively, 34% neutrally, and 0.4% negatively. Five themes were identified in the positive comments: enhancement of (1) medical decision making, (2) patient education/counseling, (3) student satisfaction/confidence, (4) patient satisfaction/confidence, and (5) clinical realism.ConclusionsOffering third year medical students the option to reflect and revisit an SP during a clinical skills examination produced a small but nontrivial increase in clinical performance. Students perceived the reflect–revisit experience as enhancing patient-centered practices (counseling, education) as well as their own medical decision making and clinical confidence.

The relationship between experience with standardized patient examinations and subsequent standardized patient examination performance: a potential problem with standardized patient exam validity.

The relationship between experience with standardized patient examinations and subsequent standardized patient examination performance: a potential problem with standardized patient exam validity.

Assessing Core Clinical Competencies Required of Medical Graduates in Taiwan

Medical students are assumed to be competent to provide basic patient care independently on graduation. However, there is a gap between what students are expected to learn and what they have actually learned. This may be due to the lack of clearly defined learning objectives, well-organized curriculum, and properly administered assessment. In an attempt to tackle this problem, we conducted a three-step study. Firstly, we identified the core clinical competencies required of medical graduates in Taiwan. Secondly, we incorporated these clinical competencies into a new medical curriculum. Finally, we identified the most appropriate assessment methods for each clinical competency. In 2004, a set of minimally required clinical competencies for medical undergraduates in Taiwan was developed, which included 92 clinical skills, four communication skills, and seven kinds of attitudes. In order to prepare 3rd and 4th year medical students at Kaohsiung Medical University (KMU) for later clinical work, the medical curriculum committee integrated the teaching and assessment of the core clinical skills identified previously into relevant organ-system blocks of the new curriculum. To identify appropriate assessment methods for each clinical skill, a structured questionnaire of assessment methods based on the Toolbox of Assessment Methods (Accreditation Council for Graduate Medical Education) and The Scottish Doctor (Scottish Deans' Medical Curriculum Group) was developed and distributed to 40 senior clinical faculty members at KMU. Simulations and Models, Standardized Patient Examination (SP), and Objective Structured Clinical Examination (OSCE) were suggested to be most suitable to assess two-thirds of the core clinical skills. These assessment methods are commonly used in American and European medical schools. We believe that the implementation of the new curriculum at KMU accompanied by the use of Simulations and Models, SP, OSCE, and other teaching and assessment methods will help 3rd and 4th year students to prepare better for clinical practice in clerkships.

Effects of Case Characteristics on Encounter Time in a High-Stakes Standardized Patient Examination

Data from national surveys indicate that patient characteristics could influence the time spent by physicians interviewing and assessing patients. The purposes of this investigation were to gather information regarding the relationship between encounter time and case characteristics for simulated clinical encounters and to provide evidence that the time provided to gather data was adequate. Timing data was extracted from United States Medical Licensing Examination Step 2 Clinical Skills. To test the relative effects of case characteristics on encounter time, an analysis of variance was conducted with encounter time as the dependent variable and case characteristics as the independent variables. Mean encounter times were computed based on the case characteristics. Station format (history only, history and physical examination, telephone cases) predicted the most variance in encounter time (16%). The extent to which examination content is balanced from administration to administration ensures a mix of cases that provides adequate time limits for examinees.

Use of Standardized Patients During a Psychiatry Clerkship

Standardized patients are used in teaching medical students and evaluating their clinical skills during the psychiatric clerkship. The purpose of this study was to evaluate whether the addition of a Psychiatry Clinical Standardized Patient Examination (PCX) during the third-year clerkship improved students' performances on the psychiatry component of the Clinical Competency Examination (CCX) that is used at our institution to prepare fourth-year students for the National Board of Medical Examiners Step 2 Clinical Skills Exam (Step 2 CSA). Scores on the CCX were analyzed by two-by-two tables (yes/no response to each clinical competency item versus took/did not take PCX) using Fisher's exact test. Of the 469 students who took the CCX between 2002 and 2004, 246 students also took the PCX. Compared to students who did not take the exam, students who took the PCX were significantly more likely (p<0.05) to receive positive ratings on the CCX items for professionalism, addressing patients' stressors at work and home and inquiring about thoughts of harm to self or others. The implementation of the PCX during the psychiatry clerkship significantly improved student performance on several important components of the CCX, which is used to prepare students for the Step 2 CSA. This suggests that the use of standardized patients during the clerkship may help students improve their interviewing and interpersonal skills.

"Oh! She doesn't speak English!" Assessing resident competence in managing linguistic and cultural barriers.

Residents must master complex skills to care for culturally and linguistically diverse patients. As part of an annual 10-station, standardized patient (SP) examination, medical residents interacted with a 50-year-old reserved, Bengali-speaking woman (SP) with a positive fecal occult blood accompanied by her bilingual brother (standardized interpreter (SI)). While the resident addressed the need for a colonoscopy, the SI did not translate word for word unless directed to, questioned medical terms, and was reluctant to tell the SP frightening information. The SP/SI, faculty observers, and the resident assessed the performance. Seventy-six residents participated. Mean faculty ratings (9-point scale) were as follows: overall 6.0, communication 6.0, knowledge 6.3. Mean SP/SI ratings (3.1, range 1.9 to 3.9) correlated with faculty ratings (overall r=.719, communication r=.639, knowledge r=.457, all P<.01). Internal reliability as measured by Cronbach's alpha coefficients for the 20 item instrument was 0.91. Poor performance on this station was associated with poor performance on other stations. Eighty-nine percent of residents stated that the educational value was moderate to high. We reliably assessed residents communication skills conducting a common clinical task across a significant language barrier. This medical education innovation provides the first steps to measuring interpreter facilitated skills in residency training.