Standardized Patient Examination Research Articles

The effects of patient characteristics and practice settings on students' participation in a primary care clerkship.

To explore the effects of practice, patient, and encounter variables on students' participation in a third-year primary care clerkship. In 1995-96, 154 students, randomly assigned to ambulatory teaching sites, completed for each patient encounter a scannable card indicating the patient's age range, gender, and insurance type, the setting of care, the type of visit, whether it was a repeat or first visit to the student, and the student's level of participation. Conventional measures of students' satisfaction (post-clerkship survey) and performance (preceptor ratings, USMLE Step 2, standardized-patient examination scores) were examined. The students reported significantly higher levels of participation for patient encounters taking place in clinic and emergency room settings than for those in private offices; for repeat rather than for first visits; for patients over 12 years old than for those 12 years old or younger; and for sick visits rather than for follow-ups or checkups. Students' participation had a modestly positive correlation with students' satisfaction and performance. Several practice and patient variables influence the level of students' participation in the care of ambulatory patients. The strongest predictor of active student participation is the clinical setting of the encounter. Monitoring students' self-reported levels of participation is an important tool for tracking the impact of practice variability on the quality of the learning environment in ambulatory clerkships.

Global Ratings of Student Performance in a Standardized Patient Examination: Is the Whole More than the Sum of the Parts?

Standardized patient examinations (SPE) are widely used in medical education to assess skills that cannot be measured with written examinations. Trained actors termed standardized patients (SPs) are used to simulate patients with specific medical problems. SPs typically use behaviorally specific checklists and rating scales to evaluate examinees. This study explored the use of faculty and SP global ratings of students' clinical and interpersonal skills in an SPE. The reliability of global ratings was found to on par with more specific behaviorally anchored ratings. Global ratings were also found to be predictive of written tests of clinical knowledge and ratings of actual clinical performance after controlling for behaviorally anchored ratings. Faculty global ratings were more reliable and more predictive of other performance than SP global ratings. These results suggest global ratings by faculty observers and possibly SPs can provide unique and useful information in these performance-based examinations.

A new vocabulary and other innovations for improving descriptive in-training evaluations

Progress in improving the credibility of teachers' descriptive evaluations of students and residents has not kept pace with the progress made in improving the credibility of more quantified methods, such as multiple-choice examinations and standardized patient examinations of clinical skills. This article addresses innovative approaches to making the ongoing in-training evaluation (ITEv) of trainees during their clinical experiences more reliable and valid. The innovations include the development of a standard vocabulary for describing the progress of trainees from "reporter" to "interpreter" to "manager" and "educator" (RIME), the use of formal evaluation sessions, and closer consideration of the unit of clinical evaluation (the case, the rotation, or the year). The author also discusses initial results of studies assessing the reliability and validity of descriptive methods, as well as the use of quantified methods to complement descriptive methods. Applying basic principles--the use of a taxonomy of professional development and statistical principles of reliability and validity--may foster research into more credible descriptive evaluation of clinical skills.

The Patient Findings Questionnaire: one solution to an important standardized patient examination problem.

To describe the Patient Findings Questionnaire (PFQ) and compare its scores and pass/fail decisions with those obtained from standardized patient (SP) examination checklists. Checklists and PFQs were used to assess data acquisition by 790 second-year medical students. PFQs were composed of multiple-choice items designed to determine whether examines had acquired key historical patient information. At the item level, the two measurement methods yielded the same decisions about data acquisition on 88% of observed occasions. Most discrepancies (74%) involved SPs rating examinees as having elicited information when the examinee was unable to answer the associated PFQ item. At the test level, the two instruments yielded the same pass/fail decision on a large majority of occasions. The PFQ and checklist yielded similar data acquisition scores and decisions at the item and test levels. Replacement of the checklist with the PFQ should result in examinees' behaving in a way more consistent with recommended interviewing practices.

Using tagged items to detect threats to security in a nationally administered standardized patient examination.

Using tagged items to detect threats to security in a nationally administered standardized patient examination.

Assessing the impacts of intra-site and inter-site checklist recording discrepancies on the reliability of scores obtained in a nationally administered standardized patient examination.

Assessing the impacts of intra-site and inter-site checklist recording discrepancies on the reliability of scores obtained in a nationally administered standardized patient examination.

Modeling the effects of security breaches on students' performances on a large-scale standardized patient examination.

Modeling the effects of security breaches on students' performances on a large-scale standardized patient examination.

White-noise field campimetry in HIV+ patients

The first morphological changes in eyes with HIV infection are microvascular disease of the retina with cotton-wool spots and microaneurysms. The study was performed to find out if evidence of disturbances of ocular microcirculation can be established by non-invasive methods. Twenty-seven patients with HIV infection and without opportunistic infections underwent thorough ophthalmologic examination with threshold-oriented, suprathreshold perimetry (TAP 2000 ct, Oculus) and white-noise field campimetry (TEC, Oculus). Visual field examination was normal in 23 out of 27 patients (85%), whereas 4 patients showed relative field defects in at least one eye. In white-noise field campimetry 13 out of 23 perimetrically unaffected patients (56%) perceived scotomas in one or both eyes. These scotomas were not stable. Three of 4 patients with relative scotomas in the visual field had cotton-wool spots in the retina and showed a stable scotoma in campimetry. Visual acuity, IOP, and cup/disc ratio were within normal ranges. White-noise field campimetry complements the standard examination of patients with HIV and might be capable of indicating disturbances of ocular microcirculation by a non-invasive method before morphological changes in the retina can be seen.

The performance of foreign medical graduates on the National Board of Medical Examiners (NBME) standardized patient examination prototype: a collaborative study of the NBME and the Educational Commission for Foreign Medical Graduates (ECFMG).

The purpose of the study was to explore foreign medical graduates' (FMGs) performance on a clinical skills (SPX) examination. The National Board of Medical Examiners (NBME) is in the process of developing an SPX for potential use in the United States Medical Licensing Examination (USMLE). The Educational Commission for Foreign Medical Graduates (ECFMG) is developing the Clinical Skills Assessment (CSA) as an additional requirement for FMGs who wish to be certified by ECFMG. Thirty-three FMGs and 151 United States medical students (USMSs) took the SPX during the winter of 1996 as part of the ongoing pilot studies conducted by the NBME. Four clinical skill areas were assessed: history-taking, physical examination, communication and interpersonal skills. The examination used in this research consisted of 12 cases. The examination utilizes standardized patients (SPs) who are trained to document examinee behaviours and evaluate the communication component of the test. The SPs were also trained to evaluate the English proficiency of the candidates. Candidates were also administered the Test of Spoken English developed by the Educational Testing Services (ETS). The examination was conducted in one medical school which served as an SPX centre for NBME pilot studies. Thirty-three foreign medical students and 151 US medical students. The indications were that the majority of candidates in both groups felt the examination was moderately fair but 78% of FMGs felt moderately pressed for time, vs. 80% of the USMSs who did not feel pressed for time. Reliabilities obtained for the various SPX components were somewhat higher for the FMGs reflecting the heterogeneity of this group. The NBME-ECFMG collaborative study yielded important information regarding the NBME SPX prototype as a performance measure for FMGs.

Learning primary care in medical school: does specialty or geographic location of the teaching site make a difference?

PURPOSE: The Liaison Committee on Medical Education mandates a core curriculum in primary care but does not specify its content or structure. In this study, we explored the question of whether primary care specialty or geographic location affects student learning and satisfaction. METHODS: From 1994 to 1996, 294 third-year medical students at one medical school in New York state were randomly assigned to multiple teaching sites for a required 5-week primary care clerkship. Independent predictor variables were primary care specialty of the preceptor (family medicine, medicine, pediatrics, or joint medicine and pediatrics) and geographic location of the site (urban, suburban, rural). Outcome measures included four areas of student satisfaction, one of patient volume, and two of student performance. RESULTS: Primary care specialty had no detectable association with the outcome measures, except for a lower rating of patient diversity in pediatric experiences ( P <0.001). Geographic location of the site had a significant association with all measures of student satisfaction and patient volume (all P values <0.001). Students at rural sites rated the experience more highly and saw on average 15 more patients per rotation. Ratings of student satisfaction remained high after adjusting for patient volume. Primary care specialty and geographic location did not influence student performance in the clerkship or scores on standardized patient examination. CONCLUSIONS: Rural geographic location of teaching site, but not primary care specialty, was associated with higher student satisfaction. However, higher student satisfaction ratings did not correspond to better student performance. Provided that all sites meet the screening criteria for inclusion in a teaching program, these findings support the continued development of high-quality, heterogeneous, interdisciplinary, primary care experiences.

Relationship between clinical competence and interpersonal and communication skills in standardized-patient assessment.

To evaluate the relationship between clinical competence and interpersonal and communication skills, in an attempt to clarify current thinking about these two dimensions as measured with standardized-patient (SP) examinations. Simple Pearson correlations were computed between total examination scores for clinical competence and interpersonal and communication skills. Three sets of different types of data involving 15 separate examinations were used to explore the generality of the findings. To control for a within-case halo effect and measurement error, corrected cross-half correlations and corrected cross-case correlations were also computed. The simple correlations and the corrected cross-half and cross-case correlations showed moderate and above relationships between these two dimensions in the clinical context. The simple correlations centered around .50, and the corrected cross-half and cross-case correlations were slightly higher, centering around .65 and .70, respectively. The authors' thinking is that the moderate relationship between clinical competence and interpersonal and communication skills is not due to a flaw in the measurement of clinical competence, as has been suggested, but rather is a natural consequence of the clinical encounter, which exacts an interdependence of these two dimensions. At least, this possibility must be seriously considered so medical educators can think and act appropriately in the assessment of clinical performance.

Development and Use of Military-Unique Standardized Gynecology Patients for Military Residents in Obstetrics and Gynecology Training Programs

Focused military curricula and readiness training are often inadequate for military resident physicians. We developed a standardized gynecologic military-unique patient scenario and examination to assess obstetrics and gynecology residents' clinical and operational problem-solving abilities. Integration of military-unique gynecologic standardized patients, clinical scenarios, and objective structured clinical examinations into obstetrics and gynecology curricula is a novel approach for realistic medical readiness training for resident physicians. This tool can become a cornerstone in the ongoing development of needed military-unique curricula.

The effect of using multiple standardized patients on the inter-case reliability of a large-scale standardized-patient examination administered over an extended testing period.

The effect of using multiple standardized patients on the inter-case reliability of a large-scale standardized-patient examination administered over an extended testing period.

Does training location influence the clinical skills of medical students?

To examine the benefits of the shift of medical education into ambulatory primary care settings by investigating whether medical students exposed to a common primary care problem (low back pain) in ambulatory care settings develop better clinical skills. In 1995, the authors categorized 420 students from all four North-Carolina medical schools into groups that had previously encountered patients with low back pain in ambulatory primary care settings, tertiary care settings, both, or neither. The clinical skills of the groups were determined using data collected during standardized-patient examination in which students took the history of, physically examined, and chose a diagnostic strategy for patients with acute, uncomplicated low back pain. In general, there was no difference between the performances of the student groups associated with the settings of their previous encounters with low back pain. On average, the students failed to ask 35% of the history items and failed to perform 35% of the physical examination items. Many students chose inappropriate diagnostic strategies. The lack of difference between the groups' clinical performances indicates a need to more rigorously define and evaluate outcomes of education in ambulatory care settings. The generally poor clinical performance of all groups suggests that the current curriculum inadequately teaches clinical skills needed to assess and manage common problems. Clearer expectations of competencies and assurances that preceptors in ambulatory care settings will help students meet those learning objectives might lead to better outcomes.

An Investigation of the Sources of Measurement Error in the Post-Encounter Written Scores from Standardized Patient Examinations.

Purpose. Post-encounter written exercises (e.g., patient notes) have been included in clinical skills assessments that use standardized patients. The purpose of this study was to estimate the generalizability of the scores from these written exercises when they are rated by various trained health professionals, including physicians.Method. The patient notes from a 10 station clinical skills examination involving 10 first year emergency medicine residents were analytically scored by four rater groups: three physicians, three nurses, three fourth year medical students, three billing clerks. Generalizability analyses were used to partition the various sources of error variance and derive reliability-like coefficients for each group of raters.Results. The generalizability analyses indicated that case-to-case variability was a major source of error variance in the patient note scores. The variance attributable to the rater or to the rater by examinee interaction was negligible. This finding was consistent across the four rater groups. Generalizability coefficients in excess of 0.80 were achieved for each of the four sets of raters. Physicians did, however, produce the most dependable scores.Conclusion. There is little advantage, from a reliability perspective, in using more than one trained physician, or other health professional who is adequately trained to score the patient note. Measurement error is introduced primarily by case sampling variability. This suggests that, if required, increases in the generalizability of the patient note scores can be made through the addition of cases, and not the addition of raters.

Assessment of Empathy in a Standardized-Patient Examination

Background: Empathy has long been thought to be an important characteristic of a good physician, and a measure of empathy is needed to provide feedback to medical students, residents, and physicians on this important aspect of their clinical performance. The standardized-patient-testing format provides a simple but intuitively convincing approach to the assessment of empathy. Purpose: To determine the extent to which 4th-year medical students were checked "empathic" by standardized patients (SPs) on a performance-based examination, to evaluate the psychometric properties of this simple empathy measure, and to see whether empathy was related to clinical performance on history taking and physical examination. Method: Analyses were performed on examination data for 1,048 senior medical students in the 8 member schools of the New York City Consortium tested at The Morch and Center of Mount Sinai School ofMedicine. Results: The percentage of students who were checked "empathic" ranged from 59% to 98% (M = 79%) across the 7 SP cases. Of the 1,048 students, 268 (26%) were checked "empathic" on all 7 cases; however, 221 (21%)were checked on ≤4 cases, 90 (9%)on ≤3 cases, and 26 (3%)on ≤2 cases. The generalizability coefficient of the overall empathy scores was. 43; the dependability index with cutoff was. 81 for detecting students checked "empathic" on fewer than half of the 7 cases. The correlations of the empathy item with the other checklist items suggest that the empathy construct refers to behaviors that make the patient feel comfortable and important. Also, students who were checked "empathic" on fewer than half of the 7 cases performed lower on history taking and physical examination. Conclusions: Empathy appears to be reasonably acceptable in this sample of students, although it is of concern that, on average, more than 200 students per case were not seen as empathic, and more than 200 were checked "empathic" on ≤4 of the 7 cases. These results show the potential usefulness of this simple measure of empathy and illustrate the need for feedback to address any problems.

Setting standards and defining quality of performance in the validation of a standardized-patient examination format

To evaluate whether written standards increase the reproducibility of a physician-facilitated station in an objective structured clinical examination (OSCE) designed to assess history, physical-examination, and communication skills. The OSCE examination at the University of Texas Medical Branch-Galveston consists of ten eight-minute stations. Six of these stations consist of three History, Physical-examination, Problem-solving, and Plan (HPPP) station pairs. Each existing clinical-problem HPPP station was given to two content experts to develop standards for faculty rating scales appropriate for the evaluation of third-year medical students. Three pairs of faculty members were used to determine interrater reliability by scoring videotapes of three HPPP stations' presentation and problem-solving components. Faculty pairs scored tapes of 15 students without using standards and tapes of 15 students using the standards developed. Differences between the reliabilities without and with the standards were tested for significance using Fisher's R to Z transformation. The reproducibility and standard error of measurement (SEM) were extrapolated for increasing amounts of testing time. The HPPP component scores were also correlated with the written examination scores and preceptors' ratings. Data were obtained from the three HPPP stations used in the 1995-96 internal medicine clerkship SP examination. In all, 196 students completed the OSCE examination. The standards developed improved interrater reliability and reached statistical significance (p < .01) for one HPPP station. Reproducibility for the presentation and problem-solving components of the HPPP stations were > .80 after five hours of testing. The problem-solving component correlated at .37 and .19 with written examinations and with ward grades, respectively. The data from this study suggest that standards increase the reproducibility of presentation and problem-solving components of an OSCE to a level as high as, or higher than, that associated with the history, physical-examination, and communication components of traditional standardized-patient examinations.

Performance of "standardized examinees" in a standardized-patient examination of clinical skills.

As a first step in testing the utility of using trained "standardized examinees" (SEs) as a quality-assurance measure for the scoring process in a standardized-patient (SP) examination, to test whether medical residents could simulate students in an SP examination and perform consistently to specified levels under test conditions. Fourth-year students from the Baltimore-Washington Consortium for SPs participated in a National Board of Medical Examiners Prototype Examination of clinical skills consisting of twelve 15-minute student-patient encounters in 1994-95. For this examination, internal medicine residents were trained to act as ordinary candidates and to achieve target scores by performing to a set level on specific checklist items used by SPs for recording interviewing, physical-examination, and communication skills. The "strong SEs" were trained to score 80% correct on six of the examination's 12 cases (study cases), and the "weak SEs" were trained to score 40% correct on the same six cases. The strong and weak SEs' checklist scores on the study cases were compared through independent, two-tailed t-tests. When there was less than 85% agreement on specific checklist items in each case between the SE training and the SP recording, videotapes of the cases were reviewed; in such cases an SE's performance was the final score agreed upon after review. Seven SEs took the SP examination and were not detected by the SPs. There was a total of 84 discrepancies between predicted and recorded checklist scores across 659 checklist items in 40 encounters scored by the SPs. After correcting the discrepancies based on videotape review, the estimated actual mean score was 77.3% for the strong SEs and 44.0% for the weak SEs, and was higher for the strong SEs in each study case. The overall fidelity of the SEs to their training was estimated to be 97%, and the overall SP accuracy was estimated to be 91%. The videotape review revealed 47 training-scoring discrepancies, most in the area of communication skills. This study suggests that SEs can be trained to specific performance levels and may be an effective internal control for a high-stakes SP examination. They may also provide a mechanism for refining scoring checklists and for exploring the validity of SP examinations.

Assessing the factor structure of a nationally administered standardized patient examination.

Assessing the factor structure of a nationally administered standardized patient examination.

Assessing the factor structure of a nationally administered standardized patient examination

Assessing the factor structure of a nationally administered standardized patient examination