High-stakes Clinical Examination Research Articles

Empirical analysis comparing the tele-objective structured clinical examination (teleOSCE) and the in-person assessment in Australia.

PurposeIt aimed to compare the use of the tele-objective structured clinical examination (teleOSCE) with in-person assessment in high-stakes clinical examination so as to determine the impact of the teleOSCE on the assessment undertaken. Discussion follows regarding what skills and domains can effectively be assessed in a teleOSCE.MethodsThis study is a retrospective observational analysis. It compares the results achieved by final year medical students in their clinical examination, assessed using the teleOSCE in 2020 (n=285), with those who were examined using the traditional in-person format in 2019 (n=280). The study was undertaken at the University of New South Wales, Australia.ResultsIn the domain of physical examination, students in 2020 scored 0.277 points higher than those in 2019 (mean difference=–0.277, P<0.001, effect size=0.332). Across all other domains, there was no significant difference in mean scores between 2019 and 2020.ConclusionThe teleOSCE does not negatively impact assessment in clinical examination in all domains except physical examination. If the teleOSCE is the future of clinical skills examination, assessment of physical examination will require concomitant workplace-based assessment.

Standardizing nurse practitioner student evaluation during high-stakes clinical examinations

Nurse practitioners (NPs) require clinical competence in advanced health assessment skills to differentiate between normal, variations of normal, and abnormal patient findings. High-stakes clinical examinations, using live hands-on simulation scenarios and standardized patients (SPs) or other human role players, are accepted teaching and evaluation tools for NP students. Providing objective, valid, and reliable feedback to students during high-stakes clinical examinations is of considerable value for ongoing skill development. The study examined opportunities to improve the quality of student evaluation in simulation testing modes. A purposive sample of 17 video recordings of health students' comprehensive examination of an SP or physical examination teaching associate (PETA) from a nursing graduate level health assessment course was evaluated. Using a standardized rubric, students were scored live and after a comprehensive examination of a SP/PETA and via a secure web-based video platform by the faculty and an independent reviewer. Evaluator group examination score comparisons revealed that distributions of examination scores within evaluator groups were not similar. Median examination scores were significantly different between groups; faculty median examination scores significantly higher than SPs/PETAs. Efficiency of student evaluation may be increased by improving reviewer training, reducing checklist length, and adopting electronic scoring. Development of an exemplary teaching video providing explanation and detail for expected student skill performance will allow reviewers to practice and improve competence in reliable scoring, reduce time and effort of scorers, and increase accuracy of scoring.

COVID-19 and programmatic assessment.

As a result of the COVID-19 pandemic, medical schools across the world have been faced with the unprecedented situation of closing their doors and cancelling clinical teaching and examinations. The traditional model of assessment relying on high-stakes clinical examination(s) is no longer possible, nor fully fit for purpose. How can medical schools still demonstrate that their students have reached the threshold for safe practice and are able to graduate? A potential solution for this issue is programmatic assessment. Programmatic assessment involves the longitudinal collection of information about a student based on multiple low-stakes assessments.1 The purpose of each individual assessment is to provide feedback, not to pass or fail a learner. By assembling feedback from many assessments, the learner monitors their progress and discusses this with a mentor. High-stakes decisions are based on all of the data and are performed by an independent competence committee. Programmatic assessment moves away from an end-of-course ‘big bang’ examination to a continuous approach of assessment that drives learning in a meaningful and self-directed way. Programmatic assessment moves away from an end-of-course ‘big bang’ examination to a continuous approach of assessment that drives learning … During the COVID-19 pandemic, medical schools have used a variety of workplace-based assessments of clinical and professional skills, as well as applied knowledge tests to collect information on learning and performance. We propose that programmatic assessment could be used in a hybrid model to complement more conventional methods of assessment to make high-stakes decisions. … programmatic assessment could be used in a hybrid model to complement more conventional methods of assessment … Programmatic assessment has been gaining traction for many years, but its implementation is often hindered by challenges around nationwide ranking, university regulations, reliability and a resistance to change from the traditional model of high-stakes, summative examinations. With the cancellation of clinical examinations and restricted student access to clinical sites, medical schools have been forced to consider alternative ways to evidence their students’ progression. There is a strong evidence base for the benefits of programmatic assessment, including studies demonstrating the ability to facilitate learning and maximise the robustness of high-stakes decisions,2 as well as identify students at risk of poor academic progress, and thereby optimise timely interventions.3 The sole use of high-stakes, summative examinations has been described as intrinsically flawed because the ‘ideal’ assessment, capable of assessing all of the necessary competencies, does not exist, and placing an emphasis on single high-stakes examinations may promote poor learning styles.2 Although concerns have been raised regarding the reliability of ‘subjective’ single assessment points, it has been shown that acceptable reliability is achievable through large numbers of assessment points, varying methods of assessment (including both standardised and non-standardised assessment) and multiple assessors.4 Although this is a challenging time for medical education, the COVID-19 crisis may in fact present an opportunity for reflection and adaptation. Here are a few considerations regarding programmatic assessment during the COVID-19 pandemic. Having assessment data over time may help to ease some of our angst about learners’ assessment decisions during such unprecedented times. With the dissemination of this message, medical schools may overcome their apprehension regarding programmatic assessment and recognise its many benefits. In the face of adversity, we have stumbled upon a unique opportunity to enrich students’ learning, and in the words of Winston Churchill, we should ‘never let a good crisis go to waste’.

Borderline grades in high stakes clinical examinations: resolving examiner uncertainty

BackgroundObjective Structured Clinical Exams are used to increase reliability and validity, yet they only achieve a modest level of reliability. This low reliability is due in part to examiner variance which is greater than the variance of students. This variance often represents indecisiveness at the cut score with apparent confusion over terms such as “borderline pass”. It is amplified by a well reported failure to fail.MethodsA borderline grade (meaning performance is neither a clear pass nor a clear fail) was introduced in a high stakes undergraduate medical clinical skills exam to replace a borderline pass grade (which was historically resolved as 50%) in a 4 point scale (distinction, pass, borderline, fail). Each Borderline grade was then resolved into a Pass or Fail grade by a formula referencing the difficulty of the station and the performance in the same domain by the student in other stations. Raw pass or fail grades were unaltered. Mean scores and 95%CI were calculated per station and per domain for the unmodified and the modified scores/grades (results are presented on error bars). To estimate the defensibility of these modifications, similar analysis took place for the P and the F grades which resulted from the modification of the B grades.ResultsOf 14,634 observations 4.69% were Borderline. Application of the formula did not impact the mean scores in each domain but the failure rate for the exam increased from 0.7 to 4.1%. Examiners and students expressed satisfaction with the Borderline grade, resolution formula and outcomes. Mean scores (by stations and by domains respectively) of students whose B grades were modified to P were significantly higher than their counterparts whose B grades were modified to F.ConclusionsThis study provides a feasible and defensible resolution to situations where the examinee’s performance is neither a clear pass nor a clear fail, demonstrating the application of the resolution of borderline formula in a high stakes exam. It does not create a new performance standard but utilises real data to make judgements about these small number of candidates. This is perceived as a fair approach to Pass/Fail decisions.

Passing MRCP (UK) PACES: a cross-sectional study examining the performance of doctors by sex and country

BackgroundThere is much discussion about the sex differences that exist in medical education. Research from the United Kingdom (UK) and United States has found female doctors earn less, and are less likely to be senior authors on academic papers, but female doctors are also less likely to be sanctioned, and have been found to perform better academically and clinically. It is also known that international medical graduates tend to perform more poorly academically compared to home-trained graduates in the UK, US, and Canada. It is uncertain whether the magnitude and direction of sex differences in doctors’ performance is variable by country. We explored the association between doctors’ sex and their performance at a large international high-stakes clinical examination: the Membership of the Royal Colleges of Physicians (UK) Practical Assessment of Clinical Examination Skills (PACES). We examined how sex differences varied by the country in which the doctor received their primary medical qualification, the country in which they took the PACES examination, and by the country in which they are registered to practise.MethodsSeven thousand six hundred seventy-one doctors attempted PACES between October 2010 and May 2013. We analysed sex differences in first time pass rates, controlling for ethnicity, in three groups: (i) UK medical graduates (N = 3574); (ii) non-UK medical graduates registered with the UK medical regulator, the General Medical Council (GMC), and thus likely to be working in the UK (N = 1067); and (iii) non-UK medical graduates without GMC registration and so legally unable to work or train in the UK (N = 2179).ResultsFemale doctors were statistically significantly more likely to pass at their first attempt in all three groups, with the greatest sex effect seen in non-UK medical graduates without GMC registration (OR = 1.99; 95% CI = 1.65-2.39; P < 0.0001) and the smallest in the UK graduates (OR = 1.18; 95% CI = 1.03-1.35; P = 0.02).ConclusionsAs found in a previous format of this examination and in other clinical examinations, female doctors outperformed male doctors. Further work is required to explore why sex differences were greater in non-UK graduates, especially those without GMC registration, and to consider how examination performance may relate to performance in practice.

Power of the policy: how the announcement of high-stakes clinical examination altered OSCE implementation at institutional level

BackgroundThe Objective Structured Clinical Examination (OSCE) has been widely applied as a high-stakes examination for assessing physicians’ clinical competency. In 1992, OSCE was first introduced in Taiwan, and the authorities announced that passing the OSCE would be a prerequisite for step-2 medical licensure examination in 2013. This study aimed to investigate the impacts of the announced national OSCE policy on implementation of OSCE at the institutional level. Further, the readiness and the recognition of barriers toward a high-stakes examination were explored.MethodsIn 2007 and 2010, the year before and after the announcement of high-stakes OSCE policy in 2008, respectively, questionnaires on the status of OSCE implementation were distributed to all hospitals with active OSCE programs in Taiwan. Information on OSCE facilities, equipment, station length, number of administrations per year, and the recognition of barriers to the success of implementing an OSCE were collected. The missing data were completed by telephone interviews. The OSCE format, administration, and facilities before and after the announcement of the nationwide OSCE policy were compared.ResultsThe data were collected from 17 hospitals in 2007 and 21 in 2010. Comparing the OSCE formats between 2007 and 2010, the number of stations increased and the station length decreased. The designated space and the equipment for OSCE were also found to have been improved. As for the awareness of OSCE implementation barriers, the hospital representatives concerned mostly about the availability and quality of standardized patients in 2007, as well as space and facilities in 2010.ConclusionsThe results of this study underscored an overall increase in the number of OSCE hospitals and changes in facilities and formats. While recruitment and training of standardized patients were the major concerns before the official disclosure of the policy, space and facilities became the focus of attention after the announcement. The study results highlighted the influence of government policy on different aspects of OSCE implementation in Taiwanese training institutes that showed high level of support as reflected in the improved hardware and the change in OSCE format to serve the summative purpose.

Performance pressure: Simulated patients and high‐stakes examinations in a regional clinical school

To investigate the effects and challenges of being a simulated patient (SP) in a high-stakes clinical examination context in a regional setting. Mixed methods, using a written survey, focus groups, and a retrospective postal survey. A university clinical school in a Victorian regional city. Nineteen SP volunteers (from an existing database of 55 people) who had been involved in mid-year, summative Objective Structured Clinical Examination (OSCE) role-play performances. Challenges of the OSCE role-play experience and the reported effects on SPs. The implications of these factors have an impact on the sustainability of SP programs in regional settings. Physical and emotional effects like exhaustion were reported, as well as empathy and concern for the medical students. The retrospective postal survey indicated that the SPs had no long-term negative effects from their high-stakes examination experiences. Participants also reported that a level of decision making and improvisation was needed in the performance of their OSCE role plays. Our study reveals the complexity and demands on SPs in performing in high-stakes clinical examinations. The results highlight that SP roles involve more than the transfer of scripted information. SPs should be considered as members of the examination team when preparing and implementing high-stakes examinations to assist in maintaining standardised performance during and across OSCE role plays. Relationships between SPs and educational institutes need to be nurtured to ensure that the ability to continue high-stakes OSCEs in a regional setting is maintained.