Objective Structured Clinical Examinations Data Research Articles

The Psychometric Quality of Objective Structured Clinical Examinations Within Psychology Programs: A Systematic Review

Background: Objective Structured Clinical Examinations (OSCEs) are a simulation-based assessment tool used extensively in medical education for evaluating clinical competence. OSCEs are widely regarded as more valid, reliable, and valuable compared to traditional assessment measures, and are now emerging within professional psychology training programs. While there is a lack of findings related to the quality of OSCEs in published psychology literature, psychometric properties can be inferred by investigating implementation. Accordingly, the current review assessed implementation of OSCEs within psychology programs against a set of Quality Assurance Guidelines (QAGs). Methods: A systematic review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) recommendations. Electronic databases including ProQuest Psychology, PsycArticles, Psychology and Behavioural Sciences Collection, PsycInfo and key indexing databases such as Scopus, ProQuest, and Web of Science were used to identify relevant articles. Twelve full-text articles met all inclusion criteria and were included in the review. Results: There was considerable heterogeneity in the quality of studies and reporting of OSCE data. Implementation of OSCEs against QAGs revealed overall adherence to be “Fair.” Conclusion: The current review consolidated what is known on psychometric quality of OSCEs within psychology programs. A further need for quantitative evidence on psychometric soundness of OSCEs within psychology training is highlighted. Furthermore, it is recommended that future training programs implement and report OSCEs in accordance with standardized guidelines.

Student knowledge, confidence, and perceptions prior to and following an inhaler and tobacco cessation simulation

Background and purposeTo assess knowledge, perceptions, and confidence of second-year pharmacy students regarding implementation of an innovative inhaler and tobacco cessation simulation. Educational activity and settingA simulation was created that required students to counsel a standardized patient concomitantly on an inhaler prescription and tobacco cessation. To assess the primary outcome of student perceptions and confidence, a survey was administered pre- and post-simulation. Survey results were compared using chi-square analysis. To assess the secondary outcome of knowledge-based improvement, students were assessed on tobacco cessation content utilizing six consistent examination questions; students also completed a tobacco cessation objective structured clinical examination (OSCE) case, with comparison to the previous student cohort. FindingsFifty-seven students (93%) completed the pre-survey, and 49 students (80%) completed the post-survey. Improvements in confidence concerning use of motivational interviewing and ability to establish a quit date via motivational interviewing were found. Examination score comparisons revealed improvements in two of the six questions but were not statistically significant. Tobacco cessation OSCE data indicated an improvement in overall student score (72%) compared to the control group (69%). A larger proportion of students achieved the tobacco cessation case cut score (97% vs. 87%). Lastly, intervention vs. control group averages improved in gathering patient information (57% vs. 45%) and developing management strategies (71% vs. 65%). SummaryThis integrated simulation was an effective learning tool that reinforced tobacco cessation concepts and increased confidence and knowledge. Simulations that require students to integrate knowledge and skills are valuable additions to pharmacy curricula.

Acceptability of the 8-case objective structured clinical examination of medical students in Korea using generalizability theory: a reliability study

This study investigated whether the reliability was acceptable when the number of cases in the objective structured clinical examination (OSCE) decreased from 12 to 8 using generalizability theory (GT). This psychometric study analyzed the OSCE data of 439 fourth-year medical students conducted in the Busan and Gyeongnam areas of South Korea from July 12 to 15, 2021. The generalizability study (G-study) considered 3 facets—students (p), cases (c), and items (i)—and designed the analysis as p×(i:c) due to items being nested in a case. The acceptable generalizability (G) coefficient was set to 0.70. The G-study and decision study (D-study) were performed using G String IV ver. 6.3.8 (Papawork, Hamilton, ON, Canada). All G coefficients except for July 14 (0.69) were above 0.70. The major sources of variance components (VCs) were items nested in cases (i:c), from 51.34% to 57.70%, and residual error (pi:c), from 39.55% to 43.26%. The proportion of VCs in cases was negligible, ranging from 0% to 2.03%. The case numbers decreased in the 2021 Busan and Gyeongnam OSCE. However, the reliability was acceptable. In the D-study, reliability was maintained at 0.70 or higher if there were more than 21 items/case in 8 cases and more than 18 items/case in 9 cases. However, according to the G-study, increasing the number of items nested in cases rather than the number of cases could further improve reliability. The consortium needs to maintain a case bank with various items to implement a reliable blueprinting combination for the OSCE.

Nailed it! Doctor of pharmacy learners' self-awareness of performance on authentic assessments

IntroductionThe primary objective of this pilot study was to determine the reliability and validity of an objective structured clinical examination (OSCE) self-awareness survey. A secondary objective was to assess learner self-awareness in OSCE performance. MethodsA retrospective review of OSCE data from a single cohort in a three-year accelerated doctor of pharmacy program occurred. The cohort completed four OSCEs with a total of 23 discrete cases. At the end of each case, before receiving feedback, learners completed an evaluation of their skills, communication, overall performance, and identified if they believed the patient/provider would return (“nailed or failed”). These self-evaluations were then compared with respective learner performance scores for each case. Content validity was assessed for the self-awareness survey via the Lawshe method. Reliability of the OSCE cases and self-awareness survey were conducted utilizing Cronbach's alpha and the Spearman-Brown formula, respectively. ResultsThe OSCE self-awareness survey was found to be face valid and reliable for the overall survey. The subsections of return to pharmacist and overall learner performance were also found to be valid and reliable. A statistical relationship was found between learner self-assessment of pass/fail and their actual grade for overall performance and return to pharmacist. ConclusionsResults from this study may guide remediation plans related to OSCE performance and development of other self-awareness activities in practice settings. Further studies are needed to evaluate the broader application and use of self-awareness tools.

Using the Many-Facet Rasch Model to analyse and evaluate the quality of objective structured clinical examination: a non-experimental cross-sectional design

ObjectivesSources of bias, such as the examiners, domains and stations, can influence the student marks in objective structured clinical examination (OSCE). This study describes the extent to which the facets...

The Reliability of 2-Station Clerkship Objective Structured Clinical Examinations in Isolation and in Aggregate.

Background:Most medical schools in the United States report having a 5- to 10-station objective structured clinical examination (OSCE) at the end of the core clerkship phase of the curriculum to assess clinical skills. We set out to investigate an alternative OSCE structure in which each clerkship has a 2-station OSCE. This study looked to determine the reliability of clerkship OSCEs in isolation to inform composite clerkship grading, as well as the reliability in aggregate, as a potential alternative to an end-of-third-year examination.Design:Clerkship OSCE data from the 2017-2018 academic year were analyzed: the generalizability coefficient (ρ2) and index of dependability (φ) were calculated for clerkships in isolation and in aggregate using variance components analysis.Results:In all, 93 students completed all examinations. The average generalizability coefficient for the individual clerkships was .47. Most often, the largest variance component was the interaction between the student and the station, indicating inconsistency in the performance of students between the 2 stations. Aggregate clerkship OSCE analysis demonstrated good reliability for consistency (ρ2 = .80). About one-third (33.8%) of the variance can be attributed to students, 8.2% can be attributed to the student by clerkship interaction, and 42.6% can be attributed to the student by block interaction, indicating that students’ relative performances varied by block.Conclusions:Two-station clerkship OSCEs have poor to fair reliability, and this should inform the weighting of the composite clerkship grade. Aggregating data results in good reliability. The largest source of variance in the aggregate was student by block, suggesting testing over several blocks may have advantages compared with a single day examination.

Peer-assisted teaching student tutors as examiners in an orthopedic surgery OSCE station - pros and cons.

Background: The OSCE (objective structured clinical examination) is composed of oral and practical examination in order to examine students’ abilities to imply clinical examination techniques and to interact with patients. The examiners for this procedure can be either lecturers or peers. The aim of this work is to evaluate the peer-assisted teaching student tutors as examiners in an orthopedic surgery OSCE station. Methods: We analyzed the OSCE data from 2013 to 2015. During this period over 300 medical students were examined each year. An evaluation was conducted at an orthopedic station and examined by peer students to assess the advantages and disadvantages of peer-assisted teaching student tutors as examiners. Results: We have noticed that student peers are more flexible regarding their schedule and they have been well trained for OSCE. Concerning the economic aspects, student peers are clearly of major economic advantage.Disadvantages were not reported in our study probably because peers were well trained and the checklists are monitored regularly. Conclusion: Student peers in OSCE are of major advantage due to their flexible time schedule and relatively low costs. They must be well trained and the checklists are to be monitored regularly. Our study shows that peer tutor examiners conducted the examination as competent as lecture examiners. However, legal restrictions on the employment of students should be considered.

Investigating disparity between global grades and checklist scores in OSCEs

Background: When measuring assessment quality, increasing focus is placed on the value of station-level metrics in the detection and remediation of problems in the assessment.Aims: This article investigates how disparity between checklist scores and global grades in an Objective Structured Clinical Examination (OSCE) can provide powerful new insights at the station level whenever such disparities occur and develops metrics to indicate when this is a problem.Method: This retrospective study uses OSCE data from multiple examinations to investigate the extent to which these new measurements of disparity complement existing station-level metrics.Results: In stations where existing metrics are poor, the new metrics provide greater understanding of the underlying sources of error. Equally importantly, stations of apparently satisfactory “quality” based on traditional metrics are shown to sometimes have problems of their own – with a tendency for checklist score “performance” to be judged stronger than would be expected from the global grades awarded.Conclusions: There is an ongoing tension in OSCE assessment between global holistic judgements and the necessarily more reductionist, but arguably more objective, checklist scores. This article develops methods to quantify the disparity between these judgements and illustrates how such analyses can inform ongoing improvement in station quality.

Advancing the objective structured clinical examination: sequential testing in theory and practice

Models of short-term remediation for failing students are typically associated with improvements in candidate performance at retest. However, the process is costly to deliver, particularly for performance retests with objective structured clinical examinations (OSCEs), and there is increasing evidence that these traditional models are associated with the longitudinal underperformance of candidates. Rather than a traditional OSCE model, sequential testing involves a shorter 'screening' format, with an additional 'sequential' test for candidates who fail to meet the screening standard. For those tested twice, overall pass/fail decisions are then based on results on the full sequence of tests. In this study, the impacts of sequential assessment on student performance, cost of assessment delivery and overall reliability were modelled using data sourced from a final graduating OSCE in an undergraduate medical degree programme. Initial modelling using pre-existing OSCE data predicted significant improvements in reliability in the critical area, reflected in pilot results: 13.5% of students (n=228) were required to sit the sequential OSCE. One student (0.4%) was identified as representing a false positive result (i.e. under the previous system this student would have passed the OSCE but failed on extended testing). Nine students (3.9%) who would have required OSCE retests under the prior system passed the full sequence and were therefore able to graduate at the normal time without loss of earnings. Overall reliability was estimated as 0.79 for the full test sequence. Significant cost savings were realised. Sequential testing in OSCEs increases reliability for borderline students because the increased number of observations implies that 'observed' student marks are closer to 'true' marks. However, the station-level quality of the assessment needs to be sufficiently high for the full benefits in terms of reliability to be achieved. The introduction of such a system has financial benefits, good validity inferences and has proved acceptable to students and other stakeholders.

An Online Management Information System for Objective Structured Clinical Examinations

Objective Structured Clinical Examinations (OSCE) are adopted for high stakes assessment in medical education. Students pass through a series of timed stations demonstrating specific skills. Examiners observe and rate students using predetermined criteria. In most OSCEs low level technology is used to capture, analyse and produce results. We describe an OSCE Management Information System (OMIS) to streamline the OSCE process and improve quality assurance. OMIS captured OSCE data in real time using a Web 2.0 platform. We compared the traditional paper trail outcome with detailed real time analyses of separate stations. Using a paper trail version only one student failed the OSCE. However, OMIS identified nineteen possibly ‘incompetent’ students. Although there are limitations to the design of the study, the results are promising and likely to lead to defendable judgements on student performance.

Rasch analysis on OSCE data : An illustrative example.

The Objective Structured Clinical Examination (OSCE) is a widely used tool for the assessment of clinical competence in health professional education. The goal of the OSCE is to make reproducible decisions on pass/fail status as well as students' levels of clinical competence according to their demonstrated abilities based on the scores. This paper explores the use of the polytomous Rasch model in evaluating the psychometric properties of OSCE scores through a case study. The authors analysed an OSCE data set (comprised of 11 stations) for 80 fourth year medical students based on the polytomous Rasch model in an effort to answer two research questions: (1) Do the clinical tasks assessed in the 11 OSCE stations map on to a common underlying construct, namely clinical competence? (2) What other insights can Rasch analysis offer in terms of scaling, item analysis and instrument validation over and above the conventional analysis based on classical test theory? The OSCE data set has demonstrated a sufficient degree of fit to the Rasch model (Χ(2) = 17.060, DF=22, p=0.76) indicating that the 11 OSCE station scores have sufficient psychometric properties to form a measure for a common underlying construct, i.e. clinical competence. Individual OSCE station scores with good fit to the Rasch model (p > 0.1 for all Χ(2) statistics) further corroborated the characteristic of unidimensionality of the OSCE scale for clinical competence. A Person Separation Index (PSI) of 0.704 indicates sufficient level of reliability for the OSCE scores. Other useful findings from the Rasch analysis that provide insights, over and above the analysis based on classical test theory, are also exemplified using the data set. The polytomous Rasch model provides a useful and supplementary approach to the calibration and analysis of OSCE examination data.

How to measure the quality of the OSCE: A review of metrics – AMEE guide no. 49

With an increasing use of criterion-based assessment techniques in both undergraduate and postgraduate healthcare programmes, there is a consequent need to ensure the quality and rigour of these assessments. The obvious question for those responsible for delivering assessment is how is this ‘quality’ measured, and what mechanisms might there be that allow improvements in assessment quality over time to be demonstrated? Whilst a small base of literature exists, few papers give more than one or two metrics as measures of quality in Objective Structured Clinical Examinations (OSCEs). In this guide, aimed at assessment practitioners, the authors aim to review the metrics that are available for measuring quality and indicate how a rounded picture of OSCE assessment quality may be constructed by using a variety of such measures, and also to consider which characteristics of the OSCE are appropriately judged by which measure(s). The authors will discuss the quality issues both at the individual station level and across the complete clinical assessment as a whole, using a series of ‘worked examples’ drawn from OSCE data sets from the authors’ institution.

Sharpening the Eye of the OSCE with Critical Action Analysis

When interpreting performance scores on an objective structured clinical examination (OSCE), are all checklist items created equal? Although assigning priority through checklist item weighting is often used to emphasize the clinical importance of selected checklist items, the authors propose the use of critical action analysis as an additional method for analyzing and discriminating clinical performance in clinical skill assessment exercises. A critical action is defined as an OSCE checklist item whose performance is critical to ensure an optimal patient outcome and avoid medical error. In this study, the authors analyzed a set of clerkship OSCE performance outcome data and compared the results of critical action analysis versus traditional checklist item performance scores. OSCE performance scores of 398 third-year clerkship students from 2003 to 2006 at the University of Virginia School of Medicine were analyzed using descriptive statistics and a logistic regression model. Through a consensus process, 10 of 25 OSCE cases were identified as containing critical actions. Students who scored above the median correctly performed the critical actions more often than those scoring lower. However, for 9 of 10 cases, 6% to 46% of higher-scoring students failed to perform the critical action correctly. Failure to address this skill assessment outcome is a missed opportunity to more fully understand and apply the results of such examinations to the clinical performance development of medical students. Including critical action analysis in OSCE data interpretation sharpens the eye of the OSCE and enhances its value in clinical skill assessment.

Communication skills knowledge, understanding and OSCE performance in medical trainees: a multivariate prospective study using structural equation modelling.

To test the stability of medical student communication skills over a period of 17 months as exhibited by performance in objective structured clinical examinations (OSCEs) and to determine the strength of prediction of these skills by initial levels of knowledge and understanding. This is a prospective study using a 2-wave cohort. Medical undergraduates (n = 383) from 2 years intake (1996 and 1997) were followed through the first 3 years of a medical curriculum. The study procedure involved the objective structured video examination (OSVE) conducted at formative and summative examinations during the first year. Two OSCE measures were employed: expert examiners and simulated patients completed the Liverpool Communication Skills Assessment Scale (LCSAS) and the Global Simulated Patient Rating Scale (GSPRS), respectively. The OSCE data were collected at Level 1 and 17 months later at Level 2 examinations. The measurement model followed prediction. A causal model using latent variables was fitted with Level 2 OSCE performance regressed on Level 1 OSCE and OSVE marks. Expert and simulated patient OSCE data were fitted separately and combined to determine strength of model fit according to professional and patient opinion of student skills. The overall fit of the models was acceptable. Communication skills performance showed a high level of stability. Some negative effect of cognitive factors on future skills performance was found. Early development of communication skills shows stable performance following an introductory course. Knowledge of communication skills has a small but significant influence on performance, depending on the time of testing. New assessments of cognitive factors are required to include both tacit and explicit knowledge.