Comparing the Borderline Regression Standard Setting Method to a Traditional Method in an Objective Structured Clinical Examination

Objective structured clinical examination (OSCE) is the main assessment tool for the end of semester (EOS) summative continuous assessment for the clinical training (Phase 2) at the Taylor’s Clinical School. OSCE is a clinical competency assessment, whereby a candidate who passes the OSCE is deemed as clinically competent. The traditional way of deciding the pass score as 50 % in any examination is arbitrary; this may lead to problem in OSCE where the score of 50 % (pass score) may not represent the actual competency required. In recent years, standard setting methods have been applied in OSCE by many medical schools so that a defensible, fair and absolute pass score is determined. The aim of this study is to describe the OSCE cut-off score by the borderline regression method (BRM) in Taylor’s Clinical School (TCS) compared to the conventional arbitrary pass mark of 50 %. This study focused on the following two research questions: (1) What is the difference in cut-off (pass mark) of OSCE if BRM standard setting is applied? (2) What is the difference in OSCE pass rate if BRM is applied? The results of EOS 5 and EOS 7 were tabulated, and the BRM standard setting was applied to these two OSCEs. The results showed that the mean score of both EOS OSCEs was significantly lower after BRM standard setting (P = 0.001). With BRM standard setting, it was able to identify more poor performers in OSCE who may have passed if the conventional arbitrary pass mark of 50 % was applied. We concluded that BRM standard setting is feasible and is a reasonable as well as defensible method of standard setting for OSCE. We recommend BRM for all OSCEs in Taylor’s Clinical School.

Medical schools in the UK set their own graduating examinations and pass marks. In a previous study we examined the equivalence of passing standards using the Angoff standard-setting method. To address the limitation this imposed on that work, we undertook further research using a standard-setting method specifically designed for objective structured clinical examinations (OSCEs). Six OSCE stations were incorporated into the graduating examinations of 3 of the medical schools that took part in the previous study. The borderline group method (BGM) or borderline regression method (BRM) was used to derive the pass marks for all stations in the OSCE. We compared passing standards at the 3 schools. We also compared the results within the schools with their previously generated Angoff pass marks. The pass marks derived using the BGM or BRM were consistent across 2 of the 3 schools, whereas the third school generated pass marks which were (with a single exception) much lower. Within-school comparisons of pass marks revealed that in 2 schools the pass marks generally did not significantly differ using either method, but for 1 school the Angoff mark was consistently and significantly lower than the BRM. The pass marks set using the BGM or BRM were more consistent across 2 of the 3 medical schools than pass marks set using the Angoff method. However, 1 medical school set significantly different pass marks from the other 2 schools. Although this study is small, we conclude that passing standards at different medical schools cannot be guaranteed to be equivalent.

Objective structured clinical examinations as an assessment method in residency training: practical considerations.

sBackgroundThis study aims to assess the feasibility, reliability and validity of the panel-based Equal Z-score (EZ) method applied to objective structural clinical examination (OSCE) of Chinese medical students and undertaking a comparison with the statistical techniques-based Borderline Regression Method (BRM).MethodsData received from two cohorts of 6th and 7th year medical students in Taiwan who set the mock OSCE as a formative assessment. Traditionally this medical school uses BRM to set the pass/fail cut-score. For the current study, 31 OSCE panellists volunteered to participate in the EZ method in parallel to the BRM.ResultsIn the conduct of this study, each panel completed this task for an OSCE exam comprising 12 stations within less than 60 min. Moreover, none of the 31 panellists, whose are busy clinicians, had indicated that the task was too difficult or too time-consuming. Although EZ method yielded higher cut-scores than the BRM it was found reliable. Intraclass correlation (ICC) measuring absolute agreement, across the three groups of panellists was .893 and .937 for the first and second rounds respectively, demonstrating high level of agreement across groups with the EZ method and the alignment between the BRM and the EZ method was visually observed. The paired t-test results identified smaller differences between the cut-scores within methods than across methods.ConclusionsOverall this study suggests that the EZ method is a feasible, reliable and valid standard setting method. The EZ method requires relatively little resources (takes about an hour to assess a 12 station OSCE); the calculation of the cut-score is simple and requires basic statistical skills; it is highly reliable even when only 10 panellists participate in the process; and its validity is supported by comparison to BRM. This study suggests that the EZ method is a feasible, reliable and valid standard setting method.

PurposeSetting standards is critical in health professions. However, appropriate standard setting methods do not always apply to the set cut score in performance assessment. The aim of this study was to compare the cut score when the standard setting is changed from the norm-referenced method to the borderline group method (BGM) and borderline regression method (BRM) in an objective structured clinical examination (OSCE) in medical school.MethodsThis was an explorative study to model the implementation of the BGM and BRM. A total of 107 fourth-year medical students attended the OSCE at 7 stations for encountering standardized patients (SPs) and at 1 station for performing skills on a manikin on July 15th, 2021. Thirty-two physician examiners evaluated the performance by completing a checklist and global rating scales.ResultsThe cut score of the norm-referenced method was lower than that of the BGM (P<0.01) and BRM (P<0.02). There was no significant difference in the cut score between the BGM and BRM (P=0.40). The station with the highest standard deviation and the highest proportion of the borderline group showed the largest cut score difference in standard setting methods.ConclusionPrefixed cut scores by the norm-referenced method without considering station contents or examinee performance can vary due to station difficulty and content, affecting the appropriateness of standard setting decisions. If there is an adequate consensus on the criteria for the borderline group, standard setting with the BRM could be applied as a practical and defensible method to determine the cut score for OSCE.

Background: In Objective Structured Clinical Examinations (OSCEs), the use of simulated patients (SPs) at many stations is a key aspect of the assessment. Often the SPs are asked to provide formal feedback (ratings) of their experience with the students under examination.Aims: This study analyses whether and how exactly SP data can be best used to enhance the robustness of the formal standard setting process.Methods: A retrospective statistical investigation into the relationship between SP ratings and those provided by the clinical assessors (criterion-based checklist scores and overall grades for each station) is presented. In addition, the paper also includes a study into the impact of the inclusion of the SP ratings in the formal standard setting process for OSCEs, particularly when pass marks are calculated using the borderline regression method.Results: The general results of the analysis, including discussion of two distinct methods for the combining of the SP ratings and assessor judgements, are presented, and demonstrate that the inclusion of this additional data can have important effects on individual student results.Conclusion: It is possible for the overall quality of the OSCE assessment process to be improved, with increased reliability, by combining assessor checklist scores and SP ratings.

Background: Objective structured clinical examinations (OSCEs) are considered the most psychometrically robust form of clinical skills assessment in the health professions. In 2014, the College of Pharmacy at Qatar University (CoP-QU) piloted the first cumulative OSCE for graduating students in collaboration with the Supreme Council of Health and the University of Toronto. Since then, interest has grown in measuring the psychometric properties of this examination to ensure adequate reliability, validity and defensibility. Objectives: This study aimed to evaluate the psychometric properties of the OSCE conducted in 2015 at the CoP-QU. A secondary objective of this study was to identify quality improvement opportunities for design, implementation, and evaluation of the OSCE. Methods: The psychometric analysis occurred as follows: We calculated cut scores and pass rates of the 10 stations being used in the OSCE assessment using borderline regression method. Predictive validity of undergraduate courses grades with OSCE grades were calculated using correlation and regression statistics. Concurrent validity of similar cumulative exams were evaluated using Pearson correlation. Risk of bias was calculated using Spearman correlation between assessors' analytical (checklist sheet of required tasks to be performed in a station) and global (the score of whole performance including communication skills on a scale from 1 to 5) scoring. Content validity was assessed quantitatively using 18 student-feedback forms and qualitatively through focus groups with OSCE participants and contributors (total of 5 assessors, 3 students, 3 administrators, 3 standardized patients). Interrater reliability was assessed using intra-class correlation coefficients (ICCs). Construct validity was evaluated by comparing interrater reliability between the first and second OSCE cycles. Cronbach's alpha was used to determine internal consistency of students' performance in all stations in terms of global and total scores. Correlation statistics were conducted at α level &lt; 0.05. Results: Out of 50% allocated for global score and 50% for analytical score per station, and based on the cut scores calculated for every station, average pass rate per analytical checklist grades in all stations was 70.4%, while average pass rate calculated for total scores in all stations was 79.2%. Four courses simulating professional skills of OSCE, two adapted undergraduate formative OSCEs, and a Medicinal Chemistry course, the control, correlated with the OSCE grades as follow, 0.72 (P &lt; 0.01), 0.47 (P &lt; 0.05), 0.43 (P&gt;0.05), 0.65 (P &lt; 0.01), 0.78 (P &lt; 0.01), 0.61 (P &lt; 0.01), and 0.36 (P&gt;0.05) respectively. OSCE grades can be moderately predicted by Professional skills course grades (52.3%) and its practical assessment (61.2%). Average correlation between analytical and global grades for all assessors was 0.52. A total of 90% of the stations were deemed to reflect practice, according to student perceptions. The average ICC of analytical checklists scores, global scores, and total scores were 0.88 (0.71-0.95), 0.61 (0.19-0.82), and 0.75 (0.45-0.88) respectively. Cronbach's alpha of students' performance in global scores across stations was 0.87, and 0.93 in terms of total scores. Conclusion: The cumulative OSCE conducted in 2015 showed acceptable validity and reliability as a high stakes examination and therefore is suitable to be implemented as a mandatory core curriculum component for student pharmacist assessment in Qatar.

Background: Implementing large-scale multi-site objective structured clinical examination (OSCEs) for national competency examination in a low resource country is challenging.Aims: To describe the first national OSCE for national competency examination of medical doctors in Indonesia and evaluate the reliability, validity, feasibility, acceptability, and educational impact.Methods: We collected electronically the OSCE scores from 49 out of 73 medical schools that participated to assess reliability and validity. We conducted electronic survey to examiners, examinees, SP trainers, and OSCE coordinators to assess feasibility, acceptability, and educational impact.Results: The Cronbach’s alpha coefficient across station was 0.79. There was strong correlation between rubric and global rating scores in each station (coefficient correlation ranges from 0.705 to 0.82). The content validity ratio was 0.97. The coefficient correlation between OSCE and MCQ was 0.335 (p = 0.00). All 49 medical schools were able to conduct OSCE simultaneously. Examiners, examinees, SP trainers, and OSCE coordinators had good perception regarding feasibility and acceptability of OSCE. Both examiners and examinees indicated good educational impact of OSCE application. The cutting score based on the borderline regression method was 61.96%. There were 67.39% of the examinees achieved similar or above the cutting score.Conclusion: With 12 stations 15 min each, the reliability coefficient across station is intermediate. Content validity is good. It is feasible and acceptable to implement large-scale multi-site OSCEs in Indonesia. Examiners and examinees perceive good educational impact on OSCE implementation.

Background: A key issue underpinning the usefulness of the OSCE assessment to medical education is standard setting, but the majority of standard-setting methods remain challenging for performance assessment because they produce varying passing marks. Several studies have compared standard-setting methods; however, most of these studies are limited by their experimental scope, or use data on examinee performance at a single OSCE station or from a single medical school. This collaborative study between 10 Australian medical schools investigated the effect of standard-setting methods on OSCE cut scores and failure rates.Methods: This research used 5256 examinee scores from seven shared OSCE stations to calculate cut scores and failure rates using two different compromise standard-setting methods, namely the Borderline Regression and Cohen’s methods.Results: The results of this study indicate that Cohen’s method yields similar outcomes to the Borderline Regression method, particularly for large examinee cohort sizes. However, with lower examinee numbers on a station, the Borderline Regression method resulted in higher cut scores and larger difference margins in the failure rates.Conclusion: Cohen’s method yields similar outcomes as the Borderline Regression method and its application for benchmarking purposes and in resource-limited settings is justifiable, particularly with large examinee numbers.

Background:One of the methods used for standard setting is the borderline regression method (BRM). This study aims to assess the reliability of BRM when the pass-fail standard in an objective structured clinical examination (OSCE) was calculated by averaging the BRM standards obtained for each station separately.Materials and Methods:In nine stations of the OSCE with direct observation the examiners gave each student a checklist score and a global score. Using a linear regression model for each station, we calculated the checklist score cut-off on the regression equation for the global scale cut-off set at 2. The OSCE pass-fail standard was defined as the average of all station's standard. To determine the reliability, the root mean square error (RMSE) was calculated. The R2 coefficient and the inter-grade discrimination were calculated to assess the quality of OSCE.Results:The mean total test score was 60.78. The OSCE pass-fail standard and its RMSE were 47.37 and 0.55, respectively. The R2 coefficients ranged from 0.44 to 0.79. The inter-grade discrimination score varied greatly among stations.Conclusion:The RMSE of the standard was very small indicating that BRM is a reliable method of setting standard for OSCE, which has the advantage of providing data for quality assurance.

Construct: Authentic standard setting methods will demonstrate high convergent validity evidence of their outcomes, that is, cutoff scores and pass/fail decisions, with most other methods when compared with each other. Background: The objective structured clinical examination (OSCE) was established for valid, reliable, and objective assessment of clinical skills in health professions education. Various standard setting methods have been proposed to identify objective, reliable, and valid cutoff scores on OSCEs. These methods may identify different cutoff scores for the same examinations. Identification of valid and reliable cutoff scores for OSCEs remains an important issue and a challenge. Approach: Thirty OSCE stations administered at least twice in the years 2010–2012 to 393 medical students in Years 2 and 3 at Aga Khan University are included. Psychometric properties of the scores are determined. Cutoff scores and pass/fail decisions of Wijnen, Cohen, Mean–1.5SD, Mean–1SD, Angoff, borderline group and borderline regression (BL-R) methods are compared with each other and with three variants of cluster analysis using repeated measures analysis of variance and Cohen's kappa. Results: The mean psychometric indices on the 30 OSCE stations are reliability coefficient = 0.76 (SD = 0.12); standard error of measurement = 5.66 (SD = 1.38); coefficient of determination = 0.47 (SD = 0.19), and intergrade discrimination = 7.19 (SD = 1.89). BL-R and Wijnen methods show the highest convergent validity evidence among other methods on the defined criteria. Angoff and Mean–1.5SD demonstrated least convergent validity evidence. The three cluster variants showed substantial convergent validity with borderline methods. Conclusions: Although there was a high level of convergent validity of Wijnen method, it lacks the theoretical strength to be used for competency-based assessments. The BL-R method is found to show the highest convergent validity evidences for OSCEs with other standard setting methods used in the present study. We also found that cluster analysis using mean method can be used for quality assurance of borderline methods. These findings should be further confirmed by studies in other settings.

BackgroundThe goal of the Objective Structured Clinical Examination (OSCE) in Competency-based Medical Education (CBME) is to establish a minimal level of competence. The purpose of this study was to 1) to determine the credibility and acceptability of the modified Angoff method of standard setting in the setting of CBME, using the Borderline Group (BG) method and the Borderline Regression (BLR) method as a reference standard; 2) to determine if it is feasible to set different standards for junior and senior residents, and 3) to determine the desired characteristics of the judges applying the modified Angoff method.MethodsThe results of a previous OSCE study (21 junior residents, 18 senior residents, and six fellows) were used. Three groups of judges performed the modified Angoff method for both junior and senior residents: 1) sports medicine surgeons, 2) non-sports medicine orthopedic surgeons, and 3) sports fellows. Judges defined a borderline resident as a resident performing at a level between competent and a novice at each station. For each checklist item, the judges answered yes or no for “will the borderline/advanced beginner examinee respond correctly to this item?” The pass mark was calculated by averaging the scores. This pass mark was compared to that created using both the BG and the BLR methods.ResultsA paired t-test showed that all examiner groups expected senior residents to get significantly higher percentage of checklist items correct compared to junior residents (all stations p < 0.001). There were no significant differences due to judge type. For senior residents, there were no significant differences between the cut scores determined by the modified Angoff method and the BG/BLR method. For junior residents, the cut scores determined by the modified Angoff method were lower than the cut scores determined by the BG/BLR Method (all p < 0.01).ConclusionThe results of this study show that the modified Angoff method is an acceptable method of setting different pass marks for senior and junior residents. The use of this method enables both senior and junior residents to sit the same OSCE, preferable in the regular assessment environment of CBME.

This study aims to compare the effectiveness of using the checklist and global rating scores to evaluate the clinical competency of medical students in Objective Structured Clinical Examinations (OSCEs). Additionally, the study assesses the appropriateness of using the borderline regression method to set standards for small-scale OSCE exams and determines if the estimated passing marks differ significantly from the university's prefixed passing score of 70%. The study also examines whether the university should utilize the borderline regression method to determine passing scores for each OSCE exam instead of a set passing score. The study analyzed medical students' grades in 11 OSCE exams in the 2022-2023 academic year at Alfaisal University, Riyadh, Saudi Arabia. Students received family medicine clerkship rotations, and after each rotation, they took an OSCE exam consisting of three stations that family medicine consultants graded. The exam included a checklist of 30 tasks and a five-level global rank scale. The study collected all the checklist marks and global rank grades and analyzed them using IBM® Statistical Package for Social Sciences (SPSS® Statistics) software. The statistical tests used were descriptive statistics, the T-test, chi-square tests, Fisher's exact test, and Pearson correlation. The study showed that students were more likely to pass when using the global rating system than the checklist scoring system. Additionally, students had a significantly lower passing rate when using the higher cut-off passing score estimated using the borderline regression method compared to the pre-set passing score of 70% established by the university (with a p-value of 0.00). Each scoring system has advantages and disadvantages, but they complement each other. Combining scoring systems can produce a more comprehensive and precise evaluation of a candidate's performance. The study also emphasizes the importance of carefully selecting and validating cut-off points in OSCE exams to ensure fairness and consistency in assessment.

BackgroundThe assessment of clinical competences during a national accreditation is a high-stakes process, and the pass mark needs to be settled through robust methods [1]. Since 2012, the MACR certifies...

Revisiting 'Assessment of clinical competence using an objective structured clinical examination (OSCE)'.