Comparing the cut score for the borderline group method and borderline regression method with norm-referenced standard setting in an objective structured clinical examination in medical school in Korea.

Song Yi Park,Ji Ho Ryu,Ki-Hwan Ji,Sang-Hwa Lee,Min-Jeong Kim

doi:10.3352/jeehp.2021.18.25

Abstract

PurposeSetting standards is critical in health professions. However, appropriate standard setting methods do not always apply to the set cut score in performance assessment. The aim of this study was to compare the cut score when the standard setting is changed from the norm-referenced method to the borderline group method (BGM) and borderline regression method (BRM) in an objective structured clinical examination (OSCE) in medical school.MethodsThis was an explorative study to model the implementation of the BGM and BRM. A total of 107 fourth-year medical students attended the OSCE at 7 stations for encountering standardized patients (SPs) and at 1 station for performing skills on a manikin on July 15th, 2021. Thirty-two physician examiners evaluated the performance by completing a checklist and global rating scales.ResultsThe cut score of the norm-referenced method was lower than that of the BGM (P<0.01) and BRM (P<0.02). There was no significant difference in the cut score between the BGM and BRM (P=0.40). The station with the highest standard deviation and the highest proportion of the borderline group showed the largest cut score difference in standard setting methods.ConclusionPrefixed cut scores by the norm-referenced method without considering station contents or examinee performance can vary due to station difficulty and content, affecting the appropriateness of standard setting decisions. If there is an adequate consensus on the criteria for the borderline group, standard setting with the BRM could be applied as a practical and defensible method to determine the cut score for OSCE.

Highlights

Background/rationale Standard setting is a process by which human judgment can be synthesized in a rational and defensible way to classify score scales into categories [1]
This study reported and provided a real example of the implementation of 2 standard setting methods for objective structured clinical examination (OSCE)
The cut score of the norm-referenced method was lower than that of the borderline group method (BGM) and borderline regression method (BRM), and there was no significant difference in the cut score between the BGM and BRM

Summary

Introduction

Background/rationale Standard setting is a process by which human judgment can be synthesized in a rational and defensible way to classify score scales into categories [1]. There is no ‘gold standard’ in regard to setting the cut score in real examinations, this activity is critical in health professions [2]. This is the minimum judgment on the per-. Criterion-referenced standard setting methods are of 2 types: test-centered and examinee-centered. The former is appealing for setting a cut score on knowledge assessment, such as multiple-choice examination, and the latter is well suited for performance assessment, such as objective structured clinical examination (OSCE) [3]. Well-known examinee-centered standard settings are the contrasting groups method and the borderline group method (BGM)

Objectives

Methods

Results

Discussion

Conclusion