Borderline Candidate Research Articles

Setting defensible minimum-stations-passed standards in OSCE-type assessments

Introduction Alongside the usual exam-level cut-score requirement, the use of a conjunctive minimum number of stations passed (MNSP) standard in OSCE-type assessments is common practice across some parts of the world. Typically, the MNSP is fixed in advance with little justification, and does not vary from one administration to another in a particular setting—which is not congruent to best assessment practice for high stakes examinations. In this paper, we investigate empirically four methods of setting such a standard in an examinee-centred (i.e. post hoc) and criterion-based way that allows the standard to vary appropriately with station and test difficulty. Methods and results Using many administrations (n = 442) from a single exam (PLAB2 in the UK), we show via mixed modelling that the total number of stations passed for each candidate has reliability close to that of the total test score (relative g-coefficient 0.73 and 0.76 respectively). We then argue that calculating the MNSP based on the predicted number of stations passed at the ‘main’ exam-level cut-score (i.e. for the borderline candidate) is conceptually, theoretically and practically preferred amongst the four approaches considered. Further analysis indicates that this standard does vary from administration to administration, but acts in a secondary way, with approximately a quarter of exam-level candidate failures resulting from application of the MNSP standard alone. Conclusion Collectively, this work suggests that employing the identified approach to setting the MNSP standard is practically possible and, in many settings, is more defensible than using a fixed number of stations set in advance.

Examiners' decision-making processes in observation-based clinical examinations.

Objective structured clinical examinations (OSCEs) are commonly used to assess the clinical skills of health professional students. Examiner judgement is one acknowledged source of variation in candidate marks. This paper reports an exploration of examiner decision making to better characterise the cognitive processes and workload associated with making judgements of clinical performance in exit-level OSCEs. Fifty-five examiners for exit-level OSCEs at five Australian medical schools completed a NASA Task Load Index (TLX) measure of cognitive load and participated in focus group interviews immediately after the OSCE session. Discussions focused on how decisions were made for borderline and clear pass candidates. Interviews were transcribed, coded and thematically analysed. NASA TLX results were quantitatively analysed. Examiners self-reported higher cognitive workload levels when assessing a borderline candidate in comparison with a clear pass candidate. Further analysis revealed five major themes considered by examiners when marking candidate performance in an OSCE: (a) use of marking criteria as a source of reassurance; (b) difficulty adhering to the marking sheet under certain conditions; (c) demeanour of candidates; (d) patient safety, and (e) calibration using a mental construct of the 'mythical [prototypical] intern'. Examiners demonstrated particularly higher mental demand when assessing borderline compared to clear pass candidates. Examiners demonstrate that judging candidate performance is a complex, cognitively difficult task, particularly when performance is of borderline or lower standard. At programme exit level, examiners intuitively want to rate candidates against a construct of a prototypical graduate when marking criteria appear not to describe both what and how a passing candidate should demonstrate when completing clinical tasks. This construct should be shared, agreed upon and aligned with marking criteria to best guide examiner training and calibration. Achieving this integration may improve the accuracy and consistency of examiner judgements and reduce cognitive workload.

Do We Know Who the Person With the Borderline Score is, in Standard-Setting and Decision-Making

PurposeWhen assessing clinical competence, health professional educators use assessments of knowledge attainment, skills acquisition, and professional development, which impact on decision-making for student’s training progression. Given the impact of progression-failure, it is critical that the expected standard of performance is derived accurately, fairly, and transparently, and that the rating of student performance is performed within the highest standards achievable. There is ongoing disagreement as to the most appropriate methods to address both standard setting and decision-making. The borderline candidate has been debated extensively in the academic and educational setting, with ongoing disagreement surrounding the concept. MethodsIn this paper, we discuss further perspectives on the use of the borderline candidate, as part of the process for standard-setting, to give insights into how we can reframe the concept more accurately and apply it more appropriately. DiscussionDrawing parallels to Kane’s validity framework, we consider the concept of the borderline candidate from four different perspectives: ‘what is’-what are the linguistics and implications behind the phrase ‘borderline candidate’; ‘who is’-who is the borderline candidate; decided ‘by whom’-who is the person making the judgement; and ‘under what circumstances’-the context of the assessment. ConclusionFinally, we translate the theoretical discussion into pragmatic and practical solutions in standard-setting practice

Fortune-tellers or content specialists: challenging the standard setting paradigm in medical education programmes

The veracity of Objective Standard Setting (OSS) as a modern approach to criterion-referenced standard setting has been reported for healthcare student assessment in the USA, while in other countries, OSS remains unrecognized. OSS upholds the foundational principle for itemized tests that judges should base their decisions on test item content. Moreover, it presents judges with a conceptually transparent decision procedure. This contrasts with the predictions concerning a hypothetical borderline candidate which typify Angoff procedures. Furthermore, the iterative process involved in the Angoff standard setting task incurs financial and administrative burdens, thus creating the potential to cut corners through recruiting fewer judges. The underlying objective of homogenizing the test standard undermines its validity, while circumventing reputable standard setting principles. While the Rasch model offers an objective approach to predicting successful outcomes, combining Rasch and Angoff procedures does not resolve the validity problem for Angoff-based pass marks. This commentary highlights the virtues of OSS relative to the modified Angoff approach in the standard setting of itemized tests. It also identifies gaps in the research literature that should be addressed to strengthen the case for using OSS on an international scale for high-stakes assessments within healthcare disciplines as a testing ground for other disciplines.

Justification and implications of the introduction of an expanded Close Marking System for the Fellowship Examination

Critical review of the close marking system used in the Fellowship Examination revealed that minor modifications to the way in which it was employed could significantly improve the quality (reliability and validity) of the examination. In addition, it could provide better information for the specialty courts to use during their discussion of the borderline candidate at their mini-court meetings. An expanded close marking system (ECMS) probably has little impact on the overall pass rate, but it does improve the ability to determine the marginal candidate. In addition, it has the capacity to provide feedback on examiner performance and provides the specialty courts with a tool to assess the quality of their questions. Analysis of data collected during several 'live trials' of ECMS has shown that the implementation of the ECMS represents a further improvement in the processes around the Fellowship Examination.

Borderline competence – from a complexity perspective: conceptualization and implementation for certifying examinations

Passing or failing a candidate is a complex decision with one essential aspect being the definition of the required minimal level of competence. Commonly used methods, such as the Angoff method, are based on the 'borderline competence' concept; however, the method describes a 'borderline candidate'--a fictitious person, rather than describing a set of clearly defined criteria that describe a real person whose competencies fall between the clearly incompetent and clearly competent--the borderline zone. We argue that competence arises from the combination of discrete abilities across the entire subject domain including the interaction of some of those abilities, where strengths in some interacting abilities could partly compensate for weaknesses in others. We further argue that scores in the borderline zone may include combinations of abilities in which deficiencies are sufficiently compensated for by areas of strength, thus ensuring competency, while other, similarly scoring combinations do not include required compensations to ensure competency. We offer evidence that standard setting judges systematically and consistently assess the level of required ability on each individual task, taking into account the characteristics of those tasks based on the significance--being a function of the severity and incidence of that condition, and the ability to help--and the difficulty of executing that task. Consequently, a borderline zone of competence implies an overlap of competence and incompetence with respect to the level of performance on tasks of varying clinical significance. We further suggest that this overlap can be conceptualized as a folding of the 'competence plane' (consistent with a cusp catastrophe model) that cuts the space defined by performance levels on clinical conditions of varying significance, with their associated skills requirements. That folding defines the borderline zone in which competent candidates will perform most of the tasks in the borderline zone correctly whereas incompetent candidates will fail most tasks.

Acute bilateral phrenic neuropathy following treatment with adalimumab

The recombinant human IgG1 monoclonal antibody specific for human TNF-a adalimumab (Humira) has been recently introduced for the treatment of moderate/severe psoriasis. Neurological diseases have been rarely described as adverse events of anti-TNF agents. A case of acute respiratory failure due to diaphragmatic weakness following adalimumab therapy for psoriasis is described. A 65-year-old female patient presented with jaundice followed 2 days later by severe dyspnea and tachypnea which worsened when patient was lying flat, 1 week after the fourth dose of adalimumab. Isoniazid and vitamin B6 were co-administered with adalimumab. A symmetric elevation of diaphragms was shown on radiography and fluoroscopy. A pulmonary restrictive defect with a prominent decline of forced vital capacity (FVC) when the patient was on supine position was recorded. In the absence of specific limb electrophysiological abnormalities, acute bilateral symmetric phrenic neuropathy was diagnosed. The patient was a borderline candidate for mechanical ventilation for 3 weeks. Conservative treatment with oxygen was administered and both respiratory and liver disorder resolved 4 weeks following admission. A causal relationship of phrenal neuropathy with adalimumab is herein discussed.

Setting Standards for Performance Tests: A Pilot Study of a Three-Level Angoff Method

The Yes/No Angoff method is a standard-setting variant in which judges estimate whether a borderline student would or would not accomplish each item. However, when there is a 50% chance of success, systematically severe or lenient ratings may skew the cut score. This study piloted a three-level Angoff in which items were assigned to "Yes," "No," or "Maybe" categories. Five judges provided three-level Angoff ratings for seven checklists used in a clinical skills exam for fourth-year students. Systematic severity and leniency were simulated by converting all "Maybe" ratings to "Yes" or "No" ratings, respectively. Forty-one percent of 121 items had at least one "Maybe" rating. Ten percent of all ratings were in the "Maybe" category. One judge accounted for 83% of all "Maybe" ratings; three did not use it at all. Case failure rates varied considerably depending on the simulated severity of ratings. Overall failure rates were not substantially impacted. The three-level Angoff retains the cognitive simplicity of the Yes/No Angoff while addressing the challenge of items midrange for the borderline candidate.

Defining competency - the role of standard setting.

The responsibility to determine just who is competent to practice medicine, and at what standard, is great. Whilst there is still a period available for potential remediation, examinations at the completion of year three of the four-year Graduate Entry Medical Programme (GEMP) at Flinders University of South Australia (FUSA) are high stakes and contain the majority of final summative assessment for the certification of student to doctor. Therefore, the medical school has recently examined its methods for certification, the clinical practice standards sought in its programme and how to determine these standards. For all assessments a standard was documented and methods employed to set these standards using specific measures of performance. A modification of the Angoff method was applied to the written examination and the Rothman method, using two criteria, was used to determine competency in the objective structured clinical examination (OSCE). These methods were used for the first time in 1998. Both methods used trained 'experts' as standard setters and both methods used the notion of the 'borderline candidate' to determine the passing standard. This paper describes these two criterion-referenced standard-setting procedures as used in this school and related examination performance. Whilst the use of standard-setting procedures goes part way to defining and measuring competence, it is time consuming and requires significant examiner training and acceptance. Using 50% to determine who is and isn't competent is simpler but not transparent, fair nor defensible.

Development of the knowledge-based standard for the written certification examination of the American Board of Anesthesiology.

In 1988 and 1989, the American Board of Anesthesiology (ABA) developed a knowledge-based standard for its written certification examination. In brief, 13 "judges" developed a construct of a "borderline candidate," i.e., a candidate who was neither ideal nor clearly failing but rather had sufficient knowledge to just pass. In 1989, this construct was applied to 90 questions from the 1989 ABA examination to estimate candidate's score on that subset. When extended to the entire examination, the use of the construct resulted in a knowledge-based standard of 57% correct. (The 1988 exercise, also using the construct of a borderline candidate but with a totally different subset of questions, produced an identical standard). This standard resulted in higher success rates among the actual examinees taking the ABA examination (84% in 1989 and 90% in 1990) than had the normative standard used previously (80%). The authors suggest that the process they describe permits development of a reproducible criterion for success that is based entirely on mastery of a relevant body of knowledge rather than on normative considerations.