Borderline Group Method Research Articles

Workshop, Assessment, and Validity Evidence for Tools Measuring Performance of Knee and Shoulder Arthrocentesis.

Musculoskeletal concerns are common, yet residents at our institution lacked arthrocentesis training. We created a workshop to teach residents knee and shoulder arthrocentesis, developed simulated assessment scenarios (SASs) with tools to measure procedural proficiency, and collected validity evidence. A multidisciplinary group conducted a modified Delphi to define content for the workshop, SASs, and assessment tools. We defined minimum thresholds for competence in knee and shoulder arthrocentesis using the modified borderline-group method. We implemented the workshop and SASs in 2020 and 2021 and analyzed assessment tool scoring for statistical reliability and validity. Our program evaluation included SAS performance, participants' survey responses, and change in the number of arthrocenteses performed in the internal medicine (IM) resident primary care clinic. Sixty-one residents (53 IM, eight physical medicine and rehabilitation [PM&R]) participated. Fifty-two (85%; 46 IM, six PM&R) completed the evaluation survey. We procured data from 48 knee and 65 shoulder SASs for validity evidence. All arthrocentesis SAS performances met the proficiency standard except one resident's shoulder SAS. Validity evidence revealed strong interrater reliability (α = .82 and .77 for knee and shoulder, respectively) and strong relational validity (p < .001 for both procedures). All participants rated workshop quality and usefulness as good or very good. The number of arthrocenteses performed at our institution's primary care clinic increased. We created a workshop to teach residents arthrocentesis and assessment tools with strong validity and reliability evidence. The workshop was well regarded by residents, who applied their arthrocentesis skills during patient care.

OSCE Standard Setting: Three Borderline Group Methods.

High-stakes assessments must discriminate between examinees who are sufficiently competent to practice in the health professions and examinees who are not. In these settings, criterion-referenced standard-setting methods are strongly preferred over norm referenced methods. While there are many criterion-referenced options, few are feasible or cost effective for objective structured clinical examinations (OSCEs). The human and financial resources required to organize OSCEs alone are often significant, leaving little in an institution's budget for additional resource-intensive standard-setting methods. The modified borderline group method introduced by Dauphinee et al. for a large-scale, multi-site OSCE is a very feasible option but is not as defensible for smaller scale OSCEs. This study compared the modified borderline group method to two adaptations that address its limitations for smaller scale OSCEs while retaining its benefits, namely feasibility. We evaluated decision accuracy and consistency of calculated cut scores derived from (1) modified, (2) regression-based, and (3) 4-facet Rasch model borderline group methods. Data were from a 12-station OSCE that assessed 112 nurses for entry to practice in a Canadian context. The three cut scores (64-65%) all met acceptable standards of accuracy and consistency; however, the modified borderline group method was the most influenced by lower scores within the borderline group, leading to the lowest cut score. The two adaptations may be more defensible than modified BGM in the context of a smaller (n < 100-150) OSCE.

American College of Surgeons Objective Assessment of Skills in Surgery (ACS OASIS): A Formative Assessment of Junior Residents’ Technical Skills

The objective assessment of technical skills of junior residents is essential in implementing competency-based training and providing specific feedback regarding areas for improvement. An innovative assessment that can be easily implemented by training programs nationwide has been developed by expert surgeon educators under the aegis of the American College of Surgeons (ACS) Division of Education. This assessment, ACS Objective Assessment of Skills in Surgery (ACS OASIS) uses eight stations to address technical skills important for junior residents within the domains of laparoscopic appendectomy, excision of lipoma, central line placement, laparoscopic cholecystectomy, trocar placement, exploratory laparotomy, repair of enterotomy, and tube thoracostomy. The purpose of this study was to implement ACS OASIS at a number of sites to study its psychometric rigor. The ACS OASIS was pre-piloted at two programs to establish feasibility and to gather information regarding implementation. Each skills station was 12 minutes long, and the faculty completed a checklist with 5 to 15 items, and a global assessment scale. The study was then repeated at three pilot sites and included 29 junior residents who were assessed by a total of 44 faculty. Psychometric data for the stations and checklists were collected and analyzed. The pre-pilot sites were Geisinger and University of Tennessee Knoxville.Data were gathered from pilot sites that included Wellspan Health, Duke University, and University of California Los Angeles. The mean checklist score for all learners was 76% (IQR of 66%-85%). The average global rating was 3.36 on a 5-point scale with a standard deviation of 0.56. The overall cut score derived using the borderline group method was at 68% with 34% of performances requiring remediation. Using this criterion, the average number of stations that were completed by each learner without need for remediation was five.The station discrimination index ranged from 0.27 to 0.65 (all above the threshold of 0.25), demonstrating solid psychometric characteristics at the station level. The internal-consistency reliability was 0.76 with SEM of 5.8%. The inter-rater reliability (intraclass correlation) was high at 0.73 with general agreement of 79% between the two raters. The station discrimination was at 0.45 (range of 0.27 to 0.65) indicating a high level of differentiation between high and low performers. Using the generalizability theory, the G-coefficient reliability was at 0.72 with the reliability projection flattening after 8 stations. Overall, 75% to 82% the faculty and learners rated ACS OASIS as realistic and beneficial. ACS OASIS is a psychometrically sound technical skills assessment tool that can provide useful information for feedback to junior residents and support efforts to remediate gaps in performance.

Comparing the cut score for the borderline group method and borderline regression method with norm-referenced standard setting in an objective structured clinical examination in medical school in Korea.

PurposeSetting standards is critical in health professions. However, appropriate standard setting methods do not always apply to the set cut score in performance assessment. The aim of this study was to compare the cut score when the standard setting is changed from the norm-referenced method to the borderline group method (BGM) and borderline regression method (BRM) in an objective structured clinical examination (OSCE) in medical school.MethodsThis was an explorative study to model the implementation of the BGM and BRM. A total of 107 fourth-year medical students attended the OSCE at 7 stations for encountering standardized patients (SPs) and at 1 station for performing skills on a manikin on July 15th, 2021. Thirty-two physician examiners evaluated the performance by completing a checklist and global rating scales.ResultsThe cut score of the norm-referenced method was lower than that of the BGM (P<0.01) and BRM (P<0.02). There was no significant difference in the cut score between the BGM and BRM (P=0.40). The station with the highest standard deviation and the highest proportion of the borderline group showed the largest cut score difference in standard setting methods.ConclusionPrefixed cut scores by the norm-referenced method without considering station contents or examinee performance can vary due to station difficulty and content, affecting the appropriateness of standard setting decisions. If there is an adequate consensus on the criteria for the borderline group, standard setting with the BRM could be applied as a practical and defensible method to determine the cut score for OSCE.

Objective Assessment and Standard Setting for Basic Flexible Ureterorenoscopy Skills Among Urology Trainees Using Simulation-Based Methods.

Objective: To objectively assess the performance of graduating urology residents performing flexible ureterorenoscopy (fURS) using a simulation-based model and to set an entrustability standard or benchmark for use across the educational spectrum. Methods: Chief urology residents and attending endourologists performed a standardized fURS task (ureterorenoscopy and repositioning of stones) using a Boston Scientific© Lithovue ureteroscope on a Cook Medical© URS model. All performances were video-recorded and blindly scored by both endourology experts and crowd-workers (C-SATS) using the Ureteroscopic Global Rating Scale, plus an overall entrustability score. Validity evidence supporting the scores was collected and categorized. The Borderline Group (BG) method was used to set absolute performance standards for the expert and crowdsourced ratings. Results: A total of 44 participants (40 chief residents, 4 faculties) completed testing. Eighty-three percent of participants had performed >50 fURS cases at the time of the study. Only 47.7% (mean score 12.6/20) and 61.4% (mean score 12.4/20) of participants were deemed "entrustable" by experts and crowd-workers, respectively. The BG method produced entrustability benchmarks of 11.8/20 for experts and 11.4/20 for crowd-worker ratings, resulting in pass rates of 56.9% and 61.4%. Conclusion: Using absolute standard setting methods, benchmark scores were set to identify trainees who could safely carry out fURS in the simulated setting. Only 60% of residents in our cohort were rated as entrustable. These findings support the use of benchmarks to earlier identify trainees requiring remediation.

Evaluation of Passing Scores in Semiotics: An Objective Structured Clinical Examination for Medical Students of Mashhad University of Medical Sciences, Mashhad, Iran, 2015

Background Numerous exams are held at different levels and in different fields of medical sciences to evaluate students’ practical knowledge. In pass-fail exams where several examiners score the students, it is important to determine “the minimum passing score” or “the passing score” to determine whether students have passed or failed; this score is sometimes called the “cut-off point” or “standard score.” The objective structured clinical examination (OSCE) method is employed for the final assessment of medical students in Semiotics I in Mashhad University of Medical Sciences, Mashhad, Iran. The commonly used standard scoring method for this lesson is the fixed score method, which sometimes results in a discrepancy between educational management and the lecturers. Hence, the current study aims to compare 4 different methods—the Cohen, borderline-group, borderline regression, and Hofstee methods—of determining the passing score in the semiotics course and comparing the results with those of the fixed score method. Methods A 6-station OSCE was used to assess Semiotics I in Mashhad University of Medical Sciences in 2015. In the current study, in order to determine a standard scale for scoring the students, two forms, Forms 1 and 2, and a checklist were completed for each student. In Form 1, a 5-option Likert scale scoring system, graded from poor to excellent, was used. Data from Form 1 were analyzed using the borderline regression and borderline-group methods. Form 2 included 4 items and the collected data were analyzed using the Hofstee method. Data collected from both forms were analyzed, after the exams, using SPSS version 16. Results The cut-off point established by the Cohen method was very close to that of the common method. In other words, there was no significant difference between the cut-off point determined by the Cohen method (11.73) and that of the common method (12). The other study methods, however, such as borderline regression and borderline-group methods proposed higher cut-off points, which were significantly different from that of the common method: more students failed Semiotics I using these methods. The Hofstee method cannot be used in the OSCE, as the results were insignificant. Conclusions Because there was a significant difference in the number of students who passed the exam based on the fixed score and Cohen methods, and on the borderline-group and borderline regression methods, it is recommended that the latter methods not be widely employed. In addition, it is suggested that different methods should be used to define a mean standard passing score because, according to the statistics, an accurate and efficient estimator with minimum variances accuracy should be employed to evaluate population parameters, and the mean estimator would benefit from such advantages.

Evaluating different standard-setting methods in an ESL placement testing context

In language programs, it is crucial to place incoming students into appropriate levels to ensure that course curriculum and materials are well targeted to their learning needs. Deciding how and where to set cutscores on placement tests is thus of central importance to programs, but previous studies in educational measurement disagree as to which standard-setting method (or methods) should be employed in different contexts. Furthermore, the results of different standard-setting methods rarely converge on a single set of cutscores, and standard-setting procedures within language program placement testing contexts specifically have been relatively understudied. This study aims to compare and evaluate three different standard-setting procedures – the Bookmark method (a test-centered approach), the Borderline group method (an examinee-centered approach), and cluster analysis (a statistical approach) – and to discuss the ways in which they do and do not provide valid and reliable information regarding placement cut-offs for an intensive English program at a large Midwestern university in the USA. As predicted, the cutscores derived from the different methods did not converge on a single solution, necessitating a means of judging between divergent results. We discuss methods of evaluating cutscores, explicate the advantages and limitations associated with each standard-setting method, recommend against using statistical approaches for most English for academic purposes (EAP) placement contexts, and demonstrate how specific psychometric qualities of the exam can affect the results obtained using those methods. Recommendations for standard setting, exam development, and cutscore use are discussed.

How Good is Good Enough?: A Comparison of Three Methods for Establishing Cut Scores on Placement Tests

The purpose of this study was to determine whether three standard-setting methods would produce sufficiently consistent results for placement decisions. The cut scores were derived from modified Angoff method, borderline group method, and cluster analysis. The results indicate that the cut scores derived from the three standard setting methods did not entirely agree with each other in assigning students into different levels, suggesting that the choice of standard setting method influenced the resulting cut scores. Specifically, the borderline-group method tended to produce lower cut scores than the other two methods. The cluster analysis yielded two cut scores, which were very similar to the cut scores derived from the modified Angoff method at the corresponding levels. The implications of the findings are discussed and avenues for further study are suggested.

Comparing the borderline group and borderline regression approaches to setting Objective Structured Clinical Examination cut scores

OBJECTIVE The aim of the study was to compare the Borderline Group and Borderline Regression approaches to setting standards on nine Objective Structured Clinical Examinations (OSCEs) undertaken in the final year of medical school. For each of nine OSCEs completed by 301 students we obtained a total score (maximum 40 marks) and a global score on a 5-point rating scale which examiners completed after scoring the OSCE using a checklist. We calculated cut scores for each OSCE using the Borderline Group method by computing the mean and the median score of the group of students whose performance was rated by examiners rated as borderline. We calculated cut scores for each OSCE using the Borderline Regression method by predicting total OSCE scores from global ratings using linear regression, and calculating the cut score by substituting the score of borderline candidates (2) into the regression equation for each OSCE. Both methods established higher standards than an arbitrary 50 per cent criterion. There was also a high degree of concordance between methods. Establishing conceptual standards on OSCEs requires ongoing consideration of appropriate methods and research modelling the effects on cut scores of applying different techniques.

Global Rating Scale for the Assessment of Paramedic Clinical Competence

Objective. The aim of this study was to develop and critically appraise a global rating scale (GRS) for the assessment of individual paramedic clinical competence at the entry-to-practice level. Methods. The development phase of this study involved task analysis by experts, contributions from a focus group, and a modified Delphi process using a national expert panel to establish evidence of content validity. The critical appraisal phase had two raters apply the GRS, developed in the first phase, to a series of sample performances from three groups: novice paramedic students (group 1), paramedic students at the entry-to-practice level (group 2), and experienced paramedics (group 3). Using data from this process, we examined the tool's reliability within each group and tested the discriminative validity hypothesis that higher scores would be associated with higher levels of training and experience. Results. The development phase resulted in a seven-dimension, seven-point adjectival GRS. The two independent blinded raters scored 81 recorded sample performances (n = 25 in group 1, n = 33 in group 2, n = 23 in group 3) using the GRS. For groups 1, 2, and 3, respectively, interrater reliability reached 0.75, 0.88, and 0.94. Intrarater reliability reached 0.94 and the internal consistency ranged from 0.53 to 0.89. Rater differences contributed 0–5.7% of the total variance. The GRS scores assigned to each group increased with level of experience, both using the overall rating (means = 2.3, 4.1, 5.0; p < 0.001) and considering each dimension separately. Applying a modified borderline group method, 54.9% of group 1, 13.4% of group 2, and 2.9% of group 3 were below the cut score. Conclusion. The results of this study provide evidence that the scores generated using this scale can be valid for the purpose of making decisions regarding paramedic clinical competence. Key words: educational measurement; clinical competence; licensure; certification; paramedics; global rating scale; rating scales

The Objective Borderline method (OBM): a probability-based model for setting up an objective pass/fail cut-off score in medical programme assessments

The decision to pass or fail a medical student is a 'high stakes' one. The aim of this study is to introduce and demonstrate the feasibility and practicality of a new objective standard-setting method for determining the pass/fail cut-off score from borderline grades. Three methods for setting up pass/fail cut-off scores were compared: the Regression Method, the Borderline Group Method, and the new Objective Borderline Method (OBM). Using Year 5 students' OSCE results from one medical school we established the pass/fail cut-off scores by the abovementioned three methods. The comparison indicated that the pass/fail cut-off scores generated by the OBM were similar to those generated by the more established methods (0.840 ≤ r ≤ 0.998; p < .0001). Based on theoretical and empirical analysis, we suggest that the OBM has advantages over existing methods in that it combines objectivity, realism, robust empirical basis and, no less importantly, is simple to use.

Simulation-Based Examinations in Physician Assistant Education: A Comparison of Two Standard-Setting Methods

this study explored the reliability of two simple standard-setting methods that are used to set passing standards for a standardized patient (SP) exam in physician assistant (PA) education. fifty-four second-year PA students participated in a multistation SP-based clinical skills exam. Cut scores were set using the Angoff and Borderline Group methods. A panel of PA faculty set cut scores using the Angoff method. A modified version of the Borderline Group method set cut scores using SP global ratings verified by faculty review. Inter-rater reliability between judges was evaluated using kappa coefficient (k) for the Angoff method and intraclass correlation coefficient (ICC) for the Borderline Group method. the Borderline Group method set an overall cut score for the exam of 76% (95% CI +/- 5) and the Angoff method set a cut score at 62% (95% CI +/- 9). Both methods demonstrated sufficient inter-rater reliability (k 0.60, ICC > 0.70; both significant at p < 0.05), although one case (preop history and physical) demonstrated poor inter-rater reliability between judges using the Borderline Group method. the Borderline Group method offered a slightly more reliable cut score when compared to the standard set by the Angoff method, but was more challenging to implement. In addition, one case demonstrated poor inter-rater reliability with the Borderline Group method. Using SPs to complete global borderline ratings offers one solution to make the Borderline Group method more feasibile, but requires a high degree of initial rater calibration and periodic measures of interrater reliability between faculty and SPs.

Setting standards for an objective structured clinical examination: the borderline group method gains ground on Angoff

Setting standards for an objective structured clinical examination: the borderline group method gains ground on Angoff

Using borderline methods to compare passing standards for OSCEs at graduation across three medical schools

Medical schools in the UK set their own graduating examinations and pass marks. In a previous study we examined the equivalence of passing standards using the Angoff standard-setting method. To address the limitation this imposed on that work, we undertook further research using a standard-setting method specifically designed for objective structured clinical examinations (OSCEs). Six OSCE stations were incorporated into the graduating examinations of 3 of the medical schools that took part in the previous study. The borderline group method (BGM) or borderline regression method (BRM) was used to derive the pass marks for all stations in the OSCE. We compared passing standards at the 3 schools. We also compared the results within the schools with their previously generated Angoff pass marks. The pass marks derived using the BGM or BRM were consistent across 2 of the 3 schools, whereas the third school generated pass marks which were (with a single exception) much lower. Within-school comparisons of pass marks revealed that in 2 schools the pass marks generally did not significantly differ using either method, but for 1 school the Angoff mark was consistently and significantly lower than the BRM. The pass marks set using the BGM or BRM were more consistent across 2 of the 3 medical schools than pass marks set using the Angoff method. However, 1 medical school set significantly different pass marks from the other 2 schools. Although this study is small, we conclude that passing standards at different medical schools cannot be guaranteed to be equivalent.

Standard Setting in a Small Scale OSCE: A Comparison of the Modified Borderline-Group Method and the Borderline Regression Method

When setting standards, administrators of small-scale OSCEs often face several challenges, including a lack of resources, a lack of available expertise in statistics, and difficulty in recruiting judges. The Modified Borderline-Group Method is a standard setting procedure that compensates for these challenges by using physician examiners and is easy to use making it a good choice for small scale OSCEs. Unfortunately, the use of this approach may introduce a new challenge. Because a small scale OSCE has a small number of examinees, there may be few examinees in the borderline range, which could introduce an unintentional bias. A standard setting method called The Borderline Regression Method will be described. This standard setting method is similar to the Modified Borderline-Group Method but incorporates a linear regression approach allowing the cut score to be set using the scores from all examinees and not from a subset. The current study uses confidence intervals to analyze the precision of cut scores derived from both approaches when applied to a small scale OSCE.

Consistency of standards and stability of pass/fail decisions with examinee-based standard-setting methods in a small-scale objective structured clinical examination.

The apparent feasibility and the face validity of the examinee-based methods such as the borderline-group methods provide support for their increasing adoption by health profession schools. Before that can occur, however, more information on the quality of the standards produced by these techniques is required. The purpose of the present study was to assess the quality of the standards produced on a small-scale objective structured clinical examination (OSCE) by the borderline-group and contrasting-groups examinee-based standard-setting procedures. These two examinee-based criterion-referenced standard-setting procedures were done for an undergraduate fourth-year surgical OSCE and the consistency of the standards and the decisions arising from the standards were assessed. Both techniques provided consistent and realistic standards. There is sufficient theoretical and empirical evidence to support the use of examinee-based standard-setting techniques in small-scale OSCEs that use expert examiners.

Making the Cut in School Districts: Alternative Methods for Setting Cutscores

School districts are under increasing pressure to demonstrate that students are competent in various skills, such as reading and mathematics. Often, demonstrating competence involves comparing performance on assessments to a standard of performance, as embodied in a test score. These scores, called cutscores, separate competent and noncompetent examinees. Because school districts have varied sources of data to inform cutscore decisions, various methods are available for suggesting cutscores. In 2 studies, we examine a selection of methods for arriving at rational and defensible cutscores in school districts. Methods examined are the Angoff (1971) method; the borderline and contrasting groups methods; and 2 new methods, 1 based on course enrollment and 1 based on expert expectations. In Study 1, the Angoff, borderline group, and course enrollment results were consistent, whereas in Study 2, the Angoff and professional judgment methods yielded suggested cutscores that were lower than the borderline group method. Suggestions for further study include the reaction of teachers to the cutscore-setting methods, the effect of different teacher attributes on the results of cutscore-setting methods, and the efficiency of and most effective order for employing the various methods.

A Comparative Study of Standard-Setting Methods

The borderline-group method and the contrasting-groups method were compared with Nedelsky's method at four schools, with Angoff's method at another four schools, and with each other at all eight schools, using tests of basic skills in reading and mathematics. The borderline-group and contrastinggroups methods produced similar results when approximately equal numbers of students were classified as masters and nonmasters. The contrasting-groups passing score was lower than the borderline-group passing score when masters greatly outnumbered nonmasters and higher when nonmasters outnumbered masters. Results involving the Nedelsky and Angoff methods were not consistent across schools. Passing scores tended to be higher at schools where students were more able.