Criterion-referenced Standard Setting Research Articles

A validity study of COMLEX-USA Level 3 with the new test design.

The National Board of Osteopathic Medical Examiners (NBOME) administers the Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA), a three-level examination designed for licensure for the practice of osteopathic medicine. The examination design for COMLEX-USA Level 3 (L3) was changed in September 2018 to a two-day computer-based examination with two components: a multiple-choice question (MCQ) component with single best answer and a clinical decision-making (CDM) case component with extended multiple-choice (EMC) and short answer (SA) questions. Continued validation of the L3 examination, especially with the new design, is essential for the appropriate interpretation and use of the test scores. The purpose of this study is to gather evidence to support the validity of the L3 examination scores under the new design utilizing sources of evidence based on Kane's validity framework. Kane's validity framework contains four components of evidence to support the validity argument: Scoring, Generalization, Extrapolation, and Implication/Decision. In this study, we gathered data from various sources and conducted analyses to provide evidence that the L3 examination is validly measuring what it is supposed to measure. These include reviewing content coverage of the L3 examination, documenting scoring and reporting processes, estimating the reliability and decision accuracy/consistency of the scores, quantifying associations between the scores from the MCQ and CDM components and between scores from different competency domains of the L3 examination, exploring the relationships between L3 scores and scores from a performance-based assessment that measures related constructs, performing subgroup comparisons, and describing and justifying the criterion-referenced standard setting process. The analysis data contains first-attempt testscores for 8,366 candidates who took the L3 examination between September 2018 and December 2019. The performance-based assessment utilized as a criterion measure in this study is COMLEX-USA Level 2 Performance Evaluation (L2-PE). All assessment forms were built through the automated test assembly (ATA) procedure to maximize parallelism in terms of content coverage and statistical properties across the forms. Scoring and reporting follows industry-standard quality-control procedures. The inter-rater reliability of SA rating, decision accuracy, and decision consistency for pass/fail classifications are all very high. There is a statistically significant positive association between the MCQ and the CDM components of the L3 examination. The patterns of associations, both within the L3 subscores and with L2-PE domain scores, fit with what is being measured. The subgroup comparisons by gender, race, and first language showed expected small differences in mean scores between the subgroups within each category and yielded findings that are consistent with those described in the literature. The L3 pass/fail standard was established through implementation of a defensible criterion-referenced procedure. This study provides some additional validity evidence for the L3 examination based on Kane's validity framework. The validity of any measurement must be established through ongoing evaluation of the related evidence. The NBOME will continue to collect evidence to support validity arguments for the COMLEX-USA examination series.

Establishing performance standards for child development: learnings from the ECDI2030

BackgroundStandards of early childhood development (ECD) are needed to determine whether children living in different contexts are developmentally on track. The Early Childhood Development Index 2030 (ECDI2030) is a population-level measure intended to be used in household surveys to collect globally comparable data on one of the indicators chosen to monitor progress toward target 4.2 of the Sustainable Development Goals: The proportion of children aged 24–59 months who are developmentally on track in health, learning and psychosocial well-being.MethodsTo define performance cut-scores for the ECDI2030 we followed a criterion-referenced standard setting exercise using the modified Angoff method. The exercise gauged the expectations from 15 global experts in ECD and was informed by representative population data collected in Mexico and the State of Palestine. The final calibrated age-specific performance cut-scores were applied to these data to estimate the proportion of children developmentally on track, disaggregated by background characteristics, including the child's sex and attendance to early childhood education.ResultsThrough a process of standard setting, we generated robust performance standards for the ECDI2030 by establishing five age-specific cut-scores to identify children as developmentally on track.ConclusionsThis paper demonstrated how the standard setting methodology, typically applied to measures in the health and education fields, could be applied to a measure of child development. By creating robust criterion-referenced standards, we have been able to ensure that the cut-scores related to age for the ECDI2030 are based on performance standards set by global experts in the ECD field for defining on and off track development.

A resource efficient and reliable standard setting method for OSCEs: Borderline regression method using standardized patients as sole raters in clinical case encounters with medical students

Finding a reliable, practical and low-cost criterion-referenced standard setting method for performance-based assessments has proved challenging. The borderline regression method of standard setting for OSCEs has been shown to estimate reliable scores in studies using faculty as raters. Standardized patients (SPs) have been shown to be reliable OSCE raters but have not been evaluated as raters using this standard setting method. Our study sought to find whether SPs could be reliably used as sole raters in an OSCE of clinical encounters using the borderline regression standard setting method. SPs were trained for on a five-point global rating scale. In an OSCE for medical students, SPs completed skills checklists and the global rating scale. The borderline regression method was used to create case passing scores. We estimated the dependability of the final pass or fail decisions and the absolute dependability coefficients for global ratings, checklist scores, and case pass-score decisions using generalizability theory. The overall dependability estimate is 0.92 for pass or fail decisions for the complete OSCE. Dependability coefficients (0.70–0.86) of individual case passing scores range demonstrated high dependability. Based on our findings, the borderline regression method of standard setting can be used with SPs as sole raters in a medical student OSCE to produce a dependable passing score. For those already using SPs as raters, this can provide a practical criterion-referenced standard setting method for no additional cost or faculty time.

Implementation and students’ perception of a criterion-referenced standard setting in a therapeutics course

Objective: To implement a criterion-referenced method to set standards for grading written tests in a didactic course and to assess students’ perceptions of the implementation. Methods: The Angoff method, a criterion-referenced method, was implemented in a therapeutics course with a letter grading system. Students were surveyed on their perceptions of the method including stress and test anxiety level after the course. Results: Of 122 students enrolled, 118 responded. More than 60% of respondents felt that the criterion-referenced method was fairer and reflected competency better than a norm-referenced method. The percent of respondents who felt that the new method increased the level of stress and test anxiety was higher than that of those who did not. Conclusions: A criterion-referenced method was successfully implemented in a pharmacy didactic course with a letter grading system. The implementation was overall favourably received by students although it may have increased the level of stress and test anxiety.

Exploring differences in individual and group judgements in standard setting.

Standard setting is critically important to assessment decisions in medical education. Recent research has demonstrated variations between medical schools in the standards set for shared items. Despite the centrality of judgement to criterion-referenced standard setting methods, little is known about the individual or group processes that underpin them. This study aimed to explore the operation and interaction of these processes in order to illuminate potential sources of variability. Using qualitative research, we purposively sampled across UK medical schools that set a low, medium or high standard on nationally shared items, collecting data by observation of graduation-level standard-setting meetings and semi-structured interviews with standard-setting judges. Data were analysed using thematic analysis based on the principles of grounded theory. Standard setting occurred through the complex interaction of institutional context, judges' individual perspectives and group interactions. Schools' procedures, panel members and atmosphere produced unique contexts. Individual judges formed varied understandings of the clinical and technical features of each question, relating these to their differing (sometimes contradictory) conceptions of minimally competent students, by balancing information and making suppositions. Conceptions of minimal competence variously comprised: limited attendance; limited knowledge; poor knowledge application; emotional responses to questions; 'test-savviness', or a strategic focus on safety. Judges experienced tensions trying to situate these abstract conceptions in reality, revealing uncertainty. Groups constructively revised scores through debate, sharing information and often constructing detailed clinical representations of cases. Groups frequently displayed conformity, illustrating a belief that outlying judges were likely to be incorrect. Less frequently, judges resisted change, using emphatic language, bargaining or, rarely, 'polarisation' to influence colleagues. Despite careful conduct through well-established procedures, standard setting is judgementally complex and involves uncertainty. Understanding whether or how these varied processes produce the previously observed variations in outcomes may offer routes to enhance equivalence of criterion-referenced standards.

Modifying Hofstee standard setting for assessments that vary in difficulty, and to determine boundaries for different levels of achievement

BackgroundFixed mark grade boundaries for non-linear assessment scales fail to account for variations in assessment difficulty. Where assessment difficulty varies more than ability of successive cohorts or the quality of the teaching, anchoring grade boundaries to median cohort performance should provide an effective method for setting standards.MethodsThis study investigated the use of a modified Hofstee (MH) method for setting unsatisfactory/satisfactory and satisfactory/excellent grade boundaries for multiple choice question-style assessments, adjusted using the cohort median to obviate the effect of subjective judgements and provision of grade quotas.ResultsOutcomes for the MH method were compared with formula scoring/correction for guessing (FS/CFG) for 11 assessments, indicating that there were no significant differences between MH and FS/CFG in either the effective unsatisfactory/satisfactory grade boundary or the proportion of unsatisfactory graded candidates (p > 0.05). However the boundary for excellent performance was significantly higher for MH (p < 0.01), and the proportion of candidates returned as excellent was significantly lower (p < 0.01). MH also generated performance profiles and pass marks that were not significantly different from those given by the Ebel method of criterion-referenced standard setting.ConclusionsThis supports MH as an objective model for calculating variable grade boundaries, adjusted for test difficulty. Furthermore, it easily creates boundaries for unsatisfactory/satisfactory and satisfactory/excellent performance that are protected against grade inflation. It could be implemented as a stand-alone method of standard setting, or as part of the post-examination analysis of results for assessments for which pre-examination criterion-referenced standard setting is employed.

Changing the guard: Staff turnover as a source of variation in test results

Due to variation in test difficulty, the use of pre-fixed cut-off scores in criterion-referenced standard setting methods may lead to variation in grades and pass rates. This paper aims to empirically investigate the strength of this relationship. To this end we examine a dataset of over 500 observations from an institution of higher education in The Netherlands over the period 2008–2013. We measure variation in test difficulty by using students’ perceptions of the validity of the examination and by recording personnel changes in the primary instructor. The latter measure is based on the considerable variation in teachers’ ability to assess test difficulty that is found in the literature. Other explanatory variables are course evaluations, instructor evaluations and self-reported study time. Variation in student quality is controlled for by measuring course results in deviation from the cohort average. We take a panel approach in estimating the effect of the explanatory variables on the variability in grades and pass rates. Our findings indicate that exam validity and instructor change are significantly related to variation in test results. The latter finding supports the hypothesis that instructors’ difficulty in assessing test difficulty may introduce subjectivity in criterion-referenced standard setting methods.

Fortune-tellers or content specialists: challenging the standard setting paradigm in medical education programmes

The veracity of Objective Standard Setting (OSS) as a modern approach to criterion-referenced standard setting has been reported for healthcare student assessment in the USA, while in other countries, OSS remains unrecognized. OSS upholds the foundational principle for itemized tests that judges should base their decisions on test item content. Moreover, it presents judges with a conceptually transparent decision procedure. This contrasts with the predictions concerning a hypothetical borderline candidate which typify Angoff procedures. Furthermore, the iterative process involved in the Angoff standard setting task incurs financial and administrative burdens, thus creating the potential to cut corners through recruiting fewer judges. The underlying objective of homogenizing the test standard undermines its validity, while circumventing reputable standard setting principles. While the Rasch model offers an objective approach to predicting successful outcomes, combining Rasch and Angoff procedures does not resolve the validity problem for Angoff-based pass marks. This commentary highlights the virtues of OSS relative to the modified Angoff approach in the standard setting of itemized tests. It also identifies gaps in the research literature that should be addressed to strengthen the case for using OSS on an international scale for high-stakes assessments within healthcare disciplines as a testing ground for other disciplines.

Standard setting of script concordance tests using an adapted Nedelsky approach

Background: Standard setting in assessment seeks to apply meaning of achievement to an assessment score. Appropriate standard setting for script concordance tests (SCTs) remains a challenge, with existing methods representing norm-referenced approaches.Aims: To develop a criterion-referenced standard setting approach for SCT using an adapted Nedelsky approach, to pilot feasibility, and to compare failure rates with two other methods.Methods: Second- and third-year medical students were administered a 45-question SCT and results collated. Standard setting was applied using three approaches: (1) norm-referenced (student cohorts), (2) expert-referenced (student cohort compared to expert mean), and (3) adapted Nedelsky approach using answer key normalization. Feasibility and failure rates were measured.Results: All standard setting approaches were feasible, with 60 additional minutes required for the Nedelsky standard setting exercise. Failure rates between the three approaches were similar (Year 2: 8.0–9.8% and Year 3: 2.1–7.6%), with the adapted Nedelsky approach representing an intermediate option (Year 2: 8.0% and Year 3: 3.5%).Conclusion: Standard setting SCT using the criterion-referenced method of an adapted Nedelsky approach was found to be both logically justifiable and logistically simple, and produced failure rates comparable to other currently utilized and less objective approaches.

A standard setting method with the best performing students as point of reference: Practical and affordable

Background: Teachers involved in test development usually prefer criterion-referenced standard setting methods using panels. Since expert panels are costly, standards are often set by a pre-fixed percentage of questions answered correctly or norm-referenced methods aimed at ranking examinees.Aim: To discuss the (dis)advantages of commonly used criterion and norm-referenced methods and present a new compromise method: standards based on a fixed cut-off score using the best scoring students as reference point.Methods: Historical data from 54 Maastricht (norm-referenced) and 52 Groningen (criterion-referenced) tests were used to demonstrate huge discrepancies and variability in cut-off scores and failure rates. Subsequently, the compromise model – known as Cohen's method – was applied to the Groningen tests.Results: The Maastricht norm-referenced method led to a large variation in required cut-off scores (15–46%), but a stable failure rate (about 17%). The Groningen method with a conventional, pre-fixed standard of 60% led to a large variation in failure rates (17–97%). The compromise method reduced variation in required cut-off scores as well as failure rates.Conclusion: Both the criterion and norm-referenced standards, used in practice, have disadvantages. The proposed compromise model reduces the disadvantages of both methods and is considered more acceptable. Last but not least, compared to standard setting methods using panels, this method is affordable.

Development and outcomes of the multiple-choice format national certification examination in nurse-midwifery and midwifery

The national certification examination in nurse-midwifery and midwifery now is offered exclusively in the multiple-choice format. The strategies used to develop equivalent examination forms are described. The manner of criterion-referenced standard setting is explained. Outcomes of the first five multiple-choice examination forms include an aggregate first-time failure rate of 8.3%. Multiple- and logistic-regression analyses indicate that examination scores of African-American and Hispanic ethnic candidates, and older candidates, are lower with regard to statistical significance; however, these differences are too modest to be of practical significance. No substantive relationship was observed between any demographic or academic variable and the probability of passing the examination. The experience gained by the ACNM Certification Council during development, administration, and evaluation of these first five multiple-choice examination forms has provided strong support for the transition to multiple-choice testing for the assessment of entry into practice of this competency-based profession.