Angoff Standard Setting Procedure Research Articles

What Is the Minimal Competency for a Clinical Ethics Consult Simulation? Setting a Standard for Use of the Assessing Clinical Ethics Skills (ACES) Tool

Background: The field of clinical ethics is examining ways of determining competency. The Assessing Clinical Ethics Skills (ACES) tool offers a new approach that identifies a range of skills necessary in the conduct of clinical ethics consultation and provides a consistent framework for evaluating these skills. Through a training website, users learn to apply the ACES tool to clinical ethics consultants (CECs) in simulated ethics consultation videos. The aim is to recognize competent and incompetent clinical ethics consultation skills by watching and evaluating a videotaped CEC performance. We report how we set a criterion cut score (i.e., minimally acceptable score) for judging the ability of users of the ACES tool to evaluate simulated CEC performances.Methods: A modified Angoff standard-setting procedure was used to establish the cut score for an end-of-life case included on the ACES training website. The standard-setting committee viewed the Futility Case and estimated the probability that a minimally competent CEC would correctly answer each item on the ACES tool. The committee further adjusted these estimates by reviewing data from 31 pilot users of the Futility Case before determining the cut score.Results: Averaging over all 31 items, the proposed proportion correct score for minimal competency was 80%, corresponding to a cut score that is between 24 and 25 points out of 31 possible points. The standard-setting committee subsequently set the minimal competency cut score to 24 points.Conclusions: The cut score for the ACES tool identifies the number of correct responses a user of the ACES tool training website must attain to “pass” and reach minimal competency in recognizing competent and incompetent skills of the CECs in the simulated ethics consultation videos. The application of the cut score to live training of CECs and other areas of practice requires further investigation.

Have OSCEs come of age in nursing education?

This article is intended to contribute to the current debate as to whether the objective structured clinical examination (OSCE) should become a standard assessment tool for undergraduate nursing education as they currently are for medicine. The authors describe how one UK university developed an OSCE for a nursing undergraduate programme with the aim of emphasising the need for nursing students to be competent in clinical skills and offering a means of standardising the assessment of these skills. There has been an increasing number of research studies carried out in this area at international level and this article's main contribution to the literature is the description of the Angoff standard-setting procedure that was used to calibrate the OSCE at this University and which makes it the first nursing OSCE in the UK to incorporate a scientific standard-setting procedure.

Requiring a Consistent Unit of Scale Between the Responses of Students and Judges in Standard Setting

One of the best-known methods for setting a benchmark standard on a test is that of Angoff and its modifications. When scored dichotomously, judges estimate the probability that a benchmark student has of answering each item correctly. As in most methods of standard setting, it is assumed implicitly that the unit of the latent scale of the responses of the judges is the same as that of the responses of the students. In an empirical example using a modified Angoff approach, this article shows that this assumption is not correct. In particular, using the dichotomous Rasch model, the article shows that the unit of the latent scale for the students is smaller than that of the judges. Accordingly, to obtain an unbiased estimate of a standard, the article shows that it is necessary to place the judges' responses onto a latent scale with the same unit as that of the students. Implications for standard setting procedures are considered, including understanding some observed difficulties reported in the literature in applying Angoff's standard setting procedure.

The Impact of Process Instructions on Judges’ Use of Examinee Performance Data in Angoff Standard Setting Exercises

Despite being widely used and frequently studied, the Angoff standard setting procedure has received little attention with respect to an integral part of the process: how judges incorporate examinee performance data in the decision-making process. Without performance data, subject matter experts have considerable difficulty accurately making the required judgments. Providing data introduces the very real possibility that judges will turn their content-based judgments into norm-referenced judgments. This article reports on three Angoff standard setting panels for which some items were randomly assigned to have incorrect performance data. Judges were informed that some of the items were accompanied by inaccurate data, but were not told which items they were. The purpose of the manipulation was to assess the extent to which changing the instructions given to the judges would impact the extent to which they relied on the performance data. The modified instructions resulted in the judges making less use of the performance data than judges participating in recent parallel studies. The relative extent of the change judges made did not appear to be substantially influenced by the accuracy of the data.

An application of Multifaceted Rasch measurement in the Yes/No Angoff standard setting procedure

When implementing standard setting procedures, there are two major concerns: variance between panelists and efficiency in conducting multiple rounds of judgments. With regard to the former, there is concern over the consistency of the cutoff scores made by different panelists. If the cut scores show an inordinately wide range then further rounds of group discussion are required to reach consensus, which in turn leads to the latter concern. The Yes/No Angoff procedure is typically implemented across several rounds. Panelists revise their original decisions for each item based on discussion with co-panelists between each round. The purpose of this paper is to demonstrate a framework for evaluating the judgments in the standard setting process. The Multifaceted Rasch model was applied as a tool to evaluate the quality of standard setting in a context of language assessment. The results indicate that the Multifaceted Rasch model offers a promising approach to examination of the variability in the standard setting procedures. In addition, this model can identify aberrant decision making for each panelist, which can be used as feedback for both standard setting designers and panelists.

Using the Many-Faceted Rasch Model to Evaluate Standard Setting Judgments

The many-faceted Rasch (MFR) model has been used to evaluate the quality of ratings on constructed response assessments; however, it can also be used to evaluate the quality of judgments from panel-based standard setting procedures. The current study illustrates the use of the MFR model for examining the quality of ratings obtained from a standard setting process employing the Multiple Yes-No, modified Angoff standard setting procedure to set the cut scores on the Advanced Placement Environmental Science exam. The panelists’ severity, judged item difficulties, rounds, and cut scores using the MFR model were examined, in addition to panelists’ characteristics (gender and level of course taught). Findings from the Rasch analyses conducted in this study provide evidence of acceptable quality of the ratings obtained from the panelists. Thus, the current study provides an important source of both internal and procedural validity evidence for the Advanced Placement Environmental Science standard setting as well as an illustration of the utility of the MFR model for evaluating standard setting judgments.

Establishing and applying performance standards for curriculum-based examinations

This paper describes how a state education system in Australia introduced standards-referenced assessments into its large-scale, high-stakes, curriculum-based examinations in a way that enables comparison of performance across time even though the examinations are different each year. It describes the multi-stage modified Angoff standard-setting procedure used to establish cut-off scores on subject examinations, and how the results from this exercise were then used to develop standards packages. These packages illustrate the performances of students at the borders between the various bands. The paper also shows how originally it was intended to use a Rasch measurement model to create the statistical feedback used in the standard-setting procedure. It also describes the modifications to the feedback that were necessary to meet the real-time constraints of this large-scale examination programme. It argues that consideration should now be given to using the Rasch model to provide this feedback instead of the current approach.

An Empirical Examination of the Impact of Group Discussion and Examinee Performance Information on Judgments Made in the Angoff Standard-Setting Procedure

Numerous studies have compared the Angoff standard-setting procedure to other standard-setting methods, but relatively few studies have evaluated the procedure based on internal criteria. This study uses a generalizability theory framework to evaluate the stability of the estimated cut score. To provide a measure of internal consistency, this study also compares the estimated proportion correct scores resulting from the Angoff exercise to empirical conditional proportion correct scores. In this research, judges made independent estimates of the proportion of minimally proficient candidates that would be expected to answer each item correctly; they then discussed discrepancies and revised their estimates. Discussion of discrepancies decreased the variance components associated with the judge and judge-by-item effects, indicating increased agreement between judges, but it did not improve the correspondence between the judgments and the empirical proportion correct estimates. The judges then were given examinee performance information for a subset of the items. Subsequent ratings showed a substantial increase in correspondence with the empirical conditional proportion correct estimates. Particular attention is given to examining the discrepancy between the judgments and empirical proportion correct estimates as a function of item difficulty.

Item Selection Strategy for Reducing the Number of Items Rated in an Angoff Standard Setting Study

In an Angoff standard setting procedure, judges estimate the probability that a hypothetical randomly selected minimally competent candidate will answer correctly each item in the test. In many cases, these item performance estimates are made twice, with information shared with the panelists between estimates. Especially for long tests, this estimation process can be time-consuming and fatiguing for the judges. This study extended an item selection strategy earlier proposed to form subsets of test items. The results of this study suggest that 40% to 50% of test items may be sufficient to estimate an equivalent passing score in an Angoff setting study when those items are selected to represent the full test in content, discrimination, and difficulty.

The Use of Subsets of Test Questions in an Angoff Standard-Setting Method

In an Angoff standard-setting procedure, judges estimate the probability that a hypothetical randomly selected minimally competent candidate will answer correctly each item constituting the test. In many cases, these item performance estimates are made twice, with information shared with the judges between estimates. Especially for long tests, this estimation process can be time consuming and fatiguing for the participants. This study addressed the possibilities of reducing the number of items individual judges consider in an Angoff standard-setting study. The results of this study suggest that 50% of test items may be sufficient to estimate an equivalent passing score in an Angoff setting study. This could result in substantial savings of time and resources not only for the agencies that have been carrying out this activity but also for the practitioners (judges) who participate in standard-setting studies.

A model for setting performance standards for standardized patient examinations.

Setting standards for complex performance assessments is difficult. This report describes and evaluates a model for setting performance standards for standardized patient examinations. A variation of a modified Angoff standard-setting procedure was used to set cutoffs for a standardized patient exercise. Analysis of the procedure was based on evaluation of the impact of the iterative process, the ability of the final standards to accurately classify trainees and the judges' confidence in the procedure and final standards. The practicality of the process also was assessed. The iterative steps in this procedure led to a reduction in the variance between the judges' ratings. All seven judges reported confidence in the final standards. The mean time commitment for each judge was 4.7 work hours. This model for setting performance standards successfully set useful standards for this exam, was practical, and can be utilized to set performance standards for other standardized patient examinations.

Panel expertise for an Angoff standard setting procedure in progress testing: item writers compared to recently graduated students.

An earlier study showed that an Angoff procedure with > or = 10 recently graduated students as judges can be used to estimate the passing score of a progress test. As the acceptability and feasibility of this approach are questionable, we conducted an Angoff procedure with test item writers as judges. This paper reports on the reliability and credibility of this procedure and compares the standards set by the two different panels. Fourteen item writers judged 146 test items. Recently graduated students had assessed these items in a previous study. Generalizability was investigated as a function of the number of items and judges. Credibility was judged by comparing the pass/fail rates associated with the Angoff standard, a relative standard and a fixed standard. The Angoff standards obtained by item writers and graduates were compared. The variance associated with consistent variability of item writers across items was 1.5% and for graduate students it was 0.4%. An acceptable error score required 39 judges. Item-Angoff estimates of the two panels and item P-values correlated highly. Failure rates of 57%, 55% and 7% were associated with the item writers' standard, the fixed standard and the graduates' standard, respectively. The graduates' and the item writers' standards differed substantially, as did the associated failure rates. A panel of 39 item writers is not feasible. The item writers' passing score appears to be less credible. The credibility of the graduates' standard needs further evaluation. The acceptability and feasibility of a panel consisting of both students and item writers may be worth investigating.

Reliability and credibility of an angoff standard setting procedure in progress testing using recent graduates as judges.

Progress testing is an assessment method that samples the complete domain of knowledge that is considered pertinent to undergraduate medical education. Because of the comprehensive nature of this test, it is very difficult to set a passing score. We obtained a progress test standard using an Angoff procedure with recent graduates as judges. This paper reports on the reliability and credibility of this approach. The Angoff procedure was applied to a sample of 146 progress test items. The items were judged by a panel of eight recently graduated students. Generalizability theory was used to investigate the reliability as a function of the number of items and judges. Credibility was judged by comparing the pass/fail rates resulting from the standard arrived at by the Angoff procedure with those obtained using a relative and a fixed standard. The results indicate that an acceptable error score can be achieved, yielding a precision within one percentage on the scoring scale, by using 10 judges on a full-length progress test (i.e. 250 items). The pass/fail rates associated with the Angoff standard came closest to those of the relative standard, which takes variations in test difficulty into account. A high correlation was found between item-Angoff estimates and the item P-values. The results of this study suggest that the Angoff procedure, using recently graduated students as judges, is an appropriate standard setting method for a progress test.

The incorporation of empirical item difficulty data into the Angoff standard-setting procedure.

This study represents an attempt at incorporating empirical item difficulty data into the Angoff standard-setting procedure without affecting the subjective judgment of the raters. The Rasch-model ability level corresponding to minimal competence was estimated for each of 536 items on the American Association of State Social Work Boards (AASSWB) social work licensure examinations from their empirical calibrations and Angoff ratings. The mean of these estimates for all items on a given examination was taken as the level of minimal competence of the entire examination. This procedure yielded raw passing scores that were 4 to 6 items lower (out of 150) and pass rates that were 7% to 15% higher than those obtained using the "standard" Angoff procedure.

The passing score in the objective structured clinical examination.

The objective structured clinical examination (OSCE) now has an established place in the assessment of the medical undergraduate. While much has been written about the reliability of the OSCE, empirical work on the determination of the passing score which represents competence on the OSCE is rarely encountered. If the OSCE is to play its role in the 'high stakes' testing of clinical competence, it is important that this passing score be set reliably and defensibly. This article illustrates how a two-session modified Angoff standard-setting procedure is used to set the passing score on a 14 station Obstetrics and Gynaecology OSCE used to assess final year students at The Queen's University of Belfast. The Angoff methodology harnesses the professional judgement of expert judges to establish defensible standards. Four university teachers, five non-academic consultants and six junior clinical staff took part in a two-session Angoff standard-setting procedure. In the first session, the judges (individually and in silence) used their professional judgement to estimate the score which a minimally competent final year obstetrics and gynaecology student should achieve on each tested element of the OSCE. In the second session they revised their session 1 judgements in the light of the OSCE scores of real students and the opportunity for structured discussion. The passing score for the OSCE is reported together with the statistical measures which assure its reliability.

A Comparison of the Nedelsky and Modified Angoff Standard-Setting Procedure On Evaluation Outcome

In criterion-referenced evaluation, several procedures for setting absolute standards of performance are available. However, it is not known if a given standard-setting procedure will yield consistent evaluation outcomes. To determine this, one of three parallel certifying examinations was administered to three classes of second year medical students. The examinations contained two item types and the standards were set by two procedures. The student responses to both item types were analyzed by the Rasch latent trait model. The results indicated that the standard-setting procedure is a significant factor in the determination of the evaluation outcome.