Standard Setting Methodologies Research Articles

The educational value of situational judgement tests (SJTs) when used during undergraduate medical training: A systematic review and narrative synthesis

Introduction Situational judgement tests (SJTs) are a recognised assessment method for admission into medical school, selection into postgraduate training programs, and postgraduate competency assessment. More recently, however, SJTs have been used during undergraduate medical training (UMT). This systematic review identifies, describes, and appraises the evidence for SJTs in UMT to determine educational associations and outcomes. Methods MEDLINE, EMBASE, ERIC, PsycINFO, SCOPUS, Web of Science, and grey literature were searched for original research studies evaluating SJTs implemented within UMT to 1 November 2022. Studies reporting evaluation outcomes were included. Narrative data syntheses were undertaken. Risk of Bias was appraised using the Quality in Prognosis Studies tool. Results 24 studies were included. National database-derived SJTs (n = 14) assessed against professionalism, postgraduate attainment, construct of medical degree, medical school admissions scores, personality attributes, and declaration of disability. In-house derived SJTs (n = 10) assessed against professionalism, clinical skills, and personality attributes. Most evidence evaluated and reported inverse SJT associations with professionalism and were moderate risk of bias. Conclusion SJTs may have utility for developing professional behaviours in medical students. However, further research testing SJT robustness, standard setting methodologies, and prospectively evaluating SJTs against objective outcome measures within the context of UMT is warranted.

Passing a Technical Skills Examination in the First Year of Surgical Residency Can Predict Future Performance.

The ability of an assessment to predict performance would be of major benefit to residency programs, allowing for early identification of residents at risk. We sought to establish whether passing the Objective Structured Assessment of Technical Skills (OSATS) examination in postgraduate year 1 (PGY-1) predicts future performance. Between 2002 and 2012, 133 PGY-1 surgery residents at the University of Toronto (Toronto, Ontario, Canada) completed an 8-station, simulated OSATS examination as a component of training. With recently set passing scores, residents were assigned a pass/fail status using 3 standards setting methods (contrasting groups, borderline group, and borderline regression). Future in-training performance was compared between residents who had passed and those who failed the OSATS, using in-training evaluation reports from resident files. A Mann-Whitney U test compared performance among groups at PGY-2 and PGY-4 levels. Residents who passed the OSATS examination outperformed those who failed, when compared during PGY-2 across all 3 standard setting methodologies (P < .05). During PGY-4, only the contrasting groups method showed a significant difference (P < .05). We found that PGY-1 surgical resident pass/fail status on a technical skills examination was associated with future performance on in-training evaluation reports in later years. This provides validity evidence for the current PGY-1 pass/fail score, and suggests that this technical skills examination may be used to predict performance and to identify residents who require remediation.

Systematic review to establish absolute standards for technical performance in surgery.

Standard setting allows educators to create benchmarks that distinguish between those who pass and those who fail an assessment. It can also be used to create standards in clinical and simulated procedural skill. The objective of this review was to perform a systematic review of the literature using absolute standard-setting methodology to create benchmarks in technical performance. A systematic review was conducted by searching MEDLINE, Embase, PsycINFO and the Cochrane Database of Systematic Reviews. Abstracts of retrieved studies were reviewed and those meeting the inclusion criteria were selected for full-text review. The quality of evidence presented in the included studies was assessed using the Medical Education Research Study Quality Instrument (MERSQI), where a score of 14 or more of 18 indicates high-quality evidence. Of 1809 studies identified, 37 used standard-setting methodology for assessment of procedural skill. Of these, 24 used participant-centred and 13 employed item-centred methods. Thirty studies took place in a simulated environment, and seven in a clinical setting. The included studies assessed residents (26 of 37), fellows (6 of 37) and staff physicians (17 of 37). Seventeen articles achieved a MERSQI score of 14 or more of 18, whereas 20 did not meet this mark. Absolute standard-setting methodologies can be used to establish cut-offs for procedural skill assessments.

Establishing Passing Scores for Technical Performance in Surgery: Lessons Learned From a 10-Year Experience of Resident Assessment

METHODS: A retrospective analysis of prospectively collected OSATS performance data from 432 PGY 1 surgical residents from 2002-2012 was used for this study. Residents completed a multistation technical skills exam at the end of their PGY 1 year. Three standard setting methodologies were applied to the dataset to set passing scores 1) contrasting groups (CG); 2) borderline group (BG); 3) borderline regression (BR).

Clinical indications for musculoskeletal ultrasound: A Delphi-based consensus paper of the European society of musculoskeletal radiology

To develop clinical guidelines for musculoskeletal ultrasound (MSKUS) referral in Europe. Sixteen musculoskeletal radiologists from seven European countries participated in a consensus-based interactive process (Delphi method) using consecutive questionnaires and consensus procedure meetings at several European radiology meetings. The evaluation of musculoskeletal diseases was established by literature reviews, followed by consensus on clinical utility in three consensus meetings. This involved a thorough, transparent, iterative approach which including interview, questionnaire, Delphi and standard setting methodologies. European MSK radiologists with a special interest in MSKUS formed two different expert groups who worked on reaching a consensus in the first two meetings. The third meeting resolved questions that did not achieve a consensus level of 67% using the first two questionnaires. On expert consensus, the use of MSKUS is indicated to detect joint synovitis, fluid and septic effusion for potential aspiration, and poorly indicated to detect loose bodies. Recommendations for most appropriate use of musculoskeletal ultrasound are reported in six areas relevant to musculoskeletal ultrasound: hand/wrist, elbow, shoulder, hip, knee and ankle/foot. A comprehensive evidence-based, expert consensus-defined educational framework on clinical ultrsound is presented. This should facilitate referrals for this important imaging technique throughout Europe. Musculoskeletal ultrasound is indicated for detecting joint synovitis, effusions and fluid collections. • Musculoskeletal ultrasound is poor at detecting loose bodies. • Musculoskeletal ultrasound is relevant for most joints.

Impact of Standard Setting Methodologies over Passing Scores

In this study impact of standard setting methodologies over passing scores is studied and the changes on students’ pass/fail scores using different setting methodologies are researched. The data of the research in the first phase is collected from test results which are developed by researcher using 25 items per Turkish Language and Mathematics exams and applied to 129, 7th grade, Elementary School students. The data for second phase are collected by calculating cut points standing on different expert judgments per different methodology. It is seen by this study that the decision on standard setting methodologies for the students are related to students group, the structure of test methods and instructors

Setting Performance Standards for Mannequin-Based Acute-Care Scenarios

In medicine, standard setting methodologies have been developed for both selected-response and performance-based assessments. For simulation-based tasks, research efforts have been directed primarily at assessments that incorporate standardized patients. Mannequin-based evaluations often demand complex, time-sensitive, hierarchically ordered, sequential actions that are difficult to evaluate and score. Moreover, collecting reliable proficiency judgments, necessary to estimate meaningful cut points, can be challenging. The purpose of this investigation was to explore whether expert judgments obtained using an examinee-centered standard setting method that was previously validated for standardized patient-based assessments could be used to set defensible standards for acute-care, mannequin-based scenarios. Nineteen physicians were recruited to serve as panelists. For each of 12 simulation scenarios, between 8 and 10 performance samples (audio-video recordings), covering the expected ability continuum, were chosen for review. The performance samples were selected from a previously administered evaluation of postgraduate trainees. Based on a consensus definition of readiness to enter unsupervised practice, the panelists made independent judgments of each performance. For each scenario, the association between the panelists' judgments and the assessment scores was summarized and used to estimate a scenario-specific cut score. For 9 of the scenarios, there was at least a moderately strong relationship between the aggregate panelists' rating and the performance scores, thus allowing for estimation of meaningful numeric standards. For the other 3 scenarios, the aggregate decision rules used by the panelists did not correspond with the achievement measures. For scenarios independently rated by split panels, the estimated cut scores were similar. An examinee-centered approach, using aggregate expert judgments of audio-video performances, was suitable for setting standards on most acute-care, mannequin-based scenarios. It is necessary, however, to have valid scores for the chosen scenarios and to sample performances across the ability spectrum.

Setting the Passing Standard for the National Council Licensure Examination for Registered Nurses

The Board of Directors of the National Council of State Boards of Nursing evaluates the NCLEX-RN examination passing standard every 3 years. In December 2006, the Board decided that the NCLEX-RN passing standard should be increased from -0.2800 to -0.2100 logits. This article provides information regarding standard-setting methodologies, empirical data used by the Board of Directors in determining the passing standard, and implications of the new passing standard for faculty.

The development of an evidence-based educational framework to facilitate the training of competent rheumatologist ultrasonographers

Musculoskeletal ultrasonography (MSKUS) has been described by some rheumatologists as the 'stethoscope of the joint'. Such enthusiasm is supported by evidence confirming validity and clinical utility in evaluation of musculoskeletal diseases. But if rheumatologist-performed MSKUS is to emulate the impact of cardiologist-performed echocardiography, a number of educational challenges need be addressed. Evaluation of current training reveals the absence of a unified educational structure, ad hoc teaching and assessment and published data are insufficient to make practice and training recommendations specific to rheumatology. Informed by developments in adult-learning theory, we have utilized a competency-based approach to develop an educational programme for rheumatologist ultrasonographers. Fundamental to this process has been accurate functional analysis of the role of the rheumatologist ultrasonographer and precise translation of these data into educational outcomes. This involved a thorough, transparent, iterative, curriculum-defining approach, employing quantitative and qualitative research techniques including interview, questionnaire, Delphi and standard setting methodologies. All relevant stakeholders were engaged, including international US experts and clinical rheumatologists. Outcomes include clarification of the role of a rheumatologist ultrasonographer; definition of appropriate knowledge and skills; establishment of competency standards; a balanced, clearly defined, clinically relevant educational outcome blueprint. Teaching and assessment approaches have been piloted as part of an accessible modular curriculum strategy. Thorough validation and evaluation confirms effectiveness, efficiency and suitability. A comprehensive evidence-based, expert consensus-defined educational framework is proposed that provides a template for teaching and learning and standards for competency assessment. This should facilitate common principles of training, uniform professional practice and a justifiable governance structure.

Alternative Approaches to Standard Setting for Licensing and Certification Examinations

In the most frequently used of Angoff's standard setting methodologies, judges develop a common understanding of minimally competent candidates and estimate the percentage of minimally competent candidates who would answer each item of an examination correctly. This method (percentage) and another Angoff method (yes-no) were explored in this study. Four separate groups of judges were provided behavioral descriptors or incidents to use when making ratings. Judges were asked to estimate the percentage of minimally competent candidates who would answer each item of an examination correctly (percentage) or to provide dichotomous ratings that affirmed or disaffirmed that minimally competent candidates would answer each item correctly (yes-no). The groups were allowed to make ratings in two stages: initial ratings without discussion and final ratings after presentation of item difficulty data and discussion with group members. Results indicated that the passing scores based on percentage estimates were stable from initial to final ratings. However, passing scores based on dichotomous (yes-no) ratings declined from initial to final ratings. Apparently, the judges using the yes-no method changed their judgments systematically after provision of item difficulty data and opportunity for discussion. The nature of the differences and the implications for setting passing scores for licensing examinations are discussed.

Developments in wastewater discharge standard approaches

This paper attempts to evaluate the new demands and developments in the field of pollution control, as they specifically relate to discharge standards. The new concepts of pollution control and the requirement of newer criteria are discussed. Existing standard setting methodologies are evaluated considering the new concepts and demands. A new methodology which is based on the reconstruction of pollution based category of industries and definition of appropriate technologies for the micropollutants is proposed.

A Validation of Ebel's Method for Performance Standard Setting through its Application with Comparison Approaches to a Selected Criterion-Referenced Test

This paper seeks to describe the research and development into performance standard setting processes used by the U.S. Navy's Strategic Weapon System Training Program for developing Criterion-Referenced System Achievement Tests (C-R SAT) for selected Naval Enlisted Classifications (NECs) or job titles in the Strategic Weapon System (SWS). The design of the C-R SAT involves identification of essential skills and knowledge for each NEC. This process entails planned workshops involving selected senior fleet and supervisory personnel drawn from the NEC. Following this test development process each C-R SAT is then subjected to a performance standard setting process. As all performance standard setting methodologies are subjective, the decision was made to validate the Ebel process and its outputs. Two alternative performance standard setting methodologies were applied to data generated from the two versions (watch stander and supervisor) of a recently administered TRIDENT Technician C-R SAT.