Frame-of-reference Training Research Articles

Faculty Perceptions of Frame of Reference Training to Improve Workplace-Based Assessment.

Workplace-based assessment (WBA) is a key assessment strategy in competency-based medical education. However, its full potential has not been actualized secondary to concerns with reliability, validity, and accuracy. Frame of reference training (FORT), a rater training technique that helps assessors distinguish between learner performance levels, can improve the accuracy and reliability of WBA, but the effect size is variable. Understanding FORT benefits and challenges help improve this rater training technique. To explore faculty's perceptions of the benefits and challenges associated with FORT. Subjects were internal medicine and family medicine physicians (n=41) who participated in a rater training intervention in 2018 consisting of in-person FORT followed by asynchronous online spaced learning. We assessed participants' perceptions of FORT in post-workshop focus groups and an end-of-study survey. Focus groups and survey free text responses were coded using thematic analysis. All subjects participated in 1 of 4 focus groups and completed the survey. Four benefits of FORT were identified: (1) opportunity to apply skills frameworks via deliberate practice; (2) demonstration of the importance of certain evidence-based clinical skills; (3) practice that improved the ability to discriminate between resident skill levels; and (4) highlighting the importance of direct observation and the dangers using proxy information in assessment. Challenges included time constraints and task repetitiveness. Participants believe that FORT training serves multiple purposes, including helping them distinguish between learner skill levels while demonstrating the impact of evidence-based clinical skills and the importance of direct observation.

Investigating the Impact of Rater Training on Rater Errors in the Process of Assessing Writing Skill

In the process of measuring and assessing high-level cognitive skills, interference of rater errors in measurements brings about a constant concern and low objectivity. The main purpose of this study was to investigate the impact of rater training on rater errors in the process of assessing individual performance. The study was conducted with a pretest-posttest control group quasi-experimental design. In this research, 45 raters were employed, 23 from the control group and 22 from the experimental group. As data collection tools, a writing task that was developed by IELTS and an analytical rubric that was developed to assess academic writing skills were used. As part of the experimental procedure, rater training was provided and this training was implemented by combining rater error training and frame of reference training. When the findings of the study were examined, it was found that the control and experimental groups were similar to each other before the experiment, however, after the experimental process, the study group made more valid and reliable measurements. As a result, it was investigated that the rater training given had an impact on rater errors such as rater severity, rater leniency, central tendency, and Halo effect. Based on the obtained findings, some suggestions were offered for researchers and future studies.

Development of a New Personality-Oriented Work Analysis Questionnaire: First Steps Towards Validation

In this paper we detail the construction and validation process for a new personality-oriented work analysis instrument, in the form of a standardized questionnaire, based on extant research that shows that personality traits are good predictors of job performance. We present the process of item development, frame of reference training, rating scale creation, and the selection of subject matter experts. By administering the instrument to three distinct positions, the interrater reliability coefficients resulted between .80 and .94. We also investigated the instrument’s ability to discriminate between the same rated positions, and the results for this indicator were quite low. Conclusions provide some possible explanations for the lower resulted discriminability. Practical and theoretical implications are discussed as well as other future research for general improvement of data quality.

Ethnic and gender bias in objective structured clinical examination: A critical review

This critical review aimed to synthesize the literature and critique the strength of the methodological quality of current evidence regarding examiner bias related to ethnicity and gender in objective structured clinical examination implemented in health professions education. The Guidelines for Critical Review (GCR) was used to critically appraise the selected studies. Ten studies were retrieved for review. The overall quality of the papers was moderate. Two studies met all the criteria of the GCR, indicating stronger evidence of their outcomes. One of them reported ethnic and gender bias potentially existing, while another found only one examiner showing consistent ethnic bias. No systematic bias was found across the studies. Nonetheless, the possibility of ethnic or gender bias by some examiners cannot be ignored. To mitigate potential examiner bias, the investigation of implicit bias training, frame of reference training, the use of multiple examiners, and combination assessments are suggested.

Rater training: Understanding effects of training content, practice ratings, and feedback

Despite the popularity of frame‐of‐reference training (FORT), it is not clear how different structural elements of FORT work in concert to improve rating accuracy. Furthermore, past rater training studies have lacked rigorous control groups leading to low thresholds for showing improvements in rating accuracy due to FORT. The current study allowed for the isolation of components of rater training that increase rating accuracy when compared to a rigorously designed control group. Results indicated that repeated rendering of practice ratings improve rating accuracy and this practice effect was amplified by practice rating feedback. Although accuracy‐based training content improved interrater agreement, it did not contribute to improvements rating accuracy over and above the control group. We discuss the implications of the findings in relation to best practices for designing rater training programs.

How faculty members experience workplace-based assessment rater training: a qualitative study.

Direct observation of clinical skills is a common approach in workplace-based assessment (WBA). Despite widespread use of the mini-clinical evaluation exercise (mini-CEX), faculty development efforts are typically required to improve assessment quality. Little consensus exists regarding the most effective training methods, and few studies explore faculty members' reactions to rater training. This study was conducted to qualitatively explore the experiences of faculty staff with two rater training approaches - performance dimension training (PDT) and amodified approach to frame of reference training (FoRT) - to elucidate how such faculty development can be optimally designed. In a qualitative study of a multifaceted intervention using complex intervention principles, 45 out-patient resident faculty preceptors from 26 US internal medicine residency programmes participated in a rater training faculty development programme. All participants were interviewed individually and in focus groups during and after the programme to elicit how the training influenced their approach to assessment. A constructivist grounded theory approach was used to analyse the data. Many participants perceived that rater training positively influenced their approach to direct observation and feedback, their ability to use entrustment as the standard for assessment, and their own clinical skills. However, barriers to implementation and change included: (i) a preference for holistic assessment over frameworks; (ii) challenges in defining competence; (iii) difficulty in changing one's approach to assessment, and (iv) concerns about institutional culture and buy-in. Rater training using PDT and a modified approach to FoRT can provide faculty staff with assessment skills that are congruent with principles of criterion-referenced assessment and entrustment, and foundational principles of competency-based education, while providing them with opportunities to reflect on their own clinical skills. However, multiple challenges to incorporating new forms of training exist. Ongoing efforts to improve WBA are needed to address institutional and cultural contexts, and systems of care delivery.

Performans Değerlendirmeye Karışan Puanlayıcı Etkilerini Azaltmanın Yollarından Biri Olarak Puanlayıcı Eğitimleri: Kuramsal Bir Analiz

Rater related factors which influence students' performance scores are called rater effects. Maintaining an acceptable reliability level in performance assessment is possible by keeping the rater effect at a minimum. In order to decrease rater effects, more than one rater should be used in scoring and scoring should be based on rubrics. Another suggestion would be to use rater training. Through rater training, the rating scales to be used in the assessment can be introduced to raters, sample scoring practices can be carried out and thus a common understanding can be formed among the raters. The present study focuses on rater training and types of rater training. Rater training is presented under four titles: rater error training, performance dimensions training, behavior observation training and frame of reference training. An analysis of the literature reveals that no study was conducted on rater training in the Turkish context. Therefore, it is thought that the present study will be of use to both teachers and researchers in the field of educational sciences in terms of introducing rater training

Evidence for the Effectiveness of an Alternative Multisource Performance Rating Methodology

Despite persistent concerns as to the quality of performance information obtained from multisource performance ratings (MSPRs), little research has sought ways to improve the psychometric properties of MSPRs. Borrowing from past methodologies designed to improve performance ratings, we present a new method of presenting items in MSPRs, frame‐of‐reference scales (FORS), and test the efficacy of this method in a field and lab study. The field study used confirmatory factor analysis to compare the FORS to traditional rating scales and revealed that FORS are associated with increased variance due to dimensions, decreased overlap among dimensions, and decreased error. The laboratory study compared rating accuracy associated with FORS relative to frame‐of‐reference training (FORT) and a control group and demonstrated that FORS are associated with higher levels of accuracy than the control group and similar levels of accuracy as FORT. Implications for the design and implementation of FORS are discussed.

Effect of Rater Training on Reliability and Accuracy of Mini-CEX Scores: A Randomized, Controlled Trial

Mini-CEX scores assess resident competence. Rater training might improve mini-CEX score interrater reliability, but evidence is lacking. Evaluate a rater training workshop using interrater reliability and accuracy. Randomized trial (immediate versus delayed workshop) and single-group pre/post study (randomized groups combined). Academic medical center. Fifty-two internal medicine clinic preceptors (31 randomized and 21 additional workshop attendees). The workshop included rater error training, performance dimension training, behavioral observation training, and frame of reference training using lecture, video, and facilitated discussion. Delayed group received no intervention until after posttest. Mini-CEX ratings at baseline (just before workshop for workshop group), and four weeks later using videotaped resident-patient encounters; mini-CEX ratings of live resident-patient encounters one year preceding and one year following the workshop; rater confidence using mini-CEX. Among 31 randomized participants, interrater reliabilities in the delayed group (baseline intraclass correlation coefficient [ICC] 0.43, follow-up 0.53) and workshop group (baseline 0.40, follow-up 0.43) were not significantly different (p = 0.19). Mean ratings were similar at baseline (delayed 4.9 [95% confidence interval 4.6-5.2], workshop 4.8 [4.5-5.1]) and follow-up (delayed 5.4 [5.0-5.7], workshop 5.3 [5.0-5.6]; p = 0.88 for interaction). For the entire cohort, rater confidence (1 = not confident, 6 = very confident) improved from mean (SD) 3.8 (1.4) to 4.4 (1.0), p = 0.018. Interrater reliability for ratings of live encounters (entire cohort) was higher after the workshop (ICC 0.34) than before (ICC 0.18) but the standard error of measurement was similar for both periods. Rater training did not improve interrater reliability or accuracy of mini-CEX scores. clinicaltrials.gov identifier NCT00667940

Frame of Reference Training for Assessment Centers: Effects on Interrater Reliability When Rating Behaviors and Ability Traits

Assessment centers have been widely criticized on the basis of measurement problems. The present study sought to present a methodological piece on the extent to which Frame of Reference (FOR) training would increase the interrater reliability associated with assessment center ratings provided by non-psychologist assessors. Five managerial assessors (with no psychological training) rated the behavior and the ability traits of a contrived participant on the basis of behaviors described in two alternative vignettes (detailing critical incidents of job performance). The ratings were obtained both before and following FOR training. It was found that agreement among assessors on their assessment of both behaviors and traits increased subsequent to the FOR training procedure. The implications of increasing the precision associated with assessment center ratings are discussed.

Frame of reference rater training issues: recall, time and behavior observation training

This study investigated three controversial issues regarding frame of reference training (FOR), a type of rater training: (1) Does it improve behavioral recall, (2) Can it be improved by incorporating behavior observation training (BOT), and (3) Can its effects persist over time? Results suggested that even though FOR training increases the number of behaviors recalled, it does not necessarily improve the quality of the recalled behaviors, but a combined FOR + BOT program does improve recall quality. Lastly, results suggested that FOR training can improve rating accuracy, even after a two‐week delay between rater training and the rating task.