Generalizability Analysis Research Articles

Implementation of the ACS/ APDS Resident Skills Curriculum reveals a need for rater training: An analysis using generalizability theory

BackgroundThe American College of Surgeons (ACS)/Association of Program Directors in Surgery (APDS) Resident Skills Curriculum includes validated task-specific checklists and global rating scales (GRS) for Objective Structured Assessment of Technical Skills (OSATS). However, it does not include instructions on use of these assessment tools. Since consistency of ratings is a key feature of assessment, we explored rater reliability for two skills. MethodsSurgical faculty assessed hand-sewn bowel and vascular anastomoses in real-time using the OSATS GRS. OSATS were video-taped and independently evaluated by a research resident and surgical attending. Rating consistency was estimated using intraclass correlation coefficients (ICC) and generalizability analysis. ResultsThree-rater ICC coefficients across 24 videos ranged from 0.12 to 0.75. Generalizability reliability coefficients ranged from 0.55 to 0.8. Percent variance attributable to raters ranged from 2.7% to 32.1%. Pairwise agreement showed considerable inconsistency for both tasks. ConclusionsVariability of ratings for these two skills indicate the need for rater training to increase scoring agreement and decrease rater variability for technical skill assessments.

The findings of a surgical hip fracture trial were generalizable to the UK national hip fracture database

ObjectiveTo estimate the generalizability of treatment effects observed in a randomized trial of hip fracture surgery implants to a broader population of people undergoing hip surgery in the United Kingdom. Study Design and SettingIn 2018, the WHiTE-3 trial (n = 958) demonstrated that a modular hemiarthroplasty implant conferred no additional benefit over the traditional monoblock implant for quality of life and length of hospital stay. We compared and weighted the trial sample against two target populations: WHiTE-cohort (n = 2,457) and UK-National Hip Fracture Database (NHFD, n = 190,894), and re-estimate expected treatment effects for the target populations. ResultsDespite differences in baseline characteristics of the trial sample and target populations, the re-estimated treatment effects were comparable. For quality of life, the differences between the trial estimate and WHiTE-cohort and NHFD estimates were 0.01 points on the EuroQol (EQ5D). For length of stay, the difference between the trial estimate and WHiTE-cohort was 0.50 days; and the difference between the trial estimate and NHFD estimate was −0.47 days. ConclusionThis generalizability analysis of the WHiTE-3 trial found that the inferences from the trial can be generalized to a wider population of individuals in the UK NHFD and the WHiTE-cohort who met the inclusion criteria for WHiTE-3.

교사평정을 활용한 초등학생의 그릿(Grit) 측정에서 나타나는 요인구조와 일반화가능도 분석

교사평정을 활용한 초등학생의 그릿(Grit) 측정에서 나타나는 요인구조와 일반화가능도 분석

Analysis of Reliability and Generalizability of One Instrument for Assessing Visual Attention Span: MenPas Mondrian Color

Attention is one skill related to processes such as memory or learning, so, its evaluation is very interesting in areas such as clinical, educational or sports. The aim of this paper is to analyze the reliability and generalizability of one online computerized tool, named MenPas Mondrian Color, that has been developed for the visual attention span assessing and training. In addition, it has been intended to determine any existing relationships among the different parameters of the tasks performed in order to check the coherence of the results obtained in the executions. In 11,540 analyzed executions of 1064 users from different American, African and European countries, 6543 of them were performed by women (56.70%) and 4997 by men (43.30%). The age distribution showed that all of the participants were aged 18–55 years, with an average of 25.50 ± 8.91 years. The analyzed tool is called MenPas Mondrian Color which is included in the MENPAS 1.0 platform. Reliability (Cronbach’s Alpha), variance components and generalizability analyses were carried out in order to analyze the quality of the data gathered by this tool. The obtained results indicated optimal scores in the analyses performed, suggesting that the data gathered are reliable, precise and statistically generalizable to a larger population. Likewise, correlation analyses indicated that the difficulty of the task is related to the effectiveness in its executions, indicating that this is a highly sensitive tool.

Extending Inferences from a Cluster‐Randomized Trial to a Target Population

Research ObjectiveIn cluster‐randomized trials, participating clusters are not necessarily representative of the target population of trial‐eligible clusters where the experimental treatments might be applied. To the extent that participating and nonparticipating clusters differ in terms of covariates that are modifiers of the treatment effect, the average treatment effect in the trial may not apply to the entire target population. We describe analyses that extend (generalize or transport) causal inferences from cluster‐randomized trials to a target population of clusters, under a general nonparametric model that allows for arbitrary within‐cluster dependence.Study DesignWe consider study designs where a subset of a cohort of clusters are invited and agree to participate in a randomized trial with cluster‐level treatment assignment (ie, nested trial designs). Treatment and outcome data need only be collected from clusters participating in the trial, but data on baseline covariates must be collected from the entire cohort. We propose doubly robust estimators of potential outcome means in the target population that exploit individual‐level data on covariates and outcomes to improve efficiency and are appropriate for use with machine learning methods.Population StudiedWe illustrate the methods using a 2 × 2 factorial cluster‐randomized trial of influenza vaccination strategies conducted in 818 nursing homes nested in a cohort of 4,475 trial‐eligible Medicare‐certified nursing homes. To be eligible for the trial, nursing homes had to be free‐standing facilities (ie, not hospital based) within 50 miles of a CDC‐reporting city, have a minimum of 50 long‐stay residents, and at least 80% of residents aged 65 years or older.Principal FindingsWe applied the methods to estimate the potential outcome means for the four vaccination strategies in the target population of trial‐eligible nursing homes. We estimated the conditional probability of trial participation (nursing home level) and the conditional probability of death from any cause (individual level, separately for each treatment group) using different methods to illustrate that our approach can easily accommodate them. One set of analyses used logistic regression models estimated by maximum likelihood methods. For comparison, we also estimated probabilities using various machine learning methods. Overall, the doubly robust estimator produced similar point estimates for the potential outcome mean under each treatment across all approaches for estimating the conditional probabilities of trial participation and death. Point estimates for generalizability analyses were largely similar to those from the randomized trial, suggesting that the trial‐only estimates were applicable to the broader population of trial‐eligible nursing homes for the outcome of all‐cause death.ConclusionsRecent work on extending causal inferences from individually randomized trials to a target population can be extended to the cluster trial setting, under a general causal model that allows for arbitrary within‐cluster dependence.Implications for Policy or PracticeWhen cluster‐randomized trials are embedded in large health care systems, investigators can use the methods we describe to extend causal inferences from the trials to the broader health care system.Primary Funding SourcePatient‐Centered Outcomes Research Institute.

Mixed-Methods Analysis of Emotional Quality in Sports Organizations: Facial Expressions of Child Users of Sports Services as Data.

Proper quality management of sports services requires knowledge of the needs, perceptions, expectations, and emotions of users. Facial expressions provide relevant information about the immediate perception of stimuli. The objective of this study was therefore to analyze how a group of child users of sports services expressed their satisfaction with the programs in which they participated through facial expressions during an in-depth interview tailored to their age. Nineteen child users of a sports organization (aged from 5 to 12 years) were interviewed, involving both qualitative and quantitative analyses. The analysis of facial expressions was performed using Face Reader v.6.1.10, which automatically analyzes eight categories of facial expressions (six basic emotions, one neutral, and one unidentified). The data collection and transformation procedures fully satisfied the requirement for flexibility and provide new perspectives for incorporating data into the study of biopsychosocial factors in everyday contexts using a Mixed-Methods approach. In addition, the generalizability analysis itself, carried out to determine the levels of reliability, generalizability and precision of the sample (with relative and absolute G indexes of between 0.86 and 0.93 for the three analyses performed), was also a Mixed-Methods procedure.

Reliable Assessment of Surgical Technical Skills Is Dependent on Context: An Exploration of Different Variables Using Generalizability Theory

Reliable assessment of surgical skills is vital for competency-based medical training. Several factors influence not only the reliability of judgments but also the number of observations needed for making judgments of competency that are both consistent and reproducible. The aim of this study was to explore the role of various conditions-through the analysis of data from large-scale, simulation-based assessments of surgical technical skills-by examining the effects of those conditions on reliability using generalizability theory. Assessment data from large-scale, simulation-based temporal bone surgical training research studies in 2012-2018 were pooled, yielding collectively 3,574 assessments of 1,723 performances. The authors conducted generalizability analyses using an unbalanced random-effects design, and they performed decision studies to explore the effect of the different variables on projections of reliability. Overall, 5 observations were needed to achieve a generalizability coefficient > 0.8. Several variables modified the projections of reliability: increased learner experience necessitated more observations (5 for medical students, 7 for residents, and 8 for experienced surgeons), the more complex cadaveric dissection required fewer observations than virtual reality simulation (2 vs 5 observations), and increased fidelity simulation graphics reduced the number of observations needed from 7 to 4. The training structure (either massed or distributed practice) and simulator-integrated tutoring had little effect on reliability. Finally, more observations were needed during initial training when the learning curve was steepest (6 observations) compared with the plateau phase (4 observations). Reliability in surgical skills assessment seems less stable than it is often reported to be. Training context and conditions influence reliability. The findings from this study highlight that medical educators should exercise caution when using a specific simulation-based assessment in other contexts.

The Development and Validation of the Diabetic Wound Assessment Learning Tool

Diabetic foot wounds are estimated to result in one major lower extremity amputation globally every 30 seconds, in addition to their economic and societal costs. Physicians and trainees encounter these patients in all specialties, particularly in medical and emergency departments. Few studies focus on how to teach and assess wound management. Given the importance of “assessment for learning” in competence-based medical education, we aimed to develop, and examine specific sources of validity evidence for, an assessment tool of wound management competencies. We organized our tool development and validation process using Kane’s framework. First, we used a nominal group process involving nine Canadian experts in diabetic wound management to develop the tool items and two 10-minute simulation-based testing scenarios. We then collected validity evidence for the tool by assessing 74 participants’ (61 physicians, 13 nonphysicians) performance during the two scenarios: 44 novices (<50 previous cases), 17 intermediates (50-500), and 13 experts (>500). Two assessors independently rated participants using our tool. We evaluated reliability using Generalizability Theory, and measured test-retest reliability with the intraclass correlation coefficient (ICC) comparing performance during each scenario. We also compared performance scores across the three experience levels using analysis of variance. Cronbach’s alpha was 0.95 for the tool, implying high internal consistency. Test-retest reliability was also excellent (ICC, 0.94; 95% confidence interval [CI], 0.91-0.96) for single measures and (ICC, 0.97; 95% CI, 0.95-0.98) for average measures. Pooled inter-rater reliability was fair for single measures (ICC, 0.71; 95% CI, 0.68-0.73) and good for average measures (ICC, 0.83; 95% CI, 0.81-0.85). The tool differentiated between novices and the other two groups well (P < .01), but not between intermediates and experts (P = .34). Our generalizability coefficient was 0.87 and our Phi coefficient was 0.87. Our generalizability analyses suggest that scores on the Diabetic Wound Assessment Learning Tool can safely be generalized over the prepared testing scenarios and can be used to provide absolute ratings as well as relative ratings of wound management skills. The accumulated validity evidence suggests our tool can reliably be used to assess novice clinicians’ competence in diabetic wound management during simulated cases.

Design, validation, and testing of an observational tool for technical and tactical analysis in the taekwondo competition at the 2016 Olympic games

Observational methodology uses validated observational tools to collect information in sports with multiple variables that interact in the sporting context. Given the importance of data quality for observational tools, the purpose of this study was to design, validate, and test the reliability of a mixed observational instrument combining field formats and category systems for analyzing technical and tactical actions in an Olympic taekwondo (TKD) tournament. The instrument collects information of six criteria and 25 categories of the tactical and technical actions, kicking zone, laterality, kicking leg, guard, and score. A total of 2 374 actions were analyzed from 10 bouts involving 12 competitors in the 2016 Rio de Janeiro Olympic Games. The coefficients derived from the generalizability analysis (G coefficients) were used to check the accuracy, validity, and reliability of the tool created, while lag-sequential and polar-coordinate analysis were used to determine technical and tactical patterns of TKD athletes. The results showed optimal G coefficients and provide us with objective, valid, and reliable information on the bout situation analyzed, allowing us to extrapolate the data from our results to the sample population. The lag-sequential and polar-coordinate analyses showed that the competitors scored one point through direct attacks with linear techniques to the chest and performed circular actions with the back leg before and after scoring one point. Three points are scored by indirect attacks and subsequent counterattacks with spinning kicks to the trunk and head (using linear and circular techniques). These results reveal technical and tactical implications derived from the latest modifications of the regulations of this sport, as well as the need to adapt athletes’ training for the next Olympic Games.

Using generalizability theory to evaluate the comparative reliability of developmental measures in neurogenetic syndrome and low-risk populations

BackgroundThe lack of available measures that can reliably characterize early developmental skills in children with neurogenetic syndromes (NGS) poses a significant challenge for research on early development in these populations. Although syndrome-specific measures may sometimes be necessary, a more cost- and time-efficient solution would be to identify existing measures that are appropriate for use in special populations or optimize existing measures to be used in these groups. Reliability is an important metric of psychometric rigor to consider when auditing and optimizing assessment tools for NGS. In this study, we use Generalizability Theory, an extension of classical test theory, as a novel approach for more comprehensively characterizing the reliability of existing measures and making decisions about their use in the field of NGS research.MethodsWe conducted generalizability analyses on a popular early social communication screener, the Communication and Symbolic Behavior Scales—Infant-Toddler Checklist (CSBS-ITC), collected on 172 children (41 Angelman syndrome, 30 Prader-Willi syndrome, 42 Williams syndrome, 59 low-risk controls).ResultsOverall, the CSBS-ITC demonstrated at least adequate reliability in the NGS groups included in this study, particularly for the Prader-Willi and Williams syndrome groups. However, the sources of systematic error variance in the CSBS-ITC varied greatly between the low-risk control and NGS groups. Moreover, as unassessed in previous research, the CSBS-ITC demonstrated substantial differences in variance sources among the NGS groups. Reliability of CSBS-ITC scores was highest when averaging across all measurement points for a given child and was generally similar or better in the NGS groups compared to the low-risk control group.ConclusionsOur findings suggest that the CSBS-ITC communicates different information about the reliability of stability versus change, in low-risk control and NGS samples, respectively, and that psychometric approaches like Generalizability Theory can provide more complete information about the reliability of existing measures and inform decisions about how measures are used in research on early development in NGS.

Exploring assessor cognition as a source of score variability in a performance assessment of practice-based competencies

BackgroundA common feature of performance assessments is the use of human assessors to render judgements on student performance. From a measurement perspective, variability among assessors when assessing students may be viewed as a concern because it negatively impacts score reliability and validity. However, from a contextual perspective, variability among assessors is considered both meaningful and expected. A qualitative examination of assessor cognition when assessing student performance can assist in exploring what components are amenable to improvement through enhanced rater training, and the extent of variability when viewing assessors as contributing their individual expertise. Therefore, the purpose of this study was to explore assessor cognition as a source of score variability in a performance assessment of practice-based competencies.MethodA mixed-method sequential explanatory study design was used where findings from the qualitative strand assisted in the interpretation of results from the quantitative strand. Scores from one objective structured clinical examination (OSCE) were obtained for 95 occupational therapy students. Two Generalizability studies were conducted to examine the relative contribution of assessors as a source of score variability and to estimate the reliability of domain and holistic scores. Think-aloud interviews were conducted with eight participants assessing a subset of student performances from the OSCE in which they participated. Findings from the analysis of think-aloud data and consideration of assessors’ background characteristics were used to assist in the interpretation of variance component estimates involving assessors, and score reliability.ResultsResults from two generalizability analyses indicated the highest-order interaction-error term involving assessors accounted for the second-highest proportion of variance, after student variation. Score reliability was higher in the holistic vs. analytic scoring framework. Verbal analysis of assessors' think-aloud interviews provided evidential support for the quantitative results.ConclusionsThis study provides insight into the nature and extent of assessor variability during a performance assessment of practice-based competencies. Study findings are interpretable from the measurement and contextual perspectives on assessor cognition. An integrated understanding is important to elucidate the meaning underlying the numerical score because the defensibility of inferences made about students’ proficiencies rely on score quality, which in turn relies on expert judgements.

LO75: Does the Ottawa emergency department shift observation tool give more useful information – assessing the utility of transitioning to a novel, entrustability based assessment tool in the emergency department

Introduction: The Ottawa Emergency Department Shift Observation Tool (O-EDShOT) was recently developed to assess a resident's ability to safely run an ED shift and is supported by multiple sources of validity evidence. The O-EDShOT uses entrustability scales, which reflect the degree of supervision required for a given task. It was found to discriminate between learners of different levels, and to differentiate between residents who were rated as able to safely run the shift and those who were not. In June 2018 we replaced norm-based daily encounter cards (DECs) with the O-EDShOT. With the ideal assessment tool, most of the score variability would be explained by variability in learners’ performances. In reality, however, much of the observed variability is explained by other factors. The purpose of this study is to determine what proportion of total score variability is accounted for by learner variability when using norm-based DECs vs the O-EDShOT. Methods: This was a prospective pre-/post-implementation study, including all daily assessments completed between July 2017 and June 2019 at The Ottawa Hospital ED. A generalizability analysis (G study) was performed to determine what proportion of total score variability is accounted for by the various factors in this study (learner, rater, form, pgy level) for both the pre- and post- implementation phases. We collected 12 months of data for each phase, because we estimated that 6-12 months would be required to observe a measurable increase in entrustment scale scores within a learner. Results: A total of 3908 and 3679 assessments were completed by 99 and 116 assessors in the pre- and post- implementation phases respectively. Our G study revealed that 21% of total score variance was explained by a combination of post-graduate year (PGY) level and the individual learner in the pre-implementation phase, compared to 59% in the post-implementation phase. An average of 51 vs 27 forms/learner are required to achieve a reliability of 0.80 in the pre- and post-implementation phases respectively. Conclusion: A significantly greater proportion of total score variability is explained by variability in learners’ performances with the O-EDShOT compared to norm-based DECs. The O-EDShOT also requires fewer assessments to generate a reliable estimate of the learner's ability. This study suggests that the O-EDShOT is a more useful assessment tool than norm-based DECs, and could be adopted in other emergency medicine training programs.

PIN82 EXTENDING INFERENCES FROM A CLUSTER RANDOMIZED TRIAL TO A TARGET POPULATION

The clusters participating in cluster randomized trials are not necessarily representative of the target population of trial-eligible clusters where the experimental treatments might be applied. To the extent that participating and non-participating clusters differ in terms of covariates that are modifiers of the treatment effect, the average treatment effect in the trial may not apply to the entire target population. We describe analyses that extend (generalize or transport) causal inferences from cluster randomized trials to a target population of clusters, under a general nonparametric model that allows for arbitrary within-cluster dependence. We consider study designs where a subset of a cohort of clusters are invited and agree to participate in a randomized trial with cluster-level treatment assignment (i.e., nested trial designs). Treatment and outcome data need only be collected from clusters participating in the trial, but data on baseline covariates must be collected from the entire cohort. We propose doubly robust estimators of potential outcome means in the target population that exploit individual- level data on covariates and outcomes to improve efficiency and are appropriate for use with machine learning. We illustrate the methods using a 2x2 factorial cluster randomized trial of influenza vaccination strategies in 818 nursing homes that were nested in a cohort of 4,475 trial- eligible Medicare-certified nursing homes. We used machine learning methods to estimate the conditional probability of participation and the conditional probability of death, to obtain the potential outcome means for the vaccination strategies in the target population of trial-eligible nursing homes. Point estimates from generalizability analyses using different methods were similar, suggesting that the trial-only estimates were applicable to the broader population of trial-eligible nursing homes for the outcome of all-cause death. When cluster randomized trials are embedded in large health-care systems, investigators can use the methods to extend causal inferences.

Effect of Instructional Format on Veterinary Students' Task Performance and Emotional State during a Simulation-Based Canine Endotracheal Intubation Laboratory: Handout versus Video.

Video- versus handout-based instructions may influence student outcomes during simulation training and competency-based assessments. Forty-five third-year veterinary students voluntarily participated in a simulation module on canine endotracheal intubation. A prospective, randomized, double-blinded study investigated the impact of video (n = 23) versus handout (n = 22) instructions on student confidence, anxiety, and task performance. Students self-scored their confidence and anxiety before and after the simulation. During the simulation laboratory, three raters independently evaluated student performance using a 20-item formal assessment tool with a 5-point global rating scale. No significant between- or within-group differences (p > .05) were found for both confidence and anxiety scores. Video-based instructions were associated with significantly higher (p < .05) total formal assessment scores compared with handout-based instructions. The video group had significantly higher scores than the handout group on 3 of the 20 individual skills (items) assessed: placement of tie to the adaptor-endotracheal tube complex (p < .05), using the anesthetic machine (p < .01), and pop-off valve management (p < .001). Inter-rater reliability as assessed by Cronbach's α (.92), and Kendall's W (.89) was excellent and almost perfect, respectively. A two-faceted crossed-design generalizability analysis yielded G coefficients for both the handout (Ep2 = .68) and the video (Ep2 = .72) groups. Video instructions may be associated with higher performance scores than handout instructions during endotracheal intubation simulation training. Further research into skill retention and learning styles is warranted.

Colonoscopy Direct Observation of Procedural Skills Assessment Tool for Evaluating Competency Development During Training.

Formative colonoscopy direct observation of procedural skills (DOPS) assessments were updated in 2016 and incorporated into UK training but lack validity evidence. We aimed to appraise the validity of DOPS assessments, benchmark performance, and evaluate competency development during training in diagnostic colonoscopy. This prospective national study identified colonoscopy DOPS submitted over an 18-month period to the UK training e-portfolio. Generalizability analyses were conducted to evaluate internal structure validity and reliability. Benchmarking was performed using receiver operator characteristic analyses. Learning curves for DOPS items and domains were studied, and multivariable analyses were performed to identify predictors of DOPS competency. Across 279 training units, 10,749 DOPS submitted for 1,199 trainees were analyzed. The acceptable reliability threshold (G > 0.70) was achieved with 3 assessors performing 2 DOPS each. DOPS competency rates correlated with the unassisted caecal intubation rate (rho 0.404, P < 0.001). Demonstrating competency in 90% of assessed items provided optimal sensitivity (90.2%) and specificity (87.2%) for benchmarking overall DOPS competence. This threshold was attained in the following order: "preprocedure" (50-99 procedures), "endoscopic nontechnical skills" and "postprocedure" (150-199), "management" (200-249), and "procedure" (250-299) domain. At item level, competency in "proactive problem solving" (rho 0.787) and "loop management" (rho 0.780) correlated strongest with the overall DOPS rating (P < 0.001) and was the last to develop. Lifetime procedure count, DOPS count, trainer specialty, easier case difficulty, and higher cecal intubation rate were significant multivariable predictors of DOPS competence. This study establishes milestones for competency acquisition during colonoscopy training and provides novel validity and reliability evidence to support colonoscopy DOPS as a competency assessment tool.

Construct validity of multiple mini interviews – Investigating the role of stations, skills, and raters using Bayesian G-theory

Background: One popular procedure in the medical student selection process are multiple mini-interviews (MMIs), which are designed to assess social skills (e.g., empathy) by means of brief interview and role-play stations. However, it remains unclear whether MMIs reliably measure desired social skills or rather general performance differences that do not depend on specific social skills. Here, we provide a detailed investigation into the construct validity of MMIs, including the identification and quantification of performance facets (social skill-specific performance, station-specific performance, general performance) and their relations with other selection measures. Methods: We used data from three MMI samples (N = 376 applicants, 144 raters) that included six interview and role-play stations and multiple assessed social skills. Results: Bayesian generalizability analyses show that, the largest amount of reliable MMI variance was accounted for by station-specific and general performance differences between applicants. Furthermore, there were low or no correlations with other selection measures. Discussion: Our findings suggest that MMI ratings are less social skill-specific than originally conceptualized and are due more to general performance differences (across and within-stations). Future research should focus on the development of skill-specific MMI stations and on behavioral analyses on the extents to which performance differences are based on desirable skills versus undesired aspects.

Number of Operative Performance Ratings Needed to Reliably Assess the Difficulty of Surgical Procedures

The profession of surgery is entering a new era of "big data," where analyses of longitudinal trainee assessment data will be used to inform ongoing efforts to improve surgical education. Given the high-stakes implications of these types of analyses, researchers must define the conditions under which estimates derived from these large datasets remain valid. With this study, we determine the number of assessments of residents' performances needed to reliably assess the difficulty of "Core" surgical procedures. Using the SIMPL smartphone application from the Procedural Learning and Safety Collaborative, 402 attending surgeons directly observed and provided workplace-based assessments for 488 categorical residents after 5259 performances of 87 Core surgical procedures performed at 14 institutions. We used these faculty ratings to construct a linear mixed model with resident performance as the outcome variable and multiple predictors including, most significantly, the operative procedure as a random effect. We interpreted the variance in performance ratings attributable to the procedure, after controlling for other variables, as the "difficulty" of performing the procedure. We conducted a generalizability analysis and decision study to estimate the number of SIMPL performance ratings needed to reliably estimate the difficulty of a typical Core procedure. Twenty-four faculty ratings of resident operative performance were necessary to reliably estimate the difficulty of a typical Core surgical procedure (mean dependability coefficient 0.80, 95% confidence interval 0.73-0.87). At least 24 operative performance ratings are required to reliably estimate the difficulty of a typical Core surgical procedure. Future research using performance ratings to establish procedure difficulty should include adequate numbers of ratings given the high-stakes implications of those results for curriculum design and policy.

How Consistent Are Challenge and Threat Evaluations? A Generalizability Analysis.

Viewing stressful situations as more of a challenge than a threat (i.e., coping resources match or exceed situational demands) has been associated with better performance and long-term health. However, to date, little research has examined if individuals have tendencies to evaluate all stressful situations as more of a challenge or threat. Thus, this study used generalizability analyses to investigate the consistency (or variability) of challenge and threat evaluations across potentially stressful situations. 1813 roller derby players (89.0% female; Mage = 33 years, SD = 7) read nine stressful vignettes (e.g., injury, non-selection, family illness), before completing self-report items assessing challenge and threat evaluations. Generalizability analyses revealed that the Athlete × Stressor interaction accounted for the greatest amount of variance in challenge and threat evaluations (51.9%), suggesting that athletes had idiosyncrasies in their tendency to view particular stressors as more of a challenge or threat. The Athlete (15.4%) and Stressor (21.9%) components also accounted for a significant amount of variance. While the Athlete component suggested some consistency in challenge and threat evaluations, and that differences existed between athletes in whether they tended to view stressors as more of a challenge or threat, the Stressor component indicated some agreement among the athletes in their tendency to view some stressors as more of a challenge or threat than others. The findings offer direct support for transactional stress theories, and have important implications for practitioners developing stress management interventions.

Main and Regional Campus Assessments of Applicants to a Rural Physician Leadership Program

While the selection of qualified applicants often relies, in part, on scores generated from a medical school pre-admission interview (MSPI), the growth of regional medical campuses (RMCs) – many with specialized rural tracks, programs, or missions – has challenged schools to accommodate a wider range of stakeholder input. This study examines the reliabilities of main (urban) and regional (rural) campus interviewers’ assessments of applicants to a Rural Physician Leadership Program (RPLP) located in the southeastern United States.&#x0D; Data from RPLP applicants completing MSPIs on two campuses from 2009-2017 (n = 232) were examined in a generalizability analysis. In two separate interviews on each campus (4 total), raters independently evaluated applicants’ overall acceptability and likelihood of practicing in a rural area of the state. Results provided campus-specific and combined (composite) estimates of obtained and projected reliabilities. &#x0D; The person-by-campus interaction accounted for 11% and 5% of the respective variance in interviewers’ ratings of overall applicant acceptability and likelihood of rural in-state practice, and the reliability of mean scores across the four independent interviews (each with a single, unique rater) was 0.73 and 0.82. Error variances were higher among main campus interviewers, but scores correlated highly between the two campuses.&#x0D; While broadening the universe of generalization often results in decreased reliability, reliability was shown to be enhanced with the addition of regional (rural) campus interviews. As the RPLP matures, an examination of graduates’ actual practice locations should yield insights into the predictive validity of these pre-admissions assessments. More generally, research may wish to explore the conditions under which increasing the diversity of stakeholder input can be accommodated without concomitant reductions in overall reliability.

Main and Regional Campus Assessments of Applicants to a Rural Physician Leadership Program

While the selection of qualified applicants often relies, in part, on scores generated from a medical school pre-admission interview (MSPI), the growth of regional medical campuses (RMCs) – many with specialized rural tracks, programs, or missions – has challenged schools to accommodate a wider range of stakeholder input. This study examines the reliabilities of main (urban) and regional (rural) campus interviewers’ assessments of applicants to a Rural Physician Leadership Program (RPLP) located in the southeastern United States.&#x0D; Data from RPLP applicants completing MSPIs on two campuses from 2009-2017 (n = 232) were examined in a generalizability analysis. In two separate interviews on each campus (4 total), raters independently evaluated applicants’ overall acceptability and likelihood of practicing in a rural area of the state. Results provided campus-specific and combined (composite) estimates of obtained and projected reliabilities. &#x0D; The person-by-campus interaction accounted for 11% and 5% of the respective variance in interviewers’ ratings of overall applicant acceptability and likelihood of rural in-state practice, and the reliability of mean scores across the four independent interviews (each with a single, unique rater) was 0.73 and 0.82. Error variances were higher among main campus interviewers, but scores correlated highly between the two campuses.&#x0D; While broadening the universe of generalization often results in decreased reliability, reliability was shown to be enhanced with the addition of regional (rural) campus interviews. As the RPLP matures, an examination of graduates’ actual practice locations should yield insights into the predictive validity of these pre-admissions assessments. More generally, research may wish to explore the conditions under which increasing the diversity of stakeholder input can be accommodated without concomitant reductions in overall reliability.