Analyzing Resident Perceptions of Current Radiation Oncology Evaluation Methods

Abstract

Background In an effort to facilitate positive change in radiation oncology (RO) resident evaluations, we collect and report resident familiarity and perceptions of current evaluation methods. We hypothesize familiarity with evaluation methods is predictive of the perceived utility of evaluations, stress or intimidation of evaluations, and behavioral changes. Methods A survey with Likert-scale questions was distributed to RO residents at 13 institutions. Questions assessed resident familiarity with evaluation methods, satisfaction and utility of different aspects of evaluations, stress, or intimidation of receiving evaluations, and likelihood of changing post-evaluation. Likert-type scales included: 1) not familiar=1; extremely familiar=5; 2) extremely unsatisfied=1; extremely satisfied=5; 3) strongly disagree=1; strongly agree=5; 4) extremely unlikely=1; extremely likely=5. Summary statistics are reported as (median, IQR). Regression analysis was used to analyze relationships between different variables. Results Eight of 13 programs responded to the survey with a response rate of 50% (52/104 residents). Surveyed residents reported being "not familiar" with the Next Accreditation System (1, 1-1) and "slightly familiar" with the six Core Competencies and the factors use to assess them (2, 2-3). Satisfaction with aspects of the resident evaluations was variable: frequency (3, 3-4), timeliness (3.5, 3-4), clarity of strengths or areas of improvement (4, 3-4). Overall, residents agreed that evaluations were useful in informing them of their competence, progress, strengths, and areas of improvement (median 4). Residents were neutral when asked whether they agreed with the statement, "receiving evaluations intimidates me," although 24/52 (46%) reported either "somewhat" or "strongly" agreeing with the statement (3, 2-4). Most residents agreed that receiving evaluations was stressful (4, 2-4). Additionally, residents indicated they were "very likely" to change behaviors and practices following evaluations (4, 4-5). Resident-reported familiarity with the evaluation methods was not found to be a significant predictor of the utility of evaluation methods (coefficient= -0.02, p=0.83), stress (coefficient= -0.11, p=0.62), or intimidation of receiving evaluations (coefficient= -0.06, p=0.792), or the likelihood of changing post evaluation (coefficient= 0.41, p=0.204). Discussion Familiarity with evaluation methods is not correlated with perceptions or behavioral changes necessitating further investigation of alternative predictor variables. Despite the low familiarity with evaluation tools, most residents reported that evaluations were useful and likely to elicit changes in their behaviors and practice, highlighting the value of current evaluation methods. This study is a part of a larger multi-institutional project that aims to consolidate evaluation methods across different institutions and develop interventions to improve the training process for the RO trainees. In an effort to facilitate positive change in radiation oncology (RO) resident evaluations, we collect and report resident familiarity and perceptions of current evaluation methods. We hypothesize familiarity with evaluation methods is predictive of the perceived utility of evaluations, stress or intimidation of evaluations, and behavioral changes. A survey with Likert-scale questions was distributed to RO residents at 13 institutions. Questions assessed resident familiarity with evaluation methods, satisfaction and utility of different aspects of evaluations, stress, or intimidation of receiving evaluations, and likelihood of changing post-evaluation. Likert-type scales included: 1) not familiar=1; extremely familiar=5; 2) extremely unsatisfied=1; extremely satisfied=5; 3) strongly disagree=1; strongly agree=5; 4) extremely unlikely=1; extremely likely=5. Summary statistics are reported as (median, IQR). Regression analysis was used to analyze relationships between different variables. Eight of 13 programs responded to the survey with a response rate of 50% (52/104 residents). Surveyed residents reported being "not familiar" with the Next Accreditation System (1, 1-1) and "slightly familiar" with the six Core Competencies and the factors use to assess them (2, 2-3). Satisfaction with aspects of the resident evaluations was variable: frequency (3, 3-4), timeliness (3.5, 3-4), clarity of strengths or areas of improvement (4, 3-4). Overall, residents agreed that evaluations were useful in informing them of their competence, progress, strengths, and areas of improvement (median 4). Residents were neutral when asked whether they agreed with the statement, "receiving evaluations intimidates me," although 24/52 (46%) reported either "somewhat" or "strongly" agreeing with the statement (3, 2-4). Most residents agreed that receiving evaluations was stressful (4, 2-4). Additionally, residents indicated they were "very likely" to change behaviors and practices following evaluations (4, 4-5). Resident-reported familiarity with the evaluation methods was not found to be a significant predictor of the utility of evaluation methods (coefficient= -0.02, p=0.83), stress (coefficient= -0.11, p=0.62), or intimidation of receiving evaluations (coefficient= -0.06, p=0.792), or the likelihood of changing post evaluation (coefficient= 0.41, p=0.204). Familiarity with evaluation methods is not correlated with perceptions or behavioral changes necessitating further investigation of alternative predictor variables. Despite the low familiarity with evaluation tools, most residents reported that evaluations were useful and likely to elicit changes in their behaviors and practice, highlighting the value of current evaluation methods. This study is a part of a larger multi-institutional project that aims to consolidate evaluation methods across different institutions and develop interventions to improve the training process for the RO trainees.