Response-shift Bias Research Articles

What Type of Self-Assessment Is Best for Your Educational Activity? A Review of Pre-Post, Now-Then, and Post-Only Designs.

In medical education, learner self-assessments are standard methods used to evaluate the impact of curricula and workshops. Due to the subjective nature of self-assessment, these measures are prone to known biases including framing, recall, social desirability, and response-shift bias. These biases can contribute to floor and ceiling effects of measurement, which can lead to false conclusions about whether the intended learning objectives were achieved. Ideal assessments of skills-based educational activities would include standardized tests and structured observations of learners demonstrating skill use before and after the educational intervention. However, educators often lack the necessary resources, time, and expertise to routinely conduct these appraisals and rely on self-assessment as a pragmatic approach to obtaining curriculum feedback and evaluation data. In this review, we describe three common designs for self-assessments: the pre-post, now-then, and post-only designs. We then give recommendations for choosing between each design to minimize bias. The choice of the best design is based on alignments with four considerations: (1) the educational objectives (e.g., demonstrate skill competency and/or change in skill level); (2) participants' prior experience and shared understanding of levels of skill performance; (3) the nature of the educational activity; and (4) available resources. For each design, we review strengths, weaknesses, and known biases and discuss examples to highlight trade-offs between options. We also discuss the use of control groups and follow-up surveys to measure retention over time as additional methods to address bias and related confounding. The guidance presented here is intended to raise educators' awareness of common pitfalls in self-assessment; minimize the impact of known biases when possible; provide evidence, examples, and rationales for optimal design choices; and increase the rigor of self-assessment evaluations.

Enhancing care: evaluating the impact of True North Sexual Health and Rehabilitation eTraining for healthcare providers working with prostate cancer patients and partners.

Educational programs that enhance healthcare providers' competence in managing the care of patients with sexual dysfunction following prostate cancer treatments are needed to facilitate comprehensive sexual health treatments for patients and their partners. In this study we evaluated the impact of a real-world online sexual health educational intervention called the True North Sexual Health and Rehabilitation eTraining Program. This program is designed to increase healthcare providers' knowledge and self-efficacy in providing sexual healthcare to prostate cancer patients and their partners. Healthcare providers were invited to join a 12-week virtual training program. Participants completed precourse surveys (n = 89), retrospective prepost surveys (n = 58), and a 3-month follow-up survey (subset n = 18) to assess retention of relevant outcomes. Additionally, a course satisfaction survey was administered to participants (n = 57) at the end of the course. The main outcomes focused on participants' perceived knowledge and self-efficacy in conducting assessments and providing interventions for various relevant physical, functional, psychological, and relational domains of sexual dysfunction in prostate cancer patients and their partners. According to the retrospective analysis of post-then-pre-survey results, graduates perceived that their knowledge of and self-efficacy in providing sexual health counseling improved after completing the course. The 3-month follow-up survey indicated that the course graduate self-efficacy remained high 3 months after the course. Furthermore, the satisfaction survey indicated that a vast majority (98.2%) of participants were satisfied with the educational intervention. This real-world sexual health educational intervention can increase self-efficacy and knowledge in healthcare providers who are supporting prostate cancer patients dealing with sexual dysfunction. The use of a retrospective post-then-pre-survey helped to mitigate response shift bias while minimizing data gaps. However, it is important to note that this investigation was not a traditional research study and lacked a control group, thus limiting causal attributions. The True North Sexual Health and Rehabilitation eTraining program acts as an accessible and effective resource for healthcare providers seeking specialized training in providing sexual healthcare for prostate cancer patients and their partners.

Response-shift bias in student self-efficacy during an actively taught physics course

Response-shift bias in student self-efficacy during an actively taught physics course

Intervention Effect or Measurement Artifact? Using Invariance Models to Reveal Response-Shift Bias in Experimental Studies

One aspect usually overlooked in studies evaluating the efficacy of educational interventions, is the extent to which posttest responses reflect actual changes in targeted latent constructs versus a more superficial shift in participants’ interpretations of assessment items after exposure to the intervention (i.e., response shift). We conducted a Monte Carlo simulation study to examine the impact of response shift on treatment effects. Our results indicate that treatment effects estimated with composite scores and latent variable models may be severely overestimated in certain conditions. We describe the use of a measurement invariance framework across groups and across time to mitigate response-shift bias. We also discuss the need to conceptualize response shift as a first step in the evaluation of interventions.

Unraveling the safety of adjuvant radiotherapy in prostate cancer: impact of older age and hypofractionated regimens on acute and late toxicity - a multicenter comprehensive analysis.

The objective of this study was to assess the impact of age and other patient and treatment characteristics on toxicity in prostate cancer patients receiving adjuvant radiotherapy (RT). This observational study (ICAROS-1) evaluated both acute (RTOG) and late (RTOG/EORTC) toxicity. Patient- (age; Charlson's comorbidity index) and treatment-related characteristics (nodal irradiation; previous TURP; use, type, and duration of ADT, RT fractionation and technique, image-guidance systems, EQD2 delivered to the prostate bed and pelvic nodes) were recorded and analyzed. A total of 381 patients were enrolled. The median EQD2 to the prostate bed (α/β=1.5) was 71.4 Gy. The majority of patients (75.4%) were treated with intensity-modulated radiation therapy (IMRT) or volumetric-modulated arc therapy (VMAT). Acute G3 gastrointestinal (GI) and genitourinary (GU) toxicity rates were 0.5% and 1.3%, respectively. No patients experienced >G3 acute toxicity. The multivariable analysis of acute toxicity (binomial logistic regression) showed a statistically significant association between older age (> 65) and decreased odds of G≥2 GI acute toxicity (OR: 0.569; 95%CI: 0.329-0.973; p: 0.040) and decreased odds of G≥2 GU acute toxicity (OR: 0.956; 95%CI: 0.918-0.996; p: 0.031). The 5-year late toxicity-free survival rates for G≥3 GI and GU toxicity were 98.1% and 94.5%, respectively. The only significant correlation found (Cox's regression model) was a reduced risk of late GI toxicity in patients undergoing hypofractionation (HR: 0.38; 95% CI: 0.18-0.78; p: 0.008). The unexpected results of this analysis could be explained by a "response shift bias" concerning the protective effect of older age and by treatment in later periods (using IMRT/VMAT) concerning the favorable effect of hypofractionation. However, overall, the study suggests that age should not be a reason to avoid adjuvant RT and that the latter is well-tolerated even with moderately hypofractionated regimens.

Psychometric sensitivity analyses can identify bias related to measurement properties in trials that use patient-reported outcome measures: a secondary analysis of a clinical trial using the disabilities of the arm, shoulder, and hand questionnaire

ObjectivesDemonstrate psychometric sensitivity analyses for testing the stability of study findings to assumptions made about patient-reported outcome measures. Study Design and SettingWe performed secondary analyses of Disability of Arm, Shoulder, and Hand (DASH) data collected within the Prevention of Shoulder Problems clinical trial, which compared upper limb function scores in women who had undergone breast cancer surgery, randomized to either an exercise program or usual care. We repeated the principal trial analyses after grouping DASH items into subscales suggested by factorial analyses in this dataset and applied item response theory to account for unequal item weighting. We checked for measurement invariance by participant age and response shift bias using established techniques. ResultsOur analyses suggested that the DASH measured two constructs: motor function and sensory symptoms. The majority of the six-month difference in DASH score was driven by motor function. With item response theory scoring, we found differences in both constructs at 12 months (P = 0.019 and P = 0.007), but in neither construct at 6 months, contrary to the original trial results. We found no differential item function by age or between baseline and 12-month measurements. ConclusionsPsychometric sensitivity analyses aid in the interpretation of the Prevention of Shoulder Problems trial’s results.

Undergraduate teaching of surgical skills in the UK: systematic review.

Students must be proficient in surgical skills according to General Medical Council and Royal College of Surgeons of England guidelines. If these skills are not appropriately taught, there is a risk of an incoming junior workforce with inadequate surgical skills. This paper aimed to review the literature relating to undergraduate teaching of surgical skills in the UK and summarize future suggested training methods. The databases MEDLINE, Embase and SCOPUS were searched, and the existing literature relating to methodology of undergraduate teaching of surgical skills in the UK over the past 10 years was summarized. The Medical Education Research Quality Instrument was used to assess research quality. A total of 19 papers were included. Cross-sectional evaluations and survey-based studies highlight a clear deficit in surgical skills teaching in the UK. Medical students are currently unable to fulfil their own learning needs and meet requirements set out by the General Medical Council. This lack of surgical teaching appears to negatively affect student desire to pursue a surgical career. The three main themes for improvement are extracurricular surgical skills days, near-peer teaching and simulation. Each method appeared to improve learning, although no studies utilized medium- to long-term follow-up to demonstrate efficacy and there lacks a clear consensus as to the 'standard' of undergraduate surgical skill education. There was also potential for selection bias and response shift bias in many of the studies assessing pre- and postintervention confidence and opinions. There is a concerning lack of surgical skills teaching that has resulted in medical students and junior doctors not having the necessary surgical skills as per General Medical Council guidance and students feel that their own learning needs are not met. This failure to address the learning deficit may be responsible for the fall in surgical competition ratios. While surgical skills teaching must be improved urgently, more robust evidence is required to evaluate the optimal ways of approaching this issue.

The Draft Report by the Institute for Quality and Efficiency in Healthcare Does Not Provide Any Evidence That Graded Exercise Therapy and Cognitive Behavioral Therapy Are Safe and Effective Treatments for Myalgic Encephalomyelitis/Chronic Fatigue Syndrome

The German Institute for Quality and Efficiency in Healthcare (IQWiG) recently published its draft report to the government about myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS). The IQWiG concluded that graded exercise therapy (GET) and cognitive behavioral therapy (CBT) should be recommended in the treatment for mild and moderate ME/CFS based on two CBT and two GET studies. In this article, we reviewed the evidence used by IQWiG to support their claims, because their conclusion is diametrically opposed to the conclusion by the British National Institute for Health and Care Excellence (NICE) in its recently updated ME/CFS guidelines. Our analysis shows that the trials IQWiG used in support suffered from serious flaws, which included badly designed control groups; relying on subjective primary outcomes in non-blinded studies; alliance and response shift bias, including patients in their trials who did not have the disease under investigation, selective reporting, making extensive endpoint changes and low to very low adherence of treatments. Our analysis also shows that the report itself used one CBT and one GET study that both examined a different treatment. The report also used a definition of CBT that does not reflect the way it is being used in ME/CFS or was tested in the studies. The report noted that one study used a wrong definition of post-exertional malaise (PEM), the main characteristic of the disease, according to the report. Yet, it ignored the consequence of this, that less than the required minimum percentage of patients had the disease under investigation in that study. It also ignored the absence of improvement on most of the subjective outcomes, as well as the fact that the IQWiG methods handbook states that one should use objective outcomes and not rely on subjective outcomes in non-blinded studies. The report concluded that both treatments did not lead to objective improvement in the six-minute walk test but then ignored that. The report did not analyze the other objective outcomes of the studies (step test and occupational and benefits status), which showed a null effect. Finally, the report states that the studies do not report on safety yet assumes that the treatments are safe based on a tendency towards small subjective improvements in fatigue and physical functioning, even though the adherence to the treatments was (very) low and the studies included many patients who did not have the disease under investigation and, consequently, did not suffer from exertion intolerance contrary to ME/CFS patients. At the same time, it ignored and downplayed all the evidence that both treatments are not safe, even when the evidence was produced by a British university. In conclusion, the studies used by the report do not provide any evidence that CBT and GET are safe and effective. Consequently, the report and the studies do not provide any support for the recommendation to use CBT and GET for ME/CFS or long COVID, which, in many cases, is the same or resembles ME/CFS, after an infection with SARS-CoV-2.

Student teachers’ perceptions of motivation, independence, and supervision preferences: An exploratory study

The student teaching experience is one of the most impactful capstone experiences for the preparation of preservice teachers. The supervisor, either a cooperating teacher or university supervisor, plays a critically important role in the student teaching experience. The purpose of this study was to explore preservice teachers' perceived motivation and independence throughout their student teaching experience. It is recommended that early in the student teaching experience, a directive supervision style should be utilized. Then, as motivation starts to decline in the middle of the student teaching experience, the focus of supervision should shift to providing moral support and encouraging commitment to the profession of teaching. Recommendations for future research include replication of this study with future cohorts of student teachers across multiple institutions so data trends can be analyzed longitudinally. Additionally, it is recommended that future iterations of this study should administer a post-then version of the quantitative plotting instrument to control response shift bias.

P089 A Retrospective Pretest-Posttest Design as an Alternative Evaluation Method for EFNEP

<h3>Background</h3> The Expanded Food and Nutrition Education Program (EFNEP) evaluates its behavioral outcomes using prospective pretest-posttests, which can be prone to response shift bias. A retrospective survey design can be a potential solution. <h3>Objective</h3> To examine program participants' understanding and preferences regarding retrospective versions of the EFNEP Adult Questionnaire (AQ). <h3>Study Design, Setting, Participants</h3> One-on-one, semi-structured interviews were conducted with 23 EFNEP adult participants/graduates in four US states/territories in 2020-2021. Participants were shown retrospective pretests and posttests in the "side-by-side" and "separate page" formats, which included one question from each of the AQ domains of diet quality, physical activity, food resource management, food safety and food security. AQ questions were modified by adding "Before EFNEP" for the pretest and "Now" for the posttest timeframes. <h3>Measurable Outcome/Analysis</h3> Demographic characteristics (e.g., age, gender, education), and AQ format preferences were collected. Audio transcripts were analyzed using template analysis and NVivo 1.5. <h3>Results</h3> Most of the participants were female (87%) and high school graduates (70%); their mean age was 35 years. For the "Before EFNEP" questions, participants mentioned "any time before the past two months" or provided the particular month before EFNEP lessons took place. Regarding the "Now" questions, emerging themes were relevant to the past weeks or past month. Some participants indicated that additional descriptions such as "in the week before EFNEP" were helpful in pinpointing the intended timeframe. Most (18/23) of the participants could correctly differentiate the time periods between the "Before" and "Now" questions. Most participants preferred the "side-by-side" (vs "separate page") format. <h3>Conclusions</h3> A retrospective AQ, especially in a side-by-side format, may be an appropriate alternative to the prospective surveys to evaluate behavioral outcomes of adult EFNEP participants. Next steps include piloting the retrospective survey to examine its criterion validity and feasibility with a larger sample. <h3>Funding</h3> Agricultural Experiment Station, NC3169 Multistate Project

Vaccination induced complacency in adherence to COVID-19 precautionary measures among oral health care professionals in India and the United States: a retrospective pretest-posttest design

ABSTRACT In the context of the COVID-19 pandemic, vaccination-induced behavioral complacency in adherence to COVID-19 appropriate behavior emerged as a significant concern. This study was conducted among a convenience sample of 540 oral health care professionals in India and the United States. This was a retrospective pretest-post-test design, a choice to eliminate response-shift bias, where the participants responded online on their adherence or otherwise to COVID-19 precautionary measures before and after vaccination. The difference between post-test and retrospective pretest scores was used in assessing the magnitude of complacency demonstrated by the individual as a function of getting vaccinated, and the process was validated using exploratory factor analysis (EFA) with principal axis factoring and confirmatory factor analysis (CFA) on two randomly split subsets of the overall sample. It was observed that there had been a decline in the adherence to all the considered COVID-19 precautionary measures from the time before vaccination to the time of achievement of the fully vaccinated status. EFA performed on the randomly split sub sample of 240 subjects returned a two factor solution with five items in factor 1 and seven items in factor 2. Items in both the factors demonstrated adequate internal consistency in reliability analysis (Cronbach’s alpha 0.84 and 0.82, respectively). The two factor solution obtained in EFA demonstrated good model fit in CFA [RMSEA (90%CI) – 0.077 (0.063–0.092); TLI – 0.872; CFI – 0.897; SRMR – 0.056]. These results highlight the vaccination-induced complacency in observing COVID-19 appropriate behavior among oral health professionals in India and the United States.

Retrospective Pretests: Recent Use in Visitor Studies Research and Ways to Make Them More Informative

Many visitor studies researchers use Retrospective Pretest (RPT) ­methods to document outcomes. Research literature compares the validity of RPT with traditional pretest-posttests. This article reviews visitor study and informal learning literature about how RPT has been used to evaluate programs in museums, aquariums, parks/recreation, zoos, and tourism. It furthers discussions about response shift bias, arguing that response shift can be either intentional or inadvertent. The type of response shift is important in determining when RPT should be used. We argue that RPT is the best choice when the program intends to shift participants’ understanding of constructs being measured, and when comparing intervention with non-intervention groups. The literature review found that RPT results usually focus on statistical significance testing. RPT data offer much more learning when the results are also examined from other perspectives. This article describes additional analyses of RPT data that can assess the applicability of programs to intended audiences.

Assessing the assessment: a comparison of the IPASS and ICCAS to measure change in interprofessional competencies following a large-scale interprofessional forum

ABSTRACT Measuring the effectiveness of interprofessional education (IPE) experiences is essential but challenging. Surveying learners before (pretest) and after (posttest) an IPE experience may result in minimal change due to response shift bias. Retrospective pretest/posttest assessment may ameliorate response shift bias but may also result in inflated change scores due to social desirability bias. We studied a cohort of 675 students from 12 health professions who completed the Interprofessional Attitudes Scale (IPAS) within the 4 weeks before (pretest) and the 3 weeks after (posttest) an IPE forum and completed the Interprofessional Collaborative Competency Attainment Survey (ICCAS) within the 3 weeks after the forum as a retrospective pretest/posttest. We found higher scores on the pretest IPAS than the retrospective pretest ICCAS and greater change in scores on the ICCAS than the IPAS, indicating potential response shift and social desirability biases. Furthermore, we found few significant correlations between change scores in subscales of the two tests, but a high number of strong and significant correlation among the ICCAS subscale change scores. Our results indicate the timing of pretest administration may impact change scores or that subscales of IPAS may be more unique than those of ICCAS. These findings suggest that educators should consider the potential impact of response shift and social desirability biases when interpreting results of the IPAS and ICCAS in response to an IPE learning experience.

A Parenting Behavior Intervention (the Strengthening Families Program) for Families: Noninferiority Trial of Different Program Delivery Methods

BackgroundThe Strengthening Families Program (SFP) is an evidence-based parent training and youth life skills and drug prevention program traditionally delivered in group settings. Families attend parent and youth classes conducted by trained facilitators. Recently, a 2-disk home-use DVD series was created with the same SFP skills as the group classes for parents and the youth to watch together at home. Additional lesson material was added that included healthy brain development, school success, anger management, dangers of alcohol and drugs, and mindfulness. The SFP DVD reduces SFP delivery costs for agencies and logistic burdens to families. Creative applications of the DVD include holding SFP DVD family discussion groups of multiple families and using SFP DVD video clips as part of a shorter 10-week group class version for parents and the youth.ObjectiveThis study aimed to examine three different DVD implementation scenarios using a noninferiority trial, contrasting target outcomes with an age-matched sample culled from a national norm database of families who completed a standard SFP 14-week class.MethodsThe partial eta-square was used to compare effect sizes between the different delivery modalities for relevant programmatic outcomes. We adjusted the effect sizes by demographic measures to determine whether there were site-specific features influencing program outcomes.ResultsFor the unadjusted effect size comparisons, 13 of the 15 indicated that the home-use DVD outperformed group norms with an average 0.13 effect size estimate difference across the comparisons (28% improvement in the effect size for DVD condition). Comparisons of the home-use DVD condition with the mixed DVD use conditions showed no discernable pattern where one condition consistently outperformed another. Adjusted effect sizes still reinforced the superiority of the DVD conditions; however, there was some shrinkage in the effect sizes as expected with the inclusion of relevant covariates.ConclusionsThe home-use DVD shows that it is possible to effectively deliver an affordable family-based intervention using alternative technology outside of the traditional group-based class format. In almost all of the comparisons, the DVD conditions outperformed the group norms, underscoring that low-cost DVDs or viewing the videos on the Web may provide a useful surrogate for costly group-based formats. Future studies may want to improve on the quasi-experimental design by examining programmatic differences based on delivery format using a randomized controlled trial, thus strengthening the causal framework regarding program effects. In addition, the assessment protocol relied on retrospective reporting, which, although this can limit response shift bias, does not separate data collection in time as with a true pre- and posttest design.

Back to the future: Considerations in use and reporting of the retrospective pretest

Retrospective pretests ask respondents to report after an intervention on their aptitudes, knowledge, or beliefs before the intervention. A primary reason to administer a retrospective pretest is that in some situations, program participants may over the course of an intervention revise or recalibrate their prior understanding of program content, with the result that their posttest scores are lower than their traditional pretest scores, even though their understanding or abilities have increased. This phenomenon is called response-shift bias. The existence of response-shift bias is undisputed, but it does not always occur, and use of the retrospective pretest in place of a traditional pretest often introduces new problems. In this commentary, I provide a brief overview of the literature on response-shift bias and discuss common pitfalls in the use and reporting of retrospective pretest results, including a failure to consider multiple factors that may affect all test scores, as well as claims that retrospective pretests are less biased than traditional pretests, provide more accurate estimates of effects, and are necessarily superior to traditional pretests in program evaluation. I comment on the article by Little et al. (2019) in this issue in light of the literature on retrospective pretests and discuss the need for a theoretical framework to guide research on response-shift bias. The goal of the commentary is to provide readers with an informed and critical lens through which to evaluate and use retrospective pretest methods.

NP13 Designing Retrospective Evaluation Tools with Low-Literacy Adults: Development Process

Objective Collecting ‘before’ and ‘after‘ ratings at the time of post-testing for program evaluation, reduces the likelihood of response-shift bias with a resulting increase in reliability. This method may be promising for program evaluation, especially where overestimation of initial behaviors are likely with a resulting improvement in intervention outcomes. Applying qualitative methods, the objective is to determine client preferences for retrospective approach and format, and implement those results in two tools of varying length. Description The Nutrition Education Evaluation Model (NEEM) was used to guide selection of approach and development of format for a non-traditional retrospective evaluation. Cognitive interviews were conducted at a food bank in 2006 and at Head Start and EFNEP sites, 2011-2018. Participants were low-income, English or bilingual Spanish speaking and available for an immediate interview. Several strategies were employed: concurrent and retrospective think alouds, paraphrasing probes, response format options and restating text. EFNEP educators offered opinions in unstructured discussions. Evaluation Study participants and EFNEP educators recommended one of four approaches: the Post then Pre. They preferred the format featuring each question on row along with two sets of response options flanking the centered question, use of a visual to portray each question, use of color to guide tool instructions, and presence of icons to support ‘before/then’ and ‘after/now’ concepts. Conclusion and Implications Two examples of the visual retrospective method were produced: Focus of Sweet Drinks…Now and Then for an English speaking low-income audience and Focus on Veggies…Now and Then for a Spanish speaking low-income audience. This qualitative method and the study results can be utilized by other researchers and practitioners.

Testing the Retrospective Pretest with High School Youth in Out-of-School Time Programs

Practitioners and evaluators face several constraints in conducting rigorous evaluations to determine program effect. Researchers have offered the retrospective pretest/posttest design as a remedy to curb response-shift bias and better estimate program effects. This article presents an example of how After School Matters (ASM) tested the use of retrospective pretest/posttest design for evaluating out-of-school time (OST) programs for high school youth participants. Differences between traditional pretest and retrospective pretest scores were statistically significant, but effect sizes were negligible, indicating that both pretests yielded similar results. Interviews with youth led to 3 key findings that have implications for ASM using retrospective pretests with youth: response-shift bias was more prominent in youth interviews than in quantitative findings, youth recommended reordering the questions so that the retrospective pretest appears first to increase comprehension, and acquiescence bias emerged in the interviews. This study demonstrates that the retrospective pretest/posttest design can be an alternative to the traditional pretest/posttest design for OST at ASM. These findings are important for ASM and other youth-serving organizations, which often have limited capacity to survey youth multiple times within 1 program session.

Incorporating LGBT Health in an Undergraduate Medical Education Curriculum Through the Construct of Social Determinants of Health.

The AAMC has provided a resource to medical schools for implementing curricular change in lesbian, gay, bisexual, and transgender (LGBT) health education. However, studies have identified that many health professionals who do not feel comfortable in their ability to provide quality care for LGBT patients do not perform complete sexual histories routinely and/or harbor bias towards these patients or their sexual practices. This situation underscores the continued need for further education on this topic. Based on a needs assessment survey of medical students and faculty, we developed a 1-hour didactic lecture to provide instruction on how social determinants of health impact the care of LGBT patients. Students were not required to have any prerequisite knowledge for the session. A content expert in LGBT health taught the lecture using Microsoft PowerPoint in a traditional medical school lecture hall. The lecture was given to 180 third-year medical students. A total of 63 students (35%) responded to the retrospective pre- and postlecture survey. After the didactic lecture, students reported a statistically significant change in their knowledge of the lecture objectives. The didactic lecture was able to increase students' knowledge of how social determinants impact the health of LGBT patients. The lecture can be incorporated into a longitudinal curriculum on LGBT health. Additional work and research are needed on increasing comfort in faculty teaching.

Effectiveness of the Adolescent Medicine Rotation in Improving Pediatric Residents Self-Assessed Skill and Confidence Caring for Youth

PurposePracticing and resident pediatricians report inadequate skill in caring for adolescents, despite adolescents comprising roughly one-quarter of most general and subspecialty practices. This study examined the effectiveness of participation in an adolescent medicine rotation at improving pediatric residents’ self-perceived skills and confidence across nine key adolescent health domains. We also evaluated the impact of didactic instruction during the rotation. MethodsResident and recent-graduate participants (n = 34) completed milestone-based self-assessment of their skill and confidence caring for adolescent patients in nine key adolescent health-related domains. This study employed a post-test then retrospective pretest, an educational study design used to minimize response-shift bias whereby participants rate their skill and confidence at the end of the intervention (post-test), and then reflect back to retrospectively rate their preintervention skill (retrospective pretest). Additionally, differences in gains between those who did and did not participate in didactic instruction were evaluated. Didactic instruction was delivered during the adolescent medicine rotation utilizing a flipped-classroom model; participants received standardized preparatory materials and participated in active-learning workshops. ResultsParticipants demonstrated a significant (p ≤ .0001) increase in self-perceived skill levels for all assessed domains after the rotation as compared to before the rotation, whether or not they received didactic instruction. Participation in didactic instruction did not yield significant (p ≤ .05) additional benefit for any of the assessed domains. ConclusionsParticipation in an adolescent medicine rotation is of value to pediatric resident trainees and leads to increased self-assessed skill and confidence in caring for youth.

Research Note—Multiple Time Point Course Evaluation and Student Learning Outcomes in an MSW Course

ABSTRACTEvaluation methods play a key role in measuring student learning outcomes. Yet traditional assessments focus on assessing student satisfaction with instructors or courses rather than their progress toward competencies. In addition, the common pretest-posttest assessment is problematic because of response-shift bias. Although multiple time point assessment is suggested, very little is known about its application and potential in social work education. This research note examines how student self-assessments of their progress on core competencies in an MSW-level social work course change across three time points (pretest, posttest, retrospective test). The findings suggest that students underrated and overrated their competencies at the pretest. We argue that using multiple time point self-assessment addresses this internal validity threat and should be considered in social work course evaluation.