Perspectives Viewpoints•Few internal medicine subinternships have a formal curriculum.•E-lectures were created for common hospitalized conditions including congestive heart failure, diabetes mellitus, deep venous thrombosis/pulmonary embolism, gastrointestinal bleeding, and chronic obstructive pulmonary disease, allowing for repeated student viewing at multiple institutions.•Students with e-lecture access did not demonstrate improved knowledge or comfort-level scores compared with the control group upon completing the subinternship or during the intern year. •Few internal medicine subinternships have a formal curriculum.•E-lectures were created for common hospitalized conditions including congestive heart failure, diabetes mellitus, deep venous thrombosis/pulmonary embolism, gastrointestinal bleeding, and chronic obstructive pulmonary disease, allowing for repeated student viewing at multiple institutions.•Students with e-lecture access did not demonstrate improved knowledge or comfort-level scores compared with the control group upon completing the subinternship or during the intern year. Most students entering an internal medicine residency complete both a core clerkship and a subinternship in internal medicine. Traditionally, the core clerkship is more structured at both the school and national levels compared with the subinternship, with only one-third of subinternship directors acknowledging a formal curriculum.1Aiyer MK Vu TR Ledford C Fischer M Durning SJ The subinternship curriculum in internal medicine: a national survey of clerkship directors.Teach Learn Med. 2008; 20: 151-156Crossref PubMed Scopus (14) Google Scholar Some institutions utilize the Alliance for Academic Internal Medicine (AAIM) Internal Medicine Subinternship Curriculum first published in 2002 by the Clerkship Directors in Internal Medicine and revised in 2018.2Harrell H et al.Primer to the Internal Medicine Subinternship.Med Ed Portal. 2012; (Available at: https://mededportal.org/doi/full/10.15766/mep_2374-8265.9277. Accessed June 15, 2021)Google Scholar,3Alliance for Academic Internal Medicine. AAIM Internal Medicine Subinternship Curriculum 2.0 Available at:https://www.im.org/resources/ume-gme-program-resources/curriculum. Accessed April 23, 2021.Google Scholar Even prior to Coronavirus disease 2019 (COVID-19)-related changes in the learning environment, creating supplemental flipped classroom material was becoming more commonplace.4Chen F Lui AM Martinelli SM A systematic review of the effectiveness of flipped classrooms in medical education.Med Educ. 2017; 51: 585-597Crossref PubMed Scopus (243) Google Scholar Asynchronous learning approaches allow for spaced learning from introduction of concepts to application, active processing, and improved material retention.5Cain LF Willey RDV The effect of spaced learning on the curve of retention.J Exp Psychol. 1939; 25: 209-214Crossref Scopus (14) Google Scholar, 6Dempster FN Spacing effects and their implications for theory and practice.Educ Psychol Rev. 1989; 1: 309-330Crossref Scopus (130) Google Scholar, 7Cepeda NJ Pashler H Vul E Wixted JT Rohrer D Distributed practice in verbal recall tasks: a review and quantitative synthesis.Psychol Bull. 2006; 132: 354-380Crossref PubMed Scopus (928) Google Scholar, 8Hser Y-I Wickens TD The effects of the spacing of test trials and study trials in paired-association learning.Educ Psychol. 1989; 9: 99-120Crossref Scopus (8) Google Scholar, 9Mayer RE Moreno R Nine ways to reduce cognitive load in multimedia learning.Educ Psychol. 2003; 38: 43-52Crossref Scopus (1914) Google Scholar Evidence supporting the efficacy of this curriculum and adjunctive online resources during the subinternship is lacking.10Garber AM Flipping out! Utilizing an online micro-lecture for asynchronous learning within the acting internship.Med Sci Educ. 2020; 30: 91-96Crossref Scopus (4) Google Scholar Medical education may be forever changed as a result of the COVID-19 pandemic, given the concerns around medical student safety in the clinical environment, which significantly limits in-person didactics and medical student clinical experiences. An online collection of educational videos may help fill the void created by decreased clinical exposure and experiences during the third- and fourth-year clinical clerkships. The purpose of this study was to deliver brief high-yield online electronic modules (e-modules) covering 5 core internal medicine conditions and to assess their ability to improve subintern knowledge and comfort. Students completing an internal medicine subinternship at the 4 participating institutions (University of Chicago Pritzker School of Medicine, University of Colorado School of Medicine, Johns Hopkins University School of Medicine, and Virginia Commonwealth University School of Medicine) during the 2018-2019 academic year were included in the study. Schools were chosen to represent students from across the United States. The primary outcomes were twofold: (1) absolute percentage of knowledge improvement pre- to post-rotation within study arms and (2) absolute percentage of knowledge improvement pre- to post-rotation between study arms. Secondary outcomes included comfort recognizing symptoms/signs of, ordering diagnostic tests for, and managing 5 core medical conditions. Students were block randomized to receive the usual curriculum (control) or the intervention curriculum such that all students completing the subinternship during a given 4-week rotation within the same institution were similarly randomized. The usual curriculum consisted of experiential learning within the internal medicine subinternship clinical learning environment, while the intervention curriculum consisted of 5 asynchronous e-module video lectures in addition to experiential learning. The 5 e-module videos were created by 2 authors (JB, AN), ranged in duration from 10 to 18 minutes, and achieved learning objectives that map to the AAIM Subinternship Curriculum 2.0. The authors chose topics highly relevant to hospital medicine: congestive heart failure (CHF), diabetes mellitus (DM), deep venous thrombosis/pulmonary embolism (DVT/PE), gastrointestinal bleeding (GIB), and chronic obstructive pulmonary disease (COPD). Viewing the e-modules was not required and authors were unable to track viewing at the individual participant level. During the first week of the rotation, all students received an e-mail link to the pre-rotation survey (Qualtrics platform [Provo, Utah]; Appendix A) that assessed medical knowledge and comfort recognizing and treating the 5 conditions represented in the e-module videos. For medical knowledge assessments, students completed the same 10 questions (2 per condition) chosen from and content-mapped to the Society Hospital of Medicine SPARK Edition 1 (9 of 10) and the American College of Physicians Medical Knowledge Self-Assessment Program (MKSAP) 17 (1 of 10) question banks. Comfort level was rated on a 5-point Likert scale (1 = very uncomfortable; 5 = very comfortable). Upon completion of the rotation, students were sent an e-mail link to the post-rotation survey (Qualtrics; Appendix B), which contained the same knowledge and comfort questions as the pre-rotation survey in addition to questions assessing usefulness of, and satisfaction with, the e-modules. A reminder e-mail was sent 1 week later to encourage survey completion. The final survey, which was identical to the post-rotation survey, was sent to the same pool of participants 5 months into their intern year. This final survey remained open from November 2019 until February 2020, and was timed so participants could better identify knowledge or management gaps after experiencing a portion of the intern year of residency. All authors agreed on the survey content and design. Student participation and survey responses were reviewed after grade determination to ensure that responses did not affect the summative grade. Students could opt out of having their responses used for research purposes. This study was approved by each member school's institutional review board. Pre-rotation exposure to the 5 medical conditions was expressed as <6 or ≥6 patients, while comfort level was characterized as comfortable (comfortable and very comfortable) or uncomfortable (unsure, uncomfortable, and very uncomfortable). Four sets of descriptive statistics were used to measure frequency, percentages, medians, and interquartile ranges for pre- and post-rotation survey items stratified by control and intervention groups. Chi-squared/Fisher's Exact tests were run comparing binary survey responses (comfortable/uncomfortable; <6 or ≥6). A pre-/post-rotation comparison of scores for the knowledge-based questions was run using signed rank tests comparing the percentage of correct answers between pre- and post-surveys stratified by control and intervention groups. Mean change in percent correct from the pre- to post-survey was also compared between control and intervention groups using a t test. Lastly, separate chi-squared and Fisher's Exact tests were run on the pre-rotation group (including those who did not complete a post-rotation survey), comparing survey responses related to comfort as well as knowledge question scores (correct/incorrect) among subinterns who cared for <6 patients or ≥6 patients with any of the 5 conditions prior to the subinternship experience. All analyses were performed using SAS 9.4 (SAS Institute, Cary, NC). During the 2018-2019 academic year, 250 students completed an internal medicine subinternship across the 4 participating medical schools. Overall, 171 of 250 students (68.4%) completed the pre-rotation survey, 97 of 250 (38.8%) completed the post-rotation survey, and 24 of 250 (9.6%) completed the intern survey. Seventeen of 171 (9.9%) of the pre-rotation, 4 of 97 (4.1%) of the post-rotation, and 1 of 24 (4.2%) of the interns elected not to have their survey answers used for research purposes. Two student responses were excluded due to inability to verify study arm. Only pre- and post-rotation surveys were included in the final pre/post-comparison analyses, given the low intern survey response rate. The control group answered a mean of 50.9% of the pre-rotation and 53.9% of the post-rotation knowledge questions correctly (P = .295), while the intervention group answered 49.2% and 50.6%, respectively (P = .544; Table). There was no statistically significant difference in percentage of mean correct knowledge question improvement when comparing the control and intervention groups prior to and after the rotation (3.0% vs 1.4%, respectively, P = .695; Table). No additional knowledge improvement was noted pre-rotation to post-rotation within the intervention group, within the control group, or when comparing the intervention and control groups.Table 1Pre/Post-Rotation Knowledge Scores (% Correct)Study ArmnPre Score MeanPre Score MedianPost Score MeanPost Score MedianControl Mean % ChangeIntervention Mean % ChangeP ValueControl2350.9%50.0%53.9%50.0%.295Intervention4849.2%50.0%50.6%50.0%.544Total713.0%1.4%.695 Open table in a new tab Prior to starting the rotation, 72.7% vs 42.2% (control vs intervention) of students were comfortable managing CHF, 68.2% vs 93.8% (control vs intervention) were comfortable recognizing symptoms and signs of COPD, and 52.2% vs 79.2% (control vs intervention) were comfortable ordering and interpreting diagnostic tests for GIB (all P < .05; data not shown). There were no other significant differences in comfort between the intervention and control groups in the pre-rotation survey and no significant differences were noted in the post-rotation survey. Comfort was ≥78% in all post-rotation modular topics (control and intervention). If a student had cared for 6 or more patients with any of the 5 medical conditions prior to starting their subinternship, they were significantly more comfortable managing each of those conditions, compared with students caring for fewer than 6 patients prior to starting the subinternship (Figure 1). Students caring for 6 or more patients prior to starting the subinternship were significantly more comfortable recognizing symptoms and signs for CHF and DM, as well as ordering and interpreting diagnostic tests for CHF, DM, and GIB. Pre-rotation student test knowledge was significantly higher for students caring for 6 or more patients vs students caring for fewer than 6 patients for only one knowledge question (diabetes mellitus type 2, 98% vs 80%; P = .0004). In the post-rotation survey, students rated which learning method most improved their knowledge and clinical skills during their subinternship. Students overwhelmingly ranked personal patient experience (56/92; 60.9%) number 1, compared with 9 of 92 (9.8%) choosing e-learning (Figure 2). More students ranked personal patient and co-intern patient experiences (62/92; 67.4% and 64/92; 69.6%, respectively) in their top 3 learning methods compared with 4 alternative learning options. Forty-three percent of the pre-rotation students preferred online learning over alternative learning modalities. Upon completion of the subinternship, 58% somewhat/strongly agreed that the e-module cases were useful, while 54% were somewhat/very satisfied with the experience. Half of the students would probably/definitely like e-module learning on other rotations, while 52% probably/definitely would recommend the e-module cases to their colleagues. Links to the 5 e-modules were sent to 138 students. YouTube video total views and average time spent were tracked: COPD 142 views with average time spent 5:10 minutes (of 16:31-minute total video time), CHF 85 views with average time spent 6:35 minutes (of 17:41-minute total video time), GIB 74 views with average time spent 5:58 minutes (of 12:02-minute total video time), DVT/PE 73 views with average time spent 6:19 minutes (of 14:17-minute total video time), and DM 59 views with average time spent 5:36 minutes (of 10:01 minute total video time). In this multi-institutional study, 5 common inpatient-based e-modules did not improve subintern knowledge or comfort compared with the control group. Even though 48% of subinterns ranked online learning as their preferred learning method pre-rotation, they overwhelmingly rated direct patient care and co-intern patient observation as modalities that contribute the most to improved knowledge and clinical skills during the subinternship. Slightly more than one-half of the students felt the e-learning modules were useful, were satisfied with the experience, and would use similar modules on other clinical rotations. While knowledge improvement did not reach statistical significance, e-modules could still benefit students that learn best using this modality or may have identified a deficiency in a specific topic. Students that had previously cared for 6 or more patients with any of the 5 medical conditions often felt significantly more comfortable recognizing signs and symptoms of, ordering and interpreting diagnostic tests for, and managing these conditions. Those students had significantly higher pre-rotation scores on only 1 of 10 knowledge-based questions (the single MKSAP question). While this observation may be random or related to the test question content, one must be aware of a possible Dunning-Kruger effect.11Kruger J Dunning D Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments.J Pers Soc Psychol. 1999; 77: 1121-1134Crossref PubMed Scopus (3248) Google Scholar It is unclear whether prior exposure to the 5 medical conditions led to higher-level aspects of the Kirkpatrick learning evaluation model, such as behavior change or improvement in patient care outcomes.12Kirkpatrick DL Evaluating Training Programs: The Four Levels. Berrett-Koehler Publishers Inc., San Francisco: CA1994Google Scholar Regardless, these results emphasize the importance of maximizing clinical exposure during both the third and fourth years of medical school, even during a pandemic. Students likely gained skills and attitudes as well as clinical knowledge through experiential learning by caring for multiple patients with each condition that were not directly measured in our study, but were reflected in their significant comfort recognizing and managing these conditions through direct exposure. The subinterns in the study placed a high value on direct patient care and observational experiences. Clerkship directors, faculty, and residents should be cognizant of this important finding. Historically, multiple factors have competed with bedside learning, including, but not limited to, duty hour rules, time constraints, competing conferences and administrative responsibilities, lack of physician confidence, and patient volume and turnover.13Rousseau M Könings KD Touchie C Overcoming the barriers of teaching physical examination at the bedside: more than just curriculum design.BMC Med Educ. 2018; 18: 302Crossref PubMed Scopus (6) Google Scholar The COVID-19 pandemic created additional obstacles to the bedside experience. Many schools have not permitted students to care for COVID-19-positive patients or those under investigation for COVID-19. Other restrictions include not seeing patients in the emergency department, virtual team rounding, and fewer opportunities for ad hoc teaching from team members due to a lack of shared workspace and less frequent bedside rounds with resident or attending physicians. This shift creates both a challenge and an opportunity for medical education leaders to create novel teaching approaches to enhance the potentially shortened or otherwise limited clinical experience. Intervention group e-module viewing was not required or tracked, so there is no guarantee that every student watched the e-modules, nor any way to track the pace and frequency of viewing per student. For most medical schools, the internal medicine subinternship is offered most months of an academic year. Students performing their subinternship later in the year may have additional fourth-year clinical experiences compared with those completing an earlier subinternship, which could affect knowledge and comfort responses. Similarly, varied alternative learning experiences between students within and across institutions both prior to and during the fourth year of medical school may affect survey responses. The post-rotation and intern survey response rates were significantly lower than the pre-rotation response rate, potentially preventing the identification of meaningful, statistically significant differences between intervention and control groups. Similarly, this analysis only included students that completed the pre- and post-rotation surveys, which may have affected pre to post comparisons of outcomes. Survey knowledge assessment items were designed to target practicing physicians rather than medical students; however, this limitation should have affected both groups equally. Given that the pre- and post-rotation knowledge test questions were the same, it is unclear why the intervention student knowledge scores did not improve. One possible explanation is that intervention students did not watch the videos or only partially watched them. Although unable to track average viewing time spent per individual, given that the overall average video view times were one-third to one-half the complete video duration, it may at least partially explain a lack of significant knowledge score improvement. Students also may have watched these videos at double speed, given most videos were viewed in roughly half of the duration. Although overall well-accepted, 5 e-learning modules did not improve subintern knowledge or comfort in 5 commonly encountered medical conditions. Students placed high value on experiential learning through personal and co-intern patient care. Educators need to develop and implement innovative strategies to get students back to the bedside. E-learning and remote learning have increased exponentially during the recent pandemic and will likely continue to play an important part in multiple facets of learning. Clerkship directors, subinternship directors, and teaching faculty must collaborate to create innovative e-learning curricula that are engaging, interactive, meaningful, longitudinal, complementary to direct patient care, and ideally, improve patient care.