Family medicine resident-led clinics in northern Saskatchewan: A programme evaluation.

Despite investment in maternal-child health programs, there has been little impact on the health outcomes of Indigenous mothers and their children, creating a need to understand how programs can be successfully implemented. Community input is essential for successful programs; however, there is little research exploring the perspectives of frontline workers providing these programs. To gain a better understanding of how to support maternal-child health program success a research partnership was formed with the KidsFirst North program in Northern Saskatchewan, Canada. Using a community-based participatory research approach, this study was codeveloped to (1) explore families', frontline workers', and administrators' perceptions of factors that contribute to the success and barriers of a program for Indigenous families; and (2) describe the current role of frontline workers within health program planning, implementation, and evaluation. From September 2019 to January 2020, data were collected through in-person meetings, focus groups, and semi-structured interviews with KidsFirst North families (n=9), frontline workers (n=18), and administrators (n=7) from 11 sites in Northern Saskatchewan. Data were analyzed using the Collective Consensual Data Analytic Procedure. The identified factors of program success included the importance of staff, where staff demonstrated certain positive characteristics and created a welcoming atmosphere for families; community events that were open to the entire community; and the integration of Indigenous culture in the program. Program barriers included jurisdictional policy that negatively impacted frontline workers, a lack of father inclusion in program activities, and community challenges such as a lack of access to other services within the community. All frontline workers had a role in program delivery, most reported involvement in program development and planning, and approximately half were included in program evaluation. Factors of success and barriers from the KidsFirst North project have illustrated elements to build on and areas to address in public health program planning, implementation, and evaluation of maternal-child health programs that serve Indigenous families. KidsFirst North has demonstrated ways a contemporary maternal-child health program can utilize frontline workers outside of program delivery to influence all aspects of health program planning, implementation, and evaluation. Contributing to the evidence base of maternal-child health programs for Indigenous families may help foster the success of public health programs; inform the role of frontline workers in health program planning, implementation, and evaluation; and positively impact the health of Indigenous children and families.

Community engagement processes conducted with the Northern Village of Pinehouse determined the need for an evaluation of the Recovery Lake Program (RLP). The study exemplifies a collaborative process of sharing experiences and stories, inclusive of key informants from varied backgrounds and expertise, affording the opportunity to weave together Indigenous knowledge, research and policy expertise in a voluntary, non-hierarchical context. The stories shared throughout the interviews offer wisdom that will inform the realignment and direction of growth of some of the facilities and programs offered by the RLP. Through these stories, themes that highlight good practices, positive components and several gaps in the program emerge. Finally, the article includes recommendations for improving the RLP’s treatment and post-program support needs for their clients.

Developing Programs That Will Change Health Professions Education and Practice: Principles of Program Evaluation Scholarship

Curriculum Management and Governance Structure Case Western Reserve University (CWRU) School of Medicine (SOM) has two collaborative and innovative educational tracks: University track and College track. ♦ University-based program (University track) was established in 1843 and Cleveland Clinic-based Cleveland Clinic Lerner College of Medicine (College track) was established in 2002. ♦ University track implemented a major curriculum revision in 2006, focusing on development of lifelong learners and incorporating themes of scholarship, clinical mastery, leadership, and civic professionalism. A dedicated four-month research requirement culminating in an MD thesis was incorporated into the four-year curriculum. A hybrid of active, student-centered PBL groups and teacher-centered large and medium group sessions form the backbone of learning venues. ♦ Five-year competency-based College track is designed to train graduates with excellent clinical skills, expertise in research, and a passion for scientific inquiry. Students complete a master's level thesis and graduate with an MD degree with Special Qualification in Biomedical Research. ♦ While course format, assessment approaches, and curricular topics beyond core content of University and College tracks differ, clinical rotations for both tracks were designed collaboratively and implemented jointly. Curriculum Management and Governance Structure ♦ SOM faculty Committee on Medical Education evaluates, reviews, and makes recommendations concerning overall goals and policies of medical education programs for University and College tracks. ♦ Curriculum Monitoring Council (University track) and Curriculum Steering Council (College track) have responsibility for (1) defining and allocating educational objectives, (2) approving teaching methods and instructional formats, (3) monitoring coordination and integration of curricular content, (4) selecting assessment methods to document student performance, (5) monitoring quality of teaching, (6) overseeing curricular and program outcomes. ♦ Basic science leadership groups for each track facilitate sharing of best educational practices among course leaders, design and implement programs to ensure basic science mastery, and facilitate smooth implementation of methods of student assessment. ♦ Joint Clinical Oversight Group (JCOG) monitors and evaluates clinical experiences at affiliated teaching hospitals of CWRU SOM. JCOG oversees curriculum design, program evaluation, educational quality, student assessment, and compliance with LCME and institutional requirements for the clinical curriculum. ♦ The Dean is chief academic officer responsible for both education tracks. ♦ The Dean delegates day-to-day responsibility for both tracks to the Vice Dean for Education and Academic Affairs. The Executive Dean for the College track reports to the Vice Dean for Education and Academic Affairs. The Executive Dean for the College track is also accountable to the Chief of Staff of the Cleveland Clinic with regard to Cleveland Clinic resources and support for the College track. Curriculum Support Offices ♦ Offices of Curricular Affairs for each track include an associate dean for curricular affairs, MD and PhD educational leaders, and administrative staff who provide support for their respective educational programs. These offices support curriculum development and implementation, ongoing quality improvement, student assessment, and program evaluation. They work collaboratively with respective offices of information technology to develop, improve, and maintain extensive electronic curricula, student assessment resources, and program evaluation methodologies. ♦ Each track has a director of undergraduate clinical education, support staff, and physician and nonphysician educators to support components of clinical curriculum for each track and collaborate on the joint clinical program. ♦ Both tracks have robust faculty development programs to support teaching and implemented initiatives to encourage educational scholarship. Financial Management of Educational Programs ♦ When the Dean committed to revision of the University track's curriculum in 2004, a new curriculum budget was established to support University track curriculum development, implementation, ongoing program maintenance, and improvement. ♦ CWRU School of Medicine has affiliation agreements with its major teaching hospitals. As a part of affiliation agreements with three teaching hospitals, faculty are expected to contribute 100 hours annually to medical school teaching activities. University track established a faculty-teaching database that provides ongoing reports about faculty teaching effort to department chairs. ♦ All College track faculty are salaried employees of a central Cleveland Clinic budget, allowing Department Chairs to allocate release time for teaching and service activities. Valuing Teaching ♦ Qualifications for faculty appointments include “a high level of teaching effectiveness,” with documentation of quality and quantity of teaching activities within teaching portfolios. ♦ Nontenure track faculty appointments are based on an area of excellence: teaching, clinical service, or research. ♦ In addition to schoolwide and departmental teaching awards, faculty are invited annually to submit applications for Scholarship in Teaching awards. Applications are peer-reviewed, and each year 20–30 faculty are recognized by the Dean at the Annual Education Retreat. ♦ Cleveland Clinic faculty are recognized for teaching as part of annual performance reviews, which are used to determine promotion and salary decisions. College track maintains a database of faculty leadership and teaching activities for all components of the curriculum; summary information is provided to faculty members and chairs as part of the reviews. Curriculum Renewal/Development Process University Track Curriculum Development ♦ Major curriculum transformation, Western Reserve2 (WR2), launched July 2006. ♦ Key Objectives for Curriculum Development Process Medical education will be experiential and emphasize skills for scholarship, critical thinking, and lifelong learning. Educational methods will stimulate an active interchange of ideas among students and faculty. Students will be immersed in a graduate school educational environment with flexibility and high expectations for independent study and self-directed learning. Learning will be fostered by weaving scientific foundations of medicine and health with clinical experiences throughout the curriculum. These scientific foundations include basic science, clinical science, population-based science, and social and behavioral sciences. Every student will have an in-depth mentored experience in research and scholarship. Recognizing obligations of physicians to society, central themes of public health, civic professionalism, and leadership will be longitudinally woven throughout the curriculum. Systems issues of patient safety, quality medical care, and health care delivery will be emphasized and integrated throughout the curriculum. Students will acquire a core set of competencies in knowledge, mastery of clinical skills, and attitudes that are prerequisite to graduate medical education. Learning Outcomes/Competencies ♦ WR2 Curriculum is based on nine core competencies with well-defined achievement levels for each that serve as educational objectives (medical knowledge, patient care, communication skills, professionalism, lifelong learning–personal development, research and scholarship, civic professionalism–health advocacy–leadership, problem-based learning and improvement, systems-based practice). Components of the Curriculum Development Process ♦ Preclerkship basic science portion of WR2 curriculum is 20 months in length with six systems-based course blocks that integrate normal and abnormal content throughout (Chart 1).CHART 1: University Track Basic Science Curriculum♦ Weekly in-class basic science teaching is limited to 16 hours a week, paired with high expectations for self-directed learning outside the classroom. ♦ A wide range of dual-degree opportunities are available for students to pursue. ♦ One week of each block is dedicated to experiences in the clinical setting. During Clinical Immersion Weeks, students have opportunities to see how basic science concepts learned in the classroom translate to and impact on patient care. ♦ First block of the curriculum, five weeks in length, is “Becoming a Doctor.” It introduces students to medical school education at the macrolevel with a focus on social and behavioral context of health and disease. ♦ Themes from the first block and additional content in Health Policy, Bioethics, Doctor–Patient relationship, and sociobehavioral medicine are continued in weekly 2-hour seminars. ♦ Case Inquiry Groups (IQ groups), based on McMaster's revised PBL format, serve as a core teaching method for six of the formal teaching hours (three 2-hour sessions per week). ♦ Anatomy, histopathology, and radiology are integrated into a longitudinal “Structure” block. ♦ Learning from multiple sources is emphasized (including a rich array of web-based resources); previous extensive written syllabus was eliminated. ♦ Weekly Foundations of Clinical Medicine Seminars (Doctoring Course) integrates with IQ cases when appropriate. ♦ Sixteen-week mentored research experience with MD thesis is required for all students. Components of the Assessment Process ♦ Goal of deep learning, synthesis, and information transfer expected in WR2 requires that assessment strategies align with curriculum objectives. ♦ CWRU SOM's longstanding pass-fail, criterion-referenced assessment system in the preclerkship curriculum was maintained. ♦ Weekly formative assessments include 20–30 multiple choice questions and 2 synthesis essay questions primarily based on content of the week; these are open book, and group work is encouraged. ♦ End of block basic science summative assessment is a 4- to 5-hour synthesis essay examination; each essay is vignette-based and incorporates multiple concepts and transfer of knowledge to new contexts. ♦ There is an end of block summative anatomy and histopathology examination with practical, short answer, and multiple choice portions. ♦ At the end of each block, students take a formative multiple choice test developed through the NBME customized assessment services. Each test has questions specific to current block of study as well as questions assessing material from previous blocks. This enables students to monitor their progress in studying for USMLE Step 1. ♦ At midpoint of each block, students engage in a personal quality improvement exercise by defining an area for improvement and developing a Professional Learning Plan of action. ♦ Students complete an end of year reflective portfolio to assess progress in nine core competencies. ♦ IQ group performance within areas of (1) contributions to group content and process, (2) skills of critical appraisal, and (3) professional behaviors are evaluated by IQ group faculty facilitators; students engage in peer assessment as well. ♦ Performance in preclerkship clinical curriculum is integrated into the overall assessment process and incorporates formative OSCEs, preceptor feedback, and student reflections. New Topics in the Curriculum Since 2000 ♦ Quality improvement and patient safety, longitudinal theme across four years ♦ Leadership and teamwork, evolving as longitudinal theme across four years ♦ Population health ♦ Increased emphasis on longitudinal development of communication skills, both preclerkship and clerkship components with opportunities for peer teaching in fourth year ♦ Health policy and health care economics Changes in Pedagogy ♦ Student-centered problem-based learning groups (Case Inquiry Groups) ♦ No formal syllabus; rich electronic curriculum guides students to multiple sources for study ♦ Increased emphasis on self-directed learning ♦ Decrease in formal classroom time ♦ Selective videotaping of educational activities; 16 hours of core classroom time not videotaped ♦ Weekly formative assessment and integrative summative assessment ♦ Summative essay examinations focus on synthesis and integration of concepts ♦ Portfolios used to assess nine core competencies ♦ Increased emphasis on clinical context of basic science concepts through IQ cases and clinical immersion activities ♦ Developmental introduction of clinical reasoning skills and case presentations through IQ groups ♦ Integration and extension of basic science curriculum into core clinical clerkships Program Evaluation ♦ Students complete online confidential surveys assessing their perceptions of course faculty, content and instructional methods, and learning resources. ♦ Curriculum committees, individual faculty instructors and course leaders, education leaders, and department chairs receive reports following each course or rotation. ♦ Curriculum effectiveness is assessed by tracking USMLE scores, residency program directors' perceptions, and graduation questionnaire responses. Scores on the Cognitive Behavior Survey, Attitudes toward Social Issues in Medicine, Learning Climate Inventory, Research and Scholarship Checklist, and Case Lifelong Learning Scale are also tracked. College Track Curriculum Development Components of Curriculum Development Process ♦ Affiliation agreement between Cleveland Clinic and CWRU in 2002 ensuring compliance with LCME strategies and appropriate oversight by CWRU. ♦ Mission to train physician investigators who will advance biomedical research and medical practice. ♦ Series of faculty retreats Identified outcomes for College track graduates: independent thinkers, self-directed learners, team players, strong clinical skills, broad-based research skills, scientific inquisitiveness. Developed curricular principles that guided curriculum development: provide a graduate school environment where students are responsible for their learning and seminars focus on application of knowledge; use active learning methods; research is major curriculum thread culminating in master's level thesis; students guided in personal and professional development of physicians and researchers by faculty dedicated to these activities; sufficient time and flexibility in curriculum to accommodate independent study and investigation; basic science, research, and clinical experience integrated into all years of the curriculum. ♦ All students participate in 9- to 10-week summer research experience in basic/translational research (year one) and clinical research (year two) in addition to two-year organ-based basic-science curriculum (Chart 2).CHART 2: College Track Basic Science Curriculum♦ Each week of the curriculum has a theme around which three 2-hour PBL sessions, 8 hours of seminars, and one hour of Advanced Research in Medicine seminars are organized. ♦ Foundations of Clinical Medicine seminars are held weekly. ♦ Thirteen Thread Leaders (representing sciences basic to medicine such as anatomy, physiology, ethics, epidemiology, and biostatistics) charged with responsibility to develop learning objectives for their discipline and work with organ-system course directors to determine best placement of these objectives in the curriculum. Curriculum Steering Council monitors implementation. ♦ Each student is assigned to a family medicine or internal medicine longitudinal preceptor during years one to two; students develop clinical skills with preceptors in every other week half-day sessions in year one and weekly sessions in year two; additional experiences include pediatrics, geriatrics, and acute care. ♦ Formal curriculum hours limited to 21 hours weekly; curriculum changes are time neutral to maintain flexibility for students. ♦ Opportunities available to earn master's degrees (Public Health, Engineering, Biomedical Investigation in Clinical Research, Nutrition, Pathology, or Biochemistry). ♦ Cleveland Clinic provides full tuition scholarships to all College track medical students to avoid significant financial debt at graduation and thereby facilitate entry into research careers. Learning Outcomes/Competencies ♦ The curriculum is based on nine competencies; seven reflecting ACGME competencies (medical knowledge, clinical skills, clinical reasoning, communication, professionalism, health care systems, reflective practice) as well as research and personal development. ♦ Each competency has three to five specific standards for students to achieve by the end of years one, two, and five. Components of the Assessment Process ♦ Goal of the College track assessment process is to help students become reflective practitioners of medicine complemented by a critical approach to self-assessment and self-improvement. ♦ Faculty developed assessment principles that require frequent, formative assessments to enhance student learning and engage students in ongoing cycles of self-assessment, supported by mentoring from physician advisors. ♦ Assessments align with the College track's nine competencies and developmentally appropriate standards; there are no grades or class rankings. ♦ Assessments are collected from multiple sources (faculty, peers, self) and multiple methods (OSCEs, faculty observations, MCQs) and contexts to provide students with frequent narrative formative feedback across the curriculum to identify strengths and areas needing improvement and document students' achievement of competency standards. ♦ In basic science courses, weekly CAPPS (concept appraisal essays) require students to integrate and apply knowledge to solve problems related to course material. ♦ All assessments collected in an ePortfolio for students and their advisors to reflect on and monitor progress longitudinally. ♦ Students use their assessment data to construct both formative and summative portfolios that document achievement of competency standards. ♦ Students' formative portfolios are reviewed with their advisors to develop learning plans; summative portfolios are reviewed by a Medical Student Promotion and Review Committee for promotion decisions. New Topics in the Curriculum Since 2006 ♦ Clinical reasoning in year two. ♦ Integrated program in medical humanities. ♦ Bioinformatics taught as a component of the Genetics Thread progressively over first two years. Changes in Pedagogy ♦ Course directors have increased use of small group sessions (eight students per group) whenever possible to improve teacher–student and student–student interactions and increase student learning. ♦ Emphasis on faculty development to assist faculty in developing skills in small group, interactive teaching, and providing useful formative feedback. Program Evaluation ♦ College track uses a systematic approach based on principles (collect only essential information, and so on) to determine fulfillment of institutional goals, identify curricular strengths and areas requiring improvement, and contribute to educational research. ♦ Student feedback is collected for all courses and track-specific innovations (portfolio-based assessment, research thesis, and so on) using multiple methods (debriefing meetings, web-based questionnaires, performance assessments, focus groups). ♦ Office of Curricular Affairs for College track summarizes evaluation evidence for each course into formal reports for course directors and related committees to identify curricular strengths and limitations. ♦ Course directors reflect on this information and prepare written reports for presentation to respective curriculum committees and approval by College track's curricular governance committee. ♦ Office of Curricular Affairs uses external, track-specific data (AAMC Matriculation and Graduation Questionnaires, USMLE reports) for bench-marking purposes. ♦ Since College track's inception, assessment and program evaluation data maintained in IRB-approved data registry to ensure ethical practices for program evaluation and educational research. Clinical Experiences ♦ Clinical instruction and rotations for both tracks designed collaboratively and shared from third year onward. ♦ Four affiliated teaching hospitals: University Hospitals/Case Medical Center, Cleveland Clinic, MetroHealth Medical Center, Veterans Affairs Medical Center. ♦ Clinical rotations begin March of second year. ♦ Required clinical rotations: 40 weeks of basic core rotations, 8 weeks of advanced core rotations, and two subinternships Three basic cores, each completed at one site Basic Core 1 (medicine and surgery, 16 weeks) Basic Core 2 (pediatrics, OB–Gyn, family medicine, 16 weeks) Basic Core 3 (neuroscience and psychiatry, 8 weeks) Two advanced cores Aging and Society (4 weeks) Undifferentiated and Emergent Care (4 weeks) ♦ Robust electronic Clinical Assessment System (CAS) combines patient logs with real time formative feedback and summative assessment in competency-based format. ♦ Track-specific curriculum one afternoon a week at students' home base University track: IQ+ program to integrate reflection, basic science content, and advanced clinical skills into clerkships College track: programs in advanced research skills and medical ethics and humanities ♦ Challenges Achieving uniformity in student assessment across broad base of faculty Students' desire for more formative assessments from faculty; challenging to encourage busy faculty to provide more frequent written narratives Supporting clinicians' time for teaching Moving faculty and students to consider new models for clinical education Highlights of University Track ♦ High expectations for active, student-centered learning ♦ Case Inquiry Groups: new PBL variant with shorter, more focused cases ♦ Clinical immersion weeks embedded within basic science blocks ♦ Integrative essays to assess basic science knowledge ♦ Sixteen-week research and scholarship requirement with MD thesis ♦ Integration of SNAPPS case presentations across four-year curriculum to enhance expression and strengthening of clinical reasoning ♦ Portfolio assessment of competencies ♦ Integration of basic science into core clerkships through weekly IQ+ program Highlights of College Track ♦ Small class size promotes collaborative learning environments, interactive teaching methods, collegial relationships with faculty ♦ Five-year integrated research curriculum and required master's level thesis ♦ Reliance on formative assessments to document student achievement of competencies; no grades, no class ranking ♦ Portfolio approach for competency-based assessment system ♦ Institutional commitment to faculty development for teaching and assessment roles and full tuition scholarships for all College track students ♦ Comprehensive program evaluation activities Highlights of Shared Clinical Curriculum ♦ Clinical Assessment System featuring continuous formative assessment ♦ Clerkship groupings in 8- to 16-week integrated blocks ♦ Advanced cores introduce new, innovative curricular content ♦ Dedicated curriculum time during clerkships for program- specific educational goals

BackgroundHealthcare systems rely on well-trained family medicine physicians who can offer continuous quality services to their communities and beyond. The American Academy of Family Physicians and the World Organization of Family Doctors recommend that medical curricula should have adequately supervised education and training of the learners in family medicine during their preclinical and clinical placements. However, some medical schools don’t have a comprehensive family medicine program to prepare graduates who can meet the community needs. This work aims to report the essential steps for the development, implementation, and evaluation of the family medicine program at the College of Medicine at the University of Sharjah in United Arab Emirates.MethodsWe used the Kern’s 6-step model to describe the development, implementation, and evaluation of the family medicine program. This includes problem identification, needs assessment, goals setting, educational strategies, implementation, and evaluation. During 2014–2022, we longitudinally collected essential information about the family medicine program from different stakeholders including the feedback of clinical coordinators, adjunct clinical faculty, and medical students at the end-of-clerkship. All responses were analysed to determine the effective implementation and evaluation of the family medicine program.ResultsOver the course of 8 academic years, 804 medical students, 49 adjunct clinical faculty and three College of Medicine faculty participated in the evaluation of the family medicine program. The majority of respondents were satisfied with various aspects of the family medicine program, including the skills gained, the organisation of program, and the variety of clinical encounters. The medical students and adjunct clinical faculty suggested the inclusion of e-clinics, faculty development program, and the expansion of more clinical sites for the effectiveness of the family medicine program.ConclusionsWe report a successful development, implementation, and evaluation of the family medicine program in United Arab Emirates with a positive and impactful learning experience. More attention should be paid towards a suitable representation of family medicine program in the medical curriculum with focused and targeted educational plans for medical students.

Are We Preparing Residents for Their Actual Practices?

McGill University Faculty of Medicine.

As of 2014, the Accreditation Council for Graduate Medical Education (ACGME) mandates initiating a Program Evaluation Committee (PEC) to guide ongoing program improvement.However, little guidance nor published reports exist about how individual PECs have undertaken this mandate. To explore how four primary care residency PECs configure their committees, review program goals and undertake program evaluation and improvement. We conducted a multiple case study between December 2022 and April 2023 of four purposively selected primary care residencies (e.g., family medicine, pediatrics, internal medicine). Data sources included semi-structured interviews with four PEC members per program and diverse program artifacts. Using a constructivist approach, we utilized qualitative coding to analyze participant interviews and content analysis for program artifacts. We then used coded transcripts and artifacts to construct logic models for each program guided by a systems theory lens. Results: Programs adapt their PEC structure, execution, and outcomes to meet short- and long-term needs based on organizational and program-unique factors such as size and local practices. They relied on multiple data sources and sought diverse stakeholder participationto complete program evaluation and improvement. Identified deficiencies were often categorized as internal versus external to delineate PEC responsibility, boundaries, and feasibility of interventions. The broad guidance provided by the ACGME for PEC configuration allows programs to adapt the committee based on individual needs. However, further instruction on program evaluation and organizational change principles would augment existing PEC efforts.

The Milestones Development team at the Accreditation Council for Graduate Medical Education (ACGME) has periodically surveyed the graduate medical education (GME) community and volunteers, during and after development of the specialty-specific Milestones 2.0, to check with members of each specialty on the implementation of the new Milestones. As we approached the conclusion of Milestones 2.0 development, it was decided to have broader survey dissemination for all specialties. Although still only 10 questions, 2 major changes were made to the quality assurance feedback questionnaire: (1) It solicited comments and impressions from the approximately 26 000-member email distribution list that receives the ACGME e-communications rather than only those who volunteer to be respondents; and (2) It had a more summative focus meant to be the start of the upcoming program evaluation stage of Milestones, rather than soliciting formative input for any further changes to Milestones 2.0. The survey was open for 6 weeks in the fall of 2022 and resulted in 215 responses, which, after clearing the data of collection errors and detectable duplications, produced 211 valid responses from 27 specialties across the GME community. At the time of distribution, email and survey tracking metrics were not available to produce a meaningful response rate. Respondents provided general descriptions of their specialties and specific roles, their program enrollment, and the membership of their Clinical Competency Committees (CCCs). They were then asked open-ended questions that allowed them to describe any challenges with Milestones 2.0 and to make suggestions for the program as a whole. The intent of this 10-item questionnaire was to collect data that will direct conversations on change and help guide this first step, as well as further program evaluation projects that explore the relationships between Milestones and the GME community. The results of the qualitative analysis of this questionnaire are based on those who chose to respond to the ACGME e-communication request for feedback. The subsequent findings should not be generalized to the GME community or be made to represent thoughts or reactions of any subpopulation therein.Of 211 valid responses, roughly half (n=105, 49.8%) identified as program directors (Table 1) from a dropdown list of options (Question [Q]3), and 46 (21.8%) indicated that they were their CCC chair (Q4). The most represented ACGME core specialties (Q1) were internal medicine (n=46, 21.8%), family medicine (n=35, 16.6%), pediatrics (n=24, 11.4%), and surgery (n=17, 8.1%). Additionally, 80 (37.9%) respondents self-identified a subspecialty (Q2) in the open text question following the specialty question.The responses to (Q5) “Indicate the number of residents or fellows you are currently assessing in your program using the Milestones” ranged from 2 to 170. This question was asked to obtain an approximate number of residents and fellows being assessed, and it was beyond the purview of this project to check a respondent's accuracy regarding this data point.When asked to describe the roles within their current CCC membership (Q6), 181 (85.8%) of the respondents listed their committee as having a program director, 162 (76.8%) indicated a program coordinator, 161 (76.3%) listed an assistant or associate program director, and 199 (94.3%) specified a core faculty member (Table 2). While the survey question had a “public member” option, no respondents indicated as such.Two open-ended questions guided the major focus of the survey: (Q8) “Indicate any challenges you have experienced with Milestones 2.0, or related concerns,” and (Q9) “What changes or resources would you like to see in the Milestones?” Finally, respondents could self-identify under (Q10) “Enter your email and full name to be notified of future Milestones feedback opportunities,” which produced a list of volunteers for future research projects.Researchers from the Milestones department used emergent, in vivo coding to categorize the open-ended responses describing challenges and providing suggestions. While responses reflected specific and individual experiences, some patterns emerged from similar data during analysis that helped paint a picture of the challenges facing the GME community around Milestones. Guided by a heuristic inquiry methodology, the research team performed the coding in the MAXQDA software (VERBI Software GmbH) to label the open-ended responses and look for the patterns using a constant comparative analysis approach.1 The most prevalent challenges and suggestions are discussed below with exact verbiage lifted from the responses.The first prominent idea uncovered in the response data involved faculty use and interpretation of the Milestones. Particularly, many respondents mentioned faculty not being fully invested in the Milestones as an assessment tool and that faculty may consider the goals of the Milestones as too lofty or unachievable.Another common topic from the data highlighted issues with specific subcompetencies, namely those that measure advocacy or well-being. Some respondents called out a few Milestones that were difficult to observe as well as directly assess, particularly those speaking to system management and the physician role in the health care system. Similarly, there were comments that described the Milestones as being too vague or subjective and too difficult to transfer to the provided scale. One comment stated: “Based on how these are worded many of our residents hit a level 3 at PGY-2 and then stay there despite continuing to improve in their abilities.” (internal medicine)Data also reflected issues with the amount of time required to adequately assess residents and fellows on each Milestones subcomeptency. This may be related to other comments where respondents felt there were too many Milestones to assess or that the Milestones were too long or wordy. “Too many Milestones! Takes way too long to complete them twice a year for all of our residents.” (diagnostic radiology)Some respondents described challenges with transitioning from Milestones 1.0 to 2.0, such as making adjustments to their assessment practices and software tools to reflect the changes. One comment pointed out that it would get easier: “There are some that don't correlate perfectly with the new milestones, so this made tracking resident improvement over time difficult. This should get easier once we use the 2.0 version for all 3 resident years.” (pediatrics)There were also a few respondents who felt as though the Milestones were not applicable to actual clinical practices. For example: “I find the wording of the Milestones 2.0 incredibly hard to apply to real world resident performance.” (internal medicine) and “The verbiage is still extremely convoluted and often does not apply to real life.” (family medicine) Others indicated that the Milestones do not capture the context of the resident's “rotation experience.” (family medicine)Similarly, data from the survey described issues with few opportunities for residents to be assessed or actually observed within specific Milestones contexts. They also spoke of other challenges: “Bias from prior milestone evaluations. Some CCC members may evaluate residents that they have not personally worked with over the past 6 months.” (surgery–surgical oncology) and “Finding ways for residents to achieve, and for us to observe and document, some milestones elements (ex. SBP2, SBP3, PROF1, ICS1), especially those in the upper levels.” (pathology–surgical pathology)There remained some responses that did not fall into a particular pattern or theme but still highlighted concerns worth mentioning:The survey respondents' suggestions were just as varied as their challenges, but some themes were evident. The most prominent area for suggestions centered around the need for clearer applications and more examples for the Milestones levels, either beyond the existing supplemental guidebooks or an updated version of them.Another common thread among the suggested changes was the inclusion of either additional Milestones or a new focus reflected within Milestones. Particularly, these responses spoke to skills development in leadership, teaching, and scholarly activities.Respondents also focused their suggestions on language simplification and applying more realistic language as well as condensing Milestones, making them shorter or even reducing the overall number of them.Another theme identified the need for more faculty development and education that address “goals, changes, and interventions to address failure to progress.” (surgery–acute care surgery) Responses also spoke to the need for evaluation forms that would be easy to use and implement into the learner's record automatically. Some respondents suggested using the Milestones as the assessment tool: “Evaluation tools would be helpful, not just the milestones but also some suggested way to assess. Or just let us use the milestones as assessment tools, because the rubrics are built out well enough now that they could be used as direct assessments.” (anesthesiology)Individual comments spoke to a wide variety of suggestions, such as the following:This survey was developed to collect general, voluntary feedback with only 2 focused questions for challenges and suggestions as an early look informing program evaluation for Milestones. While the data are not representative of any group or subgroup within GME, the broad and organic analysis shows that the GME community feels that the Milestones program should continue its focus on providing a transparent and concise assessment tool that does not overburden faculty, program directors, or the residents and fellows.2-5 The survey results also indicate the need to reiterate the validity of the Milestones assessments in the harmonized competencies as well as in patient care and medical knowledge.6,7 There is evidence supporting more emphasis on developing a concerted effort by faculty and CCCs to utilize the tools and resources that the Milestones program provides. Some survey respondents also expressed the need for Milestones to continue efforts for developing subcompetencies that could integrate with other specialty-specific assessment tools.The Milestones department has already researched and taken steps to provide GME faculty with access to resources, such as guidebooks (Milestones, CCCs, assessment, etc) and faculty development courses,8,9 and currently offers periodic courses to address faculty understanding and usage of the Milestones assessments. The department also has developed supplemental resources and has explained the rigor that goes into developing the validity and reliability of the tools.10 The data also spoke to the importance of keeping current and future resources visible, available, and accessible to the GME community.The ACGME Milestones team would like to thank those who took part in the survey and provided their valuable feedback. The resulting suggestions and conversations will give the Milestones department the foundations to better inform their upcoming evaluation research on the Milestones program. Over the next few years an intensive review of the Milestones will occur. This research will delve into the development process, content, and implementation within programs. Data will be collected through focus groups, interviews, content analyses, and targeted surveys. If you are interested in sharing your opinion, please send an email with your name, specialty, and/or subspecialty to MilestonesQA@acgme.org.

2022 ACGME Annual Educational Conference Wrap-Up.

The Ohio State University College of Medicine

The 2019 Primary Care and Family Medicine Education network (Primafamed) meeting in Kampala, Uganda, included a workshop that aimed to assess the state of postgraduate family medicine training programmes in the Primafamed network. Forty-six people from 14 African and five other countries were present. The evaluation of programmes or countries according to the stages of change model was compared to a previous assessment made 5 years ago. Most countries have remained at the same stage of change. Two countries appeared to have reversed their readiness to change as Rwanda moved from relapse to pre-contemplation and Mozambique moved from action to contemplation. Malawi, Zambia and Zimbabwe increased their readiness to change and moved from contemplation to action. Countries in the region remain quite diverse in terms of their commitment to family medicine training. Within Primafamed, it is possible for countries with a more advanced stage of change to assist countries with an earlier stage. Primafamed is also supported by a variety of partners outside of Africa. Five years after the previous country-level assessment, family medicine in Africa continues to span across all levels of the stages of change model. Stage-matched interventions aligned with the needs of individual countries should follow. Consequently, this workshop report will serve as a mandate and compass for Primafamed’s actions over the next few years, aimed at designing and delivering these interventions. As reiterated in the 2019 Kampala commitment, we should continue developing the discipline of family medicine (the medical ‘specialty’ of primary care), through alignment of our training programmes to the health needs in the African region.

Western University Schulich School of Medicine & Dentistry.

University of Alberta Faculty of Medicine and Dentistry.

Texas Tech University Health Sciences Center School of Medicine, Lubbock.