American ophthalmology graduate medical education and the web: current state of internet resource utilization

To assess the accessibility and breadth of online information regarding neurotology fellowship programs by appraising individual fellowship websites as well as two popular online databases. The American Neurotology Society Program Information Page (ANSPIP), the American Medical Association's Fellowship and Residency Electronic Interactive Database (FREIDA), and three online search engines (Google, Yahoo, Bing) were assessed for accessibility to individual fellowship websites. Each program's ANSPIP data sheet and fellowship-specific website were then evaluated for the presence of 18 characteristics of interest to the neurotology fellowship applicant. All three search engines yielded 23 (96%) Accreditation Council for Graduate Medical Education (ACGME)-accredited neurotology fellowship websites. Searching "Neurotology fellowships" in Google, Yahoo, and Bing required exploring up to nine pages of search results to identify all of the ACGME-accredited program-specific websites. Direct links to program websites were found on 12 (55%) and 3 (13%) of the ANSPIP and FREIDA fellowship pages, respectively. Of the 18 characteristics of interest, the ANSPIP data sheet and individual program websites on average delineated 8.0 and 7.8 characteristics, respectively. Ninety-one percent of fellowship websites included program coordinator information, general program description, and breadth of surgical exposure. In contrast, work/life balance, postfellowship placement, and current fellow(s) were least commonly displayed (4%, 9%, 13% respectively). One-hundred percent of ANSPIP data sheets included program coordinator information, number of fellows, and affiliated hospital(s), whereas none of the pages included selection criteria, application requirements, description of location, or work/life balance. Although most neurotology fellowship programs have websites or ANSPIP data sheets, many of them lack information that has been previously demonstrated to be valued by applicants. Furthermore, incongruence of information between these sources may lead to confusion, applicant stress, and reflect poorly on fellowship programs. Perhaps a standardized list of ACGME-required data points to be posted on websites would facilitate the application process.

ObjectiveOtolaryngology residency applicants face challenges finding accurate information about training programs. Social media platforms are not verified, and official training program websites are not standardized. Currently, the American Medical Association's Fellowship and Residency Electronic Interactive Database (FREIDA) is the most comprehensive sanctioned and verified otolaryngology residency program database. However, the exact amount of information shared by individual programs included is not presently known.MethodsHerein, we analyzed the available data on all 124 otolaryngology residency programs in FREIDA to assess the completeness of the database.ResultsWhile every program listed an address, contact email, and the name of the program director, more than half of programs (n = 65, 52.4%) did not provide additional information. Most programs (70.2%) did not include a one‐paragraph program description.ConclusionOur findings suggest that while FREIDA is the only sanctioned online database for residency programs, it is inadequately populated with detailed program information.Level of EvidenceN/A

ObjectiveMany psychiatry residency programs use Instagram to provide information about their program. This study assesses the content and engagement on psychiatry residency program Instagram accounts.MethodsA full list of psychiatry programs was gathered from the American Medical Association Fellowship and Residency Electronic Interactive Database (FREIDA). FREIDA was used to collect demographic information about a program. Program Instagram accounts were reviewed and assessed for content, and engagement scores were analyzed. Univariate and multivariate regression was used to analyze the information collected from the Instagram accounts.ResultsThere are 109 (42.9%) psychiatry residency programs with Instagram accounts; 99 (90.8%) accounts were opened in 2020. Analysis of the content of posts revealed posts with the following themes: 50.5% were departmental, 21.9% were social, 14.9% were other, 10.1% were academic and professional, and 2.6% were educational. Using multivariate regression, a higher total number of followers was correlated with the total number of accounts followed (p ≤ 0.001) and Instagram stories (p = 0.047) (R2 = 0.579). Engagement score was correlated with a total number of followers (p ≤ 0.001), program size (p = 0.048), and whether an account was active (p = 0.003) (R2 = 0.450).ConclusionResults demonstrate that 2020 led to an increase in the number of psychiatry residency program Instagram accounts and engagement with followers. Instagram provides a way to further promote a program, yet there is room for improvement to diversify the content.

BackgroundThe historic gap in gender diversity within orthopaedic surgery is widely acknowledged and continues to persist. The lack of female representation in orthopaedic surgery has been attributed to a variety of factors, including the absence of female mentors and leaders within the field. As such, we sought to examine the gender diversity among orthopaedic surgery faculty in various subspecialties at academic institutions and the distribution of female faculty in positions of leadership.MethodsThe American Medical Association Fellowship and Residency Electronic Interactive Database (FREIDA) was used to identify all allopathic orthopaedic surgery residency programs during the 2022 to 2023 academic year. The total number of faculty, and distribution of female faculty by subspecialty were collected from January to March 2023. The mean and percentage of female faculty in each subspecialty per program was calculated.ResultsThe total number of orthopaedic surgery female faculty identified was 524. The subspecialty with the highest percentage of female faculty per program was pediatrics at 26.1% (148/511). Hand (18.6%; 113/511), oncology (19.2%; 38/511), foot and ankle (13.6%; 49/511), spine (3.9%; 21/511), shoulder and elbow (7.4%; 7/511) and adult reconstruction (3.7%; 24/511) had lower percentages of female faculty per program. A total of 52 (10.2%) female section chiefs were identified across all programs. Oncology had the highest percentage of female faculty represented in section leadership at 18.4% (7) and sports medicine had the lowest at 4.8% (4).ConclusionGender diversity of faculty in orthopaedic surgery is low with adult reconstruction (3.7%), spine (3.9%), and shoulder and elbow (7.4%) having the lowest percentages of female faculty. The percentage of female faculty represented in section leadership is also lacking with a total of 52 (10.2%) female section chiefs identified across all programs. Increasing the number of females in leadership positions across all orthopaedic subspecialties may be one step in helping improve gender diversity in the field.

BackgroundToday’s residency applicants submit more applications than those in the past. To facilitate holistic review, many program directors have encouraged applicants to submit fewer applications. However, whether programs provide sufficient information to help applicants determine where to apply is unclear.ObjectiveTo evaluate the frequency of missing information on residency program websites and in the Fellowship and Residency Electronic Interactive Database (FREIDA).MethodsWe used FREIDA to identify all categorical pediatric residency programs in the United States. We noted the presence of information programs reported in each FREIDA data field. We compared information available on the program website for consistency with the information in FREIDA and additionally searched for current resident information and any description of the qualities of applicants/residents desired on the program website.ResultsTwo hundred and eleven pediatric residency programs were included in FREIDA. Approximately 25% of programs did not include basic information such as number of first year residents, salary, work hours, or consideration of applicants requiring work visas. Over half of programs did not report minimum licensing examination scores required for interview consideration. Discrepancies between information on program websites and FREIDA related to work visas occurred in 6-8% of programs. While 88% of program websites included information on current residents, only 17% included any description of the applicant attributes sought by the program.ConclusionsMany pediatric residency programs do not provide much of the information that applicants need to help determine if a program is a good fit or whether their application is competitive.

Although the number of women entering dermatology residency programs is increasing, they still encounter numerous challenges and disparities, including limited career opportunities and difficulties in balancing family planning with their professional lives. Parental leave policies have been recognized for their positive impact on maternal, fetal, and familial well-being, career satisfaction, and gender equality. However, negative perceptions and a lack of awareness surrounding these policies may discourage female residents from taking parental leave during training. This study investigates the relationship between the gender of department chairs and the advertisement of paid parental leave policies in dermatology residency programs. Data from the American Medical Association's Fellowship and Residency Electronic Interactive Database (FREIDA Online) and the Canadian Residency Match System (CaRMS) were utilized to identify accredited residency programs in the USA and Canada, respectively.Manual searches for gender of department chairs and parental leave policies in dermatology programs were performed between December 2021 and January 2022. Out of the 146 programs in the USA and 10 in Canada, most department chairs were male. Only 9.6% of programs, all from the USA, explicitly advertised paid parental leave policies specific to dermatology residents. There was a significant correlation between the gender of the department chair and the presence of specific policies, which was only observed in the northeastern region of the USA. This study highlights the lack of advertised parental leave policies in dermatology residency programs and potential barriers preventing access to this crucial information. Lack of clear policies and negative perceptions may deter female residents from considering childbearing during their training or hinder them from taking parental leave when needed. Future research should explore program-specific reasons for policy advertising or omission and their perception by applicants and current residents.

Gender Differences Among Academic Pediatric Radiology Faculty in the United States and Canada

To investigate the associations between program-level variables such as organizational structure, workload, and learning environment and residents' development of depressive symptoms during internship. Between 2012 and 2015, 1,276 internal medicine interns from 54 U.S. residency programs completed the Patient Health Questionnaire-9 (PHQ-9) before internship, and then quarterly throughout the internship. The training environment was assessed via a resident questionnaire and average weekly work hours. The authors gathered program structural variables from the American Medical Association Fellowship and Residency Electronic Interactive Database (FREIDA online) and program research rankings from Doximity. Associations between program-level variables and change in depressive symptoms were determined using stepwise linear regression modeling. Mean program PHQ-9 scores increased from 2.3 at baseline to 5.9 during internship (mean difference 3.6; SD 1.4; P < .001), with the mean increase ranging from -0.3 to 8.8 (interquartile range 1.1) among included programs. In multivariable models, faculty feedback (β = -0.37; 95% CI: -0.62, -0.12; P = .005), learning experience in inpatient rotations (β = -0.28; 95% CI: -0.54, -0.02; P = .030), work hours (β = 0.34; 95% CI: 0.13, 0.56; P = .002), and research ranking position (β = -0.25; 95% CI: -0.47, -0.03; P = .036) were associated with change in depressive symptoms. Poor faculty feedback and inpatient learning experience, long work hours, and high institutional research rankings were associated with increased depressive symptoms among internal medicine interns. These factors may be potential targets for interventions to improve wellness and mental health among these professionals.

The transition from American Osteopathic Association (AOA) and Accreditation Council for Graduate Medical Education (ACGME) residency matches to a single graduate medical education accreditation system culminated in a single match in 2020. Without AOA-accredited residency programs, which were open only to osteopathic medical (DO) graduates, it is not clear how desirable DO candidates will be in the unified match. To avoid increased costs and inefficiencies from overapplying to programs, DO applicants could benefit from knowing which specialties and ACGME-accredited programs have historically trained DO graduates. This study explores the characteristics of residency programs that report accepting DO students. Data from the American Medical Association's Fellowship and Residency Electronic Interactive Database Access were analyzed for percentage of DO residents in each program. Descriptive statistics and a logit link generalized linear model for a gamma distribution were performed. Characteristics associated with graduate medical education programs that reported a lower percentage of DO graduates as residents were surgical subspecialties, longer training, and higher US Medical Licensing Examination Step 1 scores of their residents compared with specialty average. Characteristics associated with a higher percentage of DO graduates included interviewing more candidates for first-year positions and reporting a higher percentage of female residents. Wide variation exists in the percentage of DO graduates accepted as residents among specialties and programs. This study provides valuable information about the single Match for DO graduates and their advisers and outlines education opportunities for the osteopathic profession among the specialties with low percentages of DO students as residents.

Inadequate representation of underrepresented racial and ethnic minorities (URMs) in Graduate Medical Education (GME) programs helps perpetuate growing poor respiratory health outcomes in underserved communities. The barriers that lead to racial and ethnic gaps in GME programs are a downstream result of fractured educational programs that prepare students from URM groups to matriculate into medical education programs. Challenges in the matriculation of UMRs into majors in Science, Technology, Engineering, and Math (STEM) result in the subsequent shortage of URMs in GME programs equipped to address respiratory health disparities, which then leads to inadequate diversity in the physician workforce in pulmonary and critical care medicine, a contributory factor to respiratory health disparities. Multifaceted and comprehensive approaches, encompassing all stages of education and career development, are needed to ensure adequate representation of URMs in GME programs. A diverse workforce in pediatric and adult training programs in pulmonary and critical care medicine is an essential step to achieve respiratory health equity in the United States.

On Blast: A Framework for Monitoring and Responding to Online Comments About Your Graduate Medical Education Program.

American Diagnostic Radiology Residency and Fellowship programmes are Graduate Medical Education programmes in the United States (US) equivalent to the Postgraduate Medical Education programmes in Singapore. Accreditation Council for Graduate Medical Education (ACGME) accredited diagnostic radiology residency programmes require 5 years total with Post Graduate Year (PGY) 1 year internship in a clinical specialty, e.g. Internal Medicine following medical school. PGY Years 2 to 5 are the core years which must include Radiology Physics, Radiation Biology and rotations in 9 required subspecialty rotations: Abdominal, Breast, Cardiothoracic, Musculoskeletal, Neuroradiology, Nuclear and Paediatric Radiology, Obstetric & Vascular Ultrasound and Vascular Interventional Radiology. A core curriculum of lectures must be organised by the required 9 core subspecialty faculty. All residents (PGY 2 to 4) take a yearly American College of Radiology Diagnostic In-Training Examination based on national benchmarks of medical knowledge in each subspecialty. Because the American Board of Radiology (ABR) examinations are changing, until 2012, residents have to take 3 ABR examinations: (i) ABR physics examination in the PGY 2 to 3 years, (ii) a written examination at the start of the PGY 5 year and (iii) an oral exam at the end of the PGY 5 year. Beginning in 2013, there will be only 2 examinations: (i) the physics and written examinations after PGY 4 will become a combined core radiology examination. Beginning in 2015, the final certifying examination will be given 15 months after the completion of residency. After residency, ACGME fellowships in PGY 6 are all one-year optional programmes which focus on only one subspecialty discipline. There are 4 ACGME accredited fellowships which have a Board Certifi cation Examination: Neuroradiology, Nuclear, Paediatric and Vascular Interventional Radiology. Some ACGME fellowships do not have a certifying examination: Abdominal, Endovascular Surgical Neuroradiology and Musculoskeletal Radiology. One year unaccredited fellowships can also be taken in Breast, Cardiothoracic or Women's Imaging.

The Accreditation Council for Graduate Medical Education (ACGME) updated the common core requirements for graduate medical education (GME) programs (effective July 1, 2019) to include a new provision, "The program, in partnership with its Sponsoring Institution, must engage in practices that focus on mission-driven, ongoing, systematic recruitment and retention of a diverse and inclusive workforce of residents."1 The ACGME's call for greater inclusion in GME presents an opportunity to include disability as an aspect of diversity in systemic recruitment and retention efforts. A 2016 prevalence study found that 2.7% of US MD candidates disclosed disabilities, most having nonapparent disabilities (eg, attention deficit/hyperactivity disorder, learning difficulties, or psychological disabilities).2 This represents a larger cohort of students with disabilities entering GME than previously imagined3–5 and suggests potential increases in requests for accommodation. Numerous resources exist to aid undergraduate medical education programs in disability-related recruitment and retention efforts,6–13 including guidance on technical standards,14,15 clinical accommodations,16 and inclusive assessment.17,18 The GME guidance is less robust.The ACGME advises programs that "the Sponsoring Institution must have a policy, not necessarily GME-specific, regarding accommodations for disabilities consistent with all applicable laws and regulations."19(p14) Much existing scholarship and resources focus on resident litigation and difficulty in performance or behavior.20,21 Articles that explore successful inclusion of disabled residents espouse the value of early disclosure and use of accommodation as potential mediators of success.22,23Residents with disabilities are already enrolled in training programs,6 and the pipeline of students in undergraduate medical education2 will soon transition to GME. To meet learner needs and realize the ACGME's new common core requirement, GME programs must create inclusive policies and practices, understand their responsibilities under federal law, and educate themselves regarding reasonable accommodations. Without those key elements, programs may be ill prepared to accommodate residents' disability-related needs (box 1). This perspective offers an overview of systemic barriers in GME for residents with disabilities and mechanisms to reframe those barriers as opportunities to build programs that are more inclusive.The scenario described in the box 1 is not unique. A 2018 report from the Association of American Medical Colleges identified 3 structural barriers to accessibility in GME, which included the absence of clearly defined policies and processes, a knowledgeable and identifiable point person for facilitating accessibility requests, and an understanding of the legal requirements for equal access under the Americans with Disabilities Act as amended (ADA-AA).24 To those 3 barriers, we add a fourth: unfamiliarity with the benefits of disability inclusion.Residents exist in a liminal space between student and employee, resulting in some confusion about who oversees disability-related needs. When programs fail to identify policies for requesting disability accommodations, residents lack clarity about who is responsible, or they incorrectly assume that the program does not make accommodations. The lack of explicit accommodation policies may also discourage qualified applicants.The lack of a qualified expert in disability inclusion as part of the interactive process to determine accommodations also poses a barrier. Legal requirements call for such a process, necessitating a good-faith exploration of options between the resident and the institution.Program leadership may not have a clear understanding of institutional obligations under employment provisions of the ADA-AA (Title I). The law outlines employer obligations to ensure equal access for qualified employees with disabilities, including provision of reasonable accommodations and responsibility to fund them.25Program directors may not fully appreciate the benefits of disability inclusion and may falsely believe that residents with disabilities require dedicated administrative time or add high costs to the program.Straightforward accommodation policies and statements that emphasize the value of disability as part of diversity help to destigmatize disability. These measures also aid in recruitment and retention of diverse applicants (table; box 2). Programs should provide accommodation policies in communications to prospective and matched residents, on program websites, and in resident handbooks. Programs and institutions must also develop their understanding of financial responsibilities for employee accommodations. Transparency about the accommodation process will likely facilitate earlier disclosure by residents, a potential mediator of success, regardless of whether the resident enters the program with a disability or develops one during training.22Programs should work with GME offices to establish a process for disability disclosure that includes a confidential point of contact with expert knowledge of disability law and clinical accommodations (figure) who can assist with determining reasonable accommodations (box 3).6 This should be someone other than a colleague, supervisor, or anyone else who would evaluate the resident's performance.Programs should maintain a clear understanding of their legal responsibilities, including those for reasonable accommodations. The ADA-AA defines disability broadly; thus, programs should anticipate implementing accommodations for residents with all categories of disabilities. Examples of reasonable accommodations include sign language interpreters, modified work schedules, and protected time for health-related appointments.25 While accommodations that pose undue administrative or financial hardship are not required, case law26 clarifies that the overall financial resources of the institution determine hardship. Most accommodations are not prohibitive: one study27 showed that approximately 33% of accommodations had zero cost while 50% were less than $5,000 across the entire period of residency. To date, no studies, to our knowledge, have addressed time and administrative costs of residents with disabilities. However, program directors regularly allot time to all residents for performance review, career guidance, and personal mentoring, and they meet with residents when remediation or disciplinary action is required. The proactive development of an inclusive training environment and clear policies may reduce administrative time and academic distress caused by a failure to accommodate. Furthermore, although rare, litigation may result in significant time and resource costs when the rights of learners with disabilities have not been appropriately addressed.Programs should seek to understand the benefits of disability inclusion. Physicians with disabilities inform health care practices for patients with disabilities and may reduce disparate population health outcomes.28–30 Physicians and learners have suggested their lived experiences with disability lead to greater empathy for patients and enrich the educational learning environment.6,10 Research shows that physicians with disabilities are more likely to provide care for underserved and disability-concordant populations.31 Furthermore, investing in disability-related inclusion has the potential to improve conditions for all residents, regardless of disability status.22Residency programs should prepare for an increasing number of residents with disabilities who have accessed undergraduate medical education accommodations, are knowledgeable about the law, and may request GME accommodations. In line with ACGME's focus on the inclusion of a more diverse resident population, programs should seek to develop transparent processes, identify a knowledgeable disability expert, solidify their understanding of the ADA-AA, and understand the benefits of inclusion to patients and residents. Ensuring that programs are accessible to residents with disabilities is imperative to maintaining the pipeline of physicians with disabilities, from premedical education to practice. By improving the climate for residents with disabilities, GME programs can successfully realize the promise of diversity among physicians with disabilities.

Continuing Medical Education (CME) is crucial for the professional development of physicians and allied healthcare professionals, ensuring they maintain and update their skills to provide high-quality patient care. In Graduate Medical Education (GME) programs, CME supports the training of medical residents and fellows as they transition to independent practice. However, challenges in tracking, documenting, and verifying CME activities persist, hindering the assessment of trainees’ progress and the identification of learning gaps. This study analyzes the current state of CME tracking in GME programs, exploring the challenges faced by program directors, educators, and trainees. It aims to identify strategies for standardizing and improving CME tracking, including digital platforms, centralized databases, and needs assessments. By reviewing literature, case studies, and real-world experiences, the study provides insights and recommendations for optimizing CME management. The search yielded a diverse range of research approaches, encompassing commentary/opinion articles, conference reports, literature reviews, and tracking in GME. Ensuring activities are relevant, engaging, and effective in enhancing the overall educational experience of medical trainees. However, tracking, and documenting CME activities for trainees pose significant challenges due to the lack of a standardized system. This study aims to analyze the current state of CME tracking within GME programs, identify challenges, and propose strategies for improvement. By reviewing literature, case studies, and real-world experiences, the study will explore methods such as digital platforms and centralized databases to enhance CME tracking, ultimately improving medical education and patient care.

It is common to complete evaluations of graduate medical education (GME) programs, present them at conferences, publish them in peer-reviewed journals, add them to curricula vitae (CVs), and then move on without using them to enact changes in the programs themselves. Such actions may reflect the reality that many individuals perceive and conduct program evaluations as if they were research.1 While research and program evaluation use similar methods, they have distinct purposes, timelines, audiences, and most notably, intended uses.2 Evaluations of GME programs need to be used to, for example, inform program decisions and modifications, grow program stakeholders' knowledge, stimulate organizational culture changes, or improve the quality of training.1,3,4 They need to be more than intellectual exercises resulting in accomplishments listed on CVs.5 As such, we emphasize that evaluation use is an essential consequence of program evaluation. Those involved in program evaluation should discuss it and maintain its prominence at the onset of every evaluation. We also promote the adage "use-it-or-lose-it" to stress timely program evaluation use. Yet the literature on program evaluation in GME often neglects to discuss use, including how selected evaluation approaches can influence evaluation use.1,6,7 In this article, we explain evaluation use by describing both the use of evaluation findings and process use (ie, changes resulting from engagement in the evaluation process itself).1,8 We also suggest strategies, including evaluation approaches, that faculty can use to increase evaluation use in GME.The 3 categories of use of evaluation findings are instrumental, conceptual, and symbolic. Instrumental use refers to instances where stakeholders use evaluation findings to take direct actions (eg, improvements, changes, terminations) in a program.9 For example, evaluation findings show that residents in a GME program are struggling to complete their research projects. Using the findings, the GME team implements new research training activities to assist residents in the completion of their projects. Conceptual use describes occurrences where stakeholders use evaluation findings to evolve their understandings of a program but do not take direct actions based on these findings.4 For instance, the GME team acknowledges the findings that residents are struggling to complete their research projects. These findings inform their understanding of why residents are not attending academic conferences to present their research. Lastly, symbolic use occurs when stakeholders use the sheer existence of a completed evaluation to comply with reporting requirements or justify a previously made program action.4 For example, the funding university requires the GME program to complete an evaluation to retain funding for residents' research projects. The GME team completes an evaluation and presents the report to the university. Alternatively, before the evaluation, the GME program hired a research assistant to help residents with their research projects and the subsequent evaluation findings are used to justify the hiring of the research assistant. In GME, we emphasize instrumental use, as this form of use leads to actions that can improve programs. However, the use of evaluation findings is typically a short-term consequence of evaluation because these findings are relevant only within a specific and limited timeframe (ie, use-it-or-lose it).On the other hand, process use can have ongoing influence on individuals, programs, and organizations. It recognizes that evaluation processes themselves can affect attitudes, thought processes, and behaviors.10 Process use recognizes stakeholders' learning advancements from their involvement in an evaluation as well as the effects of evaluation processes on program functioning and organizational culture.11 Process use does not require changes to a program or direct actions because of evaluation findings. There are 6 types of process use which we illustrate with examples:When stakeholders are involved in evaluation processes, they enter an evaluation culture and learn how to think and look at things through an evaluative lens. They can also use the knowledge and skills (eg, evaluation knowledge, methodological and facilitation skills) they develop to strengthen their organization's abilities to design, implement, interpret, and use evaluations and thereby build their organization's evaluation capacity. In this sense, process use is valuable throughout and following an evaluation and in various GME settings regardless of the evaluation findings or recommendations.12The Table presents strategies that faculty involved in program evaluation can employ to increase evaluation use.In closing, it is imperative to remember that evaluation use, especially process use, can occur throughout a program evaluation rather than simply at its conclusion.10 Evaluation use can start at the planning stage and continue well beyond a presentation or publication of an evaluation. Program evaluators need a use-it-or-lose-it perspective throughout the evaluation process to maximize improvements to training. This perspective will maintain stakeholders' faith in the value of evaluation, as they witness that evaluation efforts lead to timely, actionable findings and processes. Ultimately, we must embrace evaluation use to ensure that all stakeholders and programs, not only conference attendees, readers of peer-reviewed journals, or our CVs, witness the consequences (both positive and negative) of program evaluation.