Program Review Process Research Articles

754. Utilization of Internal Special Program Review to Institute Change in an Academic ID Fellowship

Abstract Background Fellowship accreditation is crucial to ensure adequate clinical education for physicians in training, yet there is limited data on use of accreditation survey results for program quality improvement. The Accreditation Council for Graduate Medical Education (ACGME) administers annual surveys to trainees and core faculty to monitor education and identify potential non-compliance with ACGME accreditation standards. Our purpose is to describe the creation of an internal program review process for program improvement based on results of nationally implemented accreditation surveys. Methods We collected data from standardized ACGME survey responses administered to core faculty and fellows at our academic Infectious Diseases fellowship program. The identification of areas scoring lower than 20% in comparison with national average prompted creation of a Special Program Review (SPR) committee comprised of a program director, a program coordinator, and a trainee from programs outside of the program under review. This committee met separately with core faculty, trainees, and program leadership to compile an assessment of each survey component and opportunities for improvement. This review process of survey results was completed annually from 2020-2022. Results Between 2020-2022, surveys were completed by 100% of fellows (7/7 or 8/8) and up to 92% of core faculty (11/12). Key findings raised concerns involving clinical education, faculty supervision, patient safety, and teamwork. Actions taken in response to the SPR included the creation of: an updated backup and weekend coverage system, new didactic curricula, a morbidity and mortality conference, documentation limits, informal fellow/faculty dinners, opt out counseling, and a division culture committee. Conclusion A non-punitive internal review process mimicking accreditation standards is valuable for program assessment and improvement as means to identify areas of risk for accreditation issues at a national level. This review process is generalizable, as all accredited programs complete the survey annually. The quality improvement interventions taken will differ depending on survey findings and areas to be addressed. Disclosures All Authors: No reported disclosures

Forming an academic program review learning community: description of a conceptual model

PurposeHigher education institutions are required to evaluate program quality through cyclical program review processes. Despite often being considered the “gold standard” of academic review, there persists dissatisfaction with the lack of integration of program review findings into other planning processes, such as budgeting, assessment and strategic planning. As a result, the notion of program review action plans “collecting dust on the shelf” is so ubiquitous that the concept is normalized as an expected outcome. The purpose of this paper is to describe a conceptual model whereby teams of faculty members receive education and training from quality assurance practitioners and educational developers, access to institutional resources, opportunities for cross-departmental collaborations and collective advocacy to increase the capacity of faculty members to implement improvement goals resulting from program reviews.Design/methodology/approachThe authors theorize that a professional learning community is a meaningful approach to program review and present a conceptual model – the Academic Program Review Learning Community (PRLC) – as an antidote to hierarchical, fragmented, compliance-oriented processes. The authors suggest that the PRLC offers a reliable institutional framework for learning through formalized structures and nested support services, including peer learning and external coaching, which can enhance the catalytic capacity of reviews.FindingsThe authors argue that postsecondary institutions should create formal structures for incorporating learning communities because, without a reliable infrastructure for collective learning, decision-making may be fragmented oridiosyncratic because of shifting demands, priorities or disconnected faculty.Originality/valueA learning community model for program review fits well with a new way to think about program review because faculty are most engaged when they feel ownership over the process. Furthermore, few models exist for conducting program review; as a result, chairs and academics often struggle to conduct reviews without a coherent framework to draw upon.

The Landscape of Career and Technical Education Faculty Data Use for Decision-Making and Program Improvement in National Science Foundation–Advanced Technical Education Programs in Washington State

Over the last ten years, in Washington community and technical colleges (CTC) there has been a great leap forward in access and use of data for setting and measuring goals, monitoring progress, and identifying equity gaps. Despite much progress, faculty members in career and technical education (CTE) programs still often have difficulty accessing and using data that would drive program improvements. The purpose of this research is to gain knowledge about the current state of affairs surrounding technical education faculty access to and use of student-level data; second, we seek to shed light on the constraints CTE faculty face in accessing and using data for data-informed decision making. Our research revealed much variation in faculty access and use of data, with some creating customized solutions for daily use and others only accessing data for external needs such as accreditation or mandated program review processes. While the need for data may be exacerbated by the COVID-19 crisis, faculty report data use on campus has dropped during the crisis and been replaced by immediate concerns over financial health, enrollment and instructional delivery challenges. Rather than continue to advance data-driven decision-making, we may be seeing colleges retrench to past organizational norms and customs.

Communicating Managed Retreat in California

California cities face growing threats from sea-level rise as increased frequency and severity of flooding and storms cause devastating erosion, infrastructure damage, and loss of property. Management plans are often designed to prevent or slow flooding with short-term, defensive strategies such as shoreline hardening, beach nourishment, and living shorelines. By contrast, managed retreat focuses on avoiding hazards and adapting to changing shorelines by relocating out of harm’s way. However, the term “managed retreat” can be controversial and has engendered heated debates, defensive protests, and steady resistance in some communities. Such responses have stymied inclusion of managed retreat in adaptation plans, and in some cases has resulted in complete abandonment of the policy review process. We examined the Local Coastal Program review process in seven California communities at imminent risk of sea-level rise and categorized each case as receptive or resistant to managed retreat. Three prominent themes distinguished the two groups: (1) inclusivity, timing, and consistency of communication, (2) property ownership, and (3) stakeholder reluctance to change. We examined use of terminology and communication strategies and provided recommendations to communicate “managed retreat” more effectively.

Sociology Departments and Program Review: Chair Perspectives on Process and Outcomes

Sociology faculty are accountable to multiple stakeholders to demonstrate that our academic programs are effective and that students are learning. Despite the ubiquity of mandated program review practices, which often include the assessment of student learning, research is lacking on the extent to which these efforts lead to improvements in departmental outcomes and student experiences. Similarly, little research exists on how department leaders experience and evaluate the utility of these efforts. This article uses data from a national survey of sociology departments to document how program review and assessment are enacted in sociology departments and to explore chairs’ perspectives on program review processes and its outcomes. The article concludes with a series of recommendations for improving these processes so that faculty time is used well and the experiences of sociology students are enhanced.

(How) Do Students Use Learning Outcomes? Results from a Small-Scale Project

Pre-specified, prescribed or intended Learning Outcomes have been in use throughout higher education programs for over two decades. There is an assumption amongst quality assurance bodies and university program approval and review processes that students engage with them. Yet, learning outcomes may constrain learning, they may not always be understood by learners and their relevance to learning has been questioned. There is anecdotal evidence from lecturers that some students do not understand them and do not use or refer to them. This paper reports on a small-scale research project investigating how university student’s use prescribed learning outcomes in their everyday learning and when producing assessed work. No clear differences were found between higher and lower achieving students, yet there were differences between first- and third-year students. Surprisingly, some were able to achieve highly without referring to the outcomes against which they were assessed.

The journey towards the best quality of MCQs test, a success story in a medical school

Multiple choice question (MCQ) test is the most popular assessment tool that used in medical schools, faculty training on construction of multiple choice questions and peer review are the main factors that contributing to have a good quality MCQ test. This study aimed to evaluate the effect of these two factors on the quality of MCQ test. An extensive training program was conducted for the staff members who were participated in teaching and assessment of undergraduate medical students of clinical biochemistry and psychiatric courses. Post-test item analysis was used to compare pre-training with post-training item analysis parameters. Chi-square test was applied, level of significance was 5%. Regarding difficulty index, it showed great improvement in number of questions which lie in acceptable range of difficulty from 16%, 25% before the training program to 61%,42% after the training program for psychiatry and clinical biochemistry courses respectively and this is significantly different (P0.05). In psychiatry course the number of questions with negatively discrimination power had been decreased from 36% before training program to 10% after it (P value 0.05). No significant improvement noticed in biochemistry course regarding this aspect. Both courses were showed improvement in the function of distracters, the percentage of test items where all distracters were functioning was increased from 11% to 41% in psychiatry course (P value=0.0002), and from 12% to 46%, in biochemistry course (P value=0.0001). The two courses which included in the study showed increased reliability after the training program and peer review processes, but this improvement is not statistically significant. A dramatic improvement was observed in the quality of MCQs after peer review and training program. We recommend adopting training program on construction of MCQs to be as teaching license for all medical school staff members in Saudi Arabia.

Comparing the Involvment of Academic Librarians in the Cycical Review Process for Graduate Programs in the Humanities and in the sciences

In Ontario, all graduate programs at publicly assisted universities must be reviewed on a cyclical basis at least once every eight years. This paper adopts a corpus-based approach to investigate and compare how academic librarians are integrated into the cyclical review process for graduate programs in the Humanities and graduate programs in the Sciences. Findings reveal that these two broad disciplinary cultures differ in the way and the degree to which they recognize libraries and involve librarians in the program review process.

Striving for Excellence in Program Outcomes Assessment

AbstractThis article describes the components of a program outcomes assessment model based on best practices that can be applied to improve the program review process in institutions of higher education.

Automated system for evaluating higher education programs

Program review has been recognized as a common practice amongst universities worldwide. This exercise usually ensures that programs are up to date, meet high quality standards, continuously responding to changes, and hence, adheres to continuous improvement practices. It is a cycle that involves different stakeholders ranging from program chairs and deans to program review committee members and internal/external examiners. Managing such multi-process program evaluation could be enhanced using an automated system that streamlines the different processes and accordingly, providing more cost-effective management and coordination activities. However, the literature indicates that very few automation efforts were implemented to support such review management system. In this research, we highlight the cost effectiveness of the different main activities and processes involved in programs review lifecycle. Hence, we propose a software tool design called Automated Program Review Management System (APRMS) to support and manage the program review process. APRMS handles different categories of reviewing activities. Moreover, it is designed to enhance decision-making by providing timely feedbacks, reminders, guidelines and recommendations and to make effective use of the contributions of the different stakeholders. The system has been evaluated using both quantitative and qualitative criteria. Particularly, a case study was depicted that used real data, collected from a program review process in United Arab Emirate University (UAEU). The results demonstrated that the suggested automated system leads to significant gains in time, effort, and deliverable’s quality.

Not Too Small to be Strategic: The State of Academic Program Review Guidelines and Instrumentation in Public Institutions

In higher educational settings, the academic program review process allows academic units to solicit feedback, make data-driven decisions regarding effectiveness and sustainability, and report to stakeholders. In order to discover trends and best practices in the current state of academic program review in higher education, examined was a sample of 53 processes representing a range of small-to-large public institutions in the United States in the categories of associate-degree granting, bachelors-degree granting, and masters-degree granting. Using a content analysis methodology, outcomes assessment and the use of data results featured prominently with 86.7% of the sample requiring their incorporation in academic program review, followed closely by judgments of resource adequacy and demonstration of program necessity and efficiency.

Investigating the participation of business librarians in academic program reviews using corpus-based methods

Prior research into the role of business librarians in academic program reviews has relied on surveys and interviews, revealing that librarians perceive that they are marginalized in the review process. Using a collection of program review documentation produced for the reviews of nine graduate programs offered at a Canadian business school, this study employs corpus-based techniques to obtain direct measures of librarian involvement. The findings provide objective confirmation that business librarians are not well integrated into program reviews overall, and that their contribution to the reviews of professional programs is even more limited than their contribution to the reviews of research-oriented programs. Based on best practices and missed opportunities observed as part of this study, seven strategies are suggested for integrating business librarians more fully in the program review process for the benefit of all program stakeholders.

An Innovative Model of Agricultural Education and Training in Guinea: Trending Toward Self-Sustainability

Exploring locally-designed Agricultural Education and Training (AET) programs provides opportunities for the entire system of AET to improve. Recognizing this potential, researchers conducted a case study of the Centre d’Apprentissage et de Formation Professionnelle Post-Primaire (i.e., Center for Post Primary Professional Training [CAFPPP]) located in Macenta, Guinea. The case study was completed in conjunction with a comprehensive program review and participant-engaged strategic planning process. Results illuminate CAFPPP faces significant challenges, including (a) limited funding, (b) insufficient teacher and administrator training, (c) limited curricular scope, and (d) systematic challenges. Additionally, three critical strengths were identified at CAFPPP, (a) utilization of an effective, practice-based educational model, (b) stakeholder support, and (c) access to fertile agricultural land. Considering both strengths and weaknesses, researchers and CAFPPP stakeholders co-created a model to propel CAFPPP toward its stated goal of becoming an “autonomous center of excellence.” The dynamic model envisions a school leveraging its strengths to systematically address identified weaknesses via intersecting approaches to funding, teacher and administrator development, and graduate support. Presentation of the model is supplemented with a discussion of, and recommendations for, application of the model at CAFPPP. Furthermore, opportunities for all AET programs to reflect upon, and evaluate, current strategies in light of the proposed model are discussed.

Preface

We are pleased to present the technical papers for IEEE VR 2018: the 25th IEEE Conference on Virtual Reality and 3D User Interfaces, held March 18-22, 2018 in Reutlingen, Germany. IEEE VR 2018 features two categories of submissions: (i) VR Journal Papers, and (ii) VR Conference Papers. Both categories have their own program committees, submission processes, and review processes. This year, 178 submissions were submitted to the Journal track from which 29 were accepted as articles to IEEE TVCG (16.3%). All of these will be presented at the IEEE VR 2018 conference, along with 6 additional papers in the VR area that were published in IEEE TVCG during the past year. Furthermore, 6 submission (3.4%) were recommended for a regular issue of TVCG with major revisions with reviewer continuity. Each of the papers in this special issue went through a rigorous two-round review process. All accepted journal papers are published in a special issue of IEEE Transactions on Visualization and Computer Graphics (TVCG). The Proceedings of the IEEE Conference on Virtual Reality and 3D User Interfaces contains all of the accepted conference papers and the poster abstracts.

Developing Leadership for Increasing Complexity: A Review of Online Graduate Leadership Programs

Leadership education must evolve to keep pace with the growing recognition that effective leadership happens in a complex environment and is as much a systemic variable as a personal one. As part of a program review process, a graduate leadership program at a private Midwestern university conducted a qualitative review of 18 online graduate programs in leadership education. In the absence of a discipline or accrediting body to govern leadership degree programs, we utilized the integrating framework of complexity leadership theory (CLT), as well as two professional societies, to understand how the curricula and competencies of online graduate education align and diverge to meet the changing assumptions and challenges of leadership.

Annual program review process: an enhanced process with outcomes

ABSTRACTThe Accreditation Council for Graduate Medical Education’s required Annual Program Review of Educational Effectiveness (APREE) has helped us improve our program and change its culture to one of continuous quality improvement. This report outlines our systematic process and describes specific outcomes it has produced over a 10-year period. We identified ways to enhance our APREE after reading articles that described various ways to conduct the process found in a PubMed and OvidSP search and relevant policies from our local Graduate Medical Education Office. After discussing options, we incorporated new ideas into our APREE and tasked our Program Evaluation Committee to track outcomes from objectives developed by faculty and residents during each APREE. Objectives from faculty and residents in 10 years of our APREE led to major improvements (e.g., increased board pass rate) in our program. In addition, the enhanced APREE process gradually changed our residency’s culture to one that embraces continuous quality improvement. A systematic APREE process can engage residents and faculty in improving specific components of a residency. Besides providing outcomes for Web Ads and Self-Study items, the APREE models quality improvement techniques to residents, involves a wide array of stakeholders, and helps program stakeholders embrace continuous quality improvement.

Gauging Satisfaction with the New Proposal Process in DEB and IOS at the NSF

The Division of Environmental Biology (DEB) and the Division of Integrative Organismal Systems (IOS) at the National Science Foundation recently made three key changes to the core program review process: a new preliminary proposal requirement, a cap on the number of proposals that principal investigators (PIs) submit per cycle, and a switch to an annual deadline. We queried the scientific community on their level of satisfaction with key facets of the new system and their perception of the consequences. We received 4736 responses (a 24% response rate). Our results show that there was no qualitative difference in reported satisfaction between the PIs in DEB and IOS. The preliminary proposal requirement was viewed most favorably, and the switch to a single annual deadline elicited the most dissatisfaction. Continued monitoring of these early trends through time will be important to ensure that funds are allocated fairly in support of the best science.

The art and science of forensic meteorology

When weather factors into a legal dispute, meteorologists use eyewitness accounts, troves of atmospheric data, and simulated reenactments to figure out what went awry.

Challenges of Engaging Local Stakeholders for Statewide Program Development Process

The University of Missouri Extension needed to develop an annual program review process that collaboratively engaged county-level stakeholders. The results from the first 2 years highlight the results, challenges, and implications of the design process. The annual review process needs to be adaptive, responsive, and reflective from year to year when engaging with local stakeholders. For example, the 2012 program review process changed from the prior year to focus on participant mind mapping to better engaged stakeholders and to produces more detailed data. The process has ensured more engagement with local stakeholders and generated information helpful to regional faculty members.

Undergraduate Program Review Processes: A Case Study in Opportunity for Academic Libraries

How can an academic library most effectively participate and expand its contributions to program reviews at the institutional level? By becoming involved in undergraduate reviews, college and university libraries can articulate new and enhanced roles for themselves on campus. Academic libraries have always contributed to a variety of institutional review processes. However, by embracing a more holistic view of its support, the library can expand beyond collection-related metrics to encompass all the ways the library interconnects with the program. Furthermore, by becoming proactively involved with the committee(s) responsible for managing institutional program reviews, libraries can contribute to the governance of this essential activity on campus. This paper describes one academic library's experience and efforts in becoming involved with undergraduate reviews at both the program and institutional levels. It is hoped that sharing our case study and the tools we have created, will benefit other academic libraries.