Program-level Assessment Research Articles

A conceptual model for program-level assessment

ABSTRACT Program-level assessment is a holistic approach for arranging assessments throughout a degree program that supports sequential development of discipline knowledge, transferable skills and career readiness. Currently, the modular arrangement of courses means that student learning is partial, limited to passing the assessment and compartmentalized into independent course learning outcomes. In effect, the current practices prevent students developing broader educational attributes that are essential for future employment opportunities. Interviews were conducted with 18 academics, Associate Deans and Program Directors, from seven Australian universities in the fields of dietetics, physiotherapy and science disciplines in order to develop a conceptual model to guide program-level assessment. The proposed conceptual model is designed to support academics’ assessment practices, enhance the quality of universities’ programs and provides students with a more coherent assessment experience.

Enablers and barriers to program-level assessment planning

ABSTRACT Assessment underpins students’ learning in higher education and provides evidence of knowledge and skills. Program-level planning refocuses the assessment from being content and siloed at the course level to being program oriented with alignment to program learning outcomes and graduate attributes. Program-level assessment has been gaining momentum and shown to create positive assessment practices that support the quality agenda driving the higher education sector; despite this, program-level assessment is not widely adopted in Australian universities. This article explores the factors that academics in health and science programs identify as enablers and barriers to adopting program-level assessment. Grounded Theory was the methodological approach used to construct the semi-structured interviews with 18 Associate Deans and Program Directors from seven universities. Thematic analysis revealed the enablers and barriers to participants’ undertaking program-level assessment. The enablers of program-level assessment included having a shared responsibility through whole-team collaboration, effective leadership, and professional development. Conversely, barriers included resource constraints, ownership issues, and siloed practices, a lack of assessment literacy and university policies. The identification of these enablers and mitigating the barriers can help facilitate program-level assessment planning to enhance the assessment experience and drive students’ purpose of assessment within a university program.

Program‐level Assessment at a Community College: Four Missteps that Can Be Easily Corrected

Assessment UpdateVolume 35, Issue 2 p. 4-5 Article Program-level Assessment at a Community College: Four Missteps that Can Be Easily Corrected Julia Wai-Yin So, Julia Wai-Yin SoSearch for more papers by this author Julia Wai-Yin So, Julia Wai-Yin SoSearch for more papers by this author First published: 30 March 2023 https://doi.org/10.1002/au.30338Read the full textAboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat No abstract is available for this article. Volume35, Issue2March/April 2023Pages 4-5 RelatedInformation

Vietnamese Higher Education Challenges: Quality Accreditation, Rankings and Ratings

A study analyzing data from an international university ranking system, a regional rating system, and quality accreditation processes revealed that challenges of Vietnamese higher education are identified in three levels: the national system, the institution level, and training programs. Analysis of the multiple datasets exposed 20 potential gaps at the educational institution and training program level assessments, which may be the reasons preventing the top five ASEAN Vietnamese higher education to further advance to higher levels. On that basis, the report proposes eight solutions to improve the quality and competitiveness of the university accreditation, ranking and rating systems.

Program-level assessment planning in Australia: the considerations and practices of university academics

Quality assessment in higher education needs to drive learning to produce graduates with disciplinary knowledge and employable skills. Performance-based funding requires universities to focus on graduate outcomes and future employment status. Research was conducted to investigate how program-level assessment planning is occurring within Australian universities, to determine academics understanding what aspects they consider and the assessment practice they identify as essential at the program-planning level. Constructivist grounded theory underpins the methodological approach and 18 academics were interviewed. Participants included Associate Deans and Program Directors (or equivalent roles) from seven Australian universities, in the accredited disciplines of dietetics and physiotherapy and non-accredited degrees in biomedical science and science. The results indicate that academics consider a myriad of factors including policies, curriculum, skills, and accreditation. Assessment practices identified as essential incorporate constructive alignment, a range of assessment tasks, authentic, and continuous assessments. The goal of program-level assessment planning is to holistically sequence assessment tasks to enhance students’ employability, so they become successful graduates and secure employment, which enables universities to acquire future funding.

“With the Kindness Shown to Me”: Assessing Remote Teaching and Learning Strategies in a Time of Pandemic

ABSTRACT In this article, the authors share the “pandemic” assessment method advanced after the start of the COVID-19 pandemic, a method based in longstanding assessment processes in their institution’s General Education program and reimagined through collegial interactions among faculty. After identifying what changed in their program-level assessment processes, and how, they identify key findings from two aspects of the assessment: student comments on end-of-term course evaluations and faculty comments from a portfolio review process. Finally, they offer their program’s takeaways from this revised assessment approach, including the elements from the pandemic year that they will continue to draw on into the future.

Empowering Metacognitive Learning in a Competency-Based Curriculum Through Comprehensive Learner Record

Health professions education is in the midst of a radical transformation from traditional lecture-based teaching to competency-based learning. This requires an equally radical transformation in the way we assess student learning and achievement of competencies. Comprehensive Learner Record (CLR) addresses this need, enabling metacognitive learning and reflection of student development of competencies as they progress through a curriculum. CLR helps students make sense of their various learning activities, and share their educational journey with future employers highlighting artifacts and achievements in personally curated portfolios. CLR helps students synthesize learning, guides supplemental supports, and empowers program-level assessment to guide curricular improvements. Reflection and feedback at milestones provide a formative assessment to support progression. As nursing and other health professions schools transition towards competency-based learning, CLR provides an essential mechanism to communicate the achievement of core competencies, as students progress and develop readiness for practice.

A Program-Level Assessment of Student Understanding of Bonding in the Chemistry Major

Assessment of student learning within a degree program is an important but challenging endeavor. One goal of a programmatic assessment is to systematically collect evidence for chemistry majors’ kn...

Collaborative Assessment: Using Self-assessment and Reflection for Student Learning and Program Development

As program-level assessment increasingly becomes an integral part of the higher-education landscape, so does the debate regarding the efficacy of current assessment methods. Traditionally, students do not participate in assessment—neither of their own learning nor of institutional or program efficacy. Our assessment process presents an alternative to traditional program-level assessment and is meant to improve student learning in two ways: (1) by asking students to reflect on their achievement of learning outcomes using evidence-based methods; (2) by providing assessment practitioners with authentic, contextualized data on which to make claims about curricula. This collaborative assessment process was designed to address the complex needs of a cross-curricular rhetoric program but responds to many general concerns about traditional assessment methods.

Moving toward student-faculty partnership in systems-level assessment: A qualitative analysis

Partnership models have been effective across many areas of higher education such as involving students as teaching and learning consultants, in course design and redesign, and as co-instructors. However, there are few systems-level (i.e., entire programs or institutions) examples of partnership work and virtually none in systems-level assessment. Systems-level assessment models, such as program-level assessment in the United States, are used to inform broad changes to academic programs. Thus, student input may be crucial. This study sought to explore the broad factors that underlie potential student-faculty partnership efforts in systems-level assessment. Participants were faculty and staff members based in the United States and the United Kingdom who engaged in student-faculty partnerships at the program and/or classroom level. Qualitative coding and analyses of interviews with participants resulted in seven primary themes. This study examines patterns evident in student-faculty partnership work across several areas of higher education and begins to lay the foundation for a theory of student-faculty partnership in systems-level assessment.

Case Study Supporting the Development of a University-wide Program-level Learning Outcomes Assessment Plan for Saudi Arabian Universities

Case Study Supporting the Development of a University-wide Program-level Learning Outcomes Assessment Plan for Saudi Arabian Universities

Leading with Empathy and Learning to Flex

When instruction shifted online as a result of COVID-19 in March 2020, Educational Effectiveness staff considered how best to serve our institution with available resources, skills, and services, while maintaining ongoing assessment. Administration of program-level assessment activities, such as overseeing the completion of program reviews, planning for upcoming program reviews, and advising faculty on annual program learning outcome assessment were opportunities to lead with increased empathy and flexibility. The unfamiliar nature of operating in the midst of a pandemic also required fresh data from students, faculty, and staff. Empathy and timeliness in our communications were essential in supporting our institution.

MP17: Evaluation of a national competency-based assessment system in emergency medicine: A CanDREAM study

Introduction: In 2018, Canadian postgraduate specialist Emergency Medicine (EM) programs began implementing a competency-based medical education (CBME) assessment system. To support improvement of this assessment program, we sought to evaluate its short-term educational outcomes nationally and within individual programs. Methods: Program-level data from the 2018 resident cohort were amalgamated and analyzed. The number of Entrustable Professional Activity (EPA) assessments (overall and for each EPA) and the timing of resident promotion through program stages was compared between programs and to the guidelines provided by the national EM specialty committee. Total EPA observations from each program were correlated with the number of EM and pediatric EM rotations. Results: Data from 15 of 17 (88.2%) EM programs containing 9,842 EPA observations from 68 of the 77 (88.3%) Canadian EM specialist residents in the 2018 cohort were analyzed. The average number of EPAs observed per resident in each program varied from 92.5 to 229.6 and correlated strongly with the number of blocks spent on EM and pediatric EM (r = 0.83, p < 0.001). Relative to the guidelines outlined by the specialty committee, residents were promoted later than expected and with fewer EPA observations than suggested. Conclusion: We present a new approach to the amalgamation of national and program-level assessment data. There was demonstrable variation in both EPA-based assessment numbers and promotion timelines between programs and with national guidelines. This evaluation data will inform the revision of local programs and national guidelines and serve as a starting point for further reaching outcome evaluation. This process could be replicated by other national assessment programs.

A qualitative analysis to identify the elements that support department level change in the life sciences: The PULSE Vision & Change Recognition Program.

The 2011 report, Vision and Change in Undergraduate Biology Education: A Call to Action, provided the impetus to mobilize the undergraduate life sciences education community to affect change in order to enhance the educational experiences of life sciences majors. The work of the appointed Partnership for Undergraduate Life Sciences Education (PULSE) Vision and Change (V&C) Leadership Fellows has focused on the development of programs and resources to support departmental change. In this report, we present a qualitative assessment of several documents generated from the PULSE V&C Leadership Fellow Recognition Team. The Recognition Team developed two initiatives to provide departments with feedback on their change process. The first initiative, the validated PULSE V&C Rubrics, enables departments to collaboratively self-assess their progress in enacting change. The second initiative, the PULSE Recognition Program, involves completion of the aforementioned Rubrics and a site-visit by two Recognition Team members to provide external insights and suggestions to foster a department’s change process. Eight departments participated in the Recognition Program in 2014. An evaluation of the documents yielded from the Recognition Program review of seven of the eight departments and a comparison of Rubric scores from before and three years following the site-visits uncovered several common elements required for successful department level change. These elements include an institutional culture that values and supports excellence in teaching and learning with resources and infrastructure, a departmental emphasis on program and course level assessment, and, most importantly, a departmental champion who actively supports endeavors that enhance teaching excellence.

Combining course- and program-level outcomes assessments through embedded performance assessments at key courses: A proposal based on the experience from a Japanese dental education program

This paper addresses how to combine the course- and program-level assessments and presents a new method illustrated by a case of dental education program in Japan. Performance assessments are considered effective for evaluating knowledge integration and higher-order skills, while placing a burden on faculty, hence their feasibility as the program-level assessment is regarded lower than standardized tests or questionnaire surveys. We have developed several performance assessments at the course level, such as Modified Triple Jump for the PBL course. Based on this experience, we propose Pivotal Embedded Performance Assessment (PEPA) as a method for combining assessment at the course and program levels. The method limits the range of performance assessment to key courses directly linked to program goals and placed at the critical juncture points of curriculum, while entrusting the assessment of other courses to expert judgment of individual teachers. PEPA consists of the following procedures: systematization of curriculum and selection of key courses; design and implementation of performance assessments by a faculty team; setting passing criteria with incorporating the function of formative assessment; certifying the completion of the degree program. PEPA thus enables maintaining assessment feasibility and compatibility with a credit system, while ensuring assessment validity and reliability.Received: 27 September 2018Accepted: 13 November 2018Published online: 29 November 2018

A Call for Programmatic Assessment of Undergraduate Students' Conceptual Understanding and Higher-Order Cognitive Skills.

In response to empirical evidence and calls for change, individual undergraduate biology instructors are reforming their pedagogical practices. To assess the effectiveness of these reforms, many instructors use course-specific or skill-specific assessments (e.g., concept inventories). We commend our colleagues’ noble efforts, yet we contend that this is only a starting point. In this Perspectives article, we argue that departments need to engage in reform and programmatic assessment to produce graduates who have both subject-matter knowledge and higher-order cognitive skills. We encourage biology education researchers to work collaboratively with content specialists to develop program-level assessments aimed at measuring students’ conceptual understanding and higher-order cognitive skills, and we encourage departments to develop longitudinal plans for monitoring their students’ development of these skills.

Using the ACRL Framework to Develop a Student-Centered Model for Program-Level Assessment

Information literacy instruction presents a difficult balance between quantity and quality, particularly for large-scale general education courses. This paper discusses the overhaul of the freshman composition instruction program at the University of Maryland Libraries, focusing on the transition from survey assessments to a student-centered and mixed-methods approach using qualitative reflections, rubrics, and the evaluation of student artifacts. The article discusses the progression from a pilot assessment program using Twitter as a data collection model to the implementation of a robust and multi-layered assessment using both qualitative feedback from students and the evaluation of student artifacts. Each assessment includes detailed collection methods and customized rubrics for evaluation of student responses.While information literacy assessment has been covered extensively in the literature, few articles discuss the use of qualitative student responses on a large scale (4,000 participants per year). The article also discusses the re-structuring of an assessment program around the ACRL Framework for Information Literacy, which is incorporated throughout the project from the pilot up through the full implementation of the final program.

Program Level Assessment: A Case Study for a University Clothing and Textile Major

Program Level Assessment: A Case Study for a University Clothing and Textile Major

Developing an Assessment Process for a Master’s of Science Degree in a Pharmaceutical Sciences Program

Objective. To develop a program-level assessment process for a master's of science degree in a pharmaceutical sciences (MSPS) program. Design. Program-level goals were created and mapped to course learning objectives. Embedded assessment tools were created by each course director and used to gather information related to program-level goals. Initial assessment iterations involved a subset of offered courses, and course directors met with the department assessment committee to review the quality of the assessment tools as well as the data collected with them. Insights from these discussions were used to improve the process. When all courses were used for collecting program-level assessment data, a modified system of guided reflection was used to reduce demands on committee members. Assessment. The first two iterations of collecting program-level assessment revealed problems with both the assessment tools and the program goals themselves. Course directors were inconsistent in the Bloom's Taxonomy level at which they assessed student achievement of program goals. Moreover, inappropriate mapping of program goals to course learning objectives were identified. These issues led to unreliable measures of how well students were doing with regard to program-level goals. Peer discussions between course directors and the assessment committee led to modification of program goals as well as improved assessment data collection tools. Conclusion. By starting with a subset of courses and using course-embedded assessment tools, a program-level assessment process was created with little difficulty. Involving all faculty members and avoiding comparisons between courses made obtaining faculty buy-in easier. Peer discussion often resulted in consensus on how to improve assessment tools.

Developing a Comprehensive Assessment Plan.

This chapter provides practical advice for developing a comprehensive assessment plan. Two examples from institutions that have created both individual and program-level assessment plans for leadership programs are shared.