Assessment Reform Research Articles

The AI Assessment Scale (AIAS) in action: A pilot implementation of GenAI-supported assessment

The rapid adoption of generative artificial intelligence (GenAI) technologies in higher education has raised concerns about academic integrity, assessment practices and student learning. Banning or blocking GenAI tools has proven ineffective, and punitive approaches ignore the potential benefits of these technologies. As a result, assessment reform has become a pressing topic in the GenAI era. This paper presents the findings of a pilot study conducted at British University Vietnam exploring the implementation of the Artificial Intelligence Assessment Scale (AIAS), a flexible framework for incorporating GenAI into educational assessments. The AIAS consists of five levels, ranging from “no AI” to “full AI,” enabling educators to design assessments that focus on areas requiring human input and critical thinking. The pilot study results indicate a significant reduction in academic misconduct cases related to GenAI and enhanced student engagement with GenAI technology. The AIAS facilitated a shift in pedagogical practices, with faculty members incorporating GenAI tools into their modules and students producing innovative multimodal submissions. The findings suggest that the AIAS can support the effective integration of GenAI in higher education, promoting academic integrity while leveraging technology’s potential to enhance learning experiences. Implications for practice or policy: Higher education institutions should adopt flexible frameworks like the AIAS to guide ethical integration of GenAI into assessment practices. Educators should design assessments that leverage GenAI capabilities, while supporting critical thinking and human input. Institutional policies related to GenAI should be developed in consultation with stakeholders and regularly updated to keep pace with technological advancements. Policymakers should prioritise research funding into the impacts of GenAI on higher education to inform evidence-based practices.

Perceived impact of generative AI on assessments: Comparing educator and student perspectives in Australia, Cyprus, and the United States

The growing use of generative AI tools built on large language models (LLMs) calls the sustainability of traditional assessment practices into question. Tools like OpenAI's ChatGPT can generate eloquent essays on any topic and in any language, write code in various programming languages, and ace most standardized tests, all within seconds. We conducted an international survey of educators and students in higher education to understand and compare their perspectives on the impact of generative AI across various assessment scenarios, building on an established framework for examining the quality of online assessments along six dimensions. Across three universities, 680 students and 87 educators, who moderately use generative AI, consider essay and coding assessments to be most impacted. Educators strongly prefer assessments that are adapted to assume the use of AI and encourage critical thinking, while students' reactions are mixed, in part due to concerns about a loss of creativity. The findings show the importance of engaging educators and students in assessment reform efforts to focus on the process of learning over its outputs, alongside higher-order thinking and authentic applications.

Generative AI and higher education: a review of claims from the first months of ChatGPT

AbstractThe release of the Artificial Intelligence (AI) chatbot ChatGPT renewed discussions about how AI would upend higher education. This paper presents a critical analysis of “grey literature” claims made in the first months after ChatGPT was made public, exploring what these discussions might mobilise in practice. We identified articles for inclusion through a systematic search of five prominent higher education sector outlets. The included articles were thematically coded for claims about generative AI and higher education. We identified ten claims: Three about the nature of ChatGPT, four about changing practices of institutions and teachers, and three about new independent practices of students. Overall, the claims present a positive perspective on AI in higher education. While being perceived as a disruption of the status quo, the authors generally frame AI as a catalyst for existing agendas, e.g. assessment reform, personalisation, or inclusion. This suggests a focus on embracing the affordances offered by AI and primarily addressing risks by including AI in curricula. Furthermore, the claims mainly portray students as either plagiarists or victims of a failing educational system. The paper proposes that a more critical interrogation of generative AI, and the involvement of students in the conversation, may be beneficial.

International trends in the implementation of assessment for learning revisited: Implications for policy and practice in a post-COVID world

This paper discusses the evolution of assessment for learning (AfL) across the globe with particular attention given to Western educational jurisdictions. Scholars from Australia, Canada, Ireland, Israel, New Zealand, Norway, and the United States discuss prominent assessment reforms within their respective countries over the last decade. Particular attention is given to the impact of the pandemic as well as technological developments for classroom assessment policies and practices. Ongoing tensions that exist between AfL and summative forms of assessment within national policy initiatives are also revisited in relation to the seminal version of this article published 10 years ago.

The death of exams? Grade inflation and student satisfaction when coursework replaces exams

A mid-sized advanced undergraduate economics module has undergone an assessment reform, where a typical 60% final exam was replaced by four scaffolded coursework assignments, each carrying a 15% weight in the final grade. As a result, student satisfaction with assessment and feedback went up at the expense of higher staff workload. In addition, the reform raised the final grades over and above the increase typically associated with an improvement in learning and engagement, effectively triggering grade inflation. The paper identifies the coursework assignment that inflated the grades. It then proceeds with grade simulations, whose purpose is to formulate an assessment reform agenda aiming to maintain high student satisfaction, disinflate grades, and stabilise staff workload. Finally, the paper evaluates the scalability of assessment reforms where final exams are targeted for replacement by coursework assessments. The evaluation suggests caution with such reforms, if they are necessary in the first place. They may both be feasible and desirable in small classes, but compelling arguments caution against a premature wholesale rollout.

Are Narrative CVs contributing towards shifting research culture? Workshop Report from the 2023 Recognition and Rewards Festival

Abstract* Background Over the past decade, calls for research assessment reform have grown, led by initiatives such as the Declaration on Research Assessment (DORA) and the Leiden Manifesto, and, more recently, the Coalition for Advancing Research Assessment (CoARA). A key element being discussed as part of research assessment reform is a shift towards more qualitative assessments, focussed on the content of research and the broad skills and competencies of researchers, and the array of contributions they make to knowledge creation and innovation. Narrative CV formats have emerged as a good practice example for enabling qualitative assessments of research projects and researchers, and are becoming more widely piloted and implemented. Methods As part of the 2023 Dutch Recognition and Rewards Festival, the authors hosted a workshop to gather perspectives on Narrative CVs, including whether and how they may contribute to shifts in research culture that are needed to support research assessment reform. Results Participants, representing research organisations and the research community, discussed both beneficial and critical aspects of narrative-style CV implementations from their experiences. The effects observed since narrative CVs have been implemented were discussed, with perspectives provided on career prospects and the empowerment of the research community to direct change. Finally, the discussion turned to expectations for the future, with workshop participants calling for focus on the roles that narrative-style CVs can play in improving research careers, recognition of collaborative work, and equality, diversity, and inclusion. A short informal survey exploring levels of implementation of narrative CVs across different research organisations was run prior to the workshop, the results of which are also presented as part of this report. Discussion The authors intend to expand this discussion to other scientific and policy conferences, and this report serves as a basis for a wider and deeper dialogue in the community.

Introducing technologies into national large-scale testing: Are we ready?

In recent decades, the potential benefits of introducing technologies into large-scale tests have been much discussed. Yet the path to effective technology use on large-scale testing has fallen short of expectations, especially when these tests have medium or high stakes for students. After a temporary cancelation of external assessment of learning due to the COVID-19 pandemic, the Portuguese government is taking a step forward. With a gradual implementation until 2025, the reintroduction of external assessment of learning in the Portuguese education system foresees all national assessment tests and all national examinations in digital format with the implementation of the External Assessment Dematerialization Project (DAVE). However, assessment reforms, such as DAVE, raise concerns within school communities. To identify and analyze these concerns, we conducted 32 semi-structured interviews with head teachers from mainland Portugal. To understand to what extent these concerns were considered and covered in DAVE’s design and implementation, a supplementary interview was conducted with the president of the Portuguese institute responsible for DAVE. In addition, a survey with 2 673 teachers was carried out to find out their degree of agreement with the implementation of DAVE. Results show that DAVE raises many concerns among head teachers, and it is not fully accepted by the teachers.

Digital Storm: How Ghana Defied Doubts in Nursing and Midwifery Assessment Reform Against the Odds

Digital Storm: How Ghana Defied Doubts in Nursing and Midwifery Assessment Reform Against the Odds

Research Progress on Curriculum Assessment Reform for Nursing Majors

This article introduces the limitations of conventional methods of curriculum assessment and the importance of reforming them. Specifically, it analyzes the present status of nursing curriculum assessment reform and highlights the issues that currently exist. Moreover, it also outlines the future development directions of nursing curriculum assessment reform research.

Why do we assess students? investigating general chemistry instructors’ conceptions of assessment purposes and their relationships to assessment practices

Assessment plays a critical role in instruction and curriculum. Existing literature on instructors’ assessment practices and related factors has been intensively focused on primary and secondary education. This study extended the contexts of previous literature to post-secondary chemistry education by exploring general chemistry instructors’ conceptions of assessment purposes and their assessment practices. Semi-structured interviews were conducted with 19 general chemistry instructors from 14 institutions across the East Coast region of the United States of America. The results demonstrate that instructors predominately perceive the purpose of Assessment of Learning (i.e., evaluation of student performance) with only few of them mentioning purposes of Assessment for Learning (i.e., assessment provides actionable feedback for both the instructors and the students) and Assessment as Learning (i.e., assessment promotes self-regulation). The use of various assessment practices is related to the number of assessment purposes instructors recognize. In addition, the study demonstrates that instructors perceive their assessment practices to be influenced by academic culture and departmental norms. This nuanced understanding can guide practical and research efforts to improve chemistry instructors’ engagement in assessment reforms.

Incongruencies and Detrimental Effects of Neoliberal Education Reform in the United Arab Emirates (UAE)

Charged with postcolonial educational planning and development activities over the last 50-70 years, the context and content of educational planning and policymaking have evolved considerably in most Arab countries. The methodologies of planning and policymaking have also changed to match changing milieus and national priorities. Despite some accomplishments, many have amassed a record of educational policy and reform undertakings that have been less successful. Moreover, many Arab countries have been shown to continually lag behind other regions with similar levels of development, such as Latin America and South Asia. However, a few regional countries in the Gulf Cooperation Council (GCC), such as the United Arab Emirates (UAE), have taken a leading role in educational change initiatives in the implementation of globalized neoliberal reforms. More precisely, they have promoted bilingual (English/Arabic) and imported curricula, the modernization of school systems, privatization, standardization, accountability, school choice, and assessment reform. This study examined the implementation of recent UAE neoliberal reforms utilizing a framework derived from postmodern (neo)institutional analysis. This study argues that many policy undertakings have included contradictions and harmful impacts impeding the achievement of intended goals. Furthermore, educational policymaking has been caught between global and centralized national prescriptions for education and its role in society and unique contextual demands, which have created administrative challenges, dissatisfaction, and public resentment.

Incongruencies and Detrimental Effects of Neoliberal Education Reform in the United Arab Emirates (UAE)

Charged with postcolonial educational planning and development activities over the last 50-70 years, the context and content of educational planning and policymaking have evolved considerably in most Arab countries. The methodologies of planning and policymaking have also changed to match changing milieus and national priorities. Despite some accomplishments, many have amassed a record of educational policy and reform undertakings that have been less successful. Moreover, many Arab countries have been shown to continually lag behind other regions with similar levels of development, such as Latin America and South Asia. However, a few regional countries in the Gulf Cooperation Council (GCC), such as the United Arab Emirates (UAE), have taken a leading role in educational change initiatives in the implementation of globalized neoliberal reforms. More precisely, they have promoted bilingual (English/Arabic) and imported curricula, the modernization of school systems, privatization, standardization, accountability, school choice, and assessment reform. This study examined the implementation of recent UAE neoliberal reforms utilizing a framework derived from postmodern (neo)institutional analysis. This study argues that many policy undertakings have included contradictions and harmful impacts impeding the achievement of intended goals. Furthermore, educational policymaking has been caught between global and centralized national prescriptions for education and its role in society and unique contextual demands, which have created administrative challenges, dissatisfaction, and public resentment.

Summary Report APE 2023, 10--12 January, Berlin, Germany Berlin Re-Visited: Building Technological Support for Scholarship and Scientific Publishing

This paper summarizes the 18th Academic Publishing in Europe (APE) Conference: Berlin Re-Visited: Building Technological Support for Scholarship and Scientific Publishing, held as a hybrid event 10 and 11 January 2023, and organized by the Berlin Institute of Scholarly Publishing (BISP), a not-for-profit organization dedicated to bringing publishers, researchers, funders, and policymakers together. This year’s conference theme “Out with the old, in with the new!” was discussed and presented in keynote speeches, the APE lecture, and several panel discussions. Current challenges within scholarly publishing, e.g., with research integrity, trust, and research assessment, have much to do with the old ways of doing things. To move science forward, new technologies and innovations, like the decentralized web, FAIR digital objects, and blockchain technology are needed to shape new paradigms. In many sessions it was stressed that not just new technologies are needed to move science forward, but human collaboration and partnerships as well. The changing role of the journal and the importance to recognize more diverse research outputs, beyond the journal article, was a topic of importance. Not only research assessment reforms and improved collaboration amongst different stakeholder groups are needed to address this, new publishing systems, better metadata, and open infrastructures as well. A session presenting different start-ups showcased how Artificial Intelligence, Natural Language Processing and software technology can be used to tackle problems in e.g., finding relevant funders and peer reviewers, and detecting image plagiarism. It was also discussed what publishers can do to help achieve the Sustainable Development Goals. Collaboration, data transparency and sharing best practices amongst researchers, funders, and policy makers are key. Another important topic during this year’s APE was how publishers can support Early Career Researchers: establishing new workflows and infrastructures that enable publication of a wider range of research outputs, and broader recognition of these outputs, will incentivize ECRs. This year’s APE was concluded with the APE Award Ceremony. The winner - Vsevolod Solovyov from Prophy - has used AI to enhance the current peer review system and made a very important contribution to improving the scholarly communication system.

Research Assessment – Navigating Pitfalls and Promoting Change

This is a podcast version of the closing panel discussion at The 18th Munin Conference on Scholarly Publishing (Tromsø, Norway, 8–10 November 2023). The main theme of the conference was scholarly communication in its myriad of forms. The push for a reform of research assessment argues for precisely the need to recognize and reward a wider set of research activities and competencies, not least in order to ensure that science – and scientics – remain in close contact with society. The demands for change will surely impact how and where research is disseminated and communicated. A key question then is how to evaluate research activities and outputs beyond academic publications. The panelists reflected on questions such as: how do we measure the impact of citizen science, or document quality in innovative research practices and scholarly outputs? And how to ensure consensus on assessment methods for a wider and more inclusive range of research activities within a diverse academic landscape? The panel consisted of champions of research assessment reform (Yensi Flores Bueso, University of Washington / University College Cork; Kirstie Whitaker, The Alan Turing Institute) and university leaders (Hervé Dole, vice-president for arts, culture and society at Université Paris-Saclay; Jan-Gunnar Winther, pro-rector for research and development, UiT). Questions were prepared by Tanja Larssen (open science adviser, UiT) in collaboration with Per Pippin Aspaas (head of library research and publishing support, UiT), who chaired the panel on the day. First published as a podcast: December 19, 2023. A video recording of the same session – with metadata providing more detailed information on each panelist – can be found at https://doi.org/10.7557/5.7276.

Research Assessment Reform, Non-Traditional Research Outputs, and Digital Repositories: An Analysis of the Declaration on Research Assessment (DORA) Signatories in the United Kingdom

Objective – The goal of this study was to better understand to what extent digital repositories at academic libraries are active in promoting the collection of non-traditional research outputs. To achieve this goal, the researcher examined the digital repositories of universities in the United Kingdom who are signatories of the Declaration on Research Assessment (DORA), which recommends broadening the range of research outputs included in assessment exercises. Methods – The researcher developed a list of 77 universities in the UK who are signatories to DORA and have institutional repositories. Using this list, the researcher consulted the public websites of these institutions using a structured protocol and collected data to 1) characterize the types of outputs collected by research repositories at DORA-signatory institutions and their ability to provide measures of potential impact, and 2) assess whether university library websites promote repositories as a venue for hosting non-traditional research outputs. Finally, the researcher surveyed repository managers to understand the nature of their involvement with supporting the aims of DORA on their campuses. Results – The analysis found that almost all (96%) of the 77 repositories reviewed contained a variety of non-traditional research outputs, although the proportion of these outputs was small compared to traditional outputs. Of these 77 repositories, 82% featured usage metrics of some kind. Most (67%) of the same repositories, however, were not minting persistent identifiers for items. Of the universities in this sample, 53% also maintained a standalone data repository. Of these data repositories, 90% featured persistent identifiers, and all of them featured metrics of some kind. In a review of university library websites promoting the use of repositories, 47% of websites mentioned non-traditional research outputs. In response to survey questions, repository managers reported that the library and the unit responsible for the repository were involved in implementing DORA, and managers perceived it to be influential on their campus. Conclusion – Repositories in this sample are relatively well positioned to support the collection and promotion of non-traditional research outputs. However, despite this positioning, and repository managers’ belief that realizing the goals of DORA is important, most libraries in this sample do not appear to be actively collecting non-traditional outputs, although they are active in other areas to promote research assessment reform.

Unpacking Continuous Assessment: Teacher Knowledge and Attitudes in Zimbabwe Rural Secondary Schools

The paper explores the implementation of continuous assessment in the context of teacher knowledge and attitudes in Zimbabwean rural secondary schools. Continuous assessment is a new curriculum assessment regime in the curriculum reform agenda in Zimbabwe introduced in the 2015 updated curriculum.The study defines continuous assessment as the mechanism where thelearner performs over timeconstitutes the final gradingof the learner.The researchers explain the teacher-related implementation challenges in continuous assessment in rural secondary schools in Zimbabwe. Ten teachers and five secondary school heads from Goromonzi District of Mashonaland East Province in Zimbabwe participated in the study. The five participating heads were conveniently selected, while theten teachers were randomly sampled to participate in the case study.Semi-structured questionnaires and in-depth interviews were used to collect qualitative data on rural teachers’ conceptualisationof continuous assessment and how this had shaped their attitudes towardthe concept. The general finding from the study was that the new assessment regime had empowered the teachers to make critical decisions about the students they teach and allow learners to co-create knowledge relevant to their life contexts.However, their concerns were around the absence of knowledgeable professionals in the area who can cascade and message the correct and accurate information regarding the implementation of the assessment reform.The study further found that this weakness in the implementation chain had resulted in negative attitudes of teachers and school heads. The study recommends that the Ministry of Primary and Secondary Education revisit the current continuous assessment to takeon board all educational stakeholders’ concerns for effective implementation of the assessment reform.

The rise of responsible metrics as a professional reform movement: A collective action frames account

Abstract Recent years have seen a rise in awareness around “responsible metrics” and calls for research assessment reforms internationally. Yet within the field of quantitative science studies and in research policy contexts, concerns about the limitations of evaluative bibliometrics are almost as old as the tools themselves. Given that many of the concerns articulated in recent reform movements go back decades, why has momentum for change grown only in the past 10 years? In this paper, we draw on analytical insights from the sociology of social movements on collective action frames to chart the emergence, development, and expansion of “responsible metrics” as a professional reform movement. Through reviewing important texts that have shaped reform efforts, we argue that hitherto, three framings have underpinned the responsible metrics reform agenda: the metrics skepticism framing, the professional-expert framing, and the reflexivity framing. We suggest that although these three framings have coexisted within the responsible metrics movement to date, cohabitation between these framings may not last indefinitely, especially as the responsible metrics movement extends into wider research assessment reform movements.

Research Assessment – Navigating Pitfalls and Promoting Change

Watch VIDEO. The main theme of the Munin conference is scholarly communication in its myriad of forms. The push for a reform of research assessment argues for precisely the need to recognize and reward a wider set of research activities and competencies. The demand – and initiative – for change will surely impact how and where research is disseminated and communicated. A key question then is how to evaluate research activities and outputs beyond academic publications. With this panel we welcome a discussion on research assessment that brings together perspectives from all the topics of this year’s Munin conference. We invite you to add your insights and diverse expertise to this panel that will reflect on questions such as: how do we measure the impact of citizen science, or document quality in innovative research practices and scholarly outputs? And how to ensure consensus on assessment methods for a wider and more inclusive range of research activities within a diverse academic landscape?

Implementing school-based assessment reforms to enhance student learning: a systematic review

Implementing school-based assessment reforms to enhance student learning: a systematic review

L’engagement et le rôle de l’ACEA dans les changements en matière d’évaluation

This study follows a network-based Assessment Redesign Project at a Canadian university to investigate engagement and sustained implementation. The following strategies were employed in the project: mini-grants, embedded support, a community of practice, and social networks. Assessment facilitators worked in discipline clusters to achieve mutual goals for assessment reform targeted at the authentic assessment of critical thinking and problem-solving. Interviews were conducted with nine of the 25 project members one-year post-implementation. The study adopted a motivational theoretical lens to investigate how the experience of the Assessment Redesign Project affected motivation and the continued adoption or propagation of assessment strategies. Participants commented on how helpful the embedded support had been in building their assessment skills or knowledge. The mini-grants were used (in some cases) to fulfil scholarship of teaching and learning (SoTL) goals. All of those engaged in SoTL demonstrated intrinsic motivation for assessment change and had propagated assessment techniques or activities into other courses. In the few cases where motivation was purely extrinsic, there was no SoTL or continuation of assessment activities. This study highlights the links between SoTL and the longer-term impact of the Assessment Redesign Project. Suggestions are provided for institutions wishing to replicate outcomes from the project.