Educational Assessment Research Articles

Special Issue: Selected papers from the AIxIA 2023 Workshops

The 2023 edition of the AIxIA Conference, held in Rome, brought together a large number of researchers and practitioners to discuss the most recent and important advancements in Artificial Intelligence (AI). The conference featured 19 workshops, organized by 77 experts, attracting 248 submissions and resulting in 16 proceedings. This special issue presents extended versions of selected papers initially showcased at these workshops. Each paper underwent rigorous review and represents a diverse array of topics, reflecting the multifaceted nature of the Italian AI community. The topics covered include ethical foundations to symbiotic AI, symbolic knowledge extraction from black-box models, creative influence prediction using graph theory, AI approaches to multidimensional poverty prediction, an assessment of AI-based supports for informal caregivers, deep learning-based EEG denoising, AI-assisted board-game-based learning, large language models for assessment and feedback in higher education, geometric reasoning in the Traveling Salesperson Problem, defeasible reasoning in weighted knowledge bases, and conditional computation in neural networks. These contributions demonstrate the innovative and interdisciplinary research within the AI community, offering valuable insights and advancing the field.

Behind the grades: co-constructing fairness to reach agreement in evaluative judgement

Assessment in higher education is often a balancing act between fairness and the inherent complexity of standards, with evaluative judgement playing a crucial role in determining the quality of student work. This study employs a sociomaterial perspective to investigate the often-obscured justifications used by examiners during real-time moderation meetings, as well as their role in shaping judgements. Through participant observation and focus group interviews, a two-stage analysis was conducted. Thematic analysis identified four categories of justifications – predefined, personal, norm-based and other justifications – highlighting a range of rationales behind examiners’ grading decisions. Subsequent interaction analysis explored how these justifications inform the evaluative judgement process, revealing their role in co-constructing an understanding of fairness to reach a compromise. A key finding is that different perspectives on fairness significantly impact evaluative judgements. The study recommends the use of the concepts ‘tinkering’ and ‘rupturing’ as analytical tools to promote flexibility and harness the strengths of subjectivity while maintaining the integrity of judgements.

Bias in Observed Assessments in Medical Education: A Scoping Review.

Observed assessments are integral to medical education but may be biased against structurally marginalized communities. Current understanding of assessment bias is limited because studies have focused on single specialties, levels of training, or social identity characteristics (SIDCs). This scoping review maps studies investigating bias in observed assessments in medical education arising from trainees' observable SIDCs at different medical training levels, with consideration of medical specialties, assessment environments, and assessment tools. MEDLINE, Embase, ERIC, PsycINFO, Scopus, Web of Science Core Collection, and Cochrane Library were searched for articles published between January 1, 2008, and March 15, 2023, on assessment bias related to 6 observable SIDCs: gender (binary), gender nonconformance, race and ethnicity, religious expression, visible disability, and age. Two authors reviewed the articles, with conflicts resolved by consensus or a third reviewer. Results were interpreted through group review and informed by consultation with experts and stakeholders. Sixty-six of 2,920 articles (2.3%) were included. These studies most frequently investigated graduate medical education (44 [66.7%]), used quantitative methods (52 [78.8%]), and explored gender bias (63 [95.5%]). No studies investigated gender nonconformance, religious expression, or visible disability. One evaluated intersectionality, with SIDCs described inconsistently. General surgery (16 [24.2%]) and internal medicine (12 [18.2%]) were the most studied specialties. Simulated environments (37 [56.0%]) were studied more frequently than clinical environments (29 [43.9%]). Bias favoring men was found more in assessments of intraoperative autonomy (5 of 9 [55.6%]), whereas clinical examination bias often favored women (15 of 19 [78.9%]). When race and ethnicity bias was identified, it consistently favored White students. This review mapped studies of gender, race, and ethnicity bias in the medical education assessment literature, finding limited studies on other SIDCs and intersectionality. These findings will guide future research by highlighting the importance of consistent terminology, unexplored SIDCs, and intersectionality.

'Do it afraid': An arts-based reflexive collective case study exploring youth responses to post-concussion communication changes in daily life.

Concussion and communication researchers have yet to study how post-concussion communication changes affect youths' daily lives. The lack of attention paid to how young people respond to communication changes during concussion recovery constitutes a significant gap in current concussion management research and practices. To explore how youth respond to the effects of post-concussion communication changes in their daily life, including (1) daily routines, (2) relationships with family members, (3) relationships with peers and (4) participation in school/work and community activities. Five youths (16-25 years) and three family members participated in this arts-based reflexive collective case study. Ecocultural theory provided the theoretical framework for study design, data collection and analysis. Cases consist of (1) pre-interview demographic information, (2) three 60-90-min virtual interviews, (3) optional family member interviews, (4) multi-media arts-based participant-generated materials representing participants' experiences of communication change and concussion, and (5) researcher observations, discussions and reflexive journal entries. Reflexive thematic analysis was used to analyse the data. Analysis yielded four themes that illustrate the ways youth navigated and adapted to post-concussion communication changes: (1) navigating changes in communication tasks, daily roles, and identity; (2) re-negotiating relationships and emotional reactions; (3) seeking control and learning to let go during recovery; and (4) helping youth adapt to post-concussion communication changes. The study findings deepen our understanding of the impact of post-concussion communication changes on youths' daily lives and underscore considerations critical to the development ofcommunication-focused concussion education programs and interventions tailored specifically for youth. What is already known on the subject Youth is a critical period of social and emotional development. Communication is integral to identity, relationships, participation in daily activities and well-being. Concussions can affect speech clarity, fluency, understanding and use of language, and social interactions. Re-engaging in routine activities and pre-injury roles can be challenging for youth experiencing communication changes as part of complex concussion recoveries. What this paper adds to the existing knowledge Findings from this research illuminate how youth navigate and adapt to communication changes post-concussion and support the development of youth-focused communication education programs, assessments and interventions. Youth participants actively managed their recoveries by developing innovative strategies to support their communication during daily activities, learning about communication, practicing communication tasks sequentially, and facing fears. Youth also reframed and challenged narrow views of 'normal communication'. What are the potential or actual clinical implications of this work? Findings highlight the need for more youth- and communication-focused education materials and programs within youth concussion management protocols. Information about the specific ecological and sociocultural factors youth encounter during concussion recovery is needed to develop targeted communication-focused education and intervention programs for youth and their families to mitigate risks of isolation, loneliness, and mental health concerns and increase youths' participation in family, community and cultural life. By learning from youth about how communication changes affected their participation in daily activities, identity and relationships, clinicians can provide information and interventions to reduce adverse listener reactions and help young people feel supported and understood.

Characteristics of Islamic religious education assessment instruments

Validity, Reliability, Differentiation, Level of difficulty is an important instrument in education so that the data generated can be used properly. Validation is divided into validation of a content, construct. Content validation is an instrument that can be said to be valid if it is in accordance with the curriculum, material and objectives. While construct validity is validity that thinks how the achievement of test items can measure something that will be measured and adjusted to the concepts that have been set. Reliability is determined by the Cronbach's alpha value, if the Cronbach's alpha value of 0.5 can be used for research purposes. Differentiating ability distinguishes the upper and lower groups in the form of questions. The level of difficulty distinguishes questions in terms of difficulty.

Enhancing trade policy education: A review of pedagogical approaches in public administration programs

This literature review rigorously examines pedagogical approaches in trade policy education within public administration programs, with the goal of enhancing educational strategies for future public administrators in the context of global trade. Recognizing the vital role of trade policy in economic development, international relations, and governance, the review explores the evolution, current state, and challenges of trade policy education. The scope includes an analysis of traditional lectures, case studies, experiential learning, interdisciplinary teaching, and the role of technology in modernizing educational practices. Using a qualitative methodology, the review synthesizes insights from literature reviews, case studies, and comparative analyses of curricula across various public administration programs. Findings indicate that while traditional lectures provide essential theoretical knowledge, their effectiveness is significantly enhanced by case studies and experiential learning methods, such as simulations and role-playing exercises. These methods foster practical application and critical thinking, which are crucial for comprehending complex trade policies. The integration of technology, including online courses and digital simulations, further improves learning outcomes by making education more interactive and accessible. Interdisciplinary teaching, which incorporates perspectives from economics, law, political science, and international relations, enriches the educational experience and prepares students for the multifaceted nature of trade policy. The review concludes that a balanced pedagogical approach, combining traditional and innovative methods, is essential for effective trade policy education. It recommends incorporating active learning strategies, continuous assessment, and professional development for educators. Additionally, fostering interdisciplinary collaboration and leveraging technological advancements are vital for enhancing the curriculum. These strategies aim to equip future public administrators with the comprehensive knowledge and skills necessary to navigate the complexities of global trade, thereby contributing to informed and effective policymaking. Keywords: Trade Policy Education, Public Administration, Pedagogical Approaches, Experiential Learning, Interdisciplinary Teaching, Technology Integration.

A Study of Components on Effective Academic Administration with Participation of Parents and Community in Small-Sized Schools under the Office of the Basic Education Commission

The research aimed to examine the components of effective academic administration involving collaborative participation from parents and the local community in small-sized schools under the authority of the Basic Education Commission. The study employed a mixed-method research approach, incorporating both quantitative and qualitative methodologies, divided into two phases. Phase 1: Examine the components of academic administration by synthesizing documents and related studies. Phase 2: Conduct interviews and verify the effectiveness of academic administration. The information providers consisted of 7 highly qualified individuals in the field of educational administration. The selection method was purposive sampling. The tools utilized included interview and assessment forms. Statistical analysis involved mean, standard deviation, and content analysis. The research findings indicated that the effective collaborative components of academic administration involving parents and communities in small-sized schools consisted of three main components: 1) The workload and scope of academic administration in small-sized schools, comprising 10 aspects, 2) Effective academic administration, consisting of 4 aspects, and 3) Collaborative management processes, involving 4 steps. The assessment results demonstrated a high level of suitability for the effective collaborative components of academic administration involving parents and communities in small-sized schools, both at the individual component level and overall. The component with the highest level was the development of an internal quality assurance system within the educational institution, including measurement, assessment, and comparison of learning outcomes. Conversely, the component with the lowest level was the development of learning resources.

Automated Answersheet Evaluation using BERT

The research introduces an innovative approach to automating answersheet evaluation. The methodology combines the power of BERT (Bidirectional Encoder Representations from Transformers) and cosine similarity within the realm of natural language inference, allowing for a sophisticated analysis of student responses. BERT’s contextual embeddings provide a deep understanding of the semantic nuances present in answers, while cosine similarity serves as a robust metric for measuring the semantic alignment between student responses and ideal answers. Natural language inference enriches the model’s capacity to decipher inferential relationships, enabling a more nuanced eval- uation that goes beyond surface-level language comprehension. Augmented by natural language inference, our model adeptly captures inferential relationships, enhancing its comprehension of nuanced contextual nuances. The results underscore its potential to redefine automated grading paradigms, offering educators a powerful tool for efficient and unbiased assessments of student answers, thereby contributing to the advancement of educational technology and assessment methodologies.

Universal Design for Learning (UDL) in the Islamic Religious Education Assessment Model

Current curriculum developments place greater emphasis on developing learning models and methods that can accommodate students' differences in learning styles and abilities. In the independent curriculum, this concept is better known as differentiated learning. One concept that can facilitate differentiated learning in the context of assessment is Universal Design for Learning (UDL). Using a Systematic Literature Review (SLR) approach, this research aims to analyze the effectiveness of implementing assessment models in PAI learning in the UDL context. The findings of this research indicate that the assessment model that is widely applied to PAI learning is the authentic assessment model. Based on educational level, this authentic assessment model is more widely applied at the junior secondary education level (SMP/MTs). Based on the results of the analysis of articles that met the inclusion criteria, no evidence was found regarding the effectiveness of its implementation which had an impact on PAI learning processes and outcomes. The findings of this research can be used as an empirical basis for conducting research that focuses on the influence of implementing assessment models on PAI learning processes and outcomes.

Education for flourishing: A social contract for foundational competencies

This essay is a commentary on Curren et al., ‘Finding consensus on well-being in education’. It acknowledges a growing international consensus that presents educational systems need to change and argues the case for consensus on flourishing as the overall purpose of education can be strengthened by drawing on economists’ work on well-being with respect to the inclusive wealth of nations. It emphasizes the need for tangible and measurable indicators that educators can use when implementing Curren et al.’s recommendations and outlines the International Science and Evidence based Education assessment’s suggestion of a ‘whole brain’ approach to education for flourishing.

Assessment regimes, data, gender haunting, and health education

ABSTRACT Most secondary (high) schools in a broad range of jurisdictions internationally engage in various forms of high stakes, standardized assessment and related qualifications. In this paper, we interrogate how educational achievement regimes – especially via the reporting of curriculum and assessment ‘data’ – continue to mobilize particular gender norms. Drawing on Derrida’s notion of haunting we explore how such regimes impose and reinscribe stable and binary gendered patterning and create what Barad has named ‘entangled relationalities of inheritance’ [https://doi.org/10.3366/drt.2010.0206,] despite young people (and many schools) moving towards greater recognition of non-binary genders. Drawing on assessment data from Aotearoa New Zealand, we look at both generalized reporting of educational achievement data along the lines of ‘male’ and ‘female’ and on reporting of a single (historically gendered) curriculum subject – health education. We argue that such systems are ‘haunted’ by stable gender categorizations and hierarchies and we ask what this means for the reporting of educational assessment data and the erasure of identities that don’t align with the binary.

Exploring the potentials of ChatGPT for instructional assessment: Lecturers' attitude and perception

Lecturers play a crucial role in the educational process, offering unique insights and perspectives within the classroom. The issue of cred­ibility in educational assessment often rests on the shoulders of lecturers, who are responsible for eval­uating students' progress. The present study aimed to investigate lecturers' attitudes and perceptions re­garding the potential of ChatGPT for instructional assessment. A correlational research design was adopted, and purposive sampling was used to select 102 lecturers from Nigerian universities who had uti­lised ChatGPT for instructional assessment. Data was collected through an online structured question­naire. The normality and homogeneity of variance assumptions were met, as evidenced by kurtosis and skewness values falling within acceptable thresh­olds. The lecturers employed the instructional assessment questionnaire utilising ChatGPT to gather and analyse the data, employing t-tests and ANOVA. The findings revealed a statistically significant difference between perception (F (3, 98) =7.168, p=0.001 <0.05) and lecturers' years of experience in using ChatGPT for instructional assessment. The study indicated that lecturers held low attitudes and had poor perception levels when it came to exploring the potential of ChatGPT. However, it is recommended that training be provided to enhance lecturers' attitudes and perception levels to fully exploit the potential of ChatGPT for instructional assessment.

Metricfy: Aplikasi Pembelajaran Numerasi berbasis Web

The low numeracy literacy among students is primarily due to two factors: a lack of interest in numeracy learning and limited learning resources. The Ministry of Education and Culture's National Assessment Programme in 2022 revealed that numeracy skills in Indonesia remain low, with only about 40,63% of students demonstrating numeracy competence. Improvement is necessary to achieve better results. The aim of this research is to develop an interactive numeracy application for student learning and assess its suitability. Development utilised Rapid Application Development (RAD) with three main phases: requirement planning, design, and implementation. Data collection involved surveys and quantitative descriptive analysis. In the final development phase, the application underwent testing to ensure functionality and effectiveness. Functional testing using black box methods showed all features performed as expected. Additionally, a System Usability Scale (SUS) survey administered to students yielded a score of 78.05, categorised as "acceptable" (good), indicating the application meets usability and effectiveness standards.

(138) ARE WE THERE YET? ASSESSING OUR CURRENT STATE OF SEX MED EDUCATION

Abstract Introduction Sexual concerns are prevalent, but studies have demonstrated that patients often feel most comfortable if the clinician initiates the conversation. Many patients are hesitant to discuss sexual concerns with their providers for a variety of reasons including fear of judgment, lack of clinician knowledge, and concerns that these issues are abnormal or unusual. Clinician education plays an important role in helping to facilitate these difficult conversations. However, there has never been a broad assessment of education across medical training programs, while the most recent assessment of MD training programs occurred approximately a decade ago. Objective We set out to create and administer a survey to assess sexual medicine training across most major medical disciplines where sexual health matters may be discussed, including nurse practitioner, midwifery, physician, and physician assistant training. Methods A subcommittee of the ISSWSH Advocacy Committee was formed with members representing the medical disciplines to be surveyed. A survey was developed to gather data on sexual medicine education offered at nursing, midwifery, physician, and physician assistant training programs. The survey included a series of questions regarding content taught during these programs (checklists, open ended responses). The survey was approved by the Wake Forest IRB and distributed using RedCap to various institutions from the above disciplines throughout the United States. Results We received a total of 33 complete responses. 88.5% of respondents stated they used traditional lecture as their teaching method, whilst only 11.5% used simulation, and 7.7% used actual mechanical models. The majority of programs that responded spent 3-5 hours total dedicated to human sexuality and sexual function education. Interestingly, respondents noted that faculty involved in sexual medicine education were mostly OB/GYN in background (63%), followed by family medicine (18.5%) and urology (18.5%). While not all respondents indicated that their students were assessed on their knowledge after teaching, it was noted that the majority of programs that did respond indicated that examination was the most common mode of assessment. Only 33% of institutions noted having a regional or nationally recognized expert in the field. Conclusion It is essential that the accrediting bodies for medical programs incorporate the input of international societies such as ISSWSH (in collaboration with other sexual medicine organizations) to formulate an official statement and recommendations for a standardized curriculum for sexual medicine.

Assessment Under Duress: Challenges to Physical Education Assessments and the Emergence of Compromised Norms

Assessment Under Duress: Challenges to Physical Education Assessments and the Emergence of Compromised Norms

A MODEL TO ENSURE QUALITY MANAGEMENT OF SCIENCE AND TECHNOLOGY FOR AUTONOMOUS INSTITUTIONS IN VIETNAM

Alongside internal quality assurance measures, external quality assessment serves as a mechanism to ensure the quality of education in Vietnam, proven to offer various opportunities (enrollment, research collaboration, technology transfer, etc.) as relevant parties become aware of and trust in the educational institution's quality. Consequently, the impact of educational quality assessment prompts systematic quality assurance activities within the university, yielding high results based on post-assessment expert recommendations. However, among the university activities in Vietnam, scientific and technological management has been identified as one of the few standards with a relatively low average score (Hien et al., 2022); Anh, 2022). Proposing a suitable research model to assess the influence of educational quality assessment on the scientific and technological management of autonomous universities in Vietnam will assist these institutions in identifying crucial elements of scientific and technological management. This will facilitate the implementation of more feasible activities to ensure the quality of these operations, contributing significantly to the overall educational quality of the institution.

Pioneering Online Assessment Solutions: Empirical Experiences from Educational Practitioners

The emergence of E-learning has shaped the university's new pedagogical, environment to guarantee business continuity in worse pandemics. Assessment is an end process of learning however, measuring it to globally accepted standards is still a nightmare in universities. It is not clear whether the assessment spells out the tasks posed to learners? illustrate an observable demonstration of the learners’ ability? or have a detailed scoring criterion? no wonder stake holders still doubt assessment online. The research examined qualitative literature regarding the innovative strategies for online educational assessment sustainable and scalable. To gather empirical qualitative data on this subject, a systematic review of literature was undertaken. The study responded to one major research question. “What are the experiences of educational practitioners reported in empirical qualitative research studies pertaining innovative strategies for online educational assessment?” We conducted a qualitative review of the scientific literature published between 2010 and 2022 using the PRISMA framework, thematic analysis was employed to generate themes and patterns. Education databases, like ERIC were used for the article search. Search phrases utilized for this systematic review included “Strategies for online assessment” and “Assessment of E-learning.” 16 articles were included in the analysis for this study. findings point to the Asynchronous Online Discussion, E-portfolios, Distance Project based assessment, Online proctored Exams, Online non-proctored exams as sustainable strategies. The study demonstrates a potential solution to online assessment in universities and restores confidence among educational partners and funders.

Curriculum Management and Educational Assessment At the Master Program in Defense Industry, Defense University, Republic of Indonesia

This study aims to describe curriculum management and higher education assessment systems in producing quality college graduates. This study used descriptive qualitative methods, and data collection techniques through observation, interviews and documentation studies. Data analysis techniques use data validity tests with source tringulation. The results of the study show that the Curriculum Management and Educational Assessment System in the Master of Defense Industry Study Program of Unhan of the Republic of Indonesia in accordance with curriculum standards and higher education assessment systems through planning, organizing, implementing and supervising refer to the vision, mission and profile of graduates in attitudes, knowledge and skills. The conclusion and suggestion in this study is that curriculum planning and educational assessment systems in the Defense Industry Master Study Program of the Indonesian Defense University are carried out through systematic curriculum design, strict assessment standards, and comprehensive assessment instruments. Curriculum organization involves learning designs prepared by course lecturers, distribution of courses each semester, and assessment instruments applied to each subject. The implementation of the curriculum and assessment system refers to the standards of the learning process and assessment that have been set, including the academic calendar and the schedule of teaching and learning activities of each lecturer. Supervision of the curriculum and assessment system is carried out through lesson plans, supervision, reports on lecturer learning activities, academic and non-academic achievements of students, and comprehensive learning evaluation. Although the curriculum includes aspects of attitudes, knowledge, and skills, the assessment of non-academic skills such as soft skills and character needs to be considered as one of the development areas in the educational assessment system to improve the quality of graduates holistically.Keywords: Curriculum Management, Educational Assessment, Defense University of Republic of Indonesia

Unveiling fairness in scoring: a thorough method for precise cutoff score calculation in education assessment

PurposeThe purpose of this research is to address the need for a robust system to accurately determine a cutoff score by using the Angoff method and leveraging the Rasch infit and outfit statistics of item response theory by detecting and removing misfitting items in a test.Design/methodology/approachResearchers in educational evaluation support the use of grades as a complete measure to evaluate students’ overall academic performance. In higher education, determining cutoff scores is crucial for assessing academic progress and achievement. Higher education institutions worldwide set predefined minimal standards of success, sometimes referred to as cutoff thresholds. These thresholds serve as benchmarks for assessing performance quality and determining eligibility for academic credit or advancement.FindingsThe proposed approach combines the Angoff method with Rasch item fit statistics to precisely determine the cutoff score by excluding items that do not fit well (misfitting items) in a test. This methodology holds the potential to enhance the fairness and accuracy of grading practices, ensuring that assessments align with educational objectives and provide meaningful feedback to students.Originality/valueThe value of this research extends to various fields where cutoff scores are used for decision-making purposes. By using Rasch item fit statistics and the Angoff method, practitioners and researchers can enhance the quality and validity of their assessments.

Tapping the alphabets of learning-oriented assessment: self-assessment, classroom climate, mindsets, trait emotional intelligence, and academic engagement are in focus

This study investigates the role of self-assessment (SA), classroom climate (CC), and psychological variables in learning-oriented assessment (LOA) within English as a foreign language (EFL) education. Utilizing a mixed-methods approach, data was collected through surveys and semi-structured interviews from EFL learners and instructors. Findings reveal that SA plays a pivotal role in promoting learner autonomy, self-regulated learning, and academic engagement (AE). By actively involving students in the assessment process and providing opportunities for reflection and feedback, educators can empower learners to take ownership of their learning journey and enhance their overall learning outcomes. Moreover, the study underscores the importance of creating a supportive CC characterized by mutual respect, collaboration, and inclusivity. Positive teacher-student relationships and inclusive learning environments contribute significantly to students’ academic and socioemotional development. Additionally, the study highlights the influence of psychological variables such as growth mindset and emotional intelligence (EI) on student learning outcomes. Learners who adopt a growth-oriented mindset and possess high levels of EI are more likely to persevere in the face of challenges and actively engage in the learning process. The implications of this study suggest the importance of adopting a holistic approach to assessment and instruction in EFL education, with stakeholders urged to incorporate strategies to promote SA, create supportive CC, and foster psychological variables for enhanced student learning and well-being.