Educational Assessment Research Articles

KOMPETENSI GURU DALAM PENYUSUNAN INSTRUMEN PENILAIAN MATA PELAJARAN PENDIDIKAN AGAMA ISLAM

Teacher competence is a very important aspect to improve the quality of education. The competencies that must be owned by a teacher consist of pedagogic, personality, professional and social competencies. Whereas PAI learning is a process of forming knowledge, attitudes and skills through fact-based cognitive performance and contextual socio-religious phenomena. The formulation of the problems in this study are: 1) What are the scopes of PAI learning?, 2) What competencies must a teacher have?, and 3) What are the principles and techniques in preparing PAI assessment instruments?. The purpose of this study was to determine the scope of Islamic religious education learning, describe the competencies that must be possessed by a teacher, and find out what principles and techniques are used in compiling Islamic religious education assessment instruments. This research is expected to provide benefits for all education stakeholders to be used as a reference in improving teacher competence. The method used in this research is library research with a qualitative approach.

Assessing Educational Quality: Comparative Analysis of Crowdsourced, Expert, and AI-Driven Rubric Applications

Exposing students to low-quality assessments such as multiple-choice questions (MCQs) and short answer questions (SAQs) is detrimental to their learning, making it essential to accurately evaluate these assessments. Existing evaluation methods are often challenging to scale and fail to consider their pedagogical value within course materials. Online crowds offer a scalable and cost-effective source of intelligence, but often lack necessary domain expertise. Advancements in Large Language Models (LLMs) offer automation and scalability, but may also lack precise domain knowledge. To explore these trade-offs, we compare the effectiveness and reliability of crowdsourced and LLM-based methods for assessing the quality of 30 MCQs and SAQs across six educational domains using two standardized evaluation rubrics. We analyzed the performance of 84 crowdworkers from Amazon's Mechanical Turk and Prolific, comparing their quality evaluations to those made by the three LLMs: GPT-4, Gemini 1.5 Pro, and Claude 3 Opus. We found that crowdworkers on Prolific consistently delivered the highest-quality assessments, and GPT-4 emerged as the most effective LLM for this task. Our study reveals that while traditional crowdsourced methods often yield more accurate assessments, LLMs can match this accuracy in specific evaluative criteria. These results provide evidence for a hybrid approach to educational content evaluation, integrating the scalability of AI with the nuanced judgment of humans. We offer feasibility considerations in using AI to supplement human judgment in educational assessment.

Addressing student non-compliance in AI use declarations: implications for academic integrity and assessment in higher education

This study examines the factors driving student non-compliance with AI use declarations in academic assessments at King’s Business School, where 74% of students failed to declare AI usage despite declaration being a requirement of a mandatory coursework coversheet. Utilising the Theory of Planned Behaviour (TPB) as a framework, the research combines service evaluation survey data and semi-structured interviews to explore how attitudes, subjective norms, and perceived behavioural control influence compliance. Findings reveal that fear of academic repercussions, ambiguous guidelines, inconsistent enforcement, and peer influence are key barriers to AI use declaration. These factors complicate the declaration process, undermine transparency, and challenge academic integrity. The study extends the TPB model by highlighting the ethical and practical dilemmas posed by generative AI, which blur traditional norms of academic integrity. This research offers critical insights for policymakers, suggesting that clear, consistent, and trust-based policies are crucial in fostering ethical AI use. The findings underscore the importance of transparent communication and supportive institutional cultures to improve compliance. Ultimately, this study informs policy development by evaluating the effectiveness of declaration mechanisms and providing actionable recommendations to promote a culture of academic integrity in the evolving landscape of AI technologies.

Comparing Scoring Consistency of Large Language Models with Faculty for Formative Assessments in Medical Education.

The Liaison Committee on Medical Education requires that medical students receive individualized feedback on their self-directed learning skills. Pre-clinical students are asked to complete multiple spaced critical appraisal assignments. However, the individual feedback requires significant faculty time. As large language models (LLMs) can score and generate feedback, we explored their use in grading formative assessments through validity and feasibility lenses. To explore the consistency and feasibility of using an LLM to assess and provide feedback for formative assessments in undergraduate medical education. This was a cross-sectional study of pre-clinical students' critical appraisal assignments at University of Illinois College of Medicine (UICOM) during the 2022-2023 academic year. An initial sample of ten assignments was used to develop a prompt. For each student entry, the de-identified assignment and prompt were provided to ChatGPT 3.5, and its scoring was compared to the existing faculty grade. Differences in scoring of individual items between ChatGPT and faculty were assessed. Scoring consistency using inter-rater reliability (IRR) was calculated as percent exact agreement. Chi-squared test was used to determine if there were significant differences in scores. Psychometric characteristics including internal-consistency reliability, area under precision-recall curve (AUCPR), and cost were studied. In this cross-sectional study, 111 pre-clinical students' faculty graded assignments were compared with that of ChatGPT and the scoring of individual items was comparable. The overall agreement between ChatGPT and faculty was 67% (OR = 2.53, P < 0.001); mean AUCPR was 0.69 (range 0.61-0.76). Internal-consistency reliability of ChatGPT was 0.64 and its use resulted in a fivefold reduction in faculty time, and potential savings of 150 faculty hours. This study of psychometric characteristics of ChatGPT demonstrates the potential role for LLMs to assist faculty in assessing and providing feedback for formative assignments.

Automatic generation of physics items with Large Language Models (LLMs)

High-quality items are essential for producing reliable and valid assessments, offering valuable insights for decision-making processes. As the demand for items with strong psychometric properties increases for both summative and formative assessments, automatic item generation (AIG) has gained prominence. Research highlights the potential of large language models (LLMs) in the AIG process, noting the positive impact of generative AI tools like ChatGPT on educational assessments, recognized for their ability to generate various item types across different languages and subjects. This study fills a research gap by exploring how AI-generated items in secondary/high school physics aligned with educational taxonomy. It utilizes Bloom's taxonomy, a well-known framework for designing and categorizing assessment items across various cognitive levels, from low to high. It focuses on a preliminary assessment of LLMs ability to generate physics items that match the Bloom’s taxonomy application level. Two leading LLMs, ChatGPT (GPT-4) and Gemini, were chosen for their strong performance in creating high-quality educational content. The research utilized various prompts to generate items at different cognitive levels based on Bloom's taxonomy. These items were assessed using multiple criteria: clarity, accuracy, absence of misleading content, appropriate complexity, correct language use, alignment with the intended level of Bloom's taxonomy, solvability, and assurance of a single correct answer. The findings indicated that both ChatGPT and Gemini were skilled at generating physics assessment items, though their effectiveness varied based on the prompting methods used. Instructional prompts, particularly, resulted in excellent outputs from both models, producing items that were clear, precise, and consistently aligned with the Application level of Bloom's taxonomy.

Evaluating Students' Academic Progress in the Role of Fuzzy Logic

This paper explores the role of fuzzy logic in the assessment of student performance in local high school, Badr, Libya, positing that it offers a more flexible and accurate alternative to traditional categorization methods of academic achievement. Conventional assessment tools, such as standardized test scores and GPAs, often yield a distorted and overly simplistic view of student performance. Utilizing a mixed-methods research design, this study compares the performance, attendance, and self-reported satisfaction of students assessed through fuzzy logic with those evaluated by traditional methods. The results demonstrate that students assessed using fuzzy logic achieved higher average scores and expressed greater satisfaction with the assessment process. Additionally, educators employing fuzzy logic provided more detailed and constructive feedback, enriching the learning experience. Thus, this research indicates that fuzzy logic can effectively mitigate the shortcomings of traditional assessment methods, enhancing the recognition of diverse student abilities. The findings advocate for the integration of fuzzy logic in educational assessment practices to promote inclusivity and improved learning outcomes.

Assessment of Hypospadias Patient Education Using Large Language Models

Assessment of Hypospadias Patient Education Using Large Language Models

Evaluating Use of Generative Artificial Intelligence in Clinical Pathology Practice: Opportunities and the Way Forward.

Generative artificial intelligence (GAI) technologies are likely to dramatically impact health care workflows in clinical pathology (CP). Applications in CP include education, data mining, decision support, result summaries, and patient trend assessments. To review use cases of GAI in CP, with a particular focus on large language models. Specific examples are provided for the applications of GAI in the subspecialties of clinical chemistry, microbiology, hematopathology, and molecular diagnostics. Additionally, the review addresses potential pitfalls of GAI paradigms. Current literature on GAI in health care was reviewed broadly. The use case scenarios for each CP subspecialty review common data sources generated in each subspecialty. The potential for utilization of CP data in the GAI context was subsequently assessed, focusing on issues such as future reporting paradigms, impact on quality metrics, and potential for translational research activities. GAI is a powerful tool with the potential to revolutionize health care for patients and practitioners alike. However, GAI must be implemented with much caution considering various shortcomings of the technology such as biases, hallucinations, practical challenges of implementing GAI in existing CP workflows, and end-user acceptance. Human-in-the-loop models of GAI implementation have the potential to revolutionize CP by delivering deeper, meaningful insights into patient outcomes both at an individual and population level.

Enhancing English Learning Through Virtual Environments and Formative Assessment: SDG Focus

Objectives: This study investigates the impact of virtual environments and formative assessments on English language learning among undergraduate students at a university in Lima, within the framework of Sustainable Development Goal (SDG) 4: Quality Education. The study aims to determine how these factors influence language skills, including listening, speaking, reading, and writing. Theoretical Framework: The research draws on the theories of connectivism, socio-constructivism, and the communicative approach to language teaching, emphasizing the role of digital tools and continuous feedback in enhancing language learning. Method: A quantitative, correlational, non-experimental design was employed, using structured questionnaires to collect data from 98 undergraduate students. Ordinal logistic regression was used to analyze the impact of the independent variables on English proficiency. Results and Discussion: The results indicate a significant influence of virtual environments and formative assessments on English language learning, with the Nagelkerke coefficient for the general model at 0.223. The findings highlight the importance of these educational strategies in improving language skills, particularly in reading and overall proficiency. Research Implications: The study underscores the need for integrating advanced educational technologies and formative assessments in language instruction, aligning with the broader goals of SDG 4. It provides practical insights for educators and policymakers seeking to enhance English language learning in a post-pandemic context. Originality/Value: This research contributes to the limited literature on the role of virtual environments and formative assessments in the Peruvian educational context, offering localized strategies for achieving quality education as outlined by SDG 4.

ANALYSIS OF SCHOOL VIOLENCE IN BRAZIL

School violence has emerged as an issue of great national relevance following the occurrence of tragic incidents involving children in different states of Brazil. These shocking events raised serious concerns about safety within educational institutions and highlighted the urgency of addressing the issue of school violence throughout the country. This article aims to analyze violence in high schools through the responses of school principals. The database is based on the results of the Basic Education Assessment System (SAEB) in the years 2019 and 2021, in the context of the Brazilian states. The data show a decrease in the types of violence reported between 2019 and 2021, with the exception of the Robbery type. For the year 2021, where new types of violence were introduced, mainly of a psychological nature, the data are worrying with regard to Bullying, Vandalism and the complete lack of perception of Sexual Harassment. By better understanding the dynamics of school violence, it will be possible to develop approaches to increase social capital, thus promoting a safer and more welcoming school environment and, consequently, an increase in human capital.

The constituents, ideas, and trends in team-based learning: a bibliometric analysis

Team-based learning (TBL) has gained significant popularity in higher education. Despite its widespread adoption, there is a lack of comprehensive understanding of the current state of TBL research. Hence, we conducted a bibliometric analysis to address this gap, reviewing 692 papers from the Web of Science database. The study revealed a notable increase in TBL research, with a peak of 87 publications in 2021. The United States emerged as the leading contributor, with medical and nursing education being the most prominent research areas. Specialized journals played a significant role in disseminating TBL research. Key themes, including TBL outcomes, non-technical competencies, and implementation principles, were identified. Moreover, it highlighted a growing interest in applying TBL in interprofessional education and formative assessment. The findings suggest the need to establish priorities and expand TBL research beyond health professions education to other fields. This study provides valuable insights into the current landscape of TBL research and offers directions for future investigations.

Impact of Multiple-choice and Essay-based Assessments on Academic Performance and Stress Levels among Medical Students: A Cross-sectional Study

Abstract Background: Balancing classroom instruction and examinations poses challenges in education, particularly in diverse student populations, necessitating educational reform and assessment strategies to mitigate stress. Medical education, like other fields, requires both knowledge acquisition and practical application, yet reliance on internet resources may exacerbate stress and hinder academic performance. Assessment tools in medical education vary based on the skills being evaluated, with written formats such as multiple-choice questions (MCQs) and essay-based questions (EBQs) used for cognitive assessments. This study compares MBBS students’ performance and stress levels in pharmacology examinations using MCQs and EBQs. Aim: To compare the performance and level of stress in the MBBS students after going through multiple choice questions and essay-based examination pattern. Methods: A cross-sectional study surveyed 100 third-semester MBBS students at AIIMS Rishikesh using a validated electronic questionnaire to assess various aspects of their experience with MCQs and EBQs in pharmacology assessments. The study collected data on demographics, stress levels, psychological effects, coping skills and student feedback, while excluding students who were not in the third semester, unwilling to participate or absent during the examination. Results: The study outcome showed no significant difference in scores between MCQs and EBQs. Stress factors during MCQ preparation included academic performance concerns and sleep issues, while course issues and sleep disturbances were primary stressors during EBQ preparation. Suggestions for examination pattern changes were common, indicating students’ desire for support. Conclusion: The study highlights the importance of addressing multifaceted stressors to support student well-being and academic success. Further research is needed to explore effective interventions.

Opportunities and challenges of using generative AI to personalize educational assessment.

In line with the positive effects of personalized learning, personalized assessments are expected to maximize learner motivation and engagement, allowing learners to show what they truly know and can do. Considering the advances in Generative Artificial Intelligence (GenAI), in this perspective article, we elaborate on the opportunities of integrating GenAI into personalized educational assessments to maximize learner engagement, performance, and access. We also draw attention to the challenges of integrating GenAI into personalized educational assessments regarding its potential risks to the assessment's core values of validity, reliability, and fairness. Finally, we discuss possible solutions and future directions.

University Teachers’ Views on the Adoption and Integration of Generative AI Tools for Student Assessment in Higher Education

This study examines the factors that may impact the adoption of generative artificial intelligence (Gen AI) tools for students’ assessment in tertiary education from the perspective of early-adopter instructors in the Middle East. It utilized a self-administered online survey and the Unified Theory of Acceptance and Use of Technology (UTAUT) model to collect data from 358 faculty members from different countries in the Middle East. The Smart PLS software 4 was used to analyze the data. The findings of this study revealed that educators developed new strategies to integrate Gen AI into assessment and used a systematic approach to develop assignments. Moreover, the study demonstrated the importance of developing institutional policies for the integration of Gen AI in education, as a driver factor influencing the use of Gen AI in assessments. Additionally, the research identified significant factors, namely performance expectancy, effort expectancy, social influences, and hedonic motivation, shaping educators’ behavioral intentions and actual use of Gen AI tools to assess students’ performance. The findings reveal both the potential advantages of Gen AI, namely enhanced student engagement and reduced instructor workloads, and challenges, including concerns over academic integrity and the possible negative impact on students’ writing and thinking skills. This study emphasizes the significance of targeted professional development and ethical criteria for the proper integration of Gen AI in educational assessment.

Hybrid Lexicon and Transformer-Based Sentiment Analysis of Student Feedback for Faculty Evaluation: A Speech-to-Text Approach

The conventional methods for assessing the performance of educators in academic settings have often been hindered by outdated feedback systems, leading to assessments that can be either skewed or in- complete. This study unveils an innovative, dual-approach methodology that fuses lexicon-driven sentiment analysis with cutting-edge Transformer architectures, specifically focusing on BERT (Bidirectional Encoder Representations from Transformers). This approach provides a more sophisticated, real-time assessment of educators based on student evaluations. One of the standout features of this study is the integration of Speech- to-Text technologies, which allows for the immediate transformation of verbal feedback into text that can be analyzed. The approach makes use of a specialized Educational Sentiment Lexicon for preliminary sentiment evaluation, which subsequently refines the performance of a pre-existing BERT model. This dual-model is proficient in scrutinizing both text-based and verbal feedback—the latter being converted into text through advanced Speech-to-Text techniques. Our results indicate that this approach significantly outperforms existing lexicon-based and machine learning methods in terms of accuracy and comprehensiveness. By providing a real-time, versatile sentiment analysis tool tailored for educational settings, this research marks a paradigm shift in the scope and quality of faculty evaluations, thereby contributing substantially to the field of educational technology and sentiment analysis.

Exploring digital literacy challenges: a comparative study of online inquiry skills among college students with reading difficulties

ABSTRACT Technology has revolutionised how we interact, work, and learn, with digital literacy offering lifelong learning opportunities. College students heavily rely on the Internet for education. As more students with developmental difficulties enter higher education, it is vital to understand how disparities impact classroom learning. This study assesses online inquiry and research skills in college students with reading difficulties compared to those with typical reading skills. We recruited 67 college students, including those with reading difficulties, in online research about the ketogenic diet, responding to open-ended questions. Findings reveal that students with reading difficulties could access basic information online like their peers but encountered challenges in identifying critical data, explaining concepts, describing processes, and constructing resource-based arguments. This study underscores the importance of educational assessments to enhance digital literacy among college students with reading difficulties, with implications for instruction and future research.

The Emergency Leadership Dashboard: A Comprehensive System for Visualizing Operations across Educational, Administrative, and Research Sectors

Introduction Managing patient care and fulfilling administrative tasks are challenges facing healthcare systems (1). Particularly at university hospitals, emergency departments (EDs) must carry out research and fulfill educational responsibilities. As a result of these responsibilities, educators can develop educator burnout (2). Providers must additionally maintain accreditation by completing continuous education and certification. Using new technology and automation, clinicians can focus on patient care and quality service, reducing administrative burdens. The goal of this study is to introduce an ED leadership dashboard that optimizes ED operational efficiency across the administrative, educational, and research areas. Methodology The digital dashboard was developed with Microsoft Teams, Forms, Excel, and Power BI. The system combines multiple datasets to create a comprehensive visual dashboard. The first database queries information about ED physicians and updates the related datasets, including ED research, administrative tasks, student evaluations, and performance metrics. The real-time updates and visualization on the dynamic Power BI dashboard facilitate the shift from the current paper-based data collection to an electronic workflow. Implementation includes utilizing standardized forms for data inputs about research projects, educational assessments, and administrative tasks, such as complaint management and credential monitoring. These inputs cause updates in the respective datasets and dashboard output in real time, which enhances the decision-making process by providing immediate access to performance metrics such as patient throughput. The dashboard ensures consistency, improves administrative efficiency, and supports a proactive approach to management and educational evaluations within the ED. Results The research dashboard monitors research activities, allows users to select by year and quarter, and displays interactive visualizations. Projects are categorized into "Original Research" and "Case Report," with Original Research being more common. The stages of research (Writing, Analysis, and Data Collection) are depicted in a donut chart, where Data Collection was found to be the most prevalent. The status of the projects is shown as "On Track," "Completed," or "Delayed," with most projects currently on track. An analysis of research types showed that Retrospective studies were the most frequent (Figure 1). The physician's involvement in each project is also tracked to distribute the project load among faculty. The training and certification dashboard shows completion rates for certifications such as Basic Life Support and Advanced Cardiovascular Life Support, with most completion rates near perfect. Safety training showed a low completion rate of 35%. 574 certifications were completed, and 10 were pending, for an overall 98% completion rate. (Figure 2) Figure 1: Research Dashboard Figure 2: Certification Dashboard Conclusion The implementation of an ED leadership dashboard in university hospital emergency departments has markedly enhanced operational efficiency across administrative, educational, and research functions. Utilizing technologies like Microsoft Teams, Forms, Excel, and Power BI, the dashboard has transitioned from paper-based systems to a streamlined electronic workflow, enabling real-time data visualization and updates. This has improved decision-making processes, optimally distributed faculty workloads, and maintained high certification completion rates, while also highlighting areas for improvement such as safety training. Overall, the dashboard has effectively managed patient care and administrative tasks, reducing burdens and supporting proactive management within the emergency department.

Enhancing Integration of Technology in Authentic Assessment for Education: A Structured Review

In the ever-evolving landscape of education, the seamless integration of technology into authentic assessment practices has become a transformative force, reshaping how students are both evaluated and engaged. Authentic assessment, characterized by its emphasis on the real-world application of knowledge and skills, has gained prominence for its ability to gauge students' comprehension and foster deeper learning experiences. As technology continues to infiltrate the educational field, a pressing need arises to assess the effectiveness of these digital tools in promoting genuine understanding and critical thinking among students. This structured review of the PRISMA approach was applied to find the primary data based on keywords such as Technology, Science, and Authentic Assessment. Based on advanced searching on SCOPUS and Web of Science, we found (n=55). Expert scholar decided to develop 3 themes, which are (1) Online Assessment and Technology- Enhanced Learning, (2) Educational Practices and Student Engagement, and (3) Assessment and Curriculum Development. The findings underscore technology's immense potential in elevating authentic assessments for students, offering new avenues for engagement and skill development. It is evident that when wielded thoughtfully and integrated into pedagogy, technology can empower young learners to explore, inquire, and excel in the fascinating world of education. This study not only bridges existing gaps in the literature but also serves as a valuable resource for educators and policymakers alike, providing insights into harnessing technology's power to enhance authentic assessment practices and, consequently, the overall educational experience.

Field-Testing Multiple-Choice Questions With AI Examinees: English Grammar Items.

Field-testing is an essential yet often resource-intensive step in the development of high-quality educational assessments. I introduce an innovative method for field-testing newly written exam items by substituting human examinees with artificially intelligent (AI) examinees. The proposed approach is demonstrated using 466 four-option multiple-choice English grammar questions. Pre-trained transformer language models are fine-tuned based on the 2-parameter logistic (2PL) item response model to respond like human test-takers. Each AI examinee is associated with a latent ability θ, and the item text is used to predict response selection probabilities for each of the four response options. For the best modeling approach identified, the overall correlation between the true and predicted 2PL correct response probabilities was .82 (bias = 0.00, root mean squared error = 0.18). The study results were promising, showing that item response data generated from AI can be used to calculate item proportion correct, item discrimination, conduct item calibration with anchors, distractor analysis, dimensionality analysis, and latent trait scoring. However, the proposed approach did not achieve the level of accuracy obtainable with human examinee response data. If further refined, potential resource savings in transitioning from human to AI field-testing could be enormous. AI could shorten the field-testing timeline, prevent examinees from seeing low-quality field-test items in real exams, shorten test lengths, eliminate test security, item exposure, and sample size concerns, reduce overall cost, and help expand the item bank. Example Python code from this study is available on Github: https://github.com/hotakamaeda/ai_field_testing1.

Teacher educators’ and pre-service teachers’ confidence toward the use of ICT in education

Todays’ teacher education needs to provide pre-service teachers with readiness to integrate ICT in education. Teacher educators are expected to serve as role models for pre-service teachers, providing them with examples and meaningful experiences of learning with ICT. The aim of this study is to provide an insight into teacher educators’ and pre-service teachers’ confidence toward using ICT in education. In this study, both groups assessed their Technological Pedagogical Knowledge (TPK). The aim was to study the possible differences between teacher educators’ and pre-service teachers’ TPK assessments, and based on their assessments, to provide insights into teacher educators’ readiness to act as role models for pre-service teachers. The participants of the study were 123 pre-service teachers and 83 teacher educators. The results indicate that the assessments of teacher educators were higher than pre-service teachers’ assessments at the sample level. The results also indicate that pre-service teachers and teacher educators can both be divided into two aligning sub-groups, with higher and lower assessments. However, the comparison of these sub-groups showed that the assessments of the pre-service teachers in the higher sub-group were higher than with the assessments of the teacher educators in the lower (modest) sub-group. This study provides evidence for variation among teacher educators and pre-service teachers in terms of their TPK. This implies a need for more tailored ways to develop TPK in teacher education.