Summative Assessment Research Articles

A new approach to continuous assessment: Moving from a stressful sum of grades to meaningful learning through self-reflection

This paper proposes a new approach of continuous assessment of a formative nature, aimed at achieving meaningful learning with lower stress levels. This approach minimizes the punitive nature of traditional summative continuous assessment and replaces typical graded activities with self-reflecting ones. To conduct the formative evaluation, a teaching model has been developed to guide the choice of feedback and feed-forward strategies to be implemented by the instructors, depending on the students’ levels of competence and confidence. To validate the effectiveness of this approach, a quasi-experimental design was employed, with the new continuous assessment method in the experimental group and the summative continuous assessment method in the control group. The experiment was conducted in a Mathematics course with undergraduate Economics and Business students. Data were collected through test scores and a perception questionnaire. The results indicate that the new evaluation method promotes meaningful learning and yields better academic results compared to the traditional continuous evaluation. Students expressed high satisfaction with the new approach, noting that it allows them to understand their formative status and address deficiencies before undertaking evaluable activities. They also believe this strategy helps reduce the stress associated with the summative evaluation itself.

A 360-degree evaluation of the professionalism and communication skills of technologists working in the radiology department of a public sector tertiary care hospital in Karachi, Pakistan

BackgroundRadiologic technologists (RTs) are at the forefront of radiology departments and require optimal skills to demonstrate professionalism and effective provider–patient communication. Hence, the objective of this study was to determine the reliability of 360-degree evaluation as a feedback tool to improve competencies in this crucial workforce.MethodsA planned 360-degree evaluation was conducted using a 10-item tool to evaluate communication skills and professionalism among radiological technologists working at Dow University Hospital, Karachi. In total, 311 evaluations of fifteen radiological technologists were conducted while performing various diagnostic radiology procedures at the radiology department of Dow University Hospital. The data were analysed using SPSS version 26 to compute the evaluation scores and the reliability of 360-degree evaluation.ResultsA total of 311 RT-patient interactions were recorded over 48 days, with scores from all three raters. The evaluation tool was found to have good internal consistency for patients, faculty, and RT, with Cronbach’s alpha values of 0.89, 087, and 0.74, respectively. The study found an intraclass correlation of 0.66 (95% CI; 0.58–0.72), showing moderate reliability of the 360-degree evaluation across different raters.ConclusionWe conclude that 360-degree evaluation is a valid and reliable tool for determining the professionalism and communication skills of radiologic technologists and should be incorporated into training programs for formative and summative assessments. However, large-scale multicenter studies are crucial for generalizing these findings and incorporating 360 evaluations in radiological technologist training programs for formative and summative assessments.

A Development of Online Problem-Based Learning Model to Promote Self-Regulated Learning among Undergraduates

This research aimed to develop and evaluate an online problem-based learning (PBL) model to enhance undergraduates' self-regulated learning (SRL). The Research and Development method was used in two phases: the first was to develop and validate the model. It began with a literature review to identify core features of effective PBL and SRL within an online environment, then set a concept framework and create the model manually. Seven experts were reviewed following the model, and it was piloted for further optimization. Second, the model will be implemented and evaluated with a sample of 52 students at Thailand National Sports University, Chumphon Campus. The main instruments used were the developed model, the model quality questionnaire, SRL assessments (pre- and post-), and after-action review forms. The results showed that (1) a developed model consisting of four sections: orientation of the model, the model of instruction, application, and student outcomes. Six core components: authentic problems, online collaborative learning, self-regulated learning, instructional scaffolding, online learning resources, and authentic assessments, which are organized through three main processes: preparation, implementation of SRL strategies, and summative assessment, and all activities using an iterative learning three stages: meeting the problem and planning, doing and checking, and presenting the result and reflecting. Experts agreed on the quality of this model, which was excellent. (M= 4.61, SD=0.52). (2) Average SRL score was significantly higher after learning with this model compared to prior us (p<0.01, large effect size; d=1.03). The findings of this study support the developed model that can improve the students' SRL effectiveness. Overall, the students agreed that an instructor plays a critical role in developing their SRL and problem-solving skills. The students demonstrated more self-assurance and were apt to use this method to learn other subjects.

Efficacy of a hybrid take home and in class summative assessment for the postsecondary physics classroom

Abstract Assigning course grades to students requires obtaining accurate measures of the students’ understanding and knowledge of the topic. The induced stress from a traditional summative assessment is known to negatively impact student grades, confounding the connection between knowledge and test grades. Documented approaches to reduce stress during examinations can lead to a distracting testing environment (two-stage tests) or are subject to cheating (take-home tests). We have developed a hybrid take-home/in-class exam that avoids such difficulties. We present herein the responses from student surveys conducted after each exam. The results of the surveys indicate that the hybrid exam method reduces test anxiety while improving students’ self-reported mastery of physics, as compared to traditional in-class summative assessments. These findings are significant at the p < 0.05 level. Lastly, we discuss the difficulties encountered in applying this test approach.

LLM-generated competence-based e-assessment items for higher education mathematics: methodology and evaluation

In this article, we explore the transformative impact of advanced, parameter-rich Large Language Models (LLMs) on the production of instructional materials in higher education, with a focus on the automated generation of both formative and summative assessments for learners in the field of mathematics. We introduce a novel LLM-driven process and application, called ItemForge, tailored specifically for the automatic generation of e-assessment items in mathematics. The approach is thoroughly aligned with the levels and hierarchy of cognitive learning objectives as developed by Anderson and Krathwohl, and takes specific mathematical concepts from the considered courses into consideration. The quality of the generated free-text items, along with their corresponding answers (sample solutions), as well as their appropriateness to the designated cognitive level and subject matter, were evaluated in a small-scale study. In this study, three mathematical experts reviewed a total of 240 generated items, providing a comprehensive analysis of their effectiveness and relevance. Our findings demonstrate that the tool is proficient in producing high-quality items that align with the chosen concepts and targeted cognitive levels, indicating its potential suitability for educational purposes. However, it was observed that the provided answers (sample solutions) occasionally exhibited inaccuracies or were not entirely complete, signalling a necessity for additional refinement of the tool's processes.

Automatic generation of physics items with Large Language Models (LLMs)

High-quality items are essential for producing reliable and valid assessments, offering valuable insights for decision-making processes. As the demand for items with strong psychometric properties increases for both summative and formative assessments, automatic item generation (AIG) has gained prominence. Research highlights the potential of large language models (LLMs) in the AIG process, noting the positive impact of generative AI tools like ChatGPT on educational assessments, recognized for their ability to generate various item types across different languages and subjects. This study fills a research gap by exploring how AI-generated items in secondary/high school physics aligned with educational taxonomy. It utilizes Bloom's taxonomy, a well-known framework for designing and categorizing assessment items across various cognitive levels, from low to high. It focuses on a preliminary assessment of LLMs ability to generate physics items that match the Bloom’s taxonomy application level. Two leading LLMs, ChatGPT (GPT-4) and Gemini, were chosen for their strong performance in creating high-quality educational content. The research utilized various prompts to generate items at different cognitive levels based on Bloom's taxonomy. These items were assessed using multiple criteria: clarity, accuracy, absence of misleading content, appropriate complexity, correct language use, alignment with the intended level of Bloom's taxonomy, solvability, and assurance of a single correct answer. The findings indicated that both ChatGPT and Gemini were skilled at generating physics assessment items, though their effectiveness varied based on the prompting methods used. Instructional prompts, particularly, resulted in excellent outputs from both models, producing items that were clear, precise, and consistently aligned with the Application level of Bloom's taxonomy.

Meningkatkan Hasil Belajar Pendidikan Agama Katolik dan Budi Pekerti Menggunakan Model Problem Based Learning Berbantuan Lembar Kerja Peserta Didik Materi Peran Kaum Awam dalam Gereja Katolik Kelas XI C SMA Stella Duce 2 Yogyakarta

This study aims to figure out the improvement of learning outcomes of Catholic Religious and Character Education of class XI SMA Stella Duce 2 Yogyakarta using the Problem Based Learning model assisted by Student Worksheets. The achievement of learning outcome indicators in the affective assessment of the Pancasila Student Profile, with the Critical Reasoning dimension and elements of obtaining and processing information and ideas in cycles I and II showed an increase where in cycle I the average value was 73%, increasing in cycle II by 84%. The implementation of the Problem Based Learning model assisted by Student Worksheets can improve learning outcomes cognitively which can be seen from the results of the final summative assessment of learning cycles I and II which showed an increase where in cycle I the average value was 74%, increasing in cycle II by 82%. The results of the study indicate that the implementation of the Problem Based Learning model assisted by Student Worksheets supports students to learn critically and systematically to improve learning outcomes.

Peningkatan Hasil Belajar Peserta Didik Kelas X SMA Stella Duce 1 Yogyakarta pada Materi Suara Hati Melalui Model Pembelajaran Problem Based Learning

This study aims to improve the learning outcomes of Catholic religious and ethics subjects education on the material of conscience through the learning model of Problem Based Learning. This class action research involves 36 students of class X-4 SMA Stella Duce 1 Yogyakarta in the 2023/2024 academic year. The action was carried out in two learning cycles. Data were collected with assessments in each cycle, and observation guidelines, and analyzed descriptively. The results of the class action research showed that in cycle I the average value of summative assessment results 1 of 36 students was 74.3 with a percentage of value categories of 11.1% feasible, 83.3% advanced and 5.6% proficient. And in cycle II the average value of summative assessment results 2 of 36 students was 91.2 with a percentage of value categories of 13.9% advanced and 86.1% proficient. Thus, applying the Problem-Based Learning model can improve students' learning outcomes.

ASSESSMENT OF ENGLISH SPEAKING AS A PEDAGOGICAL PROCESS

The main purpose of this research is to study and evaluate the assessment of English speaking skills as a basic pedagogical process of language education. The research uses a mixed method approach using qualitative and quantitative research methods. A literature review, classroom observations, and interviews with language teachers were conducted to gather insights into current assessment practices. The study showed that formative and summative assessment are important for proper assessment of students' English speaking skills. However, challenges such as subjectivity, time constraints, and environmental factors may affect the results. Assessment of English speaking skills is an important but complex pedagogical process that requires careful consideration of both methodology and criteria. The study highlights the importance of adopting diverse and flexible assessment strategies to ensure fairness and accuracy in assessment.

Script Concordance Test in Physiology: Preparation, Scoring, and Student Perceptions – A Mixed Method Study

Background: Although the curriculum has changed, assessment tools are not in alignment with the new types of teaching such as early clinical exposure (ECE) and self-directed learning. Both in summative and formative assessment most commonly used tools for assessment of cognitive domain are written formats including MCQ. However, these assessment tools such as MCQ and written essays cannot assess the higher order thinking skills and clinical reasoning skills. Materials and Methods: The present study was conducted in the department of physiology, in collaboration with medical simulation center, as a part of formative assessment for topics on pathophysiology of shock and obstructive and restrictive lung disorders taught during ECE using case scenarios. Two script concordance tests (SCTs) each on the topic of pathophysiological mechanism of different types of shock and obstructive and restrictive lung disorders each were prepared and administered to the students. Student perception to this assessment tools was obtained. Results: The Cronbach’s alpha of the 6-item SCT for respiratory and cardiovascular topic administered to the students (n = 107, 98) was 0.83 and 0.82, respectively. The inter-item correlation was respiratory and cardiovascular topic for 0.71, 0.69 suggesting that they are close and repetitive. Students were of the opinion that SCT are difficult to understand (75.8%), challenging to answer (47.2%), and it tested the clinical content better (71.4%). Conclusions: In our study, students felt though SCT tested the clinical content better when compared to conventionally used MCQ, they still find it challenging to understand and are not in favor of its use in summative and formative assessments.

Assessing Chest Tube Insertion Skills Using a Porcine Rib Model-A Validity Study.

Assessments require sufficient validity evidence before their use. The Assessment for Competence in Chest Tube Insertion (ACTION) tool evaluates proficiency in chest tube insertion (CTI), combining a rating scale and an error checklist. The aim of this study was to collect validity evidence for the ACTION tool on a porcine rib model according to the Messick framework. A rib model, consisting of a porcine hemithorax that was placed in a wooden frame, was used as simulator. Participants were recruited from the departments of surgery, pulmonology, and emergency medicine. After familiarization with the rib model and the equipment, standardized instructions and clinical context were provided. They performed 2 CTIs while being scored with the ACTION tool. All performances were assessed live by 1 rater and by 3 blinded raters using video recordings. Generalizability-analysis was performed and mean scores and errors of both groups on the first performance were compared. A pass/fail score was established using the contrasting groups' method. Nine novice and 8 experienced participants completed the study. Generalizability coefficients where high for the rating scale (0.92) and the error checklist (0.87). In the first CTI, novices scored lower than the experienced group (38.1/68 vs. 47.1/68, P = 0.042), but no difference was observed on the error checklist. A pass/fail score of 44/68 was established. A solid validity argument for the ACTION tool's rating scale on a porcine rib model is presented, allowing formative and summative assessment of procedural skills during training before patient contact.

초등학교 학생평가 정책에 대한 역사적 신제도주의 분석: 1999년 ~ 2024년을 중심으로

This study analyzed the evolution of elementary school student assessment policies from 1999, when performance assessment was introduced, through June 2024. It focused on structural, institutional, and actor contexts, dividing the analysis into three phases: the introduction of performance assessment, the abolition of summative assessment, and the development of assessment practices. Initially, both “path creation” and “path dependence” were observed, as performance assessments were introduced while traditional assessments persisted. In the abolition phase, process-oriented assessments marked a “fork in the path.” Finally, the development phase saw performance assessments becoming established, with new adaptive and customized assessments emerging, driven by edu tech companies. Although these changes haven’t led to institutional reforms yet, shifts in assessment paradigms were evident. The study recommends future policies consider the structural context and incorporate feedback from educators.

Cómo entrenar tu dragon: developing a module to build up AI literacy of HE lecturers

This case study summarised the findings from the first academic presentation of the module ‘Cómo entrenar tu dragon (How to train your dragon)’, which is part of the development and training programme offered by the Universidad de Las Palmas de Gran Canaria to new and experienced teaching staff. The module aims to equip participants with a theoretical understanding and practical skills to use Generative AI (GenAI) multimodal languages to enhance learning and to develop a critical awareness of the umbrella term ‘AI literacy’. The module covers topics such as GenAI and data protection, copyright issues, regulatory frameworks, getting familiar with GenAI large language models (LLM) and the impact of GenAI on assessment. The case study provides information on the type of learning activities undertaken by participants and the final summative assessment. Participants found the module insightful, giving it a total score of 4.813 out of 5 on a Likert scale. Participant feedback suggests that institutional guidance and policies are needed to introduce GenAI tools at the Universidad de Las Pal mas de Gran Canaria so they can use GenAI tools confidently with their students.

WASHBACK EFFECTS OF ONLINE ASSESSING STUDENTS’ READING COMPREHENSION IN NARRATIVE TEXT

The Covid-19 pandemic has altered the educational landscape. This is due to the fact that traditional classroom learning has been replaced with online learning. Even if the learning and assessment take place at home, teachers must be able to offer an effective learning process and assessment. The impact of testing on teaching and learning is known as washback. Teachers appear to play a significant part in developing different types of washback by examining the probable positive and negative washback that tests may cause at the micro and macro levels. In other words, the teachers' beliefs play an important role in influencing the washback effect. This study aims at explaining washback effect of online students’ assessment on reading comprehension in narrative text. Secondly, to describe students’ engagement in following online learning and assessing of reading comprehension. It employed qualitative method and descriptive research design. Data collection techniques used observation, online reading test, interview, and documentation. Meanwhile, data analysis techniques implemented data reduction, data display, data verification. The research participants were 32 the seventh graders of an Islamic secondary school in Semarang regency. The results show that narrative reading text used folklore digital storytelling namely The Crying Stone. The reading comprehension test consisted of forty items, they were short answer, close test, true or false, vocabulary development, matching, and references. Based on the data, there are two types of washback effects namely negative washback effect and positive washback effect. Positive and negative washback effect covered formative assessment and summative assessment. They are included assessment as learning, assessment for learning, and assessment of learning. Positive washback effects made students have eagerness to be autonomous learners. Students had high motivation and well-prepared in joining reading class. Meanwhile students had difficulties in learning and follow the reading assessment because of bad internet connection as negative washback. Sometimes some parents were not able to buy internet data package.

Improving student outcomes using automated feedback in a first-year economics class

The assessment literature advocates utilising formative assessment strategies to support and enhance student learning. This study uses online practice tests with inbuilt feedback, that students could repeatedly use, as a formative feedback strategy in a first-year introductory microeconomics module. The key question explored is whether repeated engagement with online practice tests had a statistically significant impact on students’ subsequent performance on a summative assessment. Quantitative engagement and performance data (N = 223) were collected from a cohort of first-year students. An analysis of this data reveals that engaging with the online practice test multiple times has a statistically significant impact on final performance. Controlling for the mid-term grade, on average, each cycle of ‘seeking and acting’ increases grades in the final exam by 1.3 % (p = 0.003). Hence, the data suggests that incorporating multiple opportunities to take an online practice test, which includes feedback, is an important feature of the strategy. A smaller sample of qualitative survey data (N = 131) reveals that these economics students hold traditional views of feedback with little awareness of their own role in seeking and acting on feedback. We argue that formative practice tests have the potential to support the development of student feedback literacy.

Authentic Assessment of English Language Learners Used in An Indonesia High School

The purpose of this study is to describe (1) the type of assessment used by teachers in assessing student in learning english and (2) the authentic assessment used by teachers in assessing students in learning english. The research method used is qualitative with a descriptive type. The sample technique used is purposive sampling. Two English language teachers in grades ten and eleven at SMA Muhammadiyah Special Program Kottabarat Surakarta are the subjects of this study. Interviews, documentation, and observation are the research methods used. The data analysis techniques used are data collection, data reduction, data presentation and then conclusion drawn. The researcher found that there are 4 types of assessments used by the two teachers, namely 1) formative assessment, namely in the form of assignments and try in tests, 2) summative assessment, namely in the form of midterm exams and final semester exams, 3) formal assessments, 4) informal assessments. The authentic assessment used by both teachers in English learning is an essay or portfolio.

Ability in physical education – No, I can’t discuss it! Insights from lower secondary physical education teachers in Western Australia

Ability has been positioned as a crucial concept in the construction of a subject that is inclusive of and meaningful for all students. Almost two decades since Evans ( 2004 ) called on the field to re-engage in discourse concerning ability, how ability is conceptualised by physical education (PE) teachers remains relatively unexplored. Furthermore, relatively little is known about how understandings of ability are enacted through curriculum, pedagogy and assessment in PE. This research explored these issues with lower secondary PE teachers in Western Australia through self-recorded audio responses to two research questions. Data from 38 participants revealed the struggles that PE teachers had in articulating meanings and representations of ability in PE in their curriculum, pedagogy and assessment practices. Findings illustrated a tendency for many teachers to privilege a performative view of ability and focus on the physical domain in discussing ability in PE. While some teachers made reference to skills and dispositions beyond this domain, data affirmed the dominance of narrow conceptualisations of ability in PE and pointed to limited understandings of how broad, fluid and inclusive notions of ability may be embedded in curriculum design, pedagogy and assessment practices. Summative assessment was the most frequent aspect of practice referred to in teachers’ explanations of how understanding of ability was expressed in their practice. The study points to the importance of professional learning and initial teacher education directing attention to ability as a focus for advancing equitable practices in PE and the need for further research exploring enactments of ability.

Librarian and Clinical Faculty Collaborative Use of Formative Assessment in a Longitudinal Evidence-Based Medicine Curriculum

Background The authors developed a longitudinal curriculum for teaching third- and fourth-year undergraduate medical students evidence-based medicine (EBM). This curriculum involved substantial librarian involvement, the use of formative assessment as a teaching method, and progressive repetition of skills. Methods Students in three clinical clerkships (Medicine, Pediatrics, OB/GYN) completed EBM assignments based on real-world scenarios in this quasi-experimental study. Each clerkship required students to submit an EBM plan, from which they received directed feedback from both the clerkship director and a librarian after submitting preliminary EBM plans. This study tracked student PICO and search submissions to determine if repeated exposure to EBM feedback resulted in improved summative assessments, both in isolation and longitudinally. Results Students’ PICO and searching performance improved between mid-clerkship formative feedback and end-clerkship summative assessment in all three clerkships using rubrics developed by the authors for each clerkship derived from existing literature. When examining student performance sequentially over three clerkships, there was significant improvement between the first and second clerkships, but this did not carry into the third clerkship. Discussion Our findings suggest that the significant inclusion of a librarian and feedback appears to have positive effects on student performance. While it may seem obvious that feedback results in improved outcomes, this method doesn’t appear widespread in medical education. Repetition, while not having a lasting increase in performance, may still be warranted to increase exposure to authentic cases and evidence types.

The spatiality of geography assessment questions: a case study of selected South African universities

ABSTRACT Spatial thinking is an essential skill for geographers and can be taught and learnt at any age. Spatial thinking should, therefore, form part of the Geography curriculum at every study level. An initial desktop study by the authors has indicated that the inclusion of spatial thinking in the undergraduate Geography curriculum at South African universities needs improvement. Up to now, published research papers have focused mostly on questions on spatiality included in Geography textbooks at the school and tertiary levels. None of these papers focused on the spatiality of assessment questions. This research aims to assess the spatiality of questions posed in formative and summative assessments of undergraduate geography modules at six selected South African universities. The taxonomy of spatial thinking was used to determine the spatiality of 2864 formative and summative assessment questions used at the first, second and third-year levels at six selected South African universities. The results indicate that the assessment questions, generally, do not reach a desirable level of spatiality and will, therefore, contribute towards developing students’ spatial thinking skills only in a limited way. Based on the results, recommendations are supplied to improve the spatiality of undergraduate Geography assessment questions using the taxonomy of spatial thinking.

Summative Assessments in Psychiatry Postgraduation in India: A Teacher's Perspective on Current Situation and the Way Forward.

Summative Assessments in Psychiatry Postgraduation in India: A Teacher's Perspective on Current Situation and the Way Forward.