Summative Assessment Research Articles

Examining the Impact of Experiential Learning Activities on Student Performance in an Introductory Animal Science Course

The use of experiential learning in agricultural courses within postsecondary institutions has become increasingly important as educators seek to provide students with a meaningful education that increases knowledge acquisition and achievement. Hands-on, experiential learning activities within agricultural sciences, and more directly animal sciences, have previously been used to provide students with experiences that increase performance in courses and overall development throughout a student’s education. Therefore, the purpose of this study was to examine the influence that experiential learning laboratory lessons had on students enrolled in an introduction to animal science lecture course at [University]. Researchers utilized a quasi-experimental study, in which students were placed into one of three groups, depending on the experiential learning lessons that were taught throughout the semester. This was done to determine if students who participated in specific hands-on, experiential learning lessons retained more content knowledge than those who did not receive these lessons, on summative assessments given throughout the semester. Researchers utilized the first quiz scores to examine prior knowledge between groups, and determined that there was no difference. While researchers were unable to determine significance between groups and student content acquisition due to unequal group sizes, researchers determined that students who received experiential learning lessons performed better on average, than those who did not. Researchers identified recommendations for future studies, which include replicating the study with modifications, and repeating the study with two introductory courses simultaneously, in which one course entirely utilizes experiential learning lessons and the other course utilizes traditional review sessions. Additionally, researchers recommended practitioners integrate blending experiential learning opportunities and traditional lecture and review when teaching introductory courses.

Culturally responsive assessment in compulsory schooling in Denmark and Iceland - An illusion or a reality? A comparative study of student teachers’ experiences and perspectives

This paper explores and compares student teachers’ experiences and perspectives on culturally responsive assessment in compulsory schooling in Denmark and Iceland. The study’s theoretical framework draws on scholarship on culturally responsive assessment. The data was derived from fourteen interviews with Danish student teachers and ten interviews with Icelandic student teachers. As per the selection criteria, all participants had to be in the final stages of their education process, meaning they had completed their on-site schoolteacher training and most courses in their teacher education programme. During the interviews, the participants reflected on pre-instructional, formative, and summative assessment practices with cultural diversity in mind. Furthermore, they discussed the importance of assessing children’s well-being. In both countries, the participants demonstrated positive attitudes towards cultural diversity and expressed awareness of considering children’s cultural and linguistic backgrounds when assessing them. On the other hand, most participants expressed criticism of teacher education programmes for lacking attention to the topic of culturally responsive assessment, which resulted in their rudimentary understanding of the theoretical underpinnings.

A successful intervention to improve conceptual knowledge of medical students who rely on memorization of disclosed items.

Background: The mere memorization of isolated facts without the claim of integrating them is detrimental to the desired learning outcomes in medical education. The current study was conducted within an environment where items from summative assessments are regularly disclosed by the university and consequently collected into an item bank that is shared between students. Within this environment, we aimed to quantify 1) to which extent students use disclosed items for their preparation for the summative exam, 2) a putative mismatch between (isolated factual) knowledge regarding distinct questions from the item bank and conceptual knowledge, and 3) to which extent this mismatch can be ameliorated by a project aiming to steer student learning away from the memorization of isolated facts toward the acquisition of conceptual knowledge. Methods: This steering project in the midst of the first semester consisted of the implementation of an oral exam based on selected learning objectives, preceded by two seminars. After their summative exam at the end of semester, 135 students performed a voluntary exam for study purposes. Here, authentic (i.e., presumably preknown) physiology questions taken from the item bank were used to assess students' ability to 1) recognize the correct answer in a multiple choice (MC) question, 2) recall the answer (short answer), or 3) display conceptual knowledge closely corresponding to the question presented in the other formats. Additionally, students received a questionnaire addressing their learning habits and attitudes. Results: The median reported percentage of learning time for the summative exam exclusively spent with this item bank was 80%. The results of the voluntary exam indicate that students frequently recognize and recall correct answers of included items without displaying knowledge of the underlying concept. Compared to recall of the correct answer, the probability of giving a correct answer regarding the corresponding basal physiologic concept was lower by 47 percentage points (p <0.001) for topics not included in the steering project. Regarding topics included in the steering project, this discrepancy was reduced to 25.5% (p <0.001). Conclusion: The results of this study demonstrate the influence of disclosed items on student learning and learning outcomes and suggest that a carefully implemented assessment is able to improve conceptual knowledge in physiology.

Practice to Perform: Formative Mini-Skills Checkoffs Prior to High-Stakes Summative Assessment for Practice-Ready Students.

In previous years, students expressed their concerns regarding the high-stress nature of the summative skills checkoff, and pass rates were not as high as expected. To evaluate the impact of mini-skills checkoff assessments and the use of standardized patients (SPs) on summative skills pass rates and nursing students' perceptions of this teaching approach. A mixed-methods approach was used with a convenience sample of 46 sophomore nursing students. There was a significant difference in the summative skills checkoff pass rates after including mini-skills checkoffs and SPs ( P < .01). Four dominant themes emerged: emotional impact, preparation and learning, confidence and competence, and readiness. Evidence from this pilot study suggests the inclusion of continuous, low-stakes assessments over time, with learner feedback bridges the gap between clinical practice, simulation, and classroom learning.

Medical specialty certification exams studied according to the Ottawa Quality Criteria: a systematic review

BackgroundMedical specialty certification exams are high-stakes summative assessments used to determine which doctors have the necessary skills, knowledge, and attitudes to treat patients independently. Such exams are crucial for patient safety, candidates’ career progression and accountability to the public, yet vary significantly among medical specialties and countries. It is therefore of paramount importance that the quality of specialty certification exams is studied in the scientific literature.MethodsIn this systematic literature review we used the PICOS framework and searched for papers concerning medical specialty certification exams published in English between 2000 and 2020 in seven databases using a diverse set of search term variations. Papers were screened by two researchers independently and scored regarding their methodological quality and relevance to this review. Finally, they were categorized by country, medical specialty and the following seven Ottawa Criteria of good assessment: validity, reliability, equivalence, feasibility, acceptability, catalytic and educational effect.ResultsAfter removal of duplicates, 2852 papers were screened for inclusion, of which 66 met all relevant criteria. Over 43 different exams and more than 28 different specialties from 18 jurisdictions were studied. Around 77% of all eligible papers were based in English-speaking countries, with 55% of publications centered on just the UK and USA. General Practice was the most frequently studied specialty among certification exams with the UK General Practice exam having been particularly broadly analyzed. Papers received an average of 4.2/6 points on the quality score. Eligible studies analyzed 2.1/7 Ottawa Criteria on average, with the most frequently studied criteria being reliability, validity, and acceptability.ConclusionsThe present systematic review shows a growing number of studies analyzing medical specialty certification exams over time, encompassing a wider range of medical specialties, countries, and Ottawa Criteria. Due to their reliance on multiple assessment methods and data-points, aspects of programmatic assessment suggest a promising way forward in the development of medical specialty certification exams which fulfill all seven Ottawa Criteria. Further research is needed to confirm these results, particularly analyses of examinations held outside the Anglosphere as well as studies analyzing entire certification exams or comparing multiple examination methods.

Process evaluation of paediatric fellowship training programs at a University Hospital in Pakistan

BackgroundFellowship programs offer training in a subspecialty focusing on distinct and advanced clinical/academic skills. This advanced postgraduate training allows physicians, who desire a more specialized practice, to further develop clinical, academic, research, and leadership/administrative skills. The Aga Khan University (AKU) is one of the few institutes offering paediatric sub-specialty training in Pakistan. We aimed to evaluate the current Paediatric fellowship programs at AKU.MethodsProcess evaluation of six paediatric fellowship programs (cardiology, neurology, endocrinology, critical care medicine, neonatology, and infectious disease) was conducted from September 2020 to April 2021 by senior clinicians and medical educationists. Evidence was collected through document review (using existing postgraduate medical education program information form), observation of teaching and learning support, and focused group discussions/interviews with program faculty and fellows were conducted. A review of the evaluation report was done as part of this study. This study received an exemption from the ethical review committee. The quantitative data were analyzed using SPSS (22.0) while the reports of discussion with fellows and friends underwent content analysis.ResultsAll fellowship programs met the criteria for having a robust competency-based fellowship curriculum as per the institutional and national guidelines. Formative assessment in the form of continuous evaluation was found to be integrated into all the fellowship programs, however, most of the programs were found to lack a summative assessment plan. Fellows in training and program faculty were satisfied with the opportunities for mentorship, teaching, and learning. Thematic analysis of the discussion reports with faculty and fellows revealed three key themes including, program aspects translating into strengthening the training, gaps in the training program in delivering the expectations, and making ways to reach par excellence.ConclusionsThe process evaluation of paediatric fellowship programs provided an opportunity to holistically review the current strengths and quality of the training in individual programs along with the unmet needs of the trainees. This will help the program stakeholders to prioritize, align and allocate the resources to further enhance the quality of training and outcome of individual fellowship programs to ensure wider impacts at a regional, national, and international health system level.

Development of a formative assessment instrument to determine students’ need for corrective actions in physics: Identifying students’ functional level of understanding

In physics education, most teachers provide students feedback on their problem solutions through grades on written tests. The practice of feedback after a summative test does not often meet the needs of many students to improve their problem solving. In this paper we report on the development of a formative assessment instrument to allow teachers to provide more meaningful action-oriented feedback on students’ performance on written tests.Our research and development approach comprised three phases.The first phase consisted of a literature guided cognitive analysis of effective problem-solving strategies in the physics domain. This analysis resulted in the identification of three crucial episodes in students’ problem-solving approaches during which students engage in specific cognitive activities. The second phase consisted of the design of an assessment instrument to monitor specific cognitive activities during the three crucial episodes when solving physics problems. This resulted in a rating scale with 11 levels to indicate students’ efficacy. The third phase consisted of research of the validity, reliability, and practicality of the instrument. Here we trained three teachers to trace students’ mistakes on different problems in the domain of kinematics and asked them to rate the mastery of 16 eleventh-grade pre-university students. In this phase we assessed the reliability and validity of this instrument by computing Krippendorff's alpha to indicate teachers’ inter-rater reliability. Practicality of the instrument was assessed by examining the variation in students’ level of mastery on problems of different complexity. Further research is needed to provide more detailed guidelines for how teachers can use the instrument in formative assessment (in contrast to summative assessment) to help students to develop correct solution methods and foster students’ metacognition about problem solving in related physics areas (i.e., knowledge transfer).

Effects of Transition from Closed-Book to Open-Book Assessment on Students' Scores in a Pharmacokinetics Course.

Closed-book summative assessment of student learning, common in pharmacy education, is challenging to administer in a remote setting due to the need for costly and intrusive monitoring technology. Therefore, open-book assessments without monitoring have been considered an alternative in remote settings. The present study investigated the effects of the transition from in-person closed-book to remote open-book format on the students' scores in different assessment categories in a Pharmacokinetics course. The students' performances in the transition cohort (Transition, n = 96) during the in-person and remote periods were compared with those of an in-person cohort (Control, n = 85) during the same periods. Assessments included take-home assignments, daily quizzes, and progress/final examinations. Whereas the take-home assignments were open-book for cohorts and periods, the quizzes and examinations were open-book only for the Transition cohort during the remote period. Only the quiz/examination questions that were identical for both cohorts were included in the analysis. Statistical analysis by a linear, mixed-effects model indicated that the transition did not have any significant impact on the scores of students in the assignments, which were open-book for both cohorts and both periods. However, there were significant increases in the Transition cohort's scores (mean ± SE) during the remote open-book period in both quizzes (+8.4 ± 1.9%) and examination (+6.8 ± 1.5%) questions, compared with the Control cohort who had in-person closed-book assessments. These differences amounted to Cohen's d-effect sizes of 0.61 and 0.59 for the quiz and examination questions, respectively. It is concluded that when the questions are similar, the students' scores in pharmacokinetic assessments are higher (medium effect size) in a remote open-book format compared with the in-person closed-book format.

Quality indicators of effective teacher-created summative assessment

PurposeThe current literature on school teacher-created summative assessment lacks a clear consensus regarding its definition and key principles. The purpose of this research was therefore to arrive at a cohesive understanding of what constitutes effective summative assessment.Design/methodology/approachConducting a systematic literature review of 95 studies, this research adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. The objective was to identify the core principles governing effective teacher-created summative assessments.FindingsThe study identified five key principles defining effective summative assessment creation: validity, reliability, fairness, authenticity and flexibility.Research limitations/implicationsThe expansiveness of education research is such that not all relevant studies may have been identified, particularly outside of mainstream databases. This study considered only the school environment, so contextual limitations will exist.Originality/valueTo the best of the authors’ knowledge, this study contributes original insights by proposing a holistic definition that can facilitate consensus-building in further research. The assimilation of core principles guided the development of quality indicators beneficial for teacher practice. The comprehensive definition, key principles and quality indicators offer a unique perspective on summative assessment discourse.

Impacting Career Choices of Historically Underserved Secondary Students by Designing Near-Peer Directed Acid-Base Thematic Laboratory Activities to Enhance STEM Interest.

The current study describes preliminary findings from the Xavier University of Louisiana Mobile Outreach for Laboratory Enrichment (XULA-MOLE) project, which is a collaboration between Xavier University of Louisiana (XULA), a Historically Black and Catholic University, and participating 9th-12th grade classrooms in the central New Orleans area with a historically underserved student population. The project described here is geared toward providing laboratory enrichment to enhance student learning and impact student career interest in STEM fields, especially in classrooms with a much-needed "hands-on" laboratory experience which is unavailable due to a lack of resources. In this case study, we will present and discuss the inquiry-based laboratory modules for the topic area of acids and bases. These modules were created with careful thought and revision by XULA undergraduate STEM students. The experimental modules were based on the curriculum that participating teachers were discussing in the high-school classroom during the semester. The active-learning efforts were carried out during 6 weeks of the semester to provide a sustained and impactful resource for the participating classrooms. Since both groups of students (XULA-MOLE students and the high school students) were from underrepresented groups there was a strong sense of shared interest and dynamic near-peer mentorship. The project outcomes were measured using both formative and summative assessments indicative of preliminary successes in impacting career interests and increasing the content knowledge of participating high-school students.

Editorial: There's an AI for that

In recent months, the educators and higher education institutions have responded with concern, critique, and hope, to the rise of generative artificial intelligence (AI)’s unregulated and mounting influence. Following the period of emergency remote teaching, and the great ‘snapback’ (Jandrić et al., 2022), yet another new concern has emerged, promising to revolutionise education, or threaten its existence. The gravity of the situation has reverberated across the system, as wizardry of predictive pattern recognition fundamentally threatens the validity of long-held practices of summative assessments including essays, and online quizzes. This latest quandary/crisis shows no sign of abating, as venture capitalist funding and language modelling datasets grow. The technology becomes more deeply integrated into word processing, and cloud-based applications through which much of our academic labour is conducted. Understanding and conceptualising new technology within education has long been a necessity, as we wrestle with wrangling tools into our human interactions. Higher education’s relationship with edtech has always been characterised by a cyclical response to disruptive external influences of evolving technology, whose recent developments are often underpinned by neoliberal values of competition, efficiency, market-based solutions, and the privatisation of software platforms. Recent large language model developments are proving no different, with deregulation and the free market serving as the impetus to design and create such tools. Across higher education, educators scramble to decode the GenAI black box, deciphering hallucinations, confabulations, and smooth outputs indistinguishable from original student work. Policy responses range along a continuum of ban or embrace. New AI literacies are being woven into curricula, as change continues apace. 2023 marks a year of existential crisis precipitated by a global pandemic, followed by geopolitical events and a fatigue from the continual adaptation to a new normal. Even within, we are constantly shaping our educational systems. That pull is in many different directions – to accredit, to certify, to help learners become, to socialise, to emancipate, to measure - to meet very diverse purposes and aims. The politics and power structures inherent in our system further affect our response (Kuhn et al., 2023). While the potential of AI chatbots based on natural language processing models is undeniable, it is crucial to discern the reality from the hype and to better understand how our actions and responses are shaping our educational systems in this evolving domain. This editorial examines this dilemma further, to consider the impact on our scholarship of teaching and learning and how we as a community of researchers and educators respond.

Implementasi Projek Penguatan Profil Pelajar Pancasila (P5): Kearifan Lokal Di SMP Negeri 4 Kubung Kabupaten Solok

This study aims to analyze the implementation of the Pancasila Student Profile Strengthening Project (P5) with the theme of Local Wisdom. This type of research is qualitative with a phenomenological approach and naturalistic methods. The results showed that the implementation of P5 at SMP Negeri 4 Kubung consisted of several stages: first, designing P5 with details; form the P5 team through meetings, identify the readiness of the education unit, dimensions of P5, theme of P5 namely Local Wisdom, time allocation, and develop modules modified from existing modules, as well as a strategy for reporting the results of P5. Second, managing the project from starting P5 with introduction and contextualization, optimizing implementation with P5 action, and closing the project with celebration. Third, process the assessment by summarizing formative and summative assessments. Reporting with a P5 report card once a year in the even semester. Fourth, evaluation through reflection and meetings as well as follow-up in the form of continuing batik activities on the theme of local wisdom.

Formative assessments during COVID-19 pandemic: an observational study on performance and experiences of medical students.

Background: Because of COVID-19, the 2020 written medical examinations were replaced by mandatory formative online assessments. This study aimed to determine students' performance, self-assessment of performance, and perception about the switch from a summative to a formative approach. Methods: Medical students from year 2 to 5 (n=648) were included. They could repeat each test once or twice. They rated their performance after each attempt and were then given their score. Detailed feedback was given at the end of the session. An online survey determined medical students' perception about the reorganization of education. Two items concerned the switch from summative to formative assessments Results: Formative assessments involved 2385 examinees totaling 3197 attempts. Among examinees, 30.8% made at least 2 attempts. Scores increased significantly at the second attempt (median 9.4, IQR 10.8), and duration decreased (median -31.0, IQR 48.0). More than half of examinees (54.6%) underestimated their score, female students more often than male. Low performers overestimated, while high performers underestimated their scores. Students approved of the switch to formative assessments. Stress was lessened but motivation for learning decreased. Conclusions: Medical students' better scores at a second attempt support a benefit of detailed feedback, learning time and re-test opportunity on performance. Decreased learning motivation and a minority of students repeating the formative assessments point to the positive influence of summative assessment on learning.

AN ASSESSMENT-DIRECTED, STUDENT-CENTERED, AND MASTERY-BASED MODEL OF TEACHING AND LEARNING IN MATHEMATICS

This is a descriptive article illustrating a multi-year research study in mathematics education of college students in the U.S. It focuses on the first two years in the undergraduate mathematics curriculum. The study presents the theory, practice, and results of an innovative instructional program (Keystone model) applied to algebra classes at the college. The research investigated two areas of student learning outcomes: performance in mathematics and improvements in students’ concentration skills. Under the application of this program, students not only achieved significant outcomes in mathematics, as evidenced by formative and summative assessments but also improved their concentration skills, as demonstrated in the gains in national reading comprehension tests. In the following, we present, in extensive detail, the elements of the Keystone approach and its implementation in a learning environment, such as a college mathematics classroom.

Students’ Voices: A Qualitative Study on Contextual, Motivational, and Self-Regulatory Factors Underpinning Language Achievement

A lack of qualitative studies examining adolescent students’ voices regarding the contextual, relational, and self-regulatory factors that drive their language achievement has been observed. Therefore, the present study aimed to address this issue. Sixteen face-to-face semi-structured interviews were conducted with secondary school students in Greece to document and analyse their perceptions of the factors that influence language achievement. The study was guided by social-cognitive theory and models of self-regulated learning. Computer-assisted thematic analysis was performed using abductive open coding followed by refinement of codes. Three overarching global themes were identified in line with theoretical expectations, namely personal factors (motivation and self-regulatory strategies), relational factors (teachers’ practices, parental achievement expectations, peer influences), and structural factors (the educational system). Afterwards, an abductive thematic network analysis was conducted to explore and theorise about potential relationships between the emergent themes in the data. It was found that parental expectations for higher performance and the teachers’ behaviours were associated with students’ language achievement. Task-related characteristics and peers influenced students’ regulation of effort in learning. The students believed that they were not objectively graded and frequent summative assessments created test anxiety. The findings are discussed in light of existing empirical evidence and in terms of educational implications.

Validating an EFL Summative Test in Business English Major at a Chinese University

The research did an analysis on content validity, construct validity and concurrent validity of an summative assessment for an Advanced Business Course administered by a university in Guangdong University, China. With validity-based assessment, this study provided implications for educators and practitioners on improving quality of test papers.

ASPECTS OF TRANSLATION COHESION STRENGTHENING IN COMPLEX NARRATIVE TEXTS

&lt;p&gt;&lt;span&gt;This paper aims to explain some of the aspects which strengthen and coordinate cohesion productivity in translation as one of the basic textuality standards which hold the translation text together, making it more comprehensive as well as intelligible to the reader. The research focus will be on twenty translated narrative texts of a medium textual complexity, which are product of students attending the course of translation with the home department. All twenty texts have been archived and are qualified as exam documents, thus used as a source of information for this research endeavor. The gathered database of translations will be analyticallyapproached using comparative method based upon cohesion evidence in each, aiming at the end to conduct a summative assessment of the percentage level strength in all translated texts. The analytical part will be dealing with detection of words of a wide range typology, which play the role of cohesive devices of a wide range, counting their efficiency in terms of the overall Texture Analyzer Observation i.e. Test, which finally come up with the overall Cohesion Functionality Percentage (CFP) at level of all twenty translations. The hypothesis aims to give an answer to the frequency evidence of these devices and their transfer from SL to TT. And finally, the qualitative descriptive conclusions, are supposed to create a clearer theoretical picture proving the fact that cohesion strengthening is a factor which preserves and sets rules of Semantic Consistency Balance (SCB) between the two languages in combination.&lt;/span&gt;&lt;/p&gt;&lt;p&gt;&lt;span&gt;&lt;strong&gt;Keywords:&lt;/strong&gt; Translation, narrative, cohesive, devices, strengthen.&lt;/span&gt;&lt;/p&gt;

Applying combinatorics in the design of multiversion exams

In Spring 2020, the need for redesigning online assessments to preserve integrity became a priority to many educators. Many of us found methods to proctor examinations using Zoom and proctoring software. Such examinations pose their own issues. To reduce the technical difficulties and cost, many Zoom proctored examination sessions were shortened; to create summative assessments we supplemented the short proctored sessions with short unproctored sessions. In the aftermath of the pandemic, the method of creating a large volume of examination variants is helping us design mastery-based examinations. We review the mathematics behind choosing the size of question pools to improve integrity of the exam. Finally we provide files for generating examination variants from the pools.

Penerapan Kurikulum Merdeka pada pembelajaran IPAS kelas IV di SD Negeri Tahunan Yogyakarta

Background: SD Negeri Tahunan is not yet a driving school, but SD Negeri Tahunan is one of the schools that has implemented the Phase B independent curriculum (grades I and IV), and the independent curriculum implemented in this school, namely the independent independent curriculum, has changed. Purpose: This study aims to describe the planning, implementation, and evaluation of science learning based on the independent curriculum in class IV SD Negeri Tahunan Yogyakarta. Method: The design of this study uses descriptive qualitative. Data collection techniques through observation, interviews and documentation. This research data analysis technique uses the Miles and Huberman analysis model. Findings: The results of this study indicate that: (1) the science lesson plan made by the teacher has been adjusted to the criteria for an independent curriculum, the teacher has analyzed learning outcomes, made learning objectives, learning goal lines, methods, models, approaches, materials, learning resources, assessments, then poured into teaching modules, (2) the implementation of science learning in class IV is carried out in accordance with the lesson plans that have been made previously. Teachers create collaborative, interactive, and contextual learning. Activities in science learning vary so that learning is easy to understand, not boring, fosters an attitude of independence, high responsibility and curiosity, and makes learning fun, (3) the form of evaluation of science learning in class IV SD Negeri Tahunan is carried out by carrying out diagnostic assessments, formative assessments and summative assessments. The diagnostic assessment that has been carried out is a non-cognitive diagnostic assessment. Formative assessments are given by the teacher at the end of science learning at each meeting. Summative assessments are carried out during daily assessments, midterm assessments, and end of semester assessments.

Assessing Student Performance on Open- vs Closed-Book Summative Assessments

Assessing Student Performance on Open- vs Closed-Book Summative Assessments