Formative Assessment Tool Research Articles

Evaluating OSPE as a Formative Assessment Tool for Medical Students

Introduction. Objective structured practical examination (OSPE) is a globally recognized and widely implemented assessment method used for evaluating students in practical examinations. It requires examinees to complete various tasks within a set time frame, each assessed against predetermined criteria designed to measure practical skills and attitudes. This study aimed to assess the effectiveness of the OSPE as a tool for internal assessment in the subject of Anatomy for first-year MBBS students, comparing its performance against the traditional practical examination (TPE) method. Methods. In this prospective study, an OSPE tool specific to Anatomy was developed and validated for face and content to ensure that the OSPE stations were standardized and well-constructed. The part-leaving test covered the Thorax and Brain chapters using both the TPE and OSPE methods. A Bland-Altman plot was employed to assess the agreement between the quantitative assessments of both methods. Additionally, a 5-point Likert scale was utilized to gauge the perceptions of both students and faculty members regarding the assessment methods. Results. The analysis revealed a significant difference between the scores obtained from OSPE and TPE. The intraclass correlation coefficients for reliability between TPE and OSPE were 0.81 for the Thorax chapter and 0.84 for the Brain chapter. Additionally, the Bland-Altman plots illustrating the scores from both assessment methods indicated low bias and no definite pattern. While no significant outliers were identified for the Brain part, the Thorax part exhibited outliers at a rate of 6.43%, suggesting the presence of low-impact erratic variables. The Chi-square test for student perceptions yielded significant results for the entire questionnaire. In contrast, faculty perceptions showed significant results for all questions except for two. Conclusions. Due to the high intraclass correlation coefficients between TPE and OSPE and the absence of bias, systematic errors, and erratic variables, OSPE is a promising assessment tool. Students perceived that OSPE decreased the chances of failure and that the questions asked in OSPE were more relevant compared to the traditional method. Faculty members also noted that OSPE provided reasonably high transparency and decreased subjective bias compared to TPE. Considering these factors, it can be concluded that OSPE is a viable potential tool to replace TSE.

Effectiveness of video-based teaching in surgery residents: A randomized control trial

Objective: To determine the effectiveness of video-based teaching in surgery residents by comparing it to the routine operating room traditional teaching. Methodology: The randomized control trial was done at four institutions from September 2022 to March 2023. Interventional group underwent video-based instruction of basic surgical skills whereas the control group was taught through traditional operating room teaching. Pre and post-test mean scores were analyzed on SPSS version 20 through paired t-test. Learning gain was calculated. Supervisors’ perceptions were recorded on survey form regarding direct observations and recorded video evaluations of resident skills on post-test. Results: Out of sixty (n=60), fifty-five newly inducted surgical residents completed the study including both females (n=13) and male residents (n=42). Video intervention Group (27.93±3.72) and control group (23.07±4.62) both showed improvements in their post-test scores as compared to pretest scores of 13.68±3.25 and 13.52±3.60 respectively. Mean score difference improvement was more in video intervention group (13.9±3.8) in comparison to control group (9.5±4.3) provided both groups exhibits improvements in all seven domains of the global rating system (reflected by the p <0.0001). However, learning gain of 65% was observed in intervention group as compared to learning gain (41%) of control group. Evaluators(n=5) observed that recorded video evaluations helped to provide integrated feedback, despite being time (40%) and resource intensive (60%). Conclusion: Video-based teaching has higher learning gain irrespective of the fact that both groups exhibit statistically significant results in all seven domains of the global rating system. Recorded video evaluation was found feasible and reliable tool for formative assessment. doi: https://doi.org/10.12669/pjms.40.10.7856 How to cite this: Niaz K, Mahboob U, Masood D, Maqbool A. Effectiveness of video-based teaching in surgery residents: A randomized control trial. Pak J Med Sci. 2024;40(10):2373-2378. doi: https://doi.org/10.12669/pjms.40.10.7856 This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

University-Based Coach Education: The E-Portfolio as a Formative Assessment Tool of Student-Coaches’ Learning

Current research highlights the need for more studies focused on how high-quality assessment strategies can contribute to coach learning in coach education. The use of e-portfolios, as formative assessment tools, has shown to contribute to student-coaches’ learning in university-based programs, but studies on this topic are still scarce. The aim of this action research project is to investigate the potentialities and challenges of the e-portfolio as a formative learning-oriented assessment tool in an undergraduate sport coach course in Brazil. By sharing this, from the perspective of student-coaches and the assistant-professors of the course, we reflect on the evidence with the intention to both inform colleagues doing similar work and contribute to an emerging body of assessment in coach education literature.

The impact of kahoot-based formative assessment on student learning outcomes at a Junior High School in Samarinda

This study explores the impact of using Kahoot! as a formative assessment tool on student learning outcomes in English at SMPN 22 Samarinda. The research utilized a quasi-experimental design with pre- and post-test measures on Class VII-7 B. Findings revealed that Kahoot! significantly improved students' understanding of English grammar, particularly tenses. Data analysis was conducted using SPSS software, showing a notable increase in student achievement post-intervention. The study emphasizes the importance of integrating technology like Kahoot! in the classroom to enhance learning. Recommendations for future research and teacher strategies are also provided.

Launching Cost-effective Virtual Escape Rooms as a Tool for FormativeAssessment

: This paper describes an innovative tool for formative assessment. This involves designing and implementing a Virtual Escape Room for students of Master’s in Health Professions Education (MHPE) at Gulf Medical University. Escape Room is a type of game-based education where players are constrained in a room, and they must solve a series of interconnected challenges in the form of puzzles or riddles to escape the room. Escape rooms are proved to be innovative and engaging learning strategies, although they are resource intensive. We developed cost-effective virtual escape room with the help of Google Workspace tools. Two cycles of escape rooms were conducted on different courses in similar fashion. Questions were based on the learning objectives of selected courses of Master’s in Health Professions Education. These tools were interconnected and revolved around a captivating storyline. Twelve students were grouped into teams of four and were subjected to the game. They played as teams on a smart device for a time duration of ten minutes. Post-activity students were given feedback on their performance on questions and on team-based skills. Students were then engaged in open discussion on their experience, and this was documented for analysis. They also gave feedback on google forms. Analysis of feedback revealed that escape room enhanced students’ engagement and motivation towards learning. It helped foster leadership and team-building skills along with effectively recalling the learned concepts. In addition, students learned to handle stress when faced with challenges. Faculty, on the other hand, developed the skills to plan and execute a unique learning activity. We continue to learn from this experience and will explore further opportunities to expose students and faculty to such engaging and unique learning activities.

Using Self-assessment Tools to Enhance English Language Skills in an Online Learning Environment

Self-assessment is considered one of the formative assessment tools, and it is helpful for teachers and students. The instructors use self-assessment to track students’ progress in lessons, while the students use it to identify their strengths and weaknesses and learn how to overcome the points of weakness in their learning. This study examines the impact of using self-assessment strategies to enhance English language skills in the online learning environment. Therefore, this study is significant as it investigates the advantages and efficacy of self-assessment tools for improving English language proficiency in an online learning setting. This research paper is trying to answer the main question which is to what extent using self-assessment tools could improve learners’ English language skills in the online learning environment. The current study has followed a qualitative research method to explore the instructors’ perspectives towards the self-assessment tools used to evaluate learners’ performance. The research instrument comprises semi-structured interviews conducted with six university instructors from three universities in the United Arab Emirates. The findings revealed the positive outcomes of the self-assessment tools by indicating progress in students’ performance in English language skills. These findings will be used to provide recommendations that may inspire teachers to adopt practical self-assessment tools in lectures

Serious Illness Communication Tool for Palliative Care Fellows: Development, Implementation, and Lessons Learned.

Introduction: Learning expert communication skills is a core educational goal within palliative care training, yet there are few communication-based educational tools specifically designed for specialty-trained palliative care providers. Objective: To develop and implement a tool to facilitate effective learning of serious illness communication for hospice and palliative medicine fellows. Methods: A novel formative assessment tool was developed within the UCLA Palliative Care Fellowship program, and utilized throughout the academic year on a weekly basis. Focus groups were held for fellows and faculty separately at the end of the academic year in order to gain insight into the experience and effectiveness of the tool. Focus group transcripts were analyzed through thematic analysis. Results: There was a 47% participation rate in the focus groups (n = 7). Qualitative analysis demonstrated positive impact of the tool in identifying learning goals, improving quality of feedback, and in standardizing language around advanced communication skills, with most value in the first half of the fellowship year. Some aspects of the tool were found to increase feedback anxiety, including the competency scoring component, frequency of use, and utilization with individual patient encounters. Conclusion: This novel communication tool provides an important addition to serious illness communication training of specialist palliative care providers, in creating a shared mental model for naming and organizing skills, as well as generating specific high quality learning goals and feedback. Preliminary feedback from our pilot implementation phase provided important information about how to refine the tool and remove components that unnecessarily added to feedback anxiety.

Digital Library for the Implementation of Formative Assessment into Teaching

This paper presents information about the development, implementation, and verification of efficiency of “Digilib” – a digital library of formative assessment (FA) tools. A selection of 10 different types of FA tools (e.g., self-assessment card, prediction card, and Frayer model) was conducted based on a literature search and Internet resources. Subsequently, in accordance with the educational standards and in cooperation with teachers, a total of 36 databases covering 774 FA tools for the selected thematic units and topics in science (biology, physics, chemistry), mathematics, and informatics were created for primary schools. “Digilib” application allows teachers to create and save their own FA tools (forms), which can be assigned to students to collect information about their understanding of the subject matter, i.e., to receive real-time feedback. In the 2022/2023 school year, 8 primary schools, 16 teachers, and 46 classes (grades 5 to 9) with a total of 846 students were registered in the digital library. Semi-structured interviews were conducted to collect teachers’ opinions on the implementation of FA provided by the digital library. This paper presents the results of a semi-structured interview performed at the selected primary school. The respective teacher appreciated the simple user experience, quick feedback about students’ current state of knowledge, automated creation of complex overviews of students’ answers, and archiving. The feedback provided by the self-assessment and prediction cards helps students develop an ability to perform objective self-assessment and address possible misunderstanding in learning.

Using multimodal learning analytics as a formative assessment tool: Exploring collaborative dynamics in mathematics teacher education

AbstractBackgroundTraditionally, understanding students' learning dynamics, collaboration, emotions, and their impact on performance has posed challenges in formative assessment. The complexity of monitoring and assessing these factors have often limited the depth and breadth of insights.ObjectivesThis study aims to explore the potential of multimodal learning analytics as a formative assessment tool in math education. The focus is on discerning how collaborative discourse behaviours and emotional indicators interplay with lesson evaluation performance.MethodsUsing undergraduate students' multimodal data, which includes collaboration data, facial behaviour data, and emotional data, the study explored the patterns of collaboration and emotion. Through the lens of multimodal learning analytics, we conducted exploratory data analysis to identify meaningful relationships between specific types of collaborative discourse, facial expressions, and performance indicators. Moreover, the study evaluated a machine learning model's potential to predict target learning outcomes by integrating data from multiple channels.ResultsThe analysis revealed key features from both discourse and emotion data as significant predictors. These findings underscore the potential of a multimodal analytical approach in understanding students' learning process and predicting outcomes.ConclusionsThe study emphasizes the importance and feasibility of a multimodal learning analytic approach in the context of math education. It highlights the academic and practical implications of such an approach, along with its limitations, pointing towards future research directions in this area.

Designing, introducing,and implementing objective structured practical examinations as a formative assessment tool in undergraduate medical pharmacology

Designing, introducing,and implementing objective structured practical examinations as a formative assessment tool in undergraduate medical pharmacology

DIGITAL GAME-BASED LANGUAGE LEARNING IN PRIMARY EDUCATION: EXPLORING MOTIVATIONAL FACTORS AND EDUCATIONAL OUTCOMES

The objective of this study was to assess the efficacy of Digital Game-Based Language Learning (DGBLL) in promoting greater pupil involvement in language acquisition at the primary school level. Three groups of grade four students were examined in this study: a control group that received conventional teaching strategies, an experimental group that utilized Kahoot and Quizizz, and a second experimental group that played the digital games competitively. The researchers evaluated students' motivation and attitudes towards the use of gamification in language learning via pre-tests, post-tests, and questionnaires. The experimental groups showed superior performance and increased motivation, with the strongest performance experienced in the second experimental group that added competition to the learning process. The findings suggest that the integration of DGBLL and peer competition with gamified formative assessment tools significantly enhances language learning by introducing a captivating, efficient, and flexible method for language instruction. This study offers useful insights into the possibilities of gamification in language learning and evaluation, emphasizing its importance in modern educational practices.

Benefits and Barriers on Quizizz Application as Formative Assessment Tool in Vocabulary Teaching: Students Side

As technology becomes more advanced, education is getting more sophisticated. The purpose of this study is to find out the benefits and barriers related on Quizizz Application as formative assessment tool in vocabulary teaching for students. This study is classified as qualitative research, the sample of this research is 10 students of SMA N 4 Yogyakarta. Data were collected through observation, interviews and documentation. The study revealed two main aspects, the benefits and the barriers. Analysis of research data is descriptive. The results of the study showed students' perspectives on the use of Quizizz. In conclusion, students' opinions on the usage of Quizizz as formative assessment tool in learning vocabulary are positive, the platform is extremely entertaining to use in the classroom and helps students focus more on the material that they have learned. Keywords: Assessment Tool, Quizizz Application, Vocabulary

Evaluation par le miniclinical évaluation exercise (Mini-CEX) en réanimation

Mini Clinical Evaluation Exercise (mini-CEX) is one of the assessment tools in medical education. It includes three steps: overview of clinical situation, observation and feedback. To evaluate the feasibility of mini-CEX as a formative assessment tool for medical trainees in 5th year of medicine in a teaching intensive care unit (ICU). Single-center qualitative research conducted in ICU during the 2nd semester of the academic year 2022-2023. Seven core clinical skill assessments were done, and the performance was rated on a 9-point scale. An assessment of the method was conducted with both trainees and clinical educators. We conducted six mini-CEX recorded sessions. All medical students had marks under the average of 4.5. In the first period, the highest mark was obtained for counselling skills (4.5). The best score was obtained for clinical judgement (4) in the second period and for management plan (4) in the third period. Most of medical trainees (11 sur 12) were satisfied with the method and feedback was according to them the most useful step. Ten students agreed fully to introduce this assessment tool in medical educational programs. Two medical educators out of three did not practice this method before. They agreed to include mini-CEX in the program of medical education of the faculty of medicine of Tunis. However, they did not agree to use it as a summative assessment tool. Our study demonstrates that we can use the mini-CEX in medical teaching. Both trainees and educators were satisfied with the method.

Unveiling the Power of Educational Mathematics Applications as Formative Assessment Tools: A Multiple Case Study

Unveiling the Power of Educational Mathematics Applications as Formative Assessment Tools: A Multiple Case Study

Simulation Games as an Alternative Formative Assessment Tool for Experiential Learning in VET Marketing Education: A Desk-Research Review

Simulation games continue to be widely used in marketing education. Based upon a review of simulation tools from the past decade, this study gathers insights on how simulations can be used as an effective assessment tool to enhance learning in marketing and digital marketing vocational subjects. In particular, this research looks more closely at the opportunities afforded by simulation games for experiential learning in relation to formative assessments as well as the challenges presented by the application and validity of the games. This study made use of the desk-based approach as a line of research study while making use of secondary data that is available in competent scholarly literature related to simulation games as an alternative formative assessment tool for experiential learning in VET marketing education. The methodological flow followed in this research used the five phases implementation of industry 4.0 technologies as suggested by Garrido-Lova (2021). This research is original as, through desk-research, the study gave an overview of how simulation games can contribute towards experiential learning. The study indicated that a plethora of evidence points towards the number of pedagogical effects students benefit from when assessments take a formative approach and are carried out through a simulation game. This research should be of keen interest to current and future users of marketing simulation games who might have questions about how to use them effectively in a vocational setting and for assessment purposes.

Assessing Teacher Knowledge and Skills for Competency Based Formative Assessment of Mathematical Activities in Public Pre-primary Schools in Gem Sub-county, Kenya

Competency Based Curriculum (CBC) is a reform initiative in Kenya’s education that focuses on Competency Based Formative Assessment (CBFA) practice. The conduct of CBFA requires teachers to be knowledgeable on the new assessment strategies, expected to be attained after training. However, only 42.79% of the pre-primary teachers in Gem have been trained on the CBC, a percentage lower than other sub-counties in Siaya. Further, mathematical activities in Gem registered lowest ranking in performance compared to other learning areas.This raises questions whether the teachers poses the requisite knowledge to conduct formative assessment, which aids learning and achievement. The purpose of the study was to assess teacher knowledge and skills for CBFA of mathematical activities in public pre-primary schools in Gem.Objectives were to establish teacher pedagogical content knowledge to conduct CBFA, determine teacher knowledge of feedback provision to conduct CBFA, examine teacher knowledge of goal setting to conduct CBFA and to determine teacher knowledge of ICT to conduct CBFA. The study was based on Schildkamp et al. (2020) model which places acquisition of knowledge as key requisite for competency based formative assessment of mathematical activities. The study employed descriptive survey research design using mixed methods of data collection. The sample size for the study was 76 teachers, 1 Sub-County ECD Coordinator (SCECDC) and 10 headteachers. Data was obtained using questionnaire, classroom observation checklist and interview schedule. Quantitative data was analyzed using descriptive statistics involving frequency distribution tables, percentages and means while qualitative data was analyzed using thematic categories. Findings indicate that teachers have limited knowledge on using formative assessment tools, setting lesson goal and use of Information Communication and Technology (ICT). They could not differentiate formative from summative evaluation when giving feedback. The results imply that there is need for better structured inset based on prior identification of teachers needs.

THE DESIGN AND IMPLEMENTATION OF AN ONLINE FORMATIVE ASSESSMENT TO FOSTER INTERACTIVE ENGLISH LEARNING AMONG SENIOR HIGH SCHOOL STUDENTS IN INDONESIA

In response to pandemic-related learning loss, the Indonesian government has implemented a new English language curriculum with a focus on holistic assessment. However, the effectiveness of this curriculum relies on innovative teaching practices. This study positions the author as a curriculum engineer who has developed an online formative assessment concept. The aim of this research is to create a formative assessment tool that records student learning processes, promotes interactive English learning, and overcomes traditional pedagogical practices. By recording student learning processes and integrating technology, such as mind mapping and word clouds, students are able to explore, investigate, and generate new knowledge. The study follows a Research and Development (RD) approach and the developed website highlights the two online formative assessment methods. The findings demonstrate that using the online formative assessment tool can enhance students' critical thinking and communication skills while improving their English proficiency. Based on these results, this study recommends that schools integrate formative assessment tools into their English language curriculum to promote interactive learning and overcome traditional teaching practices. The findings of this study are relevant to educators, policymakers, and curriculum developers interested in promoting interactive English learning in a technology-enhanced learning environment. This study expected to be an assessment innovation in the interactive English.

The Social Justice Educational Assessment Scale: Filling a Gap in Social Justice Education Assessment and Evaluation

Nationally recognized social justice standards guide educators in developing social justice education. Absent from the guidance are tools to conduct initial formative assessment or to measure the impact of related instruction. To fill that gap, an academic researcher and 10th-grade teacher used a 3-phased, 9-step process to develop, pilot test, and evaluate a self-assessment tool called the Social Justice Educational Assessment Scale (SJEAS). In Phase 1, the team created the SJEAS items, aligning each with the Learning for Justice 9th–12th-grade social justice learning outcomes to ensure content validity. In Phase 2, they pretested the questions, revised the scale, and administered it to 322 student participants. Next, they conducted inter-item and total-item correlation tests and principal component analysis to ensure internal consistency. In Phase 3, they evaluated the SJEAS. Preliminary results indicate the SJEAS could provide educators and curriculum coordinators with a practical formative assessment and impact evaluation tool for social justice education aligned with nationally recognized learning outcomes. Complementary data, including classroom assignments and student discussions, would further enhance its value.

Role of Case-Based Discussion as Workplace-Based Assessment Tool in Urology and General Surgery

OBJECTIVES To determine the effectiveness of case-based discussion in terms of medical record-keeping, clinical reasoning, clinical judgment, and self-reflection in the management of patients. METHODOLOGY A multidisciplinary descriptive study was conducted in the Department of Urology Institute of Kidney Diseases, Hayatabad Medical Complex Peshawar, and Department of General Surgery, Hayatabad Medical Complex Peshawar, Pakistan, from January to December 2023. A total number of 20 postgraduate residents, ten from Urology and General Surgery, participated in the study with equal male and female participants. Case-based discussion (CBD) assessments focused on medical record-keeping, clinical Decision-making, clinical judgment, management, follow-up, and reflective writing. The data was collected on structured proforma and was analyzed on SPSS-25. RESULTSThe total number of study participants was 20, and 410 Case-Based Discussion sessions were conducted. This constitutes an average of 4.8 assessments per week. Most encounters were related to medical clinical decisions and management. The case-based discussion showed significant improvement on the Likert scale of the supervisor’s rating, which extended over four quarters on all major parameters of formative assessment (p 0.001). The multimodal regression test showed significant enhancement in learning parameters on case-based discussions. R2=0.172 F ration=25.141 n=410 (p=0.001). There was no significant difference in the gender of residents as well as training specialty of residents of urology and general surgery (p >0.05). CONCLUSION A case-based discussion is an effective tool for formative assessment of Urology and General Surgery postgraduate residents.

Implementation and Impact of an Adapted Digital Perioperative Competency-Building Tool to Enhance Teaching, Learning And Feedback in Postgraduate Competency-Based Medical Education

The purpose of this educational intervention was to introduce, iteratively adapt, and implement a digital formative assessment tool in a surgical speciality. The study also evaluated the intervention's impact on perioperative teaching, learning, feedback, and surgical competency. A participatory action research model with a mixed methods approach. This study was performed over 10 months in an institutional hospital in South Africa with a general surgery department. Twelve supervising surgical trainers/faculty and 12 surgical trainees/residents consented to participate in the intervention. The first 4 months of the intervention focused on relationship building, a multi-stakeholder contextual needs assessment and training sessions to support a shared mindset and shift in the teaching and learning culture. The final adapted perioperative competency-building tool comprised a 23-item assessment with four open-text answers (Table 1). Over the following 6-month period, 48 workplace-based competency-building perioperative evaluations were completed. Most trainees took less than 5 minutes to self-assess (67%) before most trainers (67%) took less than 5 minutes to give oral feedback to the trainee after the perioperative supervised learning encounter. On average, the digital tool took 6 minutes to complete during the bidirectional perioperative teaching and learning encounter with no negative impact on the operational flow. All trainers and trainees reported the training and implementation of the digital tool to be beneficial to teaching, learning, feedback, and the development of surgical competency. Analysis of the completed tools revealed several trainees showing evidence of progression in surgical competency for index procedures within the speciality. The focus groups and interviews also showed a change in the teaching and learning culture: more positively framed, frequent, structured, and specific feedback, improved accountability, and trainee-trainer perioperative readiness for teaching. Highlighted changes included the usefulness of trainee self-assessment before perioperative trainer feedback and the tool's value in improving competency to Kirkpatrick Level 4. Implementing an adapted digital Workplace-Based Assessment (WBA) tool using a participatory action research model has proven successful in enhancing the effectiveness of supervised perioperative teaching and learning encounters. This approach has improved teaching and feedback practices, facilitated the development of surgical competency, and ultimately impacted the overall culture to Kirkpatrick level 4. Importantly, it has positively influenced the trainee-trainer relationship dynamic. Based on these positive outcomes, we recommend using this effective method and our relationship-centred framework for implementing formative competency-building tools in future studies. By doing so, larger-scale and successful implementation of Competency-Based Medical Education (CBME) could be achieved in various contexts. This approach can potentially enhance teaching and learning encounters, promote competency development, and improve the overall educational experience for surgical trainees and trainers.