Frequent Formative Assessments Research Articles

Application of Edutainment Concepts and Tracking Emotions Based on Transient Emotion Peak in Online Education Systems

Edutainment concepts such as Game-based learning (GBL) and ‘gamification’, pave the way for students to learn by experience by going beyond the traditional teacher-centred learning environment concepts. This paper is an attempt to investigate the feasibility of implementing a virtual learning environment by using “Edutainment” concepts to teach Computer Science subjects to undergraduate students in the Sri Lankan context while finding the most effective way out of GBL and gamification approaches in teaching and facilitating an online test environment to have frequent formative assessments to self-evaluate themselves. The first goal was to identify this student cohort’s preferred way out of GBL and gamification in learning. In this study, a game and a similar gamification approach were designed to teach a subject. Next, a qualitative analysis was conducted by collecting students’ feedback on each approach and a quantitative analysis was done to compare the effectiveness of the two approaches based on answers given to a quiz based on that subject. Under the second goal, it was found that effective feedback should address the emotional level of the student too. Hence, this study proposed a systematic way to track the emotional changes based on the transient emotion peak. We conclude, that adult students mostly tend to grab the core concepts rather than playing sophisticated games and though the proposed emotion tracking algorithm the emotions can be tracked at a promising accuracy level. In future work, we suggest that it would be better to use the proposed time interval to track the concentration level based on corresponding facial features.

Exploring wellbeing in first year medical students amidst a curriculum change

BackgroundThe support of student wellbeing features highly in all higher education institutional agendas. For medical students good physical and mental health can help prevent burnout, equip students for their future healthcare setting and indirectly improve patient care. At the University of Nottingham (UK), we were keen to explore undergraduate medical students perceived wellbeing before, during, and after an early years’ (years 1-3) curriculum change. A restructure of the curriculum enabled personal wellbeing sessions to be embedded and directly linked to the pastoral support system.MethodsStudents’ perceived wellbeing was assessed through a questionnaire distributed to three cohorts of first year students at the start and end of the autumn semester.ResultsThe data showed a clear improvement of perceived physical health at the end of the first semester following the curriculum change, alongside increased mood and ability to relax. A surprising outcome of this study was that students reported increased stress levels at the end of the semester, which we believe may be attributed to the change in assessment within the new curriculum. Our medical students are now facing end of year summative examinations, but are acutely aware of their progress as they undertake frequent formative assessments during the year. We propose that comparison of performance with peers is having a direct impact on perceived stress in these cohorts.ConclusionsThe study has shown that embedding wellbeing in the curriculum can have positive effects even within a changing curriculum. The importance of evolving wellbeing provision and support based on the needs of the student population is essential and probably never more in need than at this moment in time.

Assessment as Pedagogy in a Compressed-Format Summer Physics Abroad Program

This paper examines the impact of assessment and instructional practices in a compressed-format physics abroad program for life science students from a large U.S. university system. Using qualitative case study methodology, the study investigated the major pedagogical functions of assessments and their implications on student learning across three international sites. Findings from interviews, focus groups, and survey responses of international physics instructors indicated that instructors accommodated the unique program format and student cohort by fostering a highly supportive and collaborative environment for frequent formative assessments, feedback, and intervention. These pedagogical developments provide students the opportunity to learn physics intensively and gain disciplinary, metacognitive, and intercultural understanding.

Paul L. Foster School of Medicine, Texas Tech University Health Sciences Center El Paso.

Paul L. Foster School of Medicine, Texas Tech University Health Sciences Center El Paso.

Using Learning Management System to Promote Self-regulated Learning in a Flipped Classroom

The inherent structure of the flipped classroom (FC) supports a learning environment that promotes students’ self-regulated learning. In this paper, we focus on how a Learning Management System (LMS), in this case BrightSpace D2L, plays a key role in supporting all aspects of the FC. We discuss how: Pre-class assignments suggest that students give their learning some forethought and planning accordingly; frequent formative assessment and feedback permits students to monitor their progress in the classroom; students control their choices for studying the resources they use; and how students have ample opportunities to reflect on their work regularly. Further, we explore how learning logs (a form of reflective writing), with the LMS journaling and Dropbox features are used to keep track of students’ journal entries as well as demonstrate how professors use the LMS to motivate and encourage students to develop self-regulated learning. Qualitative analysis of students writing are provided to reveal gains in self-regulation.

Perceptions of scheduled vs. unscheduled directly observed visits in an internal medicine residency outpatient clinic

IntroductionLearners may subconsciously change their behavior once they know they are being observed, and this Hawthorne effect should be considered when designing assessments of learner behavior. While there is a growing body of literature to suggest direct observation is the ideal standard for formative assessment, the best method to directly observe learners is unknown. We explored scheduled and unscheduled methods of direct observation among internal medicine residents in the outpatient continuity clinic to advance the understanding of both observation methods.MethodsWe conducted a thematic analysis of faculty and internal medicine residents in an outpatient clinic setting. A semi-structured interview guide for focus group sessions was created. Focus groups were used to explore the internal medicine resident and core teaching faculty perceptions of the scheduled and unscheduled direct observation methods in the outpatient clinc. An experienced qualitative research interviewer external to the internal medicine residency was moderating the sessions. Eight peer focus groups were held. ion of themes from focus group transcripts identified resident and faculty perceptions of the different observation methods.ResultsFocus groups had 14 resident participants and 14 faculty participants. Unscheduled observations were felt to be more authentic than scheduled observations since residents perceived their behavior to be unmodified. Unscheduled observations allowed for increased numbers of observations per resident, which permitted more frequent formative assessments. Residents and faculty preferred remote video observation compared to in-room observation. Participants found direct observation a useful learning tool for high-yield, specific feedback.ConclusionsUnscheduled remote direct observation captures authentic clinical encounters while minimizing learner behavior modification. An unscheduled observation approach results in more frequent formative assessment and therefore in more instances of valuable feedback compared to scheduled observations. These findings can help guide the best practice approaches to direct clinical observation in order to enhance residents learning and experience.

Developing a Brief Behavior Rating Scale for Progress Monitoring of Depression in School Settings

Frequent formative assessment of students’ functioning, or progress monitoring, is a critical component of multi-tiered systems of support as data inform data-driven decisions about response to treatment. Progress monitoring tools for students’ academic and behavioral functioning are readily available and widely researched; however, despite the documented prevalence of depressive disorders among youth and that schools have been put forth as an ideal location for the delivery of mental health services, there are currently no progress monitoring tools to examine students’ response to interventions that target depression. To address this gap, this study sought to develop a progress monitoring assessment of students’ depressive symptoms using an empirically informed model for creating Brief Behavior Rating Scales (BBRS). Using this model, a four-item BBRS of depressive symptoms (BBRS-D) was created from the item pools of the Beck Depression Inventory for Youth (BDI-Y) and Children’s Depression Inventory (CDI) administered during a treatment study of depression in female youth; the resulting short scale corresponds well to the full-length assessments (i.e., r = .65 and r = .59); however, the BBRS-D possessed lower than adequate internal consistency (α = .50) and test–retest reliability ( r = .56). Limitations and future directions are discussed.

Direct Observation of Clinical Skills Feedback Scale: Development and Validity Evidence

ABSTRACTConstruct: This article describes the development and validity evidence behind a new rating scale to assess feedback quality in the clinical workplace. Background: Competency-based medical education has mandated a shift to learner-centeredness, authentic observation, and frequent formative assessments with a focus on the delivery of effective feedback. Because feedback has been shown to be of variable quality and effectiveness, an assessment of feedback quality in the workplace is important to ensure we are providing trainees with optimal learning opportunities. The purposes of this project were to develop a rating scale for the quality of verbal feedback in the workplace (the Direct Observation of Clinical Skills Feedback Scale [DOCS-FBS]) and to gather validity evidence for its use. Approach: Two panels of experts (local and national) took part in a nominal group technique to identify features of high-quality feedback. Through multiple iterations and review, 9 features were developed into the DOCS-FBS. Four rater types (residents n = 21, medical students n = 8, faculty n = 12, and educators n = 12) used the DOCS-FBS to rate videotaped feedback encounters of variable quality. The psychometric properties of the scale were determined using a generalizability analysis. Participants also completed a survey to gather data on a 5-point Likert scale to inform the ease of use, clarity, knowledge acquisition, and acceptability of the scale. Results: Mean video ratings ranged from 1.38 to 2.96 out of 3 and followed the intended pattern suggesting that the tool allowed raters to distinguish between examples of higher and lower quality feedback. There were no significant differences between rater type (range = 2.36–2.49), suggesting that all groups of raters used the tool in the same way. The generalizability coefficients for the scale ranged from 0.97 to 0.99. Item-total correlations were all above 0.80, suggesting some redundancy in items. Participants found the scale easy to use (M = 4.31/5) and clear (M = 4.23/5), and most would recommend its use (M = 4.15/5). Use of DOCS-FBS was acceptable to both trainees (M = 4.34/5) and supervisors (M = 4.22/5). Conclusions: The DOCS-FBS can reliably differentiate between feedback encounters of higher and lower quality. The scale has been shown to have excellent internal consistency. We foresee the DOCS-FBS being used as a means to provide objective evidence that faculty development efforts aimed at improving feedback skills can yield results through formal assessment of feedback quality.

Do OSCE progress test scores predict performance in a national high-stakes examination?

Progress tests, in which learners are repeatedly assessed on equivalent content at different times in their training and provided with feedback, would seem to lend themselves well to a competency-based framework, which requires more frequent formative assessments. The objective structured clinical examination (OSCE) progress test is a relatively new form of assessment that is used to assess the progression of clinical skills. The purpose of this study was to establish further evidence for the use of an OSCE progress test by demonstrating an association between scores from this assessment method and those from a national high-stakes examination. The results of 8 years' of data from an Internal Medicine Residency OSCE (IM-OSCE) progress test were compared with scores on the Royal College of Physicians and Surgeons of Canada Comprehensive Objective Examination in Internal Medicine (RCPSC IM examination), which is comprised of both a written and performance-based component (n = 180). Correlations between scores in the two examinations were calculated. Logistic regression analyses were performed comparing IM-OSCE progress test scores with an 'elevated risk of failure' on either component of the RCPSC IM examination. Correlations between scores from the IM-OSCE (for PGY-1 residents to PGY-4 residents) and those from the RCPSC IM examination ranged from 0.316 (p = 0.001) to 0.554 (<.001) for the performance-based component and 0.305 (p = 0.002) to 0.516 (p < 0.001) for the written component. Logistic regression models demonstrated that PGY-2 and PGY-4 scores from the IM-OSCE were predictive of an 'elevated risk of failure' on both components of the RCPSC IM examination. This study provides further evidence for the use of OSCE progress testing by demonstrating a correlation between scores from an OSCE progress test and a national high-stakes examination. Furthermore, there is evidence that OSCE progress test scores are predictive of future performance on a national high-stakes examination.

Large Lecture Transformation: Adopting Evidence-Based Practices to Increase Student Engagement and Performance in an Introductory Science Course

ABSTRACTThis study investigated students' attitudes, engagement, satisfaction, and performance in Introduction to Environmental Science after it was transformed from a typical large lecture to a student-centered learning environment. The instructors of the course collaborated with the Office of Teaching, Learning & Technology and radically redesigned the course by adopting evidence-based pedagogical methods and instructional technology to offer a rich active learning environment. Students' engagement, satisfaction, and performance in the transformed course were compared to the students in the same course that was traditionally taught in a previous semester in order to measure the effect of the transformed course on student learning. Four self-reported surveys and a focus group interview were administered during the semester, and performance scores, prior learning data, and demographic information were collected at the end of the semester. The assessment results indicate that the students in the transformed course prepared for the class significantly more, engaged in the course significantly better, and were significantly more satisfied with the course than were the students in the traditional lecture-format course. Most of all, multiple linear regression indicates that the students in the transformed course earned, on average, about three-fourths of a letter grade higher on their final grade after controlling for cumulative grade point average. The students in the transformed course reported that online learning materials and frequent formative assessments using online quizzes were helpful in their learning, and they perceived the course was relevant to real-world applications that matter to their daily life.

Examining the barriers to meaningful assessment and feedback in medical training

BackgroundRecent reports from both accreditation bodies in North America highlight problems with current assessment practices in postgraduate medical training. Previous work has shown that educators might be reluctant to report poor performance or fail underperforming trainees. This study explores the barriers perceived by medical educators to providing more meaningful assessment and feedback to trainees. MethodsSemistructured interviews were conducted with 22 physician educators. Interviews were audiotaped and transcribed verbatim. Three researchers analyzed the transcripts using a grounded theory approach. ResultsParticipants expressed a reluctance to provide poor assessments or feedback to trainees. Fifty-five percent of the participants reported passing trainees who could have benefited from additional training. Our data revealed a number of barriers which may account for these findings. Implementing more frequent formative assessments could help educators more effectively evaluate trainees and provide feedback, although a shift in the culture of medicine may be required. ConclusionIt is imperative that the barriers to effective assessment and feedback identified in this study be addressed to improve postgraduate medical training and enhance patient care.

Workplace-Based Assessment and Assessment for Learning: Threats to Validity.

Workplace-Based Assessment and Assessment for Learning: Threats to Validity.

Exploring student perceptions and experiences of ICT-enhanced formative assessment in an undergraduate management accounting course

The central role of assessment in driving student learning has been well argued in the literature. In particular, continuous formative assessment with timely feedback helps motivate students to work consistently, while the addition of just-in-time support may further motivate and support learning. Information and communication technologies have the potential both to contain the administrative burden of frequent formative assessment and enhance the provision of timely feedback. Drawing on theories of assessment and motivation this study investigates the value of combining two technological interventions, tutorial screencasts and online quizzes, for enhancing student learning in an undergraduate management accounting course. Data was collected from the usage logs of the learning management system and a survey of student experiences. The results indicate that the interventions enhanced students’ learning experiences both by motivating them to engage with the coursework and by encouraging them to do so on a continuous basis.

Progress testing: is there a role for the OSCE?

The shift from a time-based to a competency-based framework in medical education has created a need for frequent formative assessments. Many educational programmes use some form of written progress test to identify areas of strength and weakness and to promote continuous improvement in their learners. However, the role of performance-based assessments, such as objective structured clinical examinations (OSCEs), in progress testing remains unclear. The aims of this paper are to describe the use of an OSCE to assess learners at different stages of training, describe a structure for reporting scores, and provide evidence for the psychometric properties of different rating tools. A 10-station OSCE was administered to internal medicine residents in postgraduate years (PGYs) 1-4. Candidates were assessed using a checklist (CL), a global rating scale (GRS) and a training level rating scale (TLRS). Reliability was calculated for each measure using Cronbach's alpha. Differences in performance by year of training were explored using analysis of variance (anova). Correlations between scores obtained using the different rating instruments were calculated. Sixty-nine residents participated in the OSCE. Inter-station reliability was greater (0.88) using the TLRS compared with the CL (0.84) and GRS (0.79). Using all three rating instruments, scores varied significantly by year of training (p < 0.001). Scores from the different rating instruments were highly correlated: CL and GRS, r = 0.93; CL and TLRS, r = 0.90, and GRS and TLRS, r = 0.94 (p < 0.001). Candidates received feedback on their performance relative to examiner expectations for their PGY level. Scores were found to have high reliability and demonstrated significant differences in performance by year of training. This provides evidence for the validity of using scores achieved on an OSCE as markers of progress in learners at different levels of training. Future studies will focus on assessing individual progress on the OSCE over time.

Teachers' evaluation of formative assessment on summative assessment in undergraduate medical education

Background: Assessment is important in all forms of learning. Formative assessments have evolved as a means to adapt to student needs. Summative assessment designed to make judgments about students' performance and produce grades Formative assessment is the assessment that takes place during a course or programming of study whereas summative assessment is normally carried out at or towards the end of a course. Objective: To determine the teachers' view about the influence of formative assessment on the result of summative assessment in undergraduate medical course. Method: This cross sectional descriptive study was carried out from July 2009 to June 2010 over 106 teachers of selected two government and two private medical colleges. A structured and pretested questionnaire included different opinion about the influence of formative assessment on summative assessment was distributed among respondents and data were collected for analysis. Results: This study revealed that majority of the teachers (77%) considered feedback from formative assessment to students is important to close up the learning gap. Eighty two percent teachers were in agreement that through formative assessment teachers identified students' weak point and 69% supported that formative assessment motivates the students for deep learning and regular study. However, frequent formative assessment hampers students' independent learning which produces negative effects in summative exam. Conclusion: Formative assessment has got significant effect on summative assessment in various aspects. DOI: http://dx.doi.org/10.3329/bjme.v3i1.18591 Bangladesh Journal of Medical Education Vol.3(1) 2012: 18-21

Want to Reduce Guessing and Cheating While Making Students Happier? Give More Exams!

It is almost universally agreed that more frequent formative assessment (homework, clicker questions, practice tests, etc.) leads to better student performance and generally better course evaluations.1 There is, however, only anecdotal evidence that the same would be true for more frequent summative assessment (exams). There maybe many arguments against giving more exams, including the general “pain” associated with examinations, as well as reduced teaching time, since classroom sessions are dedicated to exams rather than lecturing. We present evidence that increasing the number of exams in fact does lead to better learning success, less cheating and guessing on homework, and better student course evaluations.

An Assessment of the PeerWise Student-Contributed Question System's Impact on Learning Outcomes: Evidence from a Large Enrollment Political Science Course

The academic literature on teaching and learning provides compelling evidence that active and collaborative learning strategies, as well as frequent formative assessments with corrective feedback, significantly improve student learning outcomes. A central challenge faculty face in teaching large undergraduate courses (150 students) is providing students with these types of learning opportunities without overburdening the teaching staff (instructors and their graduate student teaching assistants). To address this challenge, this study assesses the use of PeerWise, an online pedagogical tool that allows students to write, share, answer, and discuss multiple choice questions (MCQs) in a large enrollment political science course. Using both quantitative and qualitative student data, it finds that student use of PeerWise is not only positively correlated with student learning gains at statistically significant levels (p = 0.001), but also that it promotes student perceptions of learning and motivation to learn.

University of New Mexico School of Medicine

University of New Mexico School of Medicine